draft-ietf-opsawg-large-flow-load-balancing-05.txt   draft-ietf-opsawg-large-flow-load-balancing-06.txt 
OPSAWG R. Krishnan OPSAWG R. Krishnan
Internet Draft Brocade Communications Internet Draft Brocade Communications
Intended status: Informational L. Yong Intended status: Informational L. Yong
Expires: February 23, 2014 Huawei USA Expires: June 26, 2014 Huawei USA
August 23, 2013 A. Ghanwani December 26, 2013 A. Ghanwani
Dell Dell
Ning So Ning So
Tata Communications Tata Communications
S. Khanna S. Khanna
Cisco Systems Cisco Systems
B. Khasnabish B. Khasnabish
ZTE Corporation ZTE Corporation
Mechanisms for Optimal LAG/ECMP Component Link Utilization in Mechanisms for Optimal LAG/ECMP Component Link Utilization in
Networks Networks
draft-ietf-opsawg-large-flow-load-balancing-05.txt draft-ietf-opsawg-large-flow-load-balancing-06.txt
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. This document may not be modified, provisions of BCP 78 and BCP 79. This document may not be modified,
and derivative works of it may not be created, except to publish it and derivative works of it may not be created, except to publish it
as an RFC and to translate it into languages other than English. as an RFC and to translate it into languages other than English.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at page 1, line 43 skipping to change at page 1, line 43
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html http://www.ietf.org/shadow.html
This Internet-Draft will expire on February 23, 2014. This Internet-Draft will expire on June 26, 2014.
Copyright Notice Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Abstract Abstract
Demands on networking infrastructure are growing exponentially; the Demands on networking infrastructure are growing exponentially due to
drivers are bandwidth hungry rich media applications, inter-data bandwidth hungry applications such as rich media applications and
center communications, etc. In this context, it is important to inter-data center communications. In this context, it is important to
optimally use the bandwidth in wired networks that extensively use optimally use the bandwidth in wired networks that extensively use
LAG/ECMP techniques for bandwidth scaling. This draft explores some link aggregation groups and equal cost multi-paths as techniques for
of the mechanisms useful for achieving this. bandwidth scaling. This draft explores some of the mechanisms useful
for achieving this.
Table of Contents Table of Contents
1. Introduction...................................................3 1. Introduction...................................................3
1.1. Acronyms..................................................4 1.1. Acronyms..................................................4
1.2. Terminology...............................................4 1.2. Terminology...............................................4
2. Flow Categorization............................................4 2. Flow Categorization............................................4
3. Hash-based Load Distribution in LAG/ECMP.......................5 3. Hash-based Load Distribution in LAG/ECMP.......................5
4. Mechanisms for Optimal LAG/ECMP Component Link Utilization.....7 4. Mechanisms for Optimal LAG/ECMP Component Link Utilization.....7
4.1. Differences in LAG vs ECMP................................8 4.1. Differences in LAG vs ECMP................................7
4.2. Overview of the mechanism.................................9 4.2. Overview of the mechanism.................................8
4.3. Large Flow Recognition...................................10 4.3. Large Flow Recognition...................................10
4.3.1. Flow Identification.................................10 4.3.1. Flow Identification.................................10
4.3.2. Criteria for Identifying a Large Flow...............10 4.3.2. Criteria for Identifying a Large Flow...............10
4.3.3. Sampling Techniques.................................11 4.3.3. Sampling Techniques.................................11
4.3.4. Automatic Hardware Recognition......................12 4.3.4. Automatic Hardware Recognition......................12
4.4. Load Re-balancing Options................................13 4.4. Load Re-balancing Options................................12
4.4.1. Alternative Placement of Large Flows................13 4.4.1. Alternative Placement of Large Flows................13
4.4.2. Redistributing Small Flows..........................13 4.4.2. Redistributing Small Flows..........................13
4.4.3. Component Link Protection Considerations............14 4.4.3. Component Link Protection Considerations............14
4.4.4. Load Re-balancing Algorithms........................14 4.4.4. Load Re-balancing Algorithms........................14
4.4.5. Load Re-Balancing Example...........................14 4.4.5. Load Re-Balancing Example...........................14
5. Information Model for Flow Re-balancing.......................15 5. Information Model for Flow Re-balancing.......................15
5.1. Configuration Parameters for Flow Re-balancing...........16 5.1. Configuration Parameters for Flow Re-balancing...........15
5.2. System Configuration and Identification Parameters.......16 5.2. System Configuration and Identification Parameters.......16
5.3. Information for Alternative Placement of Large Flows.....17 5.3. Information for Alternative Placement of Large Flows.....17
5.4. Information for Redistribution of Small Flows............17 5.4. Information for Redistribution of Small Flows............17
5.5. Export of Flow Information...............................17 5.5. Export of Flow Information...............................17
5.6. Monitoring information...................................18 5.6. Monitoring information...................................18
5.6.1. Interface (link) utilization........................18 5.6.1. Interface (link) utilization........................18
5.6.2. Other monitoring information........................18 5.6.2. Other monitoring information........................18
6. Operational Considerations....................................18 6. Operational Considerations....................................19
6.1. Rebalancing Frequency....................................19 6.1. Rebalancing Frequency....................................19
6.2. Handling Route Changes...................................19 6.2. Handling Route Changes...................................19
7. IANA Considerations...........................................19 7. IANA Considerations...........................................19
8. Security Considerations.......................................19 8. Security Considerations.......................................19
9. Acknowledgements..............................................20 9. Acknowledgements..............................................20
10. References...................................................20 10. References...................................................20
10.1. Normative References....................................20 10.1. Normative References....................................20
10.2. Informative References..................................20 10.2. Informative References..................................20
1. Introduction 1. Introduction
Networks extensively use LAG/ECMP techniques for capacity scaling. Networks extensively use link aggregation groups (LAG) [802.1AX] and
Network traffic can be predominantly categorized into two traffic equal cost multi-paths (ECMP) [RFC 2991] as techniques for capacity
types: long-lived large flows and other flows (which include long- scaling. For the problems addressed by this document, network traffic
lived small flows, short-lived small/large flows). Stateless hash- can be predominantly categorized into two traffic types: long-lived
based techniques [ITCOM, RFC 2991, RFC 2992, RFC 6790] are often used large flows and other flows. These other flows, which include long-
to distribute both long-lived large flows and other flows over the lived small flows, short-lived small flows, and short-lived large
flows, are referred to as small flows in this document. Stateless
hash-based techniques [ITCOM, RFC 2991, RFC 2992, RFC 6790] are often
used to distribute both large flows and small flows over the
component links in a LAG/ECMP. However the traffic may not be evenly component links in a LAG/ECMP. However the traffic may not be evenly
distributed over the component links due to the traffic pattern. distributed over the component links due to the traffic pattern.
This draft describes mechanisms for optimal LAG/ECMP component link This draft describes mechanisms for optimal LAG/ECMP component link
utilization while using hash-based techniques. The mechanisms utilization while using hash-based techniques. The mechanisms
comprise the following steps -- recognizing long-lived large flows in comprise the following steps -- recognizing large flows in a router;
a router; and assigning the long-lived large flows to specific and assigning the large flows to specific LAG/ECMP component links or
LAG/ECMP component links or redistributing other flows when a redistributing the small flows when a component link on the router is
component link on the router is congested. congested.
It is useful to keep in mind that the typical use case is where the It is useful to keep in mind that in typical use cases for this
long-lived large flows are those that consume a significant amount of mechanism the large flows are those that consume a significant amount
bandwidth on a link, e.g. greater than 5% of link bandwidth. The of bandwidth on a link, e.g. greater than 5% of link bandwidth. The
number of such flows would necessarily be fairly small, e.g. on the number of such flows would necessarily be fairly small, e.g. on the
order of 10's or 100's per link. In other words, the number of long- order of 10's or 100's per LAG/ECMP. In other words, the number of
lived large flows is NOT expected to be on the order of millions of large flows is NOT expected to be on the order of millions of flows.
flows. Examples of such long-lived large flows would be IPSec Examples of such large flows would be IPSec tunnels in service
tunnels in service provider backbones or storage backup traffic in provider backbone networks or storage backup traffic in data center
data center networks. networks.
1.1. Acronyms 1.1. Acronyms
COTS: Commercial Off-the-shelf COTS: Commercial Off-the-shelf
DOS: Denial of Service DOS: Denial of Service
ECMP: Equal Cost Multi-path ECMP: Equal Cost Multi-path
GRE: Generic Routing Encapsulation GRE: Generic Routing Encapsulation
skipping to change at page 4, line 35 skipping to change at page 4, line 35
STT: Stateless Transport Tunneling STT: Stateless Transport Tunneling
TCAM: Ternary Content Addressable Memory TCAM: Ternary Content Addressable Memory
VXLAN: Virtual Extensible LAN VXLAN: Virtual Extensible LAN
1.2. Terminology 1.2. Terminology
Large flow(s): long-lived large flow(s) Large flow(s): long-lived large flow(s)
Small flow(s): long-lived small flow(s) and short-lived small/large Small flow(s): long-lived small flow(s), short-lived small flows, and
flow(s) short-lived large flow(s)
2. Flow Categorization 2. Flow Categorization
In general, based on the size and duration, a flow can be categorized In general, based on the size and duration, a flow can be categorized
into any one of the following four types, as shown in Figure 1: into any one of the following four types, as shown in Figure 1:
(a) Short-Lived Large Flow (SLLF), (a) Short-lived Large Flow (SLLF),
(b) Short-Lived Small Flow (SLSF), (b) Short-lived Small Flow (SLSF),
(c) Long-Lived Large Flow (LLLF), and (c) Long-lived Large Flow (LLLF), and
(d) Long-Lived Small Flow (LLSF). (d) Long-lived Small Flow (LLSF).
Flow Size Flow Size
^ ^
|--------------------|--------------------| |--------------------|--------------------|
| | | | | |
Large | SLLF | LLLF | Large | SLLF | LLLF |
Flow | | | Flow | | |
|--------------------|--------------------| |--------------------|--------------------|
| | | | | |
Small | SLSF | LLSF | Small | SLSF | LLSF |
Flow | | | Flow | | |
+--------------------+--------------------+---> Flow duration +--------------------+--------------------+---> Flow Duration
Short-Lived Long-Lived Short-lived Long-lived
Flow Flow Flow Flow
Figure 1: Flow Categorization Figure 1: Flow Categorization
In this document, we categorize Long-lived large flow(s) as "Large" In this document, as mentioned earlier, we categorize long-lived large
flow(s), and all of the others -- Long-lived small flow(s) and short- flows as "large flows", and all of the others -- long-lived small flows,
lived small/large flow(s) as "Small" flow(s). short-lived small flows, and short-lived large flows as "small flows".
3. Hash-based Load Distribution in LAG/ECMP 3. Hash-based Load Distribution in LAG/ECMP
Hashing techniques are often used for traffic load balancing to Hashing techniques are often used for traffic load balancing to
select among multiple available paths with LAG/ECMP. The advantages select among multiple available paths with LAG/ECMP. The advantages
of hash-based load distribution are the preservation of the packet of hash-based load distribution are the preservation of the packet
sequence in a flow and the real-time distribution without maintaining sequence in a flow and the real-time distribution without maintaining
per-flow state in the router. Hash-based techniques use a combination per-flow state in the router. Hash-based techniques use a combination
of fields in the packet's headers to identify a flow, and the hash of fields in the packet's headers to identify a flow, and the hash
function on these fields is used to generate a unique number that function on these fields is used to generate a unique number that
identifies a link/path in a LAG/ECMP. The result of the hashing identifies a link/path in a LAG/ECMP. The result of the hashing
procedure is a many-to-one mapping of flows to component links. procedure is a many-to-one mapping of flows to component links.
If the traffic load constitutes flows such that the result of the If the traffic mix constitutes flows such that the result of the hash
hash function across these flows is fairly uniform so that a similar function across these flows is fairly uniform so that a similar
number of flows is mapped to each component link, if, the individual number of flows is mapped to each component link, if the individual
flow rates are much smaller as compared to the link capacity, and if flow rates are much smaller as compared to the link capacity, and if
the rate differences are not dramatic, the hash-based algorithm the rate differences are not dramatic, the hash-based algorithm
produces good results with respect to utilization of the individual produces good results with respect to utilization of the individual
component links. However, if one or more of these conditions are not component links. However, if one or more of these conditions are not
met, hash-based techniques may result in unbalanced loads on met, hash-based techniques may result in unbalanced loads on
individual component links. individual component links.
One example is illustrated in Figure 2. In Figure 2, there are two One example is illustrated in Figure 2. In Figure 2, there are two
routers, R1 and R2, and there is a LAG between them which has 3 routers, R1 and R2, and there is a LAG between them which has 3
component links (1), (2), (3). There are a total of 10 flows that component links (1), (2), (3). There are a total of 10 flows that
skipping to change at page 7, line 5 skipping to change at page 6, line 29
o The absence of any large flow causes the component link o The absence of any large flow causes the component link
under-utilized. under-utilized.
. Component link (3) has 4 flows -- 2 small flows and 2 large . Component link (3) has 4 flows -- 2 small flows and 2 large
flows -- and the link capacity is exceeded resulting in flows -- and the link capacity is exceeded resulting in
congestion. congestion.
o The presence of 2 large flows causes congestion on this o The presence of 2 large flows causes congestion on this
component link. component link.
+-----------+ +-----------+ +-----------+ -> +-----------+
| | -> -> | | | | -> | |
| |=====> | | | | ===> | |
| (1)|--/---/-|(1) | | (1)|--------|(1) |
| | | | | | -> | |
| | | | | | -> | |
| (R1) |-> -> ->| (R2) | | (R1) | -> | (R2) |
| (2)|--/---/-|(2) | | (2)|--------|(2) |
| | | | | | -> | |
| | -> -> | | | | -> | |
| |=====> | | | | ===> | |
| |=====> | | | | ===> | |
| (3)|--/---/-|(3) | | (3)|--------|(3) |
| | | | | | | |
+-----------+ +-----------+ +-----------+ +-----------+
Where: ->-> small flows Where: -> small flow
===> large flow ===> large flow
Figure 2: Unevenly Utilized Component Links Figure 2: Unevenly Utilized Component Links
This document presents improved load distribution techniques based on This document presents improved load distribution techniques based on
the large flow awareness. The techniques compensate for unbalanced the large flow awareness. The techniques compensate for unbalanced
load distribution resulting from hashing as demonstrated in the above load distribution resulting from hashing as demonstrated in the above
example. example.
4. Mechanisms for Optimal LAG/ECMP Component Link Utilization 4. Mechanisms for Optimal LAG/ECMP Component Link Utilization
skipping to change at page 8, line 46 skipping to change at page 8, line 11
component links that are part of the ECMP group may impact traffic component links that are part of the ECMP group may impact traffic
patterns at all of the nodes that are downstream of the given router patterns at all of the nodes that are downstream of the given router
between itself and the destination. The local optimization may between itself and the destination. The local optimization may
result in congestion at a downstream node. (In its simplest form, an result in congestion at a downstream node. (In its simplest form, an
ECMP group may be used to distribute traffic on component links that ECMP group may be used to distribute traffic on component links that
are between two adjacent routers, and in that case, the ECMP group is are between two adjacent routers, and in that case, the ECMP group is
no different than a LAG for the purpose of this discussion.) no different than a LAG for the purpose of this discussion.)
To demonstrate the limitations of local optimization, consider a two- To demonstrate the limitations of local optimization, consider a two-
level fat-tree topology with three leaf nodes (L1, L2, L3) and two level fat-tree topology with three leaf nodes (L1, L2, L3) and two
spine nodes (S1, S2) and assume all of the links are 10 Gbps. Let L1 spine nodes (S1, S2) and assume all of the links are 10 Gbps.
have two flows of 4 Gbps each towards L3, and let L2 have one flow of
7 Gbps also towards L3. If L1 balances the load optimally between S1 +-----+ +-----+
and S2, and L2 sends the flow via S1, then the downlink from S1 to L3 | S1 | | S2 |
would get congested resulting in packet discards. On the other hand, +-----+ +-----+
if L1 had sent both its flows towards S1 and L2 had sent its flow / \ \ / /\
towards S2, there would have been no congestion at either S1 or S2. / +---------+ / \
/ / \ \ / \
/ / \ +------+ \
/ / \ / \ \
+-----+ +-----+ +-----+
| L1 | | L2 | | L3 |
+-----+ +-----+ +-----+
Figure 3: Two Level Fat-tree
Let L1 have two flows of 4 Gbps each towards L3, and let L2 have one
flow of 7 Gbps also towards L3. If L1 balances the load optimally
between S1 and S2, and L2 sends the flow via S1, then the downlink
from S1 to L3 would get congested resulting in packet discards. On
the other hand, if L1 had sent both its flows towards S1 and L2 had
sent its flow towards S2, there would have been no congestion at
either S1 or S2.
The other issue with applying this scheme to ECMP groups is that it The other issue with applying this scheme to ECMP groups is that it
may not apply equally to unicast and multicast traffic because of the may not apply equally to unicast and multicast traffic because of the
way multicast trees are constructed. way multicast trees are constructed.
4.2. Overview of the mechanism 4.2. Overview of the mechanism
The various steps in achieving optimal LAG/ECMP component link The various steps in achieving optimal LAG/ECMP component link
utilization in networks are detailed below: utilization in networks are detailed below:
skipping to change at page 10, line 14 skipping to change at page 9, line 46
multi-node visibility. Consider the following example. A router may multi-node visibility. Consider the following example. A router may
have 3 ECMP nexthops that lead down paths P1, P2, and P3. A couple have 3 ECMP nexthops that lead down paths P1, P2, and P3. A couple
of hops downstream on P1 may be congested, while P2 and P3 may be of hops downstream on P1 may be congested, while P2 and P3 may be
under-utilized, which the local router does not have visibility into. under-utilized, which the local router does not have visibility into.
With the help of a central management entity, the operator could With the help of a central management entity, the operator could
redistribute some of the flows from P1 to P2 and P3 resulting in a redistribute some of the flows from P1 to P2 and P3 resulting in a
more optimized flow of traffic. more optimized flow of traffic.
The techniques described above are especially useful when bundling The techniques described above are especially useful when bundling
links of different bandwidths for e.g. 10Gbps and 100Gbps as links of different bandwidths for e.g. 10Gbps and 100Gbps as
described in [I-D.ietf-rtgwg-cl-requirement]. described in [ID.ietf-rtgwg-cl-requirement].
4.3. Large Flow Recognition 4.3. Large Flow Recognition
4.3.1. Flow Identification 4.3.1. Flow Identification
A flow (large flow or small flow) can be defined as a sequence of A flow (large flow or small flow) can be defined as a sequence of
packets for which ordered delivery should be maintained. Flows are packets for which ordered delivery should be maintained. Flows are
typically identified using one or more fields from the packet header typically identified using one or more fields from the packet header,
from the following list: for example:
. Layer 2: source MAC address, destination MAC address, VLAN ID. . Layer 2: source MAC address, destination MAC address, VLAN ID.
. IP header: IP Protocol, IP source address, IP destination . IP header: IP Protocol, IP source address, IP destination
address, flow label (IPv6 only), TCP/UDP source port, TCP/UDP address, flow label (IPv6 only), TCP/UDP source port, TCP/UDP
destination port. destination port.
. MPLS Labels. . MPLS Labels.
For tunneling protocols like GRE, VXLAN, NVGRE, STT, etc., flow For tunneling protocols like GRE, VXLAN, NVGRE, STT, etc., flow
identification is possible based on inner and/or outer headers. The identification is possible based on inner and/or outer headers. The
above list is not exhaustive. The mechanisms described in this above list is not exhaustive. The mechanisms described in this
document are agnostic to the fields that are used for flow document are agnostic to the fields that are used for flow
identification. identification.
This definition of flows is consistent with that in IPFIX [RFC 7011].
4.3.2. Criteria for Identifying a Large Flow 4.3.2. Criteria for Identifying a Large Flow
From a bandwidth and time duration perspective, in order to identify From a bandwidth and time duration perspective, in order to identify
large flows we define an observation interval and observe the large flows we define an observation interval and observe the
bandwidth of the flow over that interval. A flow that exceeds a bandwidth of the flow over that interval. A flow that exceeds a
certain minimum bandwidth threshold over that observation interval certain minimum bandwidth threshold over that observation interval
would be considered a large flow. would be considered a large flow.
The two parameters -- the observation interval, and the minimum The two parameters -- the observation interval, and the minimum
bandwidth threshold over that observation interval -- should be bandwidth threshold over that observation interval -- should be
skipping to change at page 14, line 26 skipping to change at page 14, line 15
. The PBR rules for large flows (refer to Section 4.4.1) must . The PBR rules for large flows (refer to Section 4.4.1) must
have strict precedence over the LAG/ECMP table lookup result. have strict precedence over the LAG/ECMP table lookup result.
With this approach the small flows that are moved would be subject to With this approach the small flows that are moved would be subject to
reordering. reordering.
4.4.3. Component Link Protection Considerations 4.4.3. Component Link Protection Considerations
If desired, certain component links may be reserved for link If desired, certain component links may be reserved for link
protection. These reserved component links are not used for any flows protection. These reserved component links are not used for any flows
in the absence of any failures.. In the case when the component in the absence of any failures. In the case when the component
link(s) fail, all the flows on the failed component link(s) are moved link(s) fail, all the flows on the failed component link(s) are moved
to the reserved component link(s). The mapping table of large flows to the reserved component link(s). The mapping table of large flows
to component link simply replaces the failed component link with the to component link simply replaces the failed component link with the
reserved link. Likewise, the LAG/ECMP hash table replaces the failed reserved link. Likewise, the LAG/ECMP hash table replaces the failed
component link with the reserved link. component link with the reserved link.
4.4.4. Load Re-balancing Algorithms 4.4.4. Load Re-balancing Algorithms
Specific algorithms for placement of large flows are out of scope of Specific algorithms for placement of large flows are out of scope of
this document. One possibility is to formulate the problem for large this document. One possibility is to formulate the problem for large
flow placement as the well-known bin-packing problem and make use of flow placement as the well-known bin-packing problem and make use of
the various heuristics that are available for that problem [bin- the various heuristics that are available for that problem [bin-
pack]. pack].
4.4.5. Load Re-Balancing Example 4.4.5. Load Re-Balancing Example
Optimal LAG/ECMP component utilization for the use case in Figure 2 Optimal LAG/ECMP component utilization for the use case in Figure 2
is depicted below in Figure 3. The large flow rebalancing explained is depicted below in Figure 4. The large flow rebalancing explained
in Section 4.4 is used. The improved link utilization is as follows: in Section 4.4 is used. The improved link utilization is as follows:
. Component link (1) has 3 flows -- 2 small flows and 1 large . Component link (1) has 3 flows -- 2 small flows and 1 large
flow -- and the link utilization is normal. flow -- and the link utilization is normal.
. Component link (2) has 4 flows -- 3 small flows and 1 large . Component link (2) has 4 flows -- 3 small flows and 1 large
flow -- and the link utilization is normal now. flow -- and the link utilization is normal now.
. Component link (3) has 3 flows -- 2 small flows and 1 large . Component link (3) has 3 flows -- 2 small flows and 1 large
flow -- and the link utilization is normal now. flow -- and the link utilization is normal now.
+-----------+ +-----------+ +-----------+ -> +-----------+
| | -> -> | | | | -> | |
| |=====> | | | | ===> | |
| (1)|--/---/-|(1) | | (1)|--------|(1) |
| | | |
| |=====> | |
| (R1) |-> -> ->| (R2) |
| (2)|--/---/-|(2) |
| | | | | | | |
| | ===> | |
| | -> | |
| | -> | |
| (R1) | -> | (R2) |
| (2)|--------|(2) |
| | | | | | | |
| | -> -> | | | | -> | |
| |=====> | | | | -> | |
| (3)|--/---/-|(3) | | | ===> | |
| (3)|--------|(3) |
| | | | | | | |
+-----------+ +-----------+ +-----------+ +-----------+
Where: ->-> small flows Where: -> small flows
===> large flow ===> large flow
Figure 3: Evenly utilized Composite Links Figure 4: Evenly utilized Composite Links
Basically, the use of the mechanisms described in Section 4.4.1 Basically, the use of the mechanisms described in Section 4.4.1
resulted in a rebalancing of flows where one of the large flows on resulted in a rebalancing of flows where one of the large flows on
component link (3) which was previously congested was moved to component link (3) which was previously congested was moved to
component link (2) which was previously under-utilized. component link (2) which was previously under-utilized.
5. Information Model for Flow Re-balancing 5. Information Model for Flow Re-balancing
In order to support flow rebalancing in a router from an external In order to support flow rebalancing in a router from an external
system, the exchange of some information is necessary between the system, the exchange of some information is necessary between the
skipping to change at page 17, line 38 skipping to change at page 17, line 29
destination port. destination port.
. MPLS Labels. . MPLS Labels.
This list is not exhaustive. For example, with overlay protocols This list is not exhaustive. For example, with overlay protocols
such as VXLAN and NVGRE, fields from the outer and/or inner headers such as VXLAN and NVGRE, fields from the outer and/or inner headers
may be specified. In general, all fields in the packet that can be may be specified. In general, all fields in the packet that can be
used by forwarding decisions should be available for use when used by forwarding decisions should be available for use when
importing flow information from an external management station. importing flow information from an external management station.
The IPFIX information model [RFC 5101] can be leveraged for large The IPFIX information model [RFC 7011] can be leveraged for large
flow identification. The component link ID would be used to specify flow identification. The component link ID would be used to specify
the target component link for the flow. the target component link for the flow.
5.4. Information for Redistribution of Small Flows 5.4. Information for Redistribution of Small Flows
For small flows, the LAG ID and the component link IDs along with the For small flows, the LAG ID and the component link IDs along with the
percentage of traffic to be assigned to each component link ID Is percentage of traffic to be assigned to each component link ID Is
required. required.
5.5. Export of Flow Information 5.5. Export of Flow Information
Exporting large flow information is required when large flow Exporting large flow information is required when large flow
recognition is being done on a router, but the decision to rebalance recognition is being done on a router, but the decision to rebalance
is being made in an external management station. Large flow is being made in an external management station. Large flow
information includes flow identification and the component link ID information includes flow identification and the component link ID
that the flow currently is assigned to. Other information such as that the flow currently is assigned to. Other information such as
flow QoS and bandwidth may be exported too. flow QoS and bandwidth may be exported too.
The IPFIX information model [RFC 5101] can be leveraged for large The IPFIX information model [RFC 7011] can be leveraged for large
flow identification. flow identification.
5.6. Monitoring information 5.6. Monitoring information
5.6.1. Interface (link) utilization 5.6.1. Interface (link) utilization
The incoming bytes (ifInOctets), outgoing bytes (ifOutOctets) and The incoming bytes (ifInOctets), outgoing bytes (ifOutOctets) and
interface speed (ifSpeed) can be measured from the Interface table interface speed (ifSpeed) can be measured from the Interface table
(iftable) MIB [RFC 1213]. (iftable) MIB [RFC 1213].
skipping to change at page 18, line 39 skipping to change at page 18, line 32
For further scalability, it is recommended to use the counter push For further scalability, it is recommended to use the counter push
mechanism in [sflow-v5] for the interface counters; this would help mechanism in [sflow-v5] for the interface counters; this would help
avoid counter polling through the MIB interface. avoid counter polling through the MIB interface.
The outgoing link utilization of the component links within a LAG can The outgoing link utilization of the component links within a LAG can
be used to compute the imbalance threshold (See Section 5.1) for the be used to compute the imbalance threshold (See Section 5.1) for the
LAG. LAG.
5.6.2. Other monitoring information 5.6.2. Other monitoring information
Additional monitoring information includes: Additional monitoring information that is useful includes:
. Number of times rebalancing was done. . Number of times rebalancing was done.
. Time since the last rebalancing event. . Time since the last rebalancing event.
. The number of large flows currently rebalanced by the scheme.
. A list of the large flows that have been rebalanced including
o the rate of each large flow at the time of the last
rebalancing for that flow,
o the time that rebalancing was last performed for the given
large flow, and
o the interfaces that the large flows was (re)directed to.
. The settings for the weights of the interfaces within a
LAG/ECMP used by the small flows which depend on hashing.
6. Operational Considerations 6. Operational Considerations
6.1. Rebalancing Frequency 6.1. Rebalancing Frequency
Flows should be re-balanced only when the imbalance in the Flows should be re-balanced only when the imbalance in the
utilization across component links exceeds a certain threshold. utilization across component links exceeds a certain threshold.
Frequent re-balancing to achieve precise equitable utilization across Frequent re-balancing to achieve precise equitable utilization across
component links could be counter-productive as it may result in component links could be counter-productive as it may result in
moving flows back and forth between the component links impacting moving flows back and forth between the component links impacting
packet ordering and system stability. This applies regardless of packet ordering and system stability. This applies regardless of
skipping to change at page 20, line 14 skipping to change at page 20, line 12
heavy overloading of large flows to certain LAG/ECMP component heavy overloading of large flows to certain LAG/ECMP component
links. links.
9. Acknowledgements 9. Acknowledgements
The authors would like to thank the following individuals for their The authors would like to thank the following individuals for their
review and valuable feedback on earlier versions of this document: review and valuable feedback on earlier versions of this document:
Shane Amante, Curtis Villamizar, Fred Baker, Wes George, Brian Shane Amante, Curtis Villamizar, Fred Baker, Wes George, Brian
Carpenter, George Yum, Michael Fargano, Michael Bugenhagen, Jianrong Carpenter, George Yum, Michael Fargano, Michael Bugenhagen, Jianrong
Wong, Peter Phaal, Roman Krzanowski, Weifeng Zhang, Pete Moyer, Wong, Peter Phaal, Roman Krzanowski, Weifeng Zhang, Pete Moyer,
Andrew Malis, Dave McDysan, Zhen Cao, and Dan Romascanu. Andrew Malis, Dave McDysan, Zhen Cao, Dan Romascanu, and Benoit
Claise.
10. References 10. References
10.1. Normative References 10.1. Normative References
10.2. Informative References 10.2. Informative References
[I-D.ietf-rtgwg-cl-requirement] Villamizar, C. et al., "Requirements [802.1AX] IEEE Standards Association, "IEEE Std 802.1AX-2008 IEEE
for MPLS over a Composite Link," September 2013. Standard for Local and Metropolitan Area Networks - Link
Aggregation", 2008.
[RFC 6790] Kompella, K. et al., "The Use of Entropy Labels in MPLS [bin-pack] Coffman, Jr., E., M. Garey, and D. Johnson. Approximation
Forwarding," November 2012. Algorithms for Bin-Packing -- An Updated Survey. In Algorithm Design
for Computer System Design, ed. by Ausiello, Lucertini, and Serafini.
Springer-Verlag, 1984.
[CAIDA] Caida Internet Traffic Analysis, http://www.caida.org/home. [CAIDA] Caida Internet Traffic Analysis, http://www.caida.org/home.
[YONG] Yong, L., "Enhanced ECMP and Large Flow Aware Transport," [DevoFlow] Mogul, J., et al., "DevoFlow: Cost-Effective Flow
draft-yong-pwe3-enhance-ecmp-lfat-01, September 2010. Management for High Performance Enterprise Networks," Proceedings of
the ACM SIGCOMM, August 2011.
[ID.ietf-rtgwg-cl-requirement] Villamizar, C. et al., "Requirements
for MPLS over a Composite Link," September 2013.
[ITCOM] Jo, J., et al., "Internet traffic load balancing using [ITCOM] Jo, J., et al., "Internet traffic load balancing using
dynamic hashing with flow volume," SPIE ITCOM, 2002. dynamic hashing with flow volume," SPIE ITCOM, 2002.
[NDTM] Estan, C. and G. Varghese, "New directions in traffic
measurement and accounting," Proceedings of ACM SIGCOMM, August 2002.
[RFC 2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and [RFC 2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and
Multicast," November 2000. Multicast," November 2000.
[RFC 2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path [RFC 6790] Kompella, K. et al., "The Use of Entropy Labels in MPLS
Algorithm," November 2000. Forwarding," November 2012.
[RFC 5475] Zseby, T., et al., "Sampling and Filtering Techniques for [RFC 1213] McCloghrie, K., "Management Information Base for Network
IP Packet Selection," March 2009. Management of TCP/IP-based internets: MIB-II," March 1991.
[sFlow-v5] Phaal, P. and M. Lavine, "sFlow version 5," July 2004. [RFC 2992] Hopps, C., "Analysis of an Equal-Cost Multi-Path
Algorithm," November 2000.
[sFlow-LAG] Phaal, P. and A. Ghanwani, "sFlow LAG counters [RFC 3273] Waldbusser, S., "Remote Network Monitoring Management
structure," September 2012. Information Base for High Capacity Networks," July 2002.
[RFC 3954] Claise, B., "Cisco Systems NetFlow Services Export Version [RFC 3954] Claise, B., "Cisco Systems NetFlow Services Export Version
9," October 2004 9," October 2004.
[RFC 5101] Claise, B., "Specification of the IP Flow Information
Export (IPFIX) Protocol for the Exchange of IP Traffic Flow
Information," January 2008
[RFC 1213] McCloghrie, K., "Management Information Base for Network [RFC 5475] Zseby T., et al., "Sampling and Filtering Techniques for
Management of TCP/IP-based internets: MIB-II," March 1991. IP Packet Selection," March 2009.
[RFC 3273] Waldbusser, S., "Remote Network Monitoring Management [RFC 7011] Claise, B., "Specification of the IP Flow Information
Information Base for High Capacity Networks," July 2002. Export (IPFIX) Protocol for the Exchange of IP Traffic Flow
Information," September 2013.
[DevoFlow] Mogul, J., et al., "DevoFlow: Cost-Effective Flow [sFlow-LAG] Phaal, P. and A. Ghanwani, "sFlow LAG counters
Management for High Performance Enterprise Networks," Proceedings of structure," http://www.sflow.org/sflow_lag.txt, September 2012.
the ACM SIGCOMM, August 2011.
[NDTM] Estan, C. and G. Varghese, "New directions in traffic [sFlow-v5] Phaal, P. and M. Lavine, "sFlow version 5,"
measurement and accounting," Proceedings of ACM SIGCOMM, August 2002. http://www.sflow.org/sflow_version_5.txt, July 2004.
[bin-pack] Coffman, Jr., E., M. Garey, and D. Johnson. Approximation [YONG] Yong, L., "Enhanced ECMP and Large Flow Aware Transport,"
Algorithms for Bin-Packing -- An Updated Survey. In Algorithm Design draft-yong-pwe3-enhance-ecmp-lfat-01, September 2010.
for Computer System Design, ed. by Ausiello, Lucertini, and Serafini.
Springer-Verlag, 1984.
Appendix A. Internet Traffic Analysis and Load Balancing Simulation Appendix A. Internet Traffic Analysis and Load Balancing Simulation
Internet traffic [CAIDA] has been analyzed to obtain flow statistics Internet traffic [CAIDA] has been analyzed to obtain flow statistics
such as the number of packets in a flow and the flow duration. The such as the number of packets in a flow and the flow duration. The
five tuples in the packet header (IP addresses, TCP/UDP Ports, and IP five tuples in the packet header (IP addresses, TCP/UDP Ports, and IP
protocol) are used for flow identification. The analysis indicates protocol) are used for flow identification. The analysis indicates
that < ~2% of the flows take ~30% of total traffic volume while the that < ~2% of the flows take ~30% of total traffic volume while the
rest of the flows (> ~98%) contributes ~70% [YONG]. rest of the flows (> ~98%) contributes ~70% [YONG].
skipping to change at page 22, line 28 skipping to change at page 22, line 28
proves how large flow-aware hash-based distribution can effectively proves how large flow-aware hash-based distribution can effectively
compensate the uneven load balancing caused by hashing and the compensate the uneven load balancing caused by hashing and the
traffic characteristics [YONG]. traffic characteristics [YONG].
Authors' Addresses Authors' Addresses
Ram Krishnan Ram Krishnan
Brocade Communications Brocade Communications
San Jose, 95134, USA San Jose, 95134, USA
Phone: +1-408-406-7890 Phone: +1-408-406-7890
Email: ramk@brocade.com Email: ramkri123@gmail.com
Lucy Yong Lucy Yong
Huawei USA Huawei USA
5340 Legacy Drive 5340 Legacy Drive
Plano, TX 75025, USA Plano, TX 75025, USA
Phone: +1-469-277-5837 Phone: +1-469-277-5837
Email: lucy.yong@huawei.com Email: lucy.yong@huawei.com
Anoop Ghanwani Anoop Ghanwani
Dell Dell
skipping to change at page 23, line 15 skipping to change at page 23, line 15
Email: ning.so@tatacommunications.com Email: ning.so@tatacommunications.com
Sanjay Khanna Sanjay Khanna
Cisco Systems Cisco Systems
Email: sanjakha@gmail.com Email: sanjakha@gmail.com
Bhumip Khasnabish Bhumip Khasnabish
ZTE Corporation ZTE Corporation
New Jersey, 07960, USA New Jersey, 07960, USA
Phone: +1-781-752-8003 Phone: +1-781-752-8003
Email: bhumip.khasnabish@zteusa.com Email: vumip1@gmail.com
 End of changes. 52 change blocks. 
121 lines changed or deleted 166 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/