draft-ietf-mptcp-experience-01.txt   draft-ietf-mptcp-experience-02.txt 
MPTCP Working Group O. Bonaventure MPTCP Working Group O. Bonaventure
Internet-Draft C. Paasch Internet-Draft C. Paasch
Intended status: Informational UCLouvain Intended status: Informational UCLouvain
Expires: September 9, 2015 G. Detal Expires: January 7, 2016 G. Detal
UCLouvain and Tessares UCLouvain and Tessares
March 08, 2015 July 06, 2015
Experience with Multipath TCP Use Cases and Operational Experience with Multipath TCP
draft-ietf-mptcp-experience-01 draft-ietf-mptcp-experience-02
Abstract Abstract
This document discusses operational experiences of using Multipath This document discusses both use cases and operational experience
TCP in real world networks. It lists several prominent use cases for with Multipath TCP in real world networks. It lists several
which Multipath TCP has been considered and is being used. It also prominent use cases for which Multipath TCP has been considered and
gives insight in some heuristics and decisions that have helped to is being used. It also gives insight to some heuristics and
realize these use cases. Further, it presents several open issues decisions that have helped to realize these use cases.
that are yet unclear on how they can be solved.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 9, 2015. This Internet-Draft will expire on January 7, 2016.
Copyright Notice Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Middlebox interference . . . . . . . . . . . . . . . . . . . 3 2. Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1. Datacenters . . . . . . . . . . . . . . . . . . . . . . . 3
4. Congestion control . . . . . . . . . . . . . . . . . . . . . 8 2.2. Cellular/WiFi Offload . . . . . . . . . . . . . . . . . . 4
5. Subflow management . . . . . . . . . . . . . . . . . . . . . 9 2.3. Multipath TCP proxies . . . . . . . . . . . . . . . . . . 7
5.1. Implemented subflow managers . . . . . . . . . . . . . . 9 3. Operational Experience . . . . . . . . . . . . . . . . . . . 8
5.2. Subflow destination port . . . . . . . . . . . . . . . . 11 3.1. Middlebox interference . . . . . . . . . . . . . . . . . 8
5.3. Closing subflows . . . . . . . . . . . . . . . . . . . . 12 3.2. Congestion control . . . . . . . . . . . . . . . . . . . 10
6. Packet schedulers . . . . . . . . . . . . . . . . . . . . . . 13 3.3. Subflow management . . . . . . . . . . . . . . . . . . . 11
7. Segment size selection . . . . . . . . . . . . . . . . . . . 14 3.4. Implemented subflow managers . . . . . . . . . . . . . . 11
8. Interactions with the Domain Name System . . . . . . . . . . 15 3.5. Subflow destination port . . . . . . . . . . . . . . . . 13
9. Captive portals . . . . . . . . . . . . . . . . . . . . . . . 15 3.6. Closing subflows . . . . . . . . . . . . . . . . . . . . 14
10. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 16 4. Packet schedulers . . . . . . . . . . . . . . . . . . . . . . 15
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 16 5. Segment size selection . . . . . . . . . . . . . . . . . . . 16
12. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 16 6. Interactions with the Domain Name System . . . . . . . . . . 16
13. Informative References . . . . . . . . . . . . . . . . . . . 17 7. Captive portals . . . . . . . . . . . . . . . . . . . . . . . 17
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 8. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 18
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 18
10. Informative References . . . . . . . . . . . . . . . . . . . 18
Appendix A. Changelog . . . . . . . . . . . . . . . . . . . . . 23
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23
1. Introduction 1. Introduction
Multipath TCP was standardized in [RFC6824] and four implementations Multipath TCP was standardized in [RFC6824] and four implementations
have been developed [I-D.eardley-mptcp-implementations-survey]. have been developed [I-D.eardley-mptcp-implementations-survey].
Since the publication of [RFC6824], some experience has been gathered Since the publication of [RFC6824], experience has been gathered by
by various network researchers and users about the issues that arise various network researchers and users about the operational issues
when Multipath TCP is used in the Internet. that arise when Multipath TCP is used in today's Internet.
Most of the experience reported in this document comes from the When the MPTCP working group was created, several use cases for
utilization of the Multipath TCP implementation in the Linux kernel Multipath TCP were identified [RFC6182]. Since then, over use cases
[MultipathTCP-Linux]. It has been downloaded and is used by have been proposed and some have been tested and even deployed. We
thousands of users all over the world. Many of these users have describe these use cases in section Section 2.
provided direct or indirect feedback by writing documents (scientific
articles or blog messages) or posting to the mptcp-dev mailing list (
https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev ) . This
Multipath TCP implementation is actively maintained and continuously
improved. It is used on various types of hosts, ranging from
smartphones or embedded systems to high-end servers.
This is not, by far, the most widespread deployment of Multipath TCP. The second part of the document focuses on the operational experience
Since September 2013, Multipath TCP is also supported on smartphones with Multipath TCP. Most of this experience comes from the
and tablets running iOS7 [IOS7]. There are likely hundreds of utilisation of the Multipath TCP implementation in the Linux kernel
millions of Multipath TCP enabled devices. However, this particular [MultipathTCP-Linux]. This open-source implementation has been
Multipath TCP implementation is currently only used to support a downloaded and is used by thousands of users all over the world.
single application. Unfortunately, there is no public information Many of these users have provided direct or indirect feedback by
about the lessons learned from this large scale deployment. writing documents (scientific articles or blog messages) or posting
to the mptcp-dev mailing list (see https://listes-
2.sipr.ucl.ac.be/sympa/arc/mptcp-dev ). This Multipath TCP
implementation is actively maintained and continuously improved. It
is used on various types of hosts, ranging from smartphones or
embedded routers to high-end servers.
This document is organized as follows. We explain in The Multipath TCP implementation in the Linux kernel is is not, by
Section Section 2 which types of middleboxes the Linux Kernel far, the most widespread deployment of Multipath TCP. Since
September 2013, Multipath TCP is also supported on smartphones and
tablets running iOS7 [IOS7]. There are likely hundreds of millions
of Multipath TCP enabled devices. However, this particular Multipath
TCP implementation is currently only used to support a single
application. Unfortunately, there is no public information about the
lessons learned from this large scale deployment.
The second part of this is document is organized as follows.
Supporting the middleboxes was one of the difficult issues in
designing the Multipath TCP protocol. We explain in section
Section 3.1 which types of middleboxes the Linux Kernel
implementation of Multipath TCP supports and how it reacts upon implementation of Multipath TCP supports and how it reacts upon
encountering these. Next, we list several use cases of Multipath TCP encountering these. Section Section 3.2 summarises the MPTCP
in Section {{usecases}. Section {{congestion} summarises the MPTCP
specific congestion controls that have been implemented. Sections specific congestion controls that have been implemented. Sections
Section 5 and Section 6 discuss heuristics and issues with respect to Section 3.3 and Section 4 discuss heuristics and issues with respect
subflow management as well as the scheduling across the subflows. to subflow management as well as the scheduling across the subflows.
Section Section 7 explains some problems that occurred with subflows Section Section 5 explains some problems that occurred with subflows
having different MSS values. Section Section 8 presents issues with having different MSS values. Section Section 6 presents issues with
respect to content delivery networks and suggests a solution to this respect to content delivery networks and suggests a solution to this
issue. Finally, Section Section 9 shows an issue with captive issue. Finally, section Section 7 documents an issue with captive
portals where MPTCP will behave suboptimal. portals where MPTCP will behave suboptimal.
2. Middlebox interference 2. Use cases
The interference caused by various types of middleboxes has been an
important concern during the design of the Multipath TCP protocol.
Three studies on the interactions between Multipath TCP and
middleboxes are worth being discussed.
The first analysis was described in [IMC11]. This paper was the main
motivation for including inside Multipath TCP various techniques to
cope with middlebox interference. More specifically, Multipath TCP
has been designed to cope with middleboxes that : - change source or
destination addresses - change source or destination port numbers -
change TCP sequence numbers - split or coalesce segments - remove TCP
options - modify the payload of TCP segments
These middlebox interferences have all been included in the MBtest
suite [MBTest]. This test suite has been used [HotMiddlebox13] to
verify the reaction of the Multipath TCP implementation in the Linux
kernel when faced with middlebox interference. The test environment
used for this evaluation is a dual-homed client connected to a
single-homed server. The middlebox behavior can be activated on any
of the paths. The main results of this analysis are :
o the Multipath TCP implementation in the Linux kernel is not
affected by a middlebox that performs NAT or modifies TCP sequence
numbers
o when a middlebox removes the MP_CAPABLE option from the initial
SYN segment, the Multipath TCP implementation in the Linux kernel
falls back correctly to regular TCP
o when a middlebox removes the DSS option from all data segments,
the Multipath TCP implementation in the Linux kernel falls back
correctly to regular TCP
o when a middlebox performs segment coalescing, the Multipath TCP
implementation in the Linux kernel is still able to accurately
extract the data corresponding to the indicated mapping
o when a middlebox performs segment splitting, the Multipath TCP
implementation in the Linux kernel correctly reassembles the data
corresponding to the indicated mapping. [HotMiddlebox13]
documents a corner case with segment splitting that may lead to
desynchronisation between the two hosts.
The interactions between Multipath TCP and real deployed middleboxes
is also analyzed in [HotMiddlebox13] and a particular scenario with
the FTP application level gateway running on a NAT is described.
From an operational viewpoint, knowing that Multipath TCP can cope
with various types of middlebox interference is important. However,
there are situations where the network operators need to gather
information about where a particular middlebox interference occurs.
The tracebox software [tracebox] described in [IMC13a] is an
extension of the popular traceroute software that enables network
operators to check at which hop a particular field of the TCP header
(including options) is modified. It has been used by several network
operators to debug various middlebox interference problems. tracebox
includes a scripting language that enables its user to specify
precisely which packet is sent by the source. tracebox sends packets
with an increasing TTL/HopLimit and compares the information returned
in the ICMP messages with the packet that it sends. This enables
tracebox to detect any interference caused by middleboxes on a given
path. tracebox works better when routers implement the ICMP extension
defined in [RFC1812].
Users of the Multipath TCP implementation have reported some
experience with middlebox interference. The strangest scenario has
been a middlebox that accepts the Multipath TCP options in the SYN
segment but later replaces Multipath TCP options with a TCP EOL
option [StrangeMbox]. This causes Multipath TCP to perform a
fallback to regular TCP without any impact on the application.
3. Use cases Multipath TCP has been tested in several use cases. There is already
an abundant scientific literature on Multipath TCP [MPTCPBIB].
Several of the papers published in the scientific litterature have
identified possible improvements that are worth being discussed here.
Multipath TCP has been tested in several use cases. Several of the 2.1. Datacenters
papers published in the scientific litterature have identified
possible improvements that are worth being discussed here.
A first, although initially unexpected, documented use case for A first, although initially unexpected, documented use case for
Multipath TCP has been the datacenters [HotNets][SIGCOMM11]. Today's Multipath TCP has been the datacenters [HotNets][SIGCOMM11]. Today's
datacenters are designed to provide several paths between single- datacenters are designed to provide several paths between single-
homed servers. The multiplicity of these paths comes from the homed servers. The multiplicity of these paths comes from the
utilization of Equal Cost Multipath (ECMP) and other load balancing utilization of Equal Cost Multipath (ECMP) and other load balancing
techniques inside the datacenter. Most of the deployed load techniques inside the datacenter. Most of the deployed load
balancing techniques in these datacenters rely on hashes computed or balancing techniques in these datacenters rely on hashes computed or
the five tuple to ensure that all packets from the same TCP the five tuple to ensure that all packets from the same TCP
connection will follow the same path to prevent packet reordering. connection will follow the same path to prevent packet reordering.
skipping to change at page 5, line 37 skipping to change at page 4, line 26
environment where there are different paths between a pair of hosts. environment where there are different paths between a pair of hosts.
ECMP and other load balancing techniques such as LAG are widely used ECMP and other load balancing techniques such as LAG are widely used
in today's network and having multiple paths between a pair of in today's network and having multiple paths between a pair of
single-homed hosts is becoming the norm instead of the exception. single-homed hosts is becoming the norm instead of the exception.
Although these multiple paths have often the same cost (from an IGP Although these multiple paths have often the same cost (from an IGP
metrics viewpoint), they do not necessarily have the same metrics viewpoint), they do not necessarily have the same
performance. For example, [IMC13c] reports the results of a long performance. For example, [IMC13c] reports the results of a long
measurement study showing that load balanced Internet paths between measurement study showing that load balanced Internet paths between
that same pair of hosts can have huge delay differences. that same pair of hosts can have huge delay differences.
2.2. Cellular/WiFi Offload
A second use case that has been explored by several network A second use case that has been explored by several network
researchers is the cellular/WiFi offload use case. Smartphones or researchers is the cellular/WiFi offload use case. Smartphones or
other mobile devices equipped with two wireless interfaces are a very other mobile devices equipped with two wireless interfaces are a very
common use case for Multipath TCP. As of this writing, this is also common use case for Multipath TCP. As of this writing, this is also
the largest deployment of Multipath-TCP enabled devices [IOS7]. the largest deployment of Multipath-TCP enabled devices [IOS7].
Unfortunately, as there are no public measurements about this Unfortunately, as there are no public measurements about this
deployment, we can only rely on published papers that have mainly deployment, we can only rely on published papers that have mainly
used the Multipath TCP implementation in the Linux kernel for their used the Multipath TCP implementation in the Linux kernel for their
experiment. experiments.
The performance of Multipath TCP in wireless networks was briefly The performance of Multipath TCP in wireless networks was briefly
evaluated in [NSDI12]. One experiment analyzes the performance of evaluated in [NSDI12]. One experiment analyzes the performance of
Multipath TCP on a client with two wireless interfaces. This Multipath TCP on a client with two wireless interfaces. This
evaluation shows that when the receive window is large, Multipath TCP evaluation shows that when the receive window is large, Multipath TCP
can efficiently use the two available links. However, if the window can efficiently use the two available links. However, if the window
becomes smaller, then packets sent on a slow path can block the becomes smaller, then packets sent on a slow path can block the
transmission of packets on a faster path. In some cases, the transmission of packets on a faster path. In some cases, the
performance of Multipath TCP over two paths can become lower than the performance of Multipath TCP over two paths can become lower than the
performance of regular TCP over the best performing path. Two performance of regular TCP over the best performing path. Two
skipping to change at page 7, line 4 skipping to change at page 5, line 44
interfaces are considered to be active. If the WiFi interface fails, interfaces are considered to be active. If the WiFi interface fails,
then the traffic switches quickly to the cellular interface, ensuring then the traffic switches quickly to the cellular interface, ensuring
a smooth handover from the user's viewpoint [Cellnet12]. The cost of a smooth handover from the user's viewpoint [Cellnet12]. The cost of
this approach is that the WiFi and cellular interfaces likely remain this approach is that the WiFi and cellular interfaces likely remain
active all the time since all subflows are established over the two active all the time since all subflows are established over the two
interfaces. interfaces.
The single-path mode is slightly different. This mode benefits from The single-path mode is slightly different. This mode benefits from
the break-before-make capability of Multipath TCP. When an MPTCP the break-before-make capability of Multipath TCP. When an MPTCP
session is established, a subflow is created over the WiFi interface. session is established, a subflow is created over the WiFi interface.
No packet is sent over the cellular interface as long as the WiFi No packet is sent over the cellular interface as long as the WiFi
interface remains up [Cellnet12]. This implies that the cellular interface remains up [Cellnet12]. This implies that the cellular
interface can remain idle and battery capacity is preserved. When interface can remain idle and battery capacity is preserved. When
the WiFi interface fails, new subflows are established over the the WiFi interface fails, new subflows are established over the
cellular interface in order to preserve the established Multipath TCP cellular interface in order to preserve the established Multipath TCP
sessions. Compared to the backup mode described earlier, this mode sessions. Compared to the backup mode described earlier, this mode
of operation is characterized by a throughput drop while the cellular of operation is characterised by a throughput drop while the cellular
interface is brought up and the subflows are reestablished. During interface is brought up and the subflows are reestablished. During
this time, no data packet is transmitted. this time, no data packet is transmitted.
From a protocol viewpoint, [Cellnet12] discusses the problem posed by From a protocol viewpoint, [Cellnet12] discusses the problem posed by
the unreliability of the ADD_ADDR option and proposes a small the unreliability of the ADD_ADDR option and proposes a small
protocol extension to allow hosts to reliably exchange this option. protocol extension to allow hosts to reliably exchange this option.
It would be useful to analyze packet traces to understand whether the It would be useful to analyze packet traces to understand whether the
unreliability of the REMOVE_ADDR option poses an operational problem unreliability of the REMOVE_ADDR option poses an operational problem
in real deployments. in real deployments.
skipping to change at page 7, line 36 skipping to change at page 6, line 28
Multipath TCP implementation in the Linux kernel that "MPTCP provides Multipath TCP implementation in the Linux kernel that "MPTCP provides
a robust data transport and reduces variations in download a robust data transport and reduces variations in download
latencies". latencies".
A different study of the performance of Multipath TCP with two A different study of the performance of Multipath TCP with two
wireless networks is presented in [INFOCOM14]. In this study the two wireless networks is presented in [INFOCOM14]. In this study the two
networks had different qualities : a good network and a lossy networks had different qualities : a good network and a lossy
network. When using two paths with different packet loss ratios, the network. When using two paths with different packet loss ratios, the
Multipath TCP congestion control scheme moves traffic away from the Multipath TCP congestion control scheme moves traffic away from the
lossy link that is considered to be congested. However, [INFOCOM14] lossy link that is considered to be congested. However, [INFOCOM14]
documents an interesting scenario that is summarised in the figure documents an interesting scenario that is summarised in the Figure 1.
below.
client ----------- path1 -------- server client ----------- path1 -------- server
| | | |
+--------------- path2 ------------+ +--------------- path2 ------------+
Figure 1: Simple network topology Figure 1: Simple network topology
Initially, the two paths have the same quality and Multipath TCP Initially, the two paths have the same quality and Multipath TCP
distributes the load over both of them. During the transfer, the distributes the load over both of them. During the transfer, the
second path becomes lossy, e.g. because the client moves. Multipath second path becomes lossy, e.g. because the client moves. Multipath
skipping to change at page 8, line 23 skipping to change at page 7, line 15
option is probably not required. When the client detects that the option is probably not required. When the client detects that the
data transmitted over the second subflow has been acknowledged over data transmitted over the second subflow has been acknowledged over
the first subflow, it could decide to terminate the second subflow by the first subflow, it could decide to terminate the second subflow by
sending a RST segment. If the interface associated to this subflow sending a RST segment. If the interface associated to this subflow
is still up, a new subflow could be immediately reestablished. It is still up, a new subflow could be immediately reestablished. It
would then be immediately usable to send new data and would not be would then be immediately usable to send new data and would not be
forced to first retransmit the previously transmitted data. As of forced to first retransmit the previously transmitted data. As of
this writing, this dynamic management of the subflows is not yet this writing, this dynamic management of the subflows is not yet
implemented in the Multipath TCP implementation in the Linux kernel. implemented in the Multipath TCP implementation in the Linux kernel.
A third use case has been the coupling between software defined 2.3. Multipath TCP proxies
networking techniques such as Openflow and Multipath TCP. Openflow
can be used to configure different paths inside a network. Using an
international network, [TNC13] demonstrates that Multipath TCP can
achieve high throughput in the wide area. An interesting point to
note about the measurements reported in [TNC13] is that the
measurement setup used four paths through the WAN. Only two of these
paths were disjoint. When Multipath TCP was used, the congestion
control scheme ensured that only two of these paths were actually
used.
4. Congestion control As Multipath TCP is not yet widely deployed on both clients and
servers, several deployments have used various forms of proxies. Two
families solutions are currently being used or tested
[I-D.deng-mptcp-proxy].
A first use case is when a Multipath TCP enabled client wants to use
several interfaces to reach a regular TCP server. A typical use case
is a smartphone that needs to use both its WiFi and its cellular
interface to transfer data. Several types of proxies are possible
for this use case. An HTTP proxy deployed on a Multipath TCP capable
server would enable the smartphone to use Multipath TCP to access
regular web servers. Obviously, this solution only works for
applications that rely on HTTP. Another possibility is to use a
proxy that can convert any Multipath TCP connection into a regular
TCP connection. The SOCKS protocol [RFC1928] is an example of such a
protocol. Other proxies have been proposed
[I-D.wei-mptcp-proxy-mechanism] [HotMiddlebox13b]. Measurements
performed with smartphones [Mobicom15] show that popular applications
work correctly through a SOCKS proxy and Multipath TCP enabled
smartphones. Thanks to Multipath TCP, long connections can be spread
over the two available interfaces. However, for short connections,
most of the data is sent over the initial subflow that is created
over the interface corresponding to the default route and the second
subflow is almost not used.
A second use case is when Multipath TCP is used by middleboxes,
typically inside access networks. Various network operators are
discussing and evaluating solutions for hybrid access networks
[BBF-WT348]. Such networks arise when a network operator controls
two different access network technologies, e.g. DSL and LTE, and
wants to combine them to improve the bandwidth offered to the
endusers [I-D.lhwxz-hybrid-access-network-architecture]. Several
solutions are currently investigated for such networks [BBF-WT348].
Figure 2 shows the organisation of such a network. When a client
creates a normal TCP connection, it is intercepted by the Hybrid CPE
(HPCE) that converts it in a Multipath TCP connection so that it can
use the available access networks (DSL and LTE in the example). The
Hybrid Access Gateway (HAG) does the opposite to ensure that the
regular server see a normal TCP connection. Some of the solutions
that are currently discussed for those hybrid networks use Multipath
TCP on the HCPE and the HAG. Other solutions rely on tunnels between
the HCPE and the HAG [I-D.lhwxz-gre-notifications-hybrid-access].
client --- HCPE ------ dsl ------- HAG --- internet --- server
| |
+------- lte -----------+
Figure 2: Hybrid Access Network
3. Operational Experience
3.1. Middlebox interference
The interference caused by various types of middleboxes has been an
important concern during the design of the Multipath TCP protocol.
Three studies on the interactions between Multipath TCP and
middleboxes are worth being discussed.
The first analysis was described in [IMC11]. This paper was the main
motivation for including inside Multipath TCP various techniques to
cope with middlebox interference. More specifically, Multipath TCP
has been designed to cope with middleboxes that :
o change source or destination addresses
o change source or destination port numbers
o change TCP sequence numbers
o split or coalesce segments
o remove TCP options
o modify the payload of TCP segments
These middlebox interferences have all been included in the MBtest
suite [MBTest]. This test suite has been used [HotMiddlebox13] to
verify the reaction of the Multipath TCP implementation in the Linux
kernel when faced with middlebox interference. The test environment
used for this evaluation is a dual-homed client connected to a
single-homed server. The middlebox behavior can be activated on any
of the paths. The main results of this analysis are :
o the Multipath TCP implementation in the Linux kernel is not
affected by a middlebox that performs NAT or modifies TCP sequence
numbers
o when a middlebox removes the MP_CAPABLE option from the initial
SYN segment, the Multipath TCP implementation in the Linux kernel
falls back correctly to regular TCP
o when a middlebox removes the DSS option from all data segments,
the Multipath TCP implementation in the Linux kernel falls back
correctly to regular TCP
o when a middlebox performs segment coalescing, the Multipath TCP
implementation in the Linux kernel is still able to accurately
extract the data corresponding to the indicated mapping
o when a middlebox performs segment splitting, the Multipath TCP
implementation in the Linux kernel correctly reassembles the data
corresponding to the indicated mapping. [HotMiddlebox13] shows on
figure 4 in section 3.3 a corner case with segment splitting that
may lead to a desynchronisation between the two hosts.
The interactions between Multipath TCP and real deployed middleboxes
is also analyzed in [HotMiddlebox13] and a particular scenario with
the FTP application level gateway running on a NAT is described.
From an operational viewpoint, knowing that Multipath TCP can cope
with various types of middlebox interference is important. However,
there are situations where the network operators need to gather
information about where a particular middlebox interference occurs.
The tracebox software [tracebox] described in [IMC13a] is an
extension of the popular traceroute software that enables network
operators to check at which hop a particular field of the TCP header
(including options) is modified. It has been used by several network
operators to debug various middlebox interference problems. tracebox
includes a scripting language that enables its user to specify
precisely which packet is sent by the source. tracebox sends packets
with an increasing TTL/HopLimit and compares the information returned
in the ICMP messages with the packet that it sends. This enables
tracebox to detect any interference caused by middleboxes on a given
path. tracebox works better when routers implement the ICMP extension
defined in [RFC1812].
A closer look at the packets received on the multipath-tcp.org server
showed that among the 184 thousands Multipath TCP connections in the
trace, we observed only 125 of them falling back to regular TCP,
which happened with 28 different client IP addresses. These include
91 HTTP connections and 34 FTP connections. The FTP interference is
expected and due to Application Level Gateways running on NAT boxes.
The HTTP interference appeared only on the direction from server to
client and could have been caused by transparent proxies deployed in
cellular or enterprise networks.
Users of the Multipath TCP implementation have reported some
experience with middlebox interference. The strangest scenario has
been a middlebox that accepts the Multipath TCP options in the SYN
segment but later replaces Multipath TCP options with a TCP EOL
option [StrangeMbox]. This causes Multipath TCP to perform a
fallback to regular TCP without any impact on the application.
3.2. Congestion control
Congestion control has been an important problem for Multipath TCP. Congestion control has been an important problem for Multipath TCP.
The standardised congestion control scheme for Multipath TCP is The standardised congestion control scheme for Multipath TCP is
defined in [RFC6356] and [NSDI11]. This congestion control scheme defined in [RFC6356] and [NSDI11]. This congestion control scheme
has been implemented in the Linux implementation of Multipath TCP. has been implemented in the Linux implementation of Multipath TCP.
Linux uses a modular architecture to support various congestion Linux uses a modular architecture to support various congestion
control schemes. This architecture is applicable for both regular control schemes. This architecture is applicable for both regular
TCP and Multipath TCP. While the coupled congestion control scheme TCP and Multipath TCP. While the coupled congestion control scheme
defined in [RFC6356] is the default congestion control scheme in the defined in [RFC6356] is the default congestion control scheme in the
Linux implementation, other congestion control schemes have been Linux implementation, other congestion control schemes have been
added. The second congestion control scheme is OLIA [CONEXT12]. added. The second congestion control scheme is OLIA [CONEXT12].
This congestion control scheme is also an adaptation of the NewReno This congestion control scheme is also an adaptation of the NewReno
single path congestion control scheme to support multiple paths. single path congestion control scheme to support multiple paths.
Simulations and measurements have shown that it provides some Simulations and measurements have shown that it provides some
performance benefits compared to the the default congestion control performance benefits compared to the the default congestion control
scheme [CONEXT12]. Measurement over a wide range of parameters scheme [CONEXT12]. Measurement over a wide range of parameters
reported in [CONEXT13] also indicate some benefits with the OLIA reported in [CONEXT13] also indicate some benefits with the OLIA
congestion control scheme. Recently, a delay-based congestion congestion control scheme. Recently, a delay-based congestion
control scheme has been ported to the Multipath TCP implementation in control scheme has been ported to the Multipath TCP implementation in
the Linux kernel. This congestion control scheme has been evaluated the Linux kernel. This congestion control scheme has been evaluated
by using simulations in [ICNP12]. by using simulations in [ICNP12]. The fourth congestion control
scheme that has been included in the Linux implementation of
Multipath TCP is the BALIA scheme
[I-D.walid-mptcp-congestion-control].
5. Subflow management These different congestion control schemes have been compared in
several articles. [CONEXT13] and [PaaschPhD] apply an experimental
design approach to compare these algorithms in an emulated
environment. The evaluation showed that the delay-based congestion
control scheme is less able to efficiently use the available links
than the three other schemes. Reports from some users indicate that
they seem to favor OLIA.
The multipath capability of Multipath TCP comes from the utilization 3.3. Subflow management
The multipath capability of Multipath TCP comes from the utilisation
of one subflow per path. The Multipath TCP architecture [RFC6182] of one subflow per path. The Multipath TCP architecture [RFC6182]
and the protocol specification [RFC6824] define the basic usage of and the protocol specification [RFC6824] define the basic usage of
the subflows and the protocol mechanisms that are required to create the subflows and the protocol mechanisms that are required to create
and terminate them. However, there are no guidelines on how subflows and terminate them. However, there are no guidelines on how subflows
are used during the lifetime of a Multipath TCP session. Most of the are used during the lifetime of a Multipath TCP session. Most of the
experiments with Multipath TCP have been performed in controlled experiments with Multipath TCP have been performed in controlled
environments. Still, based on the experience running them and environments. Still, based on the experience running them and
discussions on the mptcp-dev mailing list, interesting lessons have discussions on the mptcp-dev mailing list, interesting lessons have
been learned about the management of these subflows. been learned about the management of these subflows.
skipping to change at page 9, line 34 skipping to change at page 11, line 30
create subflows. However in practice the existing Multipath TCP create subflows. However in practice the existing Multipath TCP
implementations [I-D.eardley-mptcp-implementations-survey] have opted implementations [I-D.eardley-mptcp-implementations-survey] have opted
for a strategy where only the client creates new subflows. The main for a strategy where only the client creates new subflows. The main
motivation for this strategy is that often the client resides behind motivation for this strategy is that often the client resides behind
a NAT or a firewall, preventing passive subflow openings on the a NAT or a firewall, preventing passive subflow openings on the
client. Although there are environments such as datacenters where client. Although there are environments such as datacenters where
this problem does not occur, as of this writing, no precise this problem does not occur, as of this writing, no precise
requirement has emerged for allowing the server to create new requirement has emerged for allowing the server to create new
subflows. subflows.
5.1. Implemented subflow managers 3.4. Implemented subflow managers
The Multipath TCP implementation in the Linux kernel includes several The Multipath TCP implementation in the Linux kernel includes several
strategies to manage the subflows that compose a Multipath TCP strategies to manage the subflows that compose a Multipath TCP
session. The basic subflow manager is the full-mesh. As the name session. The basic subflow manager is the full-mesh. As the name
implies, it creates a full-mesh of subflows between the communicating implies, it creates a full-mesh of subflows between the communicating
hosts. hosts.
The most frequent use case for this subflow manager is a multihomed The most frequent use case for this subflow manager is a multihomed
client connected to a single-homed server. In this case, one subflow client connected to a single-homed server. In this case, one subflow
is created for each interface on the client. The current is created for each interface on the client. The current
skipping to change at page 10, line 11 skipping to change at page 12, line 6
the corresponding interface), it is not always reestablished. There the corresponding interface), it is not always reestablished. There
is ongoing work to enhance the full-mesh path manager to deal with is ongoing work to enhance the full-mesh path manager to deal with
such events. such events.
When the server is multihomed, using the full-mesh subflow manager When the server is multihomed, using the full-mesh subflow manager
may lead to a large number of subflows being established. For may lead to a large number of subflows being established. For
example, consider a dual-homed client connected to a server with example, consider a dual-homed client connected to a server with
three interfaces. In this case, even if the subflows are only three interfaces. In this case, even if the subflows are only
created by the client, 6 subflows will be established. This may be created by the client, 6 subflows will be established. This may be
excessive in some environments, in particular when the client and/or excessive in some environments, in particular when the client and/or
the server have a large number of interfaces. It should be noted the server have a large number of interfaces. A recent draft has
that there have been reports on the mptcp-dev mailing indicating that proposed a Multipath TCP option to negotiate the maximum number of
users rely on Multipath TCP to aggregate more than four different subflows . However, it should be noted that there have been reports
interfaces. Thus, there is a need for supporting many interfaces on the mptcp-dev mailing indicating that users rely on Multipath TCP
efficiently. to aggregate more than four different interfaces. Thus, there is a
need for supporting many interfaces efficiently.
It should be noted that creating subflows between multihomed clients Creating subflows between multihomed clients and servers may
and servers may sometimes lead to operational issues as observed by sometimes lead to operational issues as observed by discussions on
discussions on the mptcp-dev mailing list. In some cases the network the mptcp-dev mailing list. In some cases the network operators
operators would like to have a better control on how the subflows are would like to have a better control on how the subflows are created
created by Multipath TCP. This might require the definition of by Multipath TCP [I-D.boucadair-mptcp-max-subflow]. This might
policy rules to control the operation of the subflow manager. The require the definition of policy rules to control the operation of
two scenarios below illustrate some of these requirements. the subflow manager. The two scenarios below illustrate some of
these requirements.
host1 ---------- switch1 ----- host2 host1 ---------- switch1 ----- host2
| | | | | |
+-------------- switch2 --------+ +-------------- switch2 --------+
Figure 2: Simple switched network topology Figure 3: Simple switched network topology
Consider the simple network topology shown in Figure 2. From an Consider the simple network topology shown in Figure 3. From an
operational viewpoint, a network operator could want to create two operational viewpoint, a network operator could want to create two
subflows between the communicating hosts. From a bandwidth subflows between the communicating hosts. From a bandwidth
utilization viewpoint, the most natural paths are host1-switch1-host2 utilization viewpoint, the most natural paths are host1-switch1-host2
and host1-switch2-host2. However, a Multipath TCP implementation and host1-switch2-host2. However, a Multipath TCP implementation
running on these two hosts may sometimes have difficulties to achieve running onthese two hosts may sometimes have difficulties to obtain
this result. this result.
To understand the difficulty, let us consider different allocation To understand the difficulty, let us consider different allocation
strategies for the IP addresses. A first strategy is to assign two strategies for the IP addresses. A first strategy is to assign two
subnets : subnetA (resp. subnetB) contains the IP addresses of subnets : subnetA (resp. subnetB) contains the IP addresses of
host1's interface to switch1 (resp. switch2) and host2's interface to host1's interface to switch1 (resp. switch2) and host2's interface to
switch1 (resp. switch2). In this case, a Multipath TCP subflow switch1 (resp. switch2). In this case, a Multipath TCP subflow
manager should only create one subflow per subnet. To enforce the manager should only create one subflow per subnet. To enforce the
utilization of these paths, the network operator would have to utilization of these paths, the network operator would have to
specify a policy that prefers the subflows in the same subnet over specify a policy that prefers the subflows in the same subnet over
skipping to change at page 11, line 16 skipping to change at page 13, line 12
this case, it becomes more difficult to specify a policy that this case, it becomes more difficult to specify a policy that
indicates which subflows should be established. indicates which subflows should be established.
The second subflow manager that is currently supported by the The second subflow manager that is currently supported by the
Multipath TCP implementation in the Linux kernel is the ndiffport Multipath TCP implementation in the Linux kernel is the ndiffport
subflow manager. This manager was initially created to exploit the subflow manager. This manager was initially created to exploit the
path diversity that exists between single-homed hosts due to the path diversity that exists between single-homed hosts due to the
utilization of flow-based load balancing techniques. This subflow utilization of flow-based load balancing techniques. This subflow
manager creates N subflows between the same pair of IP addresses. manager creates N subflows between the same pair of IP addresses.
The N subflows are created by the client and differ only in the The N subflows are created by the client and differ only in the
source port selected by the client. source port selected by the client. It was not designed to be used
on multihomed hosts.
5.2. Subflow destination port 3.5. Subflow destination port
The Multipath TCP protocol relies on the token contained in the The Multipath TCP protocol relies on the token contained in the
MP_JOIN option to associate a subflow to an existing Multipath TCP MP_JOIN option to associate a subflow to an existing Multipath TCP
session. This implies that there is no restriction on the source session. This implies that there is no restriction on the source
address, destination address and source or destination ports used for address, destination address and source or destination ports used for
the new subflow. The ability to use different source and destination the new subflow. The ability to use different source and destination
addresses is key to support multihomed servers and clients. The addresses is key to support multihomed servers and clients. The
ability to use different destination port numbers is worth being ability to use different destination port numbers is worth being
discussed because it has operational implications. discussed because it has operational implications.
For illustration, consider a dual-homed client that creates a second For illustration, consider a dual-homed client that creates a second
subflow to reach a single-homed server as illustrated in the subflow to reach a single-homed server as illustrated in Figure 4.
Figure 3.
client ------- r1 --- internet --- server client ------- r1 --- internet --- server
| | | |
+----------r2-------+ +----------r2-------+
Figure 3: Multihomed-client connected to single-homed server Figure 4: Multihomed-client connected to single-homed server
When the Multipath TCP implementation in the Linux kernel creates the When the Multipath TCP implementation in the Linux kernel creates the
second subflow it uses the same destination port as the initial second subflow it uses the same destination port as the initial
subflow. This choice is motivated by the fact that the server might subflow. This choice is motivated by the fact that the server might
be protected by a firewall and only accept TCP connections (including be protected by a firewall and only accept TCP connections (including
subflows) on the official port number. Using the same destination subflows) on the official port number. Using the same destination
port for all subflows is also useful for operators that rely on the port for all subflows is also useful for operators that rely on the
port numbers to track application usage in their network. port numbers to track application usage in their network.
There have been suggestions from Multipath TCP users to modify the There have been suggestions from Multipath TCP users to modify the
skipping to change at page 12, line 22 skipping to change at page 14, line 19
listening socket is handling the SYN segments that are sent on a listening socket is handling the SYN segments that are sent on a
specific port number. Demultiplexing incoming segments can thus be specific port number. Demultiplexing incoming segments can thus be
done solely by looking at the IP addresses and the port numbers. done solely by looking at the IP addresses and the port numbers.
With Multipath TCP however, incoming SYN segments may have an MP_JOIN With Multipath TCP however, incoming SYN segments may have an MP_JOIN
option with a different destination port. This means, that all option with a different destination port. This means, that all
incoming segments that did not match on an existing listening-socket incoming segments that did not match on an existing listening-socket
or an already established socket must be parsed for an eventual or an already established socket must be parsed for an eventual
MP_JOIN option. This imposes an additional cost on servers, MP_JOIN option. This imposes an additional cost on servers,
previously not existent on legacy TCP implementations. previously not existent on legacy TCP implementations.
5.3. Closing subflows 3.6. Closing subflows
client server client server
| | | |
MPTCP: established | | MPTCP: established MPTCP: established | | MPTCP: established
Sub: established | | Sub: established Sub: established | | Sub: established
| | | |
| DATA_FIN | | DATA_FIN |
MPTCP: close-wait | <------------------------ | close() (step 1) MPTCP: close-wait | <------------------------ | close() (step 1)
Sub: established | DATA_ACK | Sub: established | DATA_ACK |
| ------------------------> | MPTCP: fin-wait-2 | ------------------------> | MPTCP: fin-wait-2
skipping to change at page 12, line 48 skipping to change at page 14, line 45
MPTCP: closed | <------------------------ | MPTCP: closed | <------------------------ |
Sub: fin-wait-2 | | Sub: fin-wait-2 | |
| | | |
| subflow-FIN | | subflow-FIN |
MPTCP: closed | <------------------------ | subflow-close() MPTCP: closed | <------------------------ | subflow-close()
Sub: time-wait | subflow-ACK | Sub: time-wait | subflow-ACK |
(step 3) | ------------------------> | MPTCP: time-wait (step 3) | ------------------------> | MPTCP: time-wait
| | Sub: closed | | Sub: closed
| | | |
Figure 4: Multipath TCP may not be able to avoid time-wait state Figure 5: Multipath TCP may not be able to avoid time-wait state
(even if enforced by the application). (even if enforced by the application).
Figure 4 shows a very particular issue within Multipath TCP. Many Figure 5 shows a very particular issue within Multipath TCP. Many
high-performance applications try to avoid Time-Wait state by high-performance applications try to avoid Time-Wait state by
deferring the closure of the connection until the peer has sent a deferring the closure of the connection until the peer has sent a
FIN. That way, the client on the left of Figure 4 does a passive FIN. That way, the client on the left of Figure 5 does a passive
closure of the connection, transitioning from Close-Wait to Last-ACK closure of the connection, transitioning from Close-Wait to Last-ACK
and finally freeing the resources after reception of the ACK of the and finally freeing the resources after reception of the ACK of the
FIN. An application running on top of a Multipath TCP enabled Linux FIN. An application running on top of a Multipath TCP enabled Linux
kernel might also use this approach. The difference here is that the kernel might also use this approach. The difference here is that the
close() of the connection (Step 1 in Figure 4) only triggers the close() of the connection (Step 1 in Figure 5) only triggers the
sending of a DATA_FIN. Nothing guarantees that the kernel is ready sending of a DATA_FIN. Nothing guarantees that the kernel is ready
to combine the DATA_FIN with a subflow-FIN. The reception of the to combine the DATA_FIN with a subflow-FIN. The reception of the
DATA_FIN will make the application trigger the closure of the DATA_FIN will make the application trigger the closure of the
connection (step 2), trying to avoid Time-Wait state with this late connection (step 2), trying to avoid Time-Wait state with this late
closure. This time, the kernel might decide to combine the DATA_FIN closure. This time, the kernel might decide to combine the DATA_FIN
with a subflow-FIN. This decision will be fatal, as the subflow's with a subflow-FIN. This decision will be fatal, as the subflow's
state machine will not transition from Close-Wait to Last-Ack, but state machine will not transition from Close-Wait to Last-Ack, but
rather go through Fin-Wait-2 into Time-Wait state. The Time-Wait rather go through Fin-Wait-2 into Time-Wait state. The Time-Wait
state will consume resources on the host for at least 2 MSL (Maximum state will consume resources on the host for at least 2 MSL (Maximum
Segment Lifetime). Thus, a smart application, that tries to avoid Segment Lifetime). Thus, a smart application, that tries to avoid
skipping to change at page 13, line 38 skipping to change at page 15, line 33
avoid Time-Wait state - even on the subflows. avoid Time-Wait state - even on the subflows.
The solution to this problem lies in an optimistic assumption that a The solution to this problem lies in an optimistic assumption that a
host doing active-closure of a Multipath TCP connection by sending a host doing active-closure of a Multipath TCP connection by sending a
DATA_FIN will soon also send a FIN on all its in subflows. Thus, the DATA_FIN will soon also send a FIN on all its in subflows. Thus, the
passive closer of the connection can simply wait for the peer to send passive closer of the connection can simply wait for the peer to send
exactly this FIN - enforcing passive closure even on the subflows. exactly this FIN - enforcing passive closure even on the subflows.
Of course, to avoid consuming resources indefinitely, a timer must Of course, to avoid consuming resources indefinitely, a timer must
limit the time our implementation waits for the FIN. limit the time our implementation waits for the FIN.
6. Packet schedulers 4. Packet schedulers
In a Multipath TCP implementation, the packet scheduler is the In a Multipath TCP implementation, the packet scheduler is the
algorithm that is executed when transmitting each packet to decide on algorithm that is executed when transmitting each packet to decide on
which subflow it needs to be transmitted. The packet scheduler which subflow it needs to be transmitted. The packet scheduler
itself does not have any impact on the interoperability of Multipath itself does not have any impact on the interoperability of Multipath
TCP implementations. However, it may clearly impact the performance TCP implementations. However, it may clearly impact the performance
of Multipath TCP sessions. It is important to note that the problem of Multipath TCP sessions. The Multipath TCP implementation in the
of scheduling Multipath TCP packets among subflows is different from Linux kernel supports a pluggable architecture for the packet
the problem of scheduling SCTP messages. SCTP implementations also scheduler [PaaschPhD]. As of this writing, two schedules have been
include schedulers, but these are used to schedule the different implemented: round-robin and lowest-rtt-first. They are compared in
streams. Multipath TCP uses a single data stream. [CSWS14]. The experiments and measurements described in [CSWS14]
show that the lowest-rtt-first scheduler appears to be the best
Various researchers have explored theoretically and by simulations compromise from a performance viewpoint. Another study of the packet
the problem of scheduling packets among Multipath TCP subflows schedulers is presented in [PAMS2014]. This study relies on
simulations with the Multipath TCP implementation in the Linux
[ICC14]. Unfortunately, none of the proposed techniques have been kernel. These simulations confirm the impact of the packet scheduler
implemented and used in real deployment. A detailed analysis of the on the performance of Multipath TCP.
impact of the packet scheduler will appear in [CSWS14]. This article
proposes a pluggable architecture for the scheduler used by the
Multipath TCP implementation in the Linux kernel. This architecture
allows researchers to experiment with different types of schedulers.
Two schedulers are compared in [CSWS14] : round-robin and lowest-rtt-
first. The experiments and measurements described in [CSWS14] show
that the lowest-rtt-first scheduler appears to be the best compromise
from a performance viewpoint.
Another study of the packet schedulers is presented in [PAMS2014].
This study relies on simulations with the Multipath TCP
implementation in the Linux kernel. The simulation scenarios
discussed in [PAMS2014] confirm the impact of the packet scheduler on
the performance of Multipath TCP.
7. Segment size selection 5. Segment size selection
When an application performs a write/send system call, the kernel When an application performs a write/send system call, the kernel
allocates a packet buffer (sk_buff in Linux) to store the data the allocates a packet buffer (sk_buff in Linux) to store the data the
application wants to send. The kernel will store at most one MSS application wants to send. The kernel will store at most one MSS
(Maximum Segment Size) of data per buffer. As MSS can differ amongst (Maximum Segment Size) of data per buffer. As MSS can differ amongst
subflows, an MPTCP implementation must select carefully the MSS used subflows, an MPTCP implementation must select carefully the MSS used
to generate application data. The Linux kernel implementation had to generate application data. The Linux kernel implementation had
various ways of selecting the MSS: minimum or maximum amongst the various ways of selecting the MSS: minimum or maximum amongst the
different subflows. However, these heuristics of MSS selection can different subflows. However, these heuristics of MSS selection can
cause significant performances issues in some environment. Consider cause significant performances issues in some environment. Consider
skipping to change at page 15, line 7 skipping to change at page 16, line 39
The Linux implementation recently took another approach [DetalMSS]. The Linux implementation recently took another approach [DetalMSS].
Instead of selecting the minimum and maximum values, it now Instead of selecting the minimum and maximum values, it now
dynamically adapts the MSS based on the contribution of all the dynamically adapts the MSS based on the contribution of all the
subflows to the connection's throughput. For this it computes, for subflows to the connection's throughput. For this it computes, for
each subflow, the potential throughput achieved by selecting each MSS each subflow, the potential throughput achieved by selecting each MSS
value and by taking into account the lost space in the cwnd. It then value and by taking into account the lost space in the cwnd. It then
selects the MSS that allows to achieve the highest potential selects the MSS that allows to achieve the highest potential
throughput. throughput.
8. Interactions with the Domain Name System 6. Interactions with the Domain Name System
Multihomed clients such as smartphones could lead to operational Multihomed clients such as smartphones can send DNS queries over any
problems when interacting with the Domain Name System. When a of their interfaces. When a single-homed client performs a DNS
single-homed client performs a DNS query, it receives from its local query, it receives from its local resolver the best answer for its
resolver the best answer for its request. If the client is request. If the client is multihomed, the answer returned to the DNS
multihomed, the answer returned to the DNS query may vary with the query may vary with the interface over which it has been sent.
interface over which it has been sent.
cdn1 cdn1
| |
client -- cellular -- internet -- cdn3 client -- cellular -- internet -- cdn3
| | | |
+----- wifi --------+ +----- wifi --------+
| |
cdn2 cdn2
Figure 5: Simple network topology Figure 6: Simple network topology
If the client sends a DNS query over the WiFi interface, the answer If the client sends a DNS query over the WiFi interface, the answer
will point to the cdn2 server while the same request sent over the will point to the cdn2 server while the same request sent over the
cellular interface will point to the cdn1 server. This might cause cellular interface will point to the cdn1 server. This might cause
problems for CDN providers that locate their servers inside ISP problems for CDN providers that locate their servers inside ISP
networks and have contracts that specify that the CDN server will networks and have contracts that specify that the CDN server will
only be accessed from within this particular ISP. Assume now that only be accessed from within this particular ISP. Assume now that
both the client and the CDN servers support Multipath TCP. In this both the client and the CDN servers support Multipath TCP. In this
case, a Multipath TCP session from cdn1 or cdn2 would potentially use case, a Multipath TCP session from cdn1 or cdn2 would potentially use
both the cellular network and the WiFi network. This would violate both the cellular network and the WiFi network. This would violate
the contract between the CDN provider and the network operators. A the contract between the CDN provider and the network operators. A
possible solution to prevent this problem would be to modify the DNS possible solution to prevent this problem would be to modify the DNS
resolution on the client. The client subnet EDNS extension defined resolution on the client. The client subnet EDNS extension defined
in [I-D.vandergaast-edns-client-subnet] could be used for this in [I-D.vandergaast-edns-client-subnet] could be used for this
purpose. When the client sends a DNS query from its WiFi interface, purpose. When the client sends a DNS query from its WiFi interface,
it should also send the client subnet corresponding to the cellular it should also send the client subnet corresponding to the cellular
interface in this request. This would indicate to the resolver that interface in this request. This would indicate to the resolver that
the answer should be valid for both the WiFi and the cellular the answer should be valid for both the WiFi and the cellular
interfaces (e.g., the cdn3 server). interfaces (e.g., the cdn3 server).
9. Captive portals 7. Captive portals
Multipath TCP enables a host to use different interfaces to reach a Multipath TCP enables a host to use different interfaces to reach a
server. In theory, this should ensure connectivity when at least one server. In theory, this should ensure connectivity when at least one
of the interfaces is active. In practice however, there are some of the interfaces is active. In practice however, there are some
particular scenarios with captive portals that may cause operational particular scenarios with captive portals that may cause operational
problems. The reference environment is the following : problems. The reference environment is shown in Figure 7.
client ----- network1 client ----- network1
| |
+------- internet ------------- server +------- internet ------------- server
Figure 6: Issue with captive portal Figure 7: Issue with captive portal
The client is attached to two networks : network1 that provides The client is attached to two networks : network1 that provides
limited connectivity and the entire Internet through the second limited connectivity and the entire Internet through the second
network interface. In practice, this scenario corresponds to an open network interface. In practice, this scenario corresponds to an open
WiFi network with a captive portal for network1 and a cellular WiFi network with a captive portal for network1 and a cellular
service for the second interface. On many smartphones, the WiFi service for the second interface. On many smartphones, the WiFi
interface is preferred over the cellular interface. If the interface is preferred over the cellular interface. If the
smartphone learns a default route via both interfaces, it will smartphone learns a default route via both interfaces, it will
typically prefer to use the WiFi interface to send its DNS request typically prefer to use the WiFi interface to send its DNS request
and create the first subflow. This is not optimal with Multipath and create the first subflow. This is not optimal with Multipath
TCP. A better approach would probably be to try a few attempts on TCP. A better approach would probably be to try a few attempts on
the WiFi interface and then try to use the second interface for the the WiFi interface and then try to use the second interface for the
initial subflow as well. initial subflow as well.
10. Conclusion 8. Conclusion
In this document, we have documented a few years of experience with In this document, we have documented a few years of experience with
Multipath TCP. The information presented in this document was Multipath TCP. The information presented in this document was
gathered from scientific publications and discussions with various gathered from scientific publications and discussions with various
users of the Multipath TCP implementation in the Linux kernel. users of the Multipath TCP implementation in the Linux kernel.
11. Acknowledgements 9. Acknowledgements
This work was partially supported by the FP7-Trilogy2 project. We This work was partially supported by the FP7-Trilogy2 project. We
would like to thank all the implementers and users of the Multipath would like to thank all the implementers and users of the Multipath
TCP implementation in the Linux kernel. TCP implementation in the Linux kernel. This document has benefited
from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley and
12. Changelog Jaehyun Hwang.
o initial version : September 16th, 2014 : Added section Section 7
that discusses some performance problems that appeared with the
Linux implementation when using subflows having different MSS
values
o update with a description of the middlebox that replaces an 10. Informative References
unknown TCP option with EOL [StrangeMbox]
13. Informative References [BBF-WT348]
Fabregas (Ed), G., "WT-348 - Hybrid Access for Broadband
Networks", Broadband Forum, contribution bbf2014.1139.04 ,
June 2015.
[CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP", [CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP",
Communications of the ACM, 57(4):51-57 , April 2014, Communications of the ACM, 57(4):51-57 , April 2014,
<http://inl.info.ucl.ac.be/publications/multipath-tcp>. <http://inl.info.ucl.ac.be/publications/multipath-tcp>.
[CONEXT12] [CONEXT12]
Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J.
Leboudec, "MPTCP is not pareto-optimal performance issues Leboudec, "MPTCP is not pareto-optimal performance issues
and a possible solution", Proceedings of the 8th and a possible solution", Proceedings of the 8th
international conference on Emerging networking international conference on Emerging networking
skipping to change at page 18, line 5 skipping to change at page 19, line 38
2.sipr.ucl.ac.be/sympa/arc/mptcp-dev/2014-09/ 2.sipr.ucl.ac.be/sympa/arc/mptcp-dev/2014-09/
msg00130.html>. msg00130.html>.
[HotMiddlebox13] [HotMiddlebox13]
Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O. Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O.
Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT
workshop HotMiddlebox , December 2013, workshop HotMiddlebox , December 2013,
<http://inl.info.ucl.ac.be/publications/ <http://inl.info.ucl.ac.be/publications/
are-tcp-extensions-middlebox-proof>. are-tcp-extensions-middlebox-proof>.
[HotMiddlebox13b]
Detal, G., Paasch, C., and O. Bonaventure, "Multipath in
the Middle(Box)", HotMiddlebox'13 , December 2013,
<http://inl.info.ucl.ac.be/publications/
multipath-middlebox>.
[HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A., [HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A.,
Wischik, D., and M. Handley, "Data center networking with Wischik, D., and M. Handley, "Data center networking with
multipath TCP", Proceedings of the 9th ACM SIGCOMM multipath TCP", Proceedings of the 9th ACM SIGCOMM
Workshop on Hot Topics in Networks (Hotnets-IX) , 2010, Workshop on Hot Topics in Networks (Hotnets-IX) , 2010,
<http://doi.acm.org/10.1145/1868447.1868457>. <http://doi.acm.org/10.1145/1868447.1868457>.
[I-D.boucadair-mptcp-max-subflow]
Boucadair, M. and C. Jacquenet, "Negotiating the Maximum
Number of MPTCP Subflows", draft-boucadair-mptcp-max-
subflow-00 (work in progress), June 2015.
[I-D.deng-mptcp-proxy]
Lingli, D., Liu, D., Sun, T., Boucadair, M., and G.
Cauchie, "Use-cases and Requirements for MPTCP Proxy in
ISP Networks", draft-deng-mptcp-proxy-01 (work in
progress), October 2014.
[I-D.eardley-mptcp-implementations-survey] [I-D.eardley-mptcp-implementations-survey]
Eardley, P., "Survey of MPTCP Implementations", draft- Eardley, P., "Survey of MPTCP Implementations", draft-
eardley-mptcp-implementations-survey-02 (work in eardley-mptcp-implementations-survey-02 (work in
progress), July 2013. progress), July 2013.
[I-D.lhwxz-gre-notifications-hybrid-access]
Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M.
Zhang, "GRE Notifications for Hybrid Access", draft-lhwxz-
gre-notifications-hybrid-access-01 (work in progress),
January 2015.
[I-D.lhwxz-hybrid-access-network-architecture]
Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M.
Zhang, "Hybrid Access Network Architecture", draft-lhwxz-
hybrid-access-network-architecture-02 (work in progress),
January 2015.
[I-D.vandergaast-edns-client-subnet] [I-D.vandergaast-edns-client-subnet]
Contavalli, C., Gaast, W., Leach, S., and E. Lewis, Contavalli, C., Gaast, W., Leach, S., and E. Lewis,
"Client Subnet in DNS Requests", draft-vandergaast-edns- "Client Subnet in DNS Requests", draft-vandergaast-edns-
client-subnet-02 (work in progress), July 2013. client-subnet-02 (work in progress), July 2013.
[ICC14] Kuhn, N., Lochin, E., Mifdaoui, A., Sarwar, G., Mehani, [I-D.walid-mptcp-congestion-control]
O., and R. Boreli, "DAPS Intelligent Delay-Aware Packet Walid, A., Peng, Q., Hwang, J., and S. Low, "Balanced
Scheduling For Multipath Transport", IEEE ICC 2014 , 2014. Linked Adaptation Congestion Control Algorithm for MPTCP",
draft-walid-mptcp-congestion-control-02 (work in
progress), January 2015.
[I-D.wei-mptcp-proxy-mechanism]
Wei, X., Xiong, C., and E. Ed, "MPTCP proxy mechanisms",
draft-wei-mptcp-proxy-mechanism-02 (work in progress),
June 2015.
[ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion [ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion
control for multipath TCP", 20th IEEE International control for multipath TCP", 20th IEEE International
Conference on Network Protocols (ICNP) , 2012. Conference on Network Protocols (ICNP) , 2012.
[IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., [IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A.,
Handley, M., and H. Tokuda, "Is it still possible to Handley, M., and H. Tokuda, "Is it still possible to
extend TCP?", Proceedings of the 2011 ACM SIGCOMM extend TCP?", Proceedings of the 2011 ACM SIGCOMM
conference on Internet measurement conference (IMC '11) , conference on Internet measurement conference (IMC '11) ,
2011, <http://doi.acm.org/10.1145/2068816.2068834>. 2011, <http://doi.acm.org/10.1145/2068816.2068834>.
skipping to change at page 19, line 16 skipping to change at page 21, line 41
Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee, Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee,
"Cross-Layer Path Management in Multi-path Transport "Cross-Layer Path Management in Multi-path Transport
Protocol for Mobile Devices", IEEE INFOCOM'14 , 2014. Protocol for Mobile Devices", IEEE INFOCOM'14 , 2014.
[IOS7] "Multipath TCP Support in iOS 7", January 2014, [IOS7] "Multipath TCP Support in iOS 7", January 2014,
<http://support.apple.com/kb/HT5977>. <http://support.apple.com/kb/HT5977>.
[MBTest] Hesmans, B., "MBTest", 2013, [MBTest] Hesmans, B., "MBTest", 2013,
<https://bitbucket.org/bhesmans/mbtest>. <https://bitbucket.org/bhesmans/mbtest>.
[MPTCPBIB]
Bonaventure, O., "Multipath TCP - An annotated
bibliography", Technical report , April 2015,
<https://github.com/obonaventure/mptcp-bib>.
[Mobicom15]
De Coninck, Q., Baerts, M., Hesmans, B., and O.
Bonaventure, "Poster - Evaluating Android Applications
with Multipath TCP", Mobicom 2015 (Poster) , September
2015.
[MultipathTCP-Linux] [MultipathTCP-Linux]
Paasch, C., Barre, S., and . et al, "Multipath TCP Paasch, C., Barre, S., and . et al, "Multipath TCP
implementation in the Linux kernel", n.d., implementation in the Linux kernel", n.d.,
<http://www.multipath-tcp.org>. <http://www.multipath-tcp.org>.
[NSDI11] Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley, [NSDI11] Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley,
"Design, implementation and evaluation of congestion "Design, implementation and evaluation of congestion
control for Multipath TCP", In Proceedings of the 8th control for Multipath TCP", In Proceedings of the 8th
USENIX conference on Networked systems design and USENIX conference on Networked systems design and
implementation (NSDI11) , 2011. implementation (NSDI11) , 2011.
skipping to change at page 19, line 40 skipping to change at page 22, line 29
Multipath TCP", USENIX Symposium of Networked Systems Multipath TCP", USENIX Symposium of Networked Systems
Design and Implementation (NSDI12) , April 2012, Design and Implementation (NSDI12) , April 2012,
<http://inl.info.ucl.ac.be/publications/how-hard-can-it- <http://inl.info.ucl.ac.be/publications/how-hard-can-it-
be-designing-and-implementing-deployable-multipath-tcp>. be-designing-and-implementing-deployable-multipath-tcp>.
[PAMS2014] [PAMS2014]
Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo, Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo,
"Impact of Path Selection and Scheduling Policies on MPTCP "Impact of Path Selection and Scheduling Policies on MPTCP
Performance", PAMS2014 , 2014. Performance", PAMS2014 , 2014.
[PaaschPhD]
Paasch, C., "Improving Multipath TCP", Ph.D. Thesis ,
November 2014, <http://inl.info.ucl.ac.be/publications/
improving-multipath-tcp>.
[RFC1812] Baker, F., "Requirements for IP Version 4 Routers", RFC [RFC1812] Baker, F., "Requirements for IP Version 4 Routers", RFC
1812, June 1995. 1812, June 1995.
[RFC1928] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and
L. Jones, "SOCKS Protocol Version 5", RFC 1928, March
1996.
[RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J. [RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J.
Iyengar, "Architectural Guidelines for Multipath TCP Iyengar, "Architectural Guidelines for Multipath TCP
Development", RFC 6182, March 2011. Development", RFC 6182, March 2011.
[RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled [RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled
Congestion Control for Multipath Transport Protocols", RFC Congestion Control for Multipath Transport Protocols", RFC
6356, October 2011. 6356, October 2011.
[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, [RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure,
"TCP Extensions for Multipath Operation with Multiple "TCP Extensions for Multipath Operation with Multiple
skipping to change at page 20, line 22 skipping to change at page 23, line 18
performance and robustness with multipath TCP", performance and robustness with multipath TCP",
Proceedings of the ACM SIGCOMM 2011 conference , n.d., Proceedings of the ACM SIGCOMM 2011 conference , n.d.,
<http://doi.acm.org/10.1145/2018436.2018467>. <http://doi.acm.org/10.1145/2018436.2018467>.
[StrangeMbox] [StrangeMbox]
Bonaventure, O., "Multipath TCP through a strange Bonaventure, O., "Multipath TCP through a strange
middlebox", Blog post , January 2015, middlebox", Blog post , January 2015,
<http://blog.multipath-tcp.org/blog/html/2015/01/30/ <http://blog.multipath-tcp.org/blog/html/2015/01/30/
multipath_tcp_through_a_strange_middlebox.html>. multipath_tcp_through_a_strange_middlebox.html>.
[TNC13] van der Pol, R., Bredel, M., and A. Barczyk, "Experiences
with MPTCP in an intercontinental multipathed OpenFlow
network", TNC2013 , 2013.
[tracebox] [tracebox]
Detal, G., "tracebox", 2013, <http://www.tracebox.org>. Detal, G., "tracebox", 2013, <http://www.tracebox.org>.
Appendix A. Changelog
o initial version : September 16th, 2014 : Added section Section 5
that discusses some performance problems that appeared with the
Linux implementation when using subflows having different MSS
values
o update with a description of the middlebox that replaces an
unknown TCP option with EOL [StrangeMbox]
o version ietf-02 : July 2015, answer to last call comments
* Reorganised text to better separate use cases and operational
experience
* New use case on Multipath TCP proxies in Section 2.3
* Added some text on middleboxes in Section 3.1
* Removed the discussion on SDN
* Restructured text and improved writing in some parts
Authors' Addresses Authors' Addresses
Olivier Bonaventure Olivier Bonaventure
UCLouvain UCLouvain
Email: Olivier.Bonaventure@uclouvain.be Email: Olivier.Bonaventure@uclouvain.be
Christoph Paasch Christoph Paasch
UCLouvain UCLouvain
Email: Christoph.Paasch@uclouvain.be Email: Christoph.Paasch@gmail.com
Gregory Detal Gregory Detal
UCLouvain and Tessares UCLouvain and Tessares
Email: Gregory.Detal@tessares.net Email: Gregory.Detal@tessares.net
 End of changes. 66 change blocks. 
237 lines changed or deleted 382 lines changed or added

This html diff was produced by rfcdiff 1.42. The latest version is available from http://tools.ietf.org/tools/rfcdiff/