draft-ietf-mptcp-experience-07.txt   rfc8041.txt 
MPTCP Working Group O. Bonaventure Internet Engineering Task Force (IETF) O. Bonaventure
Internet-Draft UCLouvain Request for Comments: 8041 UCLouvain
Intended status: Informational C. Paasch Category: Informational C. Paasch
Expires: April 30, 2017 Apple, Inc. ISSN: 2070-1721 Apple, Inc.
G. Detal G. Detal
Tessares Tessares
October 27, 2016 January 2017
Use Cases and Operational Experience with Multipath TCP Use Cases and Operational Experience with Multipath TCP
draft-ietf-mptcp-experience-07
Abstract Abstract
This document discusses both use cases and operational experience This document discusses both use cases and operational experience
with Multipath TCP in real networks. It lists several prominent use with Multipath TCP (MPTCP) in real networks. It lists several
cases where Multipath TCP has been considered and is being used. It prominent use cases where Multipath TCP has been considered and is
also gives insight to some heuristics and decisions that have helped being used. It also gives insight to some heuristics and decisions
to realize these use cases and suggests possible improvements. that have helped to realize these use cases and suggests possible
improvements.
Status of this Memo
This Internet-Draft is submitted in full conformance with the Status of This Memo
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering This document is not an Internet Standards Track specification; it is
Task Force (IETF). Note that other groups may also distribute published for informational purposes.
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on April 30, 2017. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc8041.
Copyright Notice Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction ....................................................3
2. Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Use Cases .......................................................4
2.1. Datacenters . . . . . . . . . . . . . . . . . . . . . . . 5 2.1. Datacenters ................................................4
2.2. Cellular/WiFi Offload . . . . . . . . . . . . . . . . . . 5 2.2. Cellular/WiFi Offload ......................................5
2.3. Multipath TCP proxies . . . . . . . . . . . . . . . . . . 9 2.3. Multipath TCP Proxies ......................................8
3. Operational Experience . . . . . . . . . . . . . . . . . . . . 11 3. Operational Experience ..........................................9
3.1. Middlebox interference . . . . . . . . . . . . . . . . . . 11 3.1. Middlebox Interference .....................................9
3.2. Congestion control . . . . . . . . . . . . . . . . . . . . 13 3.2. Congestion Control ........................................11
3.3. Subflow management . . . . . . . . . . . . . . . . . . . . 13 3.3. Subflow Management ........................................12
3.4. Implemented subflow managers . . . . . . . . . . . . . . . 14 3.4. Implemented Subflow Managers ..............................13
3.5. Subflow destination port . . . . . . . . . . . . . . . . . 16 3.5. Subflow Destination Port ..................................15
3.6. Closing subflows . . . . . . . . . . . . . . . . . . . . . 17 3.6. Closing Subflows ..........................................16
3.7. Packet schedulers . . . . . . . . . . . . . . . . . . . . 18 3.7. Packet Schedulers .........................................17
3.8. Segment size selection . . . . . . . . . . . . . . . . . . 19 3.8. Segment Size Selection ....................................18
3.9. Interactions with the Domain Name System . . . . . . . . . 19 3.9. Interactions with the Domain Name System ..................19
3.10. Captive portals . . . . . . . . . . . . . . . . . . . . . 20 3.10. Captive Portals ..........................................20
3.11. Stateless webservers . . . . . . . . . . . . . . . . . . . 21 3.11. Stateless Webservers .....................................20
3.12. Loadbalanced server farms . . . . . . . . . . . . . . . . 22 3.12. Load-Balanced Server Farms ...............................21
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 4. Security Considerations ........................................21
5. Security Considerations . . . . . . . . . . . . . . . . . . . 24 5. References .....................................................23
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 26 5.1. Normative References ......................................23
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.2. Informative References ....................................23
7.1. Normative References . . . . . . . . . . . . . . . . . . . 27 Acknowledgements ..................................................30
7.2. Informative References . . . . . . . . . . . . . . . . . . 27 Authors' Addresses ................................................30
Appendix A. Changelog . . . . . . . . . . . . . . . . . . . . . . 34
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 36
1. Introduction 1. Introduction
Multipath TCP was specified in [RFC6824] and five independent Multipath TCP was specified in [RFC6824] and five independent
implementations have been developed. As of November 2016, Multipath implementations have been developed. As of November 2016, Multipath
TCP has been or is being implemented on the following platforms: TCP has been or is being implemented on the following platforms:
o Linux kernel [MultipathTCP-Linux] o Linux kernel [MultipathTCP-Linux]
o Apple iOS and macOS [Apple-MPTCP] o Apple iOS and macOS
o Citrix load balancers o Citrix load balancers
o FreeBSD [FreeBSD-MPTCP] o FreeBSD [FreeBSD-MPTCP]
o Oracle Solaris o Oracle Solaris
The first three implementations are known to interoperate. Three of The first three implementations are known to interoperate. Three of
these implementations are open-source (Linux kernel, FreeBSD and these implementations are open source (Linux kernel, FreeBSD and
Apple's iOS and macOS). Apple's implementation is widely deployed. Apple's iOS and macOS). Apple's implementation is widely deployed.
Since the publication of [RFC6824] as an experimental RFC, experience Since the publication of [RFC6824] as an Experimental RFC, experience
has been gathered by various network researchers and users about the has been gathered by various network researchers and users about the
operational issues that arise when Multipath TCP is used in today's operational issues that arise when Multipath TCP is used in today's
Internet. Internet.
When the MPTCP working group was created, several use cases for When the MPTCP working group was created, several use cases for
Multipath TCP were identified [RFC6182]. Since then, other use cases Multipath TCP were identified [RFC6182]. Since then, other use cases
have been proposed and some have been tested and even deployed. We have been proposed and some have been tested and even deployed. We
describe these use cases in Section 2. describe these use cases in Section 2.
Section 3 focuses on the operational experience with Multipath TCP. Section 3 focuses on the operational experience with Multipath TCP.
Most of this experience comes from the utilization of the Multipath Most of this experience comes from the utilization of the Multipath
TCP implementation in the Linux kernel [MultipathTCP-Linux]. This TCP implementation in the Linux kernel [MultipathTCP-Linux]. This
open-source implementation has been downloaded and is used by open-source implementation has been downloaded and implemented by
thousands of users all over the world. Many of these users have thousands of users all over the world. Many of these users have
provided direct or indirect feedback by writing documents (scientific provided direct or indirect feedback by writing documents (scientific
articles or blog messages) or posting to the mptcp-dev mailing list articles or blog messages) or posting to the mptcp-dev mailing list
(see https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev ). This (see https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev). This
Multipath TCP implementation is actively maintained and continuously Multipath TCP implementation is actively maintained and continuously
improved. It is used on various types of hosts, ranging from improved. It is used on various types of hosts, ranging from
smartphones or embedded routers to high-end servers. smartphones or embedded routers to high-end servers.
The Multipath TCP implementation in the Linux kernel is not, by far, The Multipath TCP implementation in the Linux kernel is not, by far,
the most widespread deployment of Multipath TCP. Since September the most widespread deployment of Multipath TCP. Since September
2013, Multipath TCP is also supported on smartphones and tablets 2013, Multipath TCP is also supported on smartphones and tablets
since iOS7 [IOS7]. There are likely hundreds of millions of beginning with iOS7 [IETFJ]. There are likely hundreds of millions
Multipath TCP enabled devices. This Multipath TCP implementation is of MPTCP-enabled devices. This Multipath TCP implementation is
currently only used to support the Siri voice recognition/control currently only used to support the Siri voice recognition/control
application. Some lessons learned from this deployment are described application. Some lessons learned from this deployment are described
in [IETFJ]. in [IETFJ].
Section 3 is organized as follows. Supporting the middleboxes was Section 3 is organized as follows. Supporting the middleboxes was
one of the difficult issues in designing the Multipath TCP protocol. one of the difficult issues in designing the Multipath TCP protocol.
We explain in Section 3.1 which types of middleboxes the Linux Kernel We explain in Section 3.1 which types of middleboxes the Linux Kernel
implementation of Multipath TCP supports and how it reacts upon implementation of Multipath TCP supports and how it reacts upon
encountering these. Section 3.2 summarizes the MPTCP specific encountering these. Section 3.2 summarizes the MPTCP-specific
congestion controls that have been implemented. Section 3.3 to congestion controls that have been implemented. Sections 3.3 to 3.7
Section 3.7 discuss heuristics and issues with respect to subflow discuss heuristics and issues with respect to subflow management as
management as well as the scheduling across the subflows. well as the scheduling across the subflows. Section 3.8 explains
Section 3.8 explains some problems that occurred with subflows having some problems that occurred with subflows having different Maximum
different Maximum Segment Size (MSS) values. Section 3.9 presents Segment Size (MSS) values. Section 3.9 presents issues with respect
issues with respect to content delivery networks and suggests a to content delivery networks and suggests a solution to this issue.
solution to this issue. Finally, Section 3.10 documents an issue Finally, Section 3.10 documents an issue with captive portals where
with captive portals where MPTCP will behave sub optimally. MPTCP will behave suboptimally.
2. Use cases 2. Use Cases
Multipath TCP has been tested in several use cases. There is already Multipath TCP has been tested in several use cases. There is already
an abundant scientific literature on Multipath TCP [MPTCPBIB]. an abundant amount of scientific literature on Multipath TCP
Several of the papers published in the scientific literature have [MPTCPBIB]. Several of the papers published in the scientific
identified possible improvements that are worth being discussed here. literature have identified possible improvements that are worth being
discussed here.
2.1. Datacenters 2.1. Datacenters
A first, although initially unexpected, documented use case for A first, although initially unexpected, documented use case for
Multipath TCP has been in datacenters [HotNets][SIGCOMM11]. Today's Multipath TCP has been in datacenters [HotNets][SIGCOMM11]. Today's
datacenters are designed to provide several paths between single- datacenters are designed to provide several paths between single-
homed servers. The multiplicity of these paths comes from the homed servers. The multiplicity of these paths comes from the
utilization of Equal Cost Multipath (ECMP) and other load balancing utilization of Equal-Cost Multipath (ECMP) and other load-balancing
techniques inside the datacenter. Most of the deployed load techniques inside the datacenter. Most of the deployed load-
balancing techniques in datacenters rely on hashes computed over the balancing techniques in datacenters rely on hashes computed over the
five tuple. Thus all packets from the same TCP connection follow the five tuple. Thus, all packets from the same TCP connection follow
same path and so are not reordered. The results in [HotNets] the same path: so they are not reordered. The results in [HotNets]
demonstrate by simulations that Multipath TCP can achieve a better demonstrate by simulations that Multipath TCP can achieve a better
utilization of the available network by using multiple subflows for utilization of the available network by using multiple subflows for
each Multipath TCP session. Although [RFC6182] assumes that at least each Multipath TCP session. Although [RFC6182] assumes that at least
one of the communicating hosts has several IP addresses, [HotNets] one of the communicating hosts has several IP addresses, [HotNets]
demonstrates that Multipath TCP is beneficial when both hosts are demonstrates that Multipath TCP is beneficial when both hosts are
single-homed. This idea is analyzed in more details in [SIGCOMM11] single-homed. This idea is analyzed in more details in [SIGCOMM11],
where the Multipath TCP implementation in the Linux kernel is where the Multipath TCP implementation in the Linux kernel is
modified to be able to use several subflows from the same IP address. modified to be able to use several subflows from the same IP address.
Measurements in a public datacenter show the quantitative benefits of Measurements in a public datacenter show the quantitative benefits of
Multipath TCP [SIGCOMM11] in this environment. Multipath TCP [SIGCOMM11] in this environment.
Although ECMP is widely used inside datacenters, this is not the only Although ECMP is widely used inside datacenters, this is not the only
environment where there are different paths between a pair of hosts. environment where there are different paths between a pair of hosts.
ECMP and other load balancing techniques such as Link Aggregation ECMP and other load-balancing techniques such as Link Aggregation
Groups (LAG) are widely used in today's networks and having multiple Groups (LAGs) are widely used in today's networks; having multiple
paths between a pair of single-homed hosts is becoming the norm paths between a pair of single-homed hosts is becoming the norm
instead of the exception. Although these multiple paths have often instead of the exception. Although these multiple paths often have
the same cost (from an IGP metrics viewpoint), they do not the same cost (from an IGP metrics viewpoint), they do not
necessarily have the same performance. For example, [IMC13c] reports necessarily have the same performance. For example, [IMC13c] reports
the results of a long measurement study showing that load balanced the results of a long measurement study showing that load-balanced
Internet paths between that same pair of hosts can have huge delay Internet paths between that same pair of hosts can have huge delay
differences. differences.
2.2. Cellular/WiFi Offload 2.2. Cellular/WiFi Offload
A second use case that has been explored by several network A second use case that has been explored by several network
researchers is the cellular/WiFi offload use case. Smartphones or researchers is the cellular/WiFi offload use case. Smartphones or
other mobile devices equipped with two wireless interfaces are a very other mobile devices equipped with two wireless interfaces are a very
common use case for Multipath TCP. In September 2015, this is also common use case for Multipath TCP. In September 2015, this is also
the largest deployment of Multipath-TCP enabled devices [IOS7]. It the largest deployment of MPTCP-enabled devices [IETFJ]. It has been
has been briefly discussed during IETF88 [ietf88], but there is no briefly discussed during IETF 88 [IETF88], but there is no published
published paper or report that analyses this deployment. For this paper or report that analyses this deployment. For this reason, we
reason, we only discuss published papers that have mainly used the only discuss published papers that have mainly used the Multipath TCP
Multipath TCP implementation in the Linux kernel for their implementation in the Linux kernel for their experiments.
experiments.
The performance of Multipath TCP in wireless networks was briefly The performance of Multipath TCP in wireless networks was briefly
evaluated in [NSDI12]. One experiment analyzes the performance of evaluated in [NSDI12]. One experiment analyzes the performance of
Multipath TCP on a client with two wireless interfaces. This Multipath TCP on a client with two wireless interfaces. This
evaluation shows that when the receive window is large, Multipath TCP evaluation shows that when the receive window is large, Multipath TCP
can efficiently use the two available links. However, if the window can efficiently use the two available links. However, if the window
becomes smaller, then packets sent on a slow path can block the becomes smaller, then packets sent on a slow path can block the
transmission of packets on a faster path. In some cases, the transmission of packets on a faster path. In some cases, the
performance of Multipath TCP over two paths can become lower than the performance of Multipath TCP over two paths can become lower than the
performance of regular TCP over the best performing path. Two performance of regular TCP over the best performing path. Two
skipping to change at page 6, line 45 skipping to change at page 6, line 15
from WiFi to cellular or the opposite when the user moves. from WiFi to cellular or the opposite when the user moves.
Measurements presented in [CACM14] show that the handover from one Measurements presented in [CACM14] show that the handover from one
wireless network to another is not an abrupt process. When a host wireless network to another is not an abrupt process. When a host
moves, there are regions where the quality of one of the wireless moves, there are regions where the quality of one of the wireless
networks is weaker than the other, but the host considers this networks is weaker than the other, but the host considers this
wireless network to still be up. When a mobile host enters such wireless network to still be up. When a mobile host enters such
regions, its ability to send packets over another wireless network is regions, its ability to send packets over another wireless network is
important to ensure a smooth handover. This is clearly illustrated important to ensure a smooth handover. This is clearly illustrated
from the packet trace discussed in [CACM14]. from the packet trace discussed in [CACM14].
Many cellular networks use volume-based pricing and users often Many cellular networks use volume-based pricing; users often prefer
prefer to use unmetered WiFi networks when available instead of to use unmetered WiFi networks when available instead of metered
metered cellular networks. [Cellnet12] implements support for the cellular networks. [Cellnet12] implements support for the MP_PRIO
MP_PRIO option to explore two other modes of operation. option to explore two other modes of operation.
In the backup mode, Multipath TCP opens a TCP subflow over each In the backup mode, Multipath TCP opens a TCP subflow over each
interface, but the cellular interface is configured in backup mode. interface, but the cellular interface is configured in backup mode.
This implies that data flows only over the WiFi interface when both This implies that data flows only over the WiFi interface when both
interfaces are considered to be active. If the WiFi interface fails, interfaces are considered to be active. If the WiFi interface fails,
then the traffic switches quickly to the cellular interface, ensuring then the traffic switches quickly to the cellular interface, ensuring
a smooth handover from the user's viewpoint [Cellnet12]. The cost of a smooth handover from the user's viewpoint [Cellnet12]. The cost of
this approach is that the WiFi and cellular interfaces are likely to this approach is that the WiFi and cellular interfaces are likely to
remain active all the time since all subflows are established over remain active all the time since all subflows are established over
the two interfaces. the two interfaces.
The single-path mode is slightly different. This mode benefits from The single-path mode is slightly different. This mode benefits from
the break-before-make capability of Multipath TCP. When an MPTCP the break-before-make capability of Multipath TCP. When an MPTCP
skipping to change at page 7, line 38 skipping to change at page 7, line 10
protocol extension to allow hosts to reliably exchange this option. protocol extension to allow hosts to reliably exchange this option.
It would be useful to analyze packet traces to understand whether the It would be useful to analyze packet traces to understand whether the
unreliability of the REMOVE_ADDR option poses an operational problem unreliability of the REMOVE_ADDR option poses an operational problem
in real deployments. in real deployments.
Another study of the performance of Multipath TCP in wireless Another study of the performance of Multipath TCP in wireless
networks was reported in [IMC13b]. This study uses laptops connected networks was reported in [IMC13b]. This study uses laptops connected
to various cellular ISPs and WiFi hotspots. It compares various file to various cellular ISPs and WiFi hotspots. It compares various file
transfer scenarios. [IMC13b] observes that 4-path MPTCP outperforms transfer scenarios. [IMC13b] observes that 4-path MPTCP outperforms
2-path MPTCP, especially for larger files. However, for three 2-path MPTCP, especially for larger files. However, for three
congestion control algorithms (LIA, OLIA and Reno - see Section 3.2), congestion-control algorithms (LIA, OLIA, and Reno -- see
there is no significant performance difference for file sizes smaller Section 3.2), there is no significant performance difference for file
than 4MB. sizes smaller than 4 MB.
A different study of the performance of Multipath TCP with two A different study of the performance of Multipath TCP with two
wireless networks is presented in [INFOCOM14]. In this study the two wireless networks is presented in [INFOCOM14]. In this study the two
networks had different qualities : a good network and a lossy networks had different qualities: a good network and a lossy network.
network. When using two paths with different packet loss ratios, the When using two paths with different packet-loss ratios, the Multipath
Multipath TCP congestion control scheme moves traffic away from the TCP congestion-control scheme moves traffic away from the lossy link
lossy link that is considered to be congested. However, [INFOCOM14] that is considered to be congested. However, [INFOCOM14] documents
documents an interesting scenario that is summarized hereafter. an interesting scenario that is summarized hereafter.
client ----------- path1 -------- server client ----------- path1 -------- server
| | | |
+--------------- path2 ------------+ +--------------- path2 ------------+
Figure 1: Simple network topology Figure 1: Simple network topology
Initially, the two paths in Figure 1 have the same quality and Initially, the two paths in Figure 1 have the same quality and
Multipath TCP distributes the load over both of them. During the Multipath TCP distributes the load over both of them. During the
transfer, the path2 becomes lossy, e.g. because the client moves. transfer, the path2 becomes lossy, e.g., because the client moves.
Multipath TCP detects the packet losses and they are retransmitted Multipath TCP detects the packet losses and they are retransmitted
over path1. This enables the data transfer to continue over this over path1. This enables the data transfer to continue over this
path. However, the subflow over path2 is still up and transmits one path. However, the subflow over path2 is still up and transmits one
packet from time to time. Although the N packets have been packet from time to time. Although the N packets have been
acknowledged over the first subflow (at the MPTCP level), they have acknowledged over the first subflow (at the MPTCP level), they have
not been acknowledged at the TCP level over the second subflow. To not been acknowledged at the TCP level over the second subflow. To
preserve the continuity of the sequence numbers over the second preserve the continuity of the sequence numbers over the second
subflow, TCP will continue to retransmit these segments until either subflow, TCP will continue to retransmit these segments until either
they are acknowledged or the maximum number of retransmissions is they are acknowledged or the maximum number of retransmissions is
reached. This behavior is clearly inefficient and may lead to reached. This behavior is clearly inefficient and may lead to
skipping to change at page 8, line 45 skipping to change at page 8, line 13
implemented in the Multipath TCP implementation in the Linux kernel. implemented in the Multipath TCP implementation in the Linux kernel.
Some studies have started to analyze the performance of Multipath TCP Some studies have started to analyze the performance of Multipath TCP
on smartphones with real applications. In contrast with the bulk on smartphones with real applications. In contrast with the bulk
transfers that are used by many publications, many deployed transfers that are used by many publications, many deployed
applications do not exchange huge amounts of data and mainly use applications do not exchange huge amounts of data and mainly use
small connections. [COMMAG2016] proposes a software testing small connections. [COMMAG2016] proposes a software testing
framework that allows to automate Android applications to study their framework that allows to automate Android applications to study their
interactions with Multipath TCP. [PAM2016] analyses a one-month interactions with Multipath TCP. [PAM2016] analyses a one-month
packet trace of all the packets exchanged by a dozen of smartphones packet trace of all the packets exchanged by a dozen of smartphones
used by regular users. This analysis reveals that short connections utilized by regular users. This analysis reveals that short
are important on smartphones and that the main benefit of using connections are important on smartphones and that the main benefit of
Multipath TCP on smartphones is the ability to perform seamless using Multipath TCP on smartphones is the ability to perform seamless
handovers between different wireless networks. Long connections handovers between different wireless networks. Long connections
benefit from these handovers. benefit from these handovers.
2.3. Multipath TCP proxies 2.3. Multipath TCP Proxies
As Multipath TCP is not yet widely deployed on both clients and As Multipath TCP is not yet widely deployed on both clients and
servers, several deployments have used various forms of proxies. Two servers, several deployments have used various forms of proxies. Two
families of solutions are currently being used or tested. families of solutions are currently being used or tested.
A first use case is when a Multipath TCP enabled client wants to use A first use case is when an MPTCP-enabled client wants to use several
several interfaces to reach a regular TCP server. A typical use case interfaces to reach a regular TCP server. A typical use case is a
is a smartphone that needs to use both its WiFi and its cellular smartphone that needs to use both its WiFi and its cellular interface
interface to transfer data. Several types of proxies are possible to transfer data. Several types of proxies are possible for this use
for this use case. An HTTP proxy deployed on a Multipath TCP capable case. An HTTP proxy deployed on a MPTCP-capable server would enable
server would enable the smartphone to use Multipath TCP to access the smartphone to use Multipath TCP to access regular web servers.
regular web servers. Obviously, this solution only works for Obviously, this solution only works for applications that rely on
applications that rely on HTTP. Another possibility is to use a HTTP. Another possibility is to use a proxy that can convert any
proxy that can convert any Multipath TCP connection into a regular Multipath TCP connection into a regular TCP connection. MPTCP-
TCP connection. Multipath TCP-specific proxies have been proposed specific proxies have been proposed [HotMiddlebox13b] [HAMPEL].
[HotMiddlebox13b] [HAMPEL].
Another possibility leverages the SOCKS protocol [RFC1928]. SOCKS is Another possibility leverages the SOCKS protocol [RFC1928]. SOCKS is
often used in enterprise networks to allow clients to reach external often used in enterprise networks to allow clients to reach external
servers. For this, the client opens a TCP connection to the SOCKS servers. For this, the client opens a TCP connection to the SOCKS
server that relays it to the final destination. If both the client server that relays it to the final destination. If both the client
and the SOCKS server use Multipath TCP, but not the final and the SOCKS server use Multipath TCP, but not the final
destination, then Multipath TCP can still be used on the path between destination, then Multipath TCP can still be used on the path between
the clients and the SOCKS server. At IETF'93, Korea Telecom the clients and the SOCKS server. At IETF 93, Korea Telecom
announced that they have deployed in June 2015 a commercial service announced that they have deployed (in June 2015) a commercial service
that uses Multipath TCP on smartphones. These smartphones access that uses Multipath TCP on smartphones. These smartphones access
regular TCP servers through a SOCKS proxy. This enables them to regular TCP servers through a SOCKS proxy. This enables them to
achieve throughputs of up to 850 Mbps [KT]. achieve throughputs of up to 850 Mbps [KT].
Measurements performed with Android smartphones [Mobicom15] show that Measurements performed with Android smartphones [Mobicom15] show that
popular applications work correctly through a SOCKS proxy and popular applications work correctly through a SOCKS proxy and MPTCP-
Multipath TCP enabled smartphones. Thanks to Multipath TCP, long- enabled smartphones. Thanks to Multipath TCP, long-lived connections
lived connections can be spread over the two available interfaces. can be spread over the two available interfaces. However, for short-
However, for short-lived connections, most of the data is sent over lived connections, most of the data is sent over the initial subflow
the initial subflow that is created over the interface corresponding that is created over the interface corresponding to the default route
to the default route and the second subflow is almost not used and the second subflow is almost not used [PAM2016].
[PAM2016].
A second use case is when Multipath TCP is used by middleboxes, A second use case is when Multipath TCP is used by middleboxes,
typically inside access networks. Various network operators are typically inside access networks. Various network operators are
discussing and evaluating solutions for hybrid access networks discussing and evaluating solutions for hybrid access networks
[TR-348]. Such networks arise when a network operator controls two [TR-348]. Such networks arise when a network operator controls two
different access network technologies, e.g. wired and cellular, and different access network technologies, e.g., wired and cellular, and
wants to combine them to improve the bandwidth offered to the wants to combine them to improve the bandwidth offered to the end
endusers [I-D.lhwxz-hybrid-access-network-architecture]. Several users [HYA-ARCH]. Several solutions are currently investigated for
solutions are currently investigated for such networks [TR-348]. such networks [TR-348]. Figure 2 shows the organization of such a
Figure 2 shows the organization of such a network. When a client network. When a client creates a normal TCP connection, it is
creates a normal TCP connection, it is intercepted by the Hybrid CPE intercepted by the Hybrid CPE (HPCE) that converts it in a Multipath
(HPCE) that converts it in a Multipath TCP connection so that it can TCP connection so that it can use the available access networks (DSL
use the available access networks (DSL and LTE in the example). The and LTE in the example). The Hybrid Access Gateway (HAG) does the
Hybrid Access Gateway (HAG) does the opposite to ensure that the opposite to ensure that the regular server sees a normal TCP
regular server sees a normal TCP connection. Some of the solutions connection. Some of the solutions currently discussed for hybrid
currently discussed for hybrid networks use Multipath TCP on the HCPE networks use Multipath TCP on the HCPE and the HAG. Other solutions
and the HAG. Other solutions rely on tunnels between the HCPE and rely on tunnels between the HCPE and the HAG [GRE-NOTIFY].
the HAG [I-D.lhwxz-gre-notifications-hybrid-access].
client --- HCPE ------ DSL ------- HAG --- internet --- server client --- HCPE ------ DSL ------- HAG --- internet --- server
| | | |
+------- LTE -----------+ +------- LTE -----------+
Figure 2: Hybrid Access Network Figure 2: Hybrid Access Network
3. Operational Experience 3. Operational Experience
3.1. Middlebox interference 3.1. Middlebox Interference
The interference caused by various types of middleboxes has been an The interference caused by various types of middleboxes has been an
important concern during the design of the Multipath TCP protocol. important concern during the design of the Multipath TCP protocol.
Three studies on the interactions between Multipath TCP and Three studies on the interactions between Multipath TCP and
middleboxes are worth discussing. middleboxes are worth discussing.
The first analysis appears in [IMC11]. This paper was the main The first analysis appears in [IMC11]. This paper was the main
motivation for Multipath TCP incorporating various techniques to cope motivation for Multipath TCP incorporating various techniques to cope
with middlebox interference. More specifically, Multipath TCP has with middlebox interference. More specifically, Multipath TCP has
been designed to cope with middleboxes that : been designed to cope with middleboxes that:
o change source or destination addresses o change source or destination addresses
o change source or destination port numbers o change source or destination port numbers
o change TCP sequence numbers o change TCP sequence numbers
o split or coalesce segments o split or coalesce segments
o remove TCP options o remove TCP options
o modify the payload of TCP segments o modify the payload of TCP segments
These middlebox interferences have all been included in the MBtest These middlebox interferences have all been included in the MBtest
suite [MBTest]. This test suite is used in [HotMiddlebox13] to suite [MBTest]. This test suite is used in [HotMiddlebox13] to
verify the reaction of the Multipath TCP implementation in the Linux verify the reaction of the Multipath TCP implementation in the Linux
kernel [MultipathTCP-Linux] when faced with middlebox interference. kernel [MultipathTCP-Linux] when faced with middlebox interference.
The test environment used for this evaluation is a dual-homed client The test environment used for this evaluation is a dual-homed client
connected to a single-homed server. The middlebox behavior can be connected to a single-homed server. The middlebox behavior can be
activated on any of the paths. The main results of this analysis are activated on any of the paths. The main results of this analysis
: are:
o the Multipath TCP implementation in the Linux kernel, is not o the Multipath TCP implementation in the Linux kernel is not
affected by a middlebox that performs NAT or modifies TCP sequence affected by a middlebox that performs NAT or modifies TCP sequence
numbers numbers
o when a middlebox removes the MP_CAPABLE option from the initial o when a middlebox removes the MP_CAPABLE option from the initial
SYN segment, the Multipath TCP implementation in the Linux kernel SYN segment, the Multipath TCP implementation in the Linux kernel
falls back correctly to regular TCP falls back correctly to regular TCP
o when a middlebox removes the DSS option from all data segments, o when a middlebox removes the DSS option from all data segments,
the Multipath TCP implementation in the Linux kernel falls back the Multipath TCP implementation in the Linux kernel falls back
correctly to regular TCP correctly to regular TCP
o when a middlebox performs segment coalescing, the Multipath TCP o when a middlebox performs segment coalescing, the Multipath TCP
implementation in the Linux kernel is still able to accurately implementation in the Linux kernel is still able to accurately
extract the data corresponding to the indicated mapping extract the data corresponding to the indicated mapping
o when a middlebox performs segment splitting, the Multipath TCP o when a middlebox performs segment splitting, the Multipath TCP
implementation in the Linux kernel correctly reassembles the data implementation in the Linux kernel correctly reassembles the data
corresponding to the indicated mapping. [HotMiddlebox13] shows on corresponding to the indicated mapping. [HotMiddlebox13] shows,
figure 4 in section 3.3 a corner case with segment splitting that in Figure 4 in Section 3.3, a corner case with segment splitting
may lead to a desynchronization between the two hosts. that may lead to a desynchronization between the two hosts.
The interactions between Multipath TCP and real deployed middleboxes The interactions between Multipath TCP and real deployed middleboxes
is also analyzed in [HotMiddlebox13] and a particular scenario with are also analyzed in [HotMiddlebox13]; a particular scenario with the
the FTP application level gateway running on a NAT is described. FTP Application Level Gateway running on a NAT is described.
Middlebox interference can also be detected by analyzing packet Middlebox interference can also be detected by analyzing packet
traces on Multipath TCP enabled servers. A closer look at the traces on MPTCP-enabled servers. A closer look at the packets
packets received on the multipath-tcp.org server [TMA2015] shows that received on the multipath-tcp.org server [TMA2015] shows that among
among the 184,000 Multipath TCP connections, only 125 of them were the 184,000 Multipath TCP connections, only 125 of them were falling
falling back to regular TCP. These connections originated from 28 back to regular TCP. These connections originated from 28 different
different client IP addresses. These include 91 HTTP connections and client IP addresses. These include 91 HTTP connections and 34 FTP
34 FTP connections. The FTP interference is expected since connections. The FTP interference is expected since Application
Application Level Gateways used for FTP modify the TCP payload and Level Gateways used for FTP modify the TCP payload and the DSS
the DSS Checksum detects these modifications. The HTTP interference Checksum detects these modifications. The HTTP interference appeared
appeared only on the direction from server to client and could have only on the direction from server to client and could have been
been caused by transparent proxies deployed in cellular or enterprise caused by transparent proxies deployed in cellular or enterprise
networks. A longer trace is discussed in [COMCOM2016] and similar networks. A longer trace is discussed in [COMCOM2016] and similar
conclusions about the middlebox interference are provided. conclusions about the middlebox interference are provided.
From an operational viewpoint, knowing that Multipath TCP can cope From an operational viewpoint, knowing that Multipath TCP can cope
with various types of middlebox interference is important. However, with various types of middlebox interference is important. However,
there are situations where the network operators need to gather there are situations where the network operators need to gather
information about where a particular middlebox interference occurs. information about where a particular middlebox interference occurs.
The tracebox software [tracebox] described in [IMC13a] is an The tracebox software [tracebox] described in [IMC13a] is an
extension of the popular traceroute software that enables network extension of the popular traceroute software that enables network
operators to check at which hop a particular field of the TCP header operators to check at which hop a particular field of the TCP header
skipping to change at page 13, line 5 skipping to change at page 11, line 43
defined in [RFC1812] makes it easier to debug middlebox problems in defined in [RFC1812] makes it easier to debug middlebox problems in
IPv4 networks. IPv4 networks.
Users of the Multipath TCP implementation have reported some Users of the Multipath TCP implementation have reported some
experience with middlebox interference. The strangest scenario has experience with middlebox interference. The strangest scenario has
been a middlebox that accepts the Multipath TCP options in the SYN been a middlebox that accepts the Multipath TCP options in the SYN
segment but later replaces Multipath TCP options with a TCP EOL segment but later replaces Multipath TCP options with a TCP EOL
option [StrangeMbox]. This causes Multipath TCP to perform a option [StrangeMbox]. This causes Multipath TCP to perform a
fallback to regular TCP without any impact on the application. fallback to regular TCP without any impact on the application.
3.2. Congestion control 3.2. Congestion Control
Congestion control has been an important challenge for Multipath TCP. Congestion control has been an important challenge for Multipath TCP.
The congestion control scheme specified for Multipath TCP is defined The coupled congestion-control scheme defined in [RFC6356] in an
in [RFC6356]. A detailed description of this algorithm is provided adaptation of the NewReno algorithm. A detailed description of this
in [NSDI11]. This congestion control scheme has been implemented in coupled algorithm is provided in [NSDI11]. It is the default scheme
the Linux implementation of Multipath TCP. Linux uses a modular in the Linux implementation of Multipath TCP, but Linux supports
architecture to support various congestion control schemes. This other schemes.
architecture is applicable for both regular TCP and Multipath TCP.
While the coupled congestion control scheme defined in [RFC6356] is
the default congestion control scheme in the Linux implementation,
other congestion control schemes have been added. The second
congestion control scheme is OLIA [CONEXT12]. This congestion
control scheme is also an adaptation of the NewReno single path
congestion control scheme to support multiple paths. Simulations and
measurements have shown that it provides some performance benefits
compared to the default congestion control scheme [CONEXT12].
Measurements over a wide range of parameters reported in [CONEXT13]
also indicate some benefits with the OLIA congestion control scheme.
Recently, a delay-based congestion control scheme has been ported to
the Multipath TCP implementation in the Linux kernel. This
congestion control scheme has been evaluated by using simulations in
[ICNP12] and measurements in [PaaschPhD]. The fourth congestion
control scheme that has been included in the Linux implementation of
Multipath TCP is the BALIA scheme that provides a better balance
between TCP friendliness, responsiveness, and window oscillation
[BALIA].
These different congestion control schemes have been compared in The second congestion-control scheme is OLIA [CONEXT12]. It is also
an adaptation of the NewReno single path congestion-control scheme to
support multiple paths. Simulations [CONEXT12] and measurements
[CONEXT13] have shown that it provides some performance benefits
compared to the default coupled congestion-control scheme.
The delay-based scheme proposed in [ICNP12] has also been ported to
the Multipath TCP implementation in the Linux kernel. It has been
evaluated by using simulations [ICNP12] and measurements [PaaschPhD].
BALIA, defined in [BALIA], provides a better balance between TCP
friendliness, responsiveness, and window oscillation.
These different congestion-control schemes have been compared in
several articles. [CONEXT13] and [PaaschPhD] compare these several articles. [CONEXT13] and [PaaschPhD] compare these
algorithms in an emulated environment. The evaluation showed that algorithms in an emulated environment. The evaluation showed that
the delay-based congestion control scheme is less able to efficiently the delay-based congestion-control scheme is less able to efficiently
use the available links than the three other schemes. use the available links than the three other schemes.
3.3. Subflow management 3.3. Subflow Management
The multipath capability of Multipath TCP comes from the utilization The multipath capability of Multipath TCP comes from the utilization
of one subflow per path. The Multipath TCP architecture [RFC6182] of one subflow per path. The Multipath TCP architecture [RFC6182]
and the protocol specification [RFC6824] define the basic usage of and the protocol specification [RFC6824] define the basic usage of
the subflows and the protocol mechanisms that are required to create the subflows and the protocol mechanisms that are required to create
and terminate them. However, there are no guidelines on how subflows and terminate them. However, there are no guidelines on how subflows
are used during the lifetime of a Multipath TCP session. Most of the are used during the lifetime of a Multipath TCP session. Most of the
published experiments with Multipath TCP have been performed in published experiments with Multipath TCP have been performed in
controlled environments. Still, based on the experience running them controlled environments. Still, based on the experience running them
and discussions on the mptcp-dev mailing list, interesting lessons and discussions on the mptcp-dev mailing list, interesting lessons
have been learned about the management of these subflows. have been learned about the management of these subflows.
From a subflow viewpoint, the Multipath TCP protocol is completely From a subflow viewpoint, the Multipath TCP protocol is completely
symmetrical. Both the clients and the server have the capability to symmetrical. Both the clients and the server have the capability to
create subflows. However in practice the existing Multipath TCP create subflows. However, in practice, the existing Multipath TCP
implementations have opted for a strategy where only the client implementations have opted for a strategy where only the client
creates new subflows. The main motivation for this strategy is that creates new subflows. The main motivation for this strategy is that
often the client resides behind a NAT or a firewall, preventing often the client resides behind a NAT or a firewall, preventing
passive subflow openings on the client. Although there are passive subflow openings on the client. Although there are
environments such as datacenters where this problem does not occur, environments such as datacenters where this problem does not occur,
as of this writing, no precise requirement has emerged for allowing as of this writing, no precise requirement has emerged for allowing
the server to create new subflows. the server to create new subflows.
3.4. Implemented subflow managers 3.4. Implemented Subflow Managers
The Multipath TCP implementation in the Linux kernel includes several The Multipath TCP implementation in the Linux kernel includes several
strategies to manage the subflows that compose a Multipath TCP strategies to manage the subflows that compose a Multipath TCP
session. The basic subflow manager is the full-mesh. As the name session. The basic subflow manager is the full-mesh. As the name
implies, it creates a full-mesh of subflows between the communicating implies, it creates a full-mesh of subflows between the communicating
hosts. hosts.
The most frequent use case for this subflow manager is a multihomed The most frequent use case for this subflow manager is a multihomed
client connected to a single-homed server. In this case, one subflow client connected to a single-homed server. In this case, one subflow
is created for each interface on the client. The current is created for each interface on the client. The current
implementation of the full-mesh subflow manager is static. The implementation of the full-mesh subflow manager is static. The
subflows are created immediately after the creation of the initial subflows are created immediately after the creation of the initial
subflow. If one subflow fails during the lifetime of the Multipath subflow. If one subflow fails during the lifetime of the Multipath
TCP session (e.g. due to excessive retransmissions, or the loss of TCP session (e.g., due to excessive retransmissions or the loss of
the corresponding interface), it is not always reestablished. There the corresponding interface), it is not always reestablished. There
is ongoing work to enhance the full-mesh path manager to deal with is ongoing work to enhance the full-mesh path manager to deal with
such events. such events.
When the server is multihomed, using the full-mesh subflow manager When the server is multihomed, using the full-mesh subflow manager
may lead to a large number of subflows being established. For may lead to a large number of subflows being established. For
example, consider a dual-homed client connected to a server with example, consider a dual-homed client connected to a server with
three interfaces. In this case, even if the subflows are only three interfaces. In this case, even if the subflows are only
created by the client, 6 subflows will be established. This may be created by the client, six subflows will be established. This may be
excessive in some environments, in particular when the client and/or excessive in some environments, in particular when the client and/or
the server have a large number of interfaces. Implementations should the server have a large number of interfaces. Implementations should
limit the number of subflows that are used. limit the number of subflows that are used.
Creating subflows between multihomed clients and servers may Creating subflows between multihomed clients and servers may
sometimes lead to operational issues as observed by discussions on sometimes lead to operational issues as observed by discussions on
the mptcp-dev mailing list. In some cases the network operators the mptcp-dev mailing list. In some cases, the network operators
would like to have a better control on how the subflows are created would like to have a better control on how the subflows are created
by Multipath TCP [I-D.boucadair-mptcp-max-subflow]. This might by Multipath TCP [MPTCP-MAX-SUB]. This might require the definition
require the definition of policy rules to control the operation of of policy rules to control the operation of the subflow manager. The
the subflow manager. The two scenarios below illustrate some of two scenarios below illustrate some of these requirements.
these requirements.
host1 ---------- switch1 ----- host2 host1 ---------- switch1 ----- host2
| | | | | |
+-------------- switch2 --------+ +-------------- switch2 --------+
Figure 3: Simple switched network topology Figure 3: Simple Switched Network Topology
Consider the simple network topology shown in Figure 3. From an Consider the simple network topology shown in Figure 3. From an
operational viewpoint, a network operator could want to create two operational viewpoint, a network operator could want to create two
subflows between the communicating hosts. From a bandwidth subflows between the communicating hosts. From a bandwidth
utilization viewpoint, the most natural paths are host1-switch1-host2 utilization viewpoint, the most natural paths are host1-switch1-host2
and host1-switch2-host2. However, a Multipath TCP implementation and host1-switch2-host2. However, a Multipath TCP implementation
running on these two hosts may sometimes have difficulties to obtain running on these two hosts may sometimes have difficulties to obtain
this result. this result.
To understand the difficulty, let us consider different allocation To understand the difficulty, let us consider different allocation
strategies for the IP addresses. A first strategy is to assign two strategies for the IP addresses. A first strategy is to assign two
subnets : subnetA (resp. subnetB) contains the IP addresses of subnets: subnetA (resp. subnetB) contains the IP addresses of host1's
host1's interface to switch1 (resp. switch2) and host2's interface to interface to switch1 (resp. switch2) and host2's interface to switch1
switch1 (resp. switch2). In this case, a Multipath TCP subflow (resp. switch2). In this case, a Multipath TCP subflow manager
manager should only create one subflow per subnet. To enforce the should only create one subflow per subnet. To enforce the
utilization of these paths, the network operator would have to utilization of these paths, the network operator would have to
specify a policy that prefers the subflows in the same subnet over specify a policy that prefers the subflows in the same subnet over
subflows between addresses in different subnets. It should be noted subflows between addresses in different subnets. It should be noted
that the policy should probably also specify how the subflow manager that the policy should probably also specify how the subflow manager
should react when an interface or subflow fails. should react when an interface or subflow fails.
A second strategy is to use a single subnet for all IP addresses. In A second strategy is to use a single subnet for all IP addresses. In
this case, it becomes more difficult to specify a policy that this case, it becomes more difficult to specify a policy that
indicates which subflows should be established. indicates which subflows should be established.
The second subflow manager that is currently supported by the The second subflow manager that is currently supported by the
Multipath TCP implementation in the Linux kernel is the ndiffport Multipath TCP implementation in the Linux kernel is the ndiffport
subflow manager. This manager was initially created to exploit the subflow manager. This manager was initially created to exploit the
path diversity that exists between single-homed hosts due to the path diversity that exists between single-homed hosts due to the
utilization of flow-based load balancing techniques [SIGCOMM11]. utilization of flow-based load-balancing techniques [SIGCOMM11].
This subflow manager creates N subflows between the same pair of IP This subflow manager creates N subflows between the same pair of IP
addresses. The N subflows are created by the client and differ only addresses. The N subflows are created by the client and differ only
in the source port selected by the client. It was not designed to be in the source port selected by the client. It was not designed to be
used on multihomed hosts. used on multihomed hosts.
A more flexible subflow manager has been proposed, implemented and A more flexible subflow manager has been proposed, implemented and
evaluated in [CONEXT15]. This subflow manager exposes various kernel evaluated in [CONEXT15]. This subflow manager exposes various kernel
events to a user space daemon that decides when subflows need to be events to a user space daemon that decides when subflows need to be
created and terminated based on various policies. created and terminated based on various policies.
3.5. Subflow destination port 3.5. Subflow Destination Port
The Multipath TCP protocol relies on the token contained in the The Multipath TCP protocol relies on the token contained in the
MP_JOIN option to associate a subflow to an existing Multipath TCP MP_JOIN option to associate a subflow to an existing Multipath TCP
session. This implies that there is no restriction on the source session. This implies that there is no restriction on the source
address, destination address and source or destination ports used for address, destination address and source or destination ports used for
the new subflow. The ability to use different source and destination the new subflow. The ability to use different source and destination
addresses is key to support multihomed servers and clients. The addresses is key to support multihomed servers and clients. The
ability to use different destination port numbers is worth discussing ability to use different destination port numbers is worth discussing
because it has operational implications. because it has operational implications.
For illustration, consider a dual-homed client that creates a second For illustration, consider a dual-homed client that creates a second
subflow to reach a single-homed server as illustrated in Figure 4. subflow to reach a single-homed server as illustrated in Figure 4.
client ------- r1 --- internet --- server client ------- r1 --- internet --- server
| | | |
+----------r2-------+ +----------r2-------+
Figure 4: Multihomed-client connected to single-homed server Figure 4: Multihomed-Client Connected to Single-Homed Server
When the Multipath TCP implementation in the Linux kernel creates the When the Multipath TCP implementation in the Linux kernel creates the
second subflow it uses the same destination port as the initial second subflow, it uses the same destination port as the initial
subflow. This choice is motivated by the fact that the server might subflow. This choice is motivated by the fact that the server might
be protected by a firewall and only accept TCP connections (including be protected by a firewall and only accept TCP connections (including
subflows) on the official port number. Using the same destination subflows) on the official port number. Using the same destination
port for all subflows is also useful for operators that rely on the port for all subflows is also useful for operators that rely on the
port numbers to track application usage in their network. port numbers to track application usage in their network.
There have been suggestions from Multipath TCP users to modify the There have been suggestions from Multipath TCP users to modify the
implementation to allow the client to use different destination ports implementation to allow the client to use different destination ports
to reach the server. This suggestion seems mainly motivated by to reach the server. This suggestion seems mainly motivated by
traffic shaping middleboxes that are used in some wireless networks. traffic-shaping middleboxes that are used in some wireless networks.
In networks where different shaping rates are associated to different In networks where different shaping rates are associated with
destination port numbers, this could allow Multipath TCP to reach a different destination port numbers, this could allow Multipath TCP to
higher performance. This behavior is valid according to the reach a higher performance. This behavior is valid according to the
Multipath TCP specification [RFC6824]. An application could used an Multipath TCP specification [RFC6824]. An application could use an
enhanced socket API [SOCKET] to behave in this way. enhanced socket API [SOCKET] to behave in this way.
However, from an implementation point-of-view supporting different However, from an implementation point-of-view supporting different
destination ports for the same Multipath TCP connection can cause destination ports for the same Multipath TCP connection can cause
some issues. A legacy implementation of a TCP stack creates a some issues. A legacy implementation of a TCP stack creates a
listening socket to react upon incoming SYN segments. The listening listening socket to react upon incoming SYN segments. The listening
socket is handling the SYN segments that are sent on a specific port socket is handling the SYN segments that are sent on a specific port
number. Demultiplexing incoming segments can thus be done solely by number. Demultiplexing incoming segments can thus be done solely by
looking at the IP addresses and the port numbers. With Multipath TCP looking at the IP addresses and the port numbers. With Multipath TCP
however, incoming SYN segments may have an MP_JOIN option with a however, incoming SYN segments may have an MP_JOIN option with a
different destination port. This means, that all incoming segments different destination port. This means that all incoming segments
that did not match on an existing listening-socket or an already that did not match on an existing listening-socket or an already
established socket must be parsed for an eventual MP_JOIN option. established socket must be parsed for an eventual MP_JOIN option.
This imposes an additional cost on servers, previously not existent This imposes an additional cost on servers, previously not existent
on legacy TCP implementations. on legacy TCP implementations.
3.6. Closing subflows 3.6. Closing Subflows
client server client server
| | | |
MPTCP: established | | MPTCP: established MPTCP: ESTABLISHED | | MPTCP: ESTABLISHED
Sub: established | | Sub: established Sub: ESTABLISHED | | Sub: ESTABLISHED
| | | |
| DATA_FIN | | DATA_FIN |
MPTCP: close-wait | <------------------------ | close() (step 1) MPTCP: CLOSE-WAIT | <------------------------ | close() (step 1)
Sub: established | DATA_ACK | Sub: ESTABLISHED | DATA_ACK |
| ------------------------> | MPTCP: fin-wait-2 | ------------------------> | MPTCP: FIN-WAIT-2
| | Sub: established | | Sub: ESTABLISHED
| | | |
| DATA_FIN + subflow-FIN | | DATA_FIN + subflow-FIN |
close()/shutdown() | ------------------------> | MPTCP: time-wait close()/shutdown() | ------------------------> | MPTCP: TIME-WAIT
(step 2) | DATA_ACK | Sub: close-wait (step 2) | DATA_ACK | Sub: CLOSE-WAIT
MPTCP: closed | <------------------------ | MPTCP: CLOSED | <------------------------ |
Sub: fin-wait-2 | | Sub: FIN-WAIT-2 | |
| | | |
| subflow-FIN | | subflow-FIN |
MPTCP: closed | <------------------------ | subflow-close() MPTCP: CLOSED | <------------------------ | subflow-close()
Sub: time-wait | subflow-ACK | Sub: TIME-WAIT | subflow-ACK |
(step 3) | ------------------------> | MPTCP: time-wait (step 3) | ------------------------> | MPTCP: TIME-WAIT
| | Sub: closed | | Sub: CLOSED
| | | |
Figure 5: Multipath TCP may not be able to avoid time-wait state on Figure 5: Multipath TCP may not be able to avoid time-wait state on
the subflow (indicated as Sub in the drawing), even if enforced by the subflow (indicated as Sub in the drawing), even if enforced by
the application on the client-side. the application on the client-side.
Figure 5 shows a very particular issue within Multipath TCP. Many Figure 5 shows a very particular issue within Multipath TCP. Many
high-performance applications try to avoid Time-Wait state by high-performance applications try to avoid TIME-WAIT state by
deferring the closure of the connection until the peer has sent a deferring the closure of the connection until the peer has sent a
FIN. That way, the client on the left of Figure 5 does a passive FIN. That way, the client on the left of Figure 5 does a passive
closure of the connection, transitioning from Close-Wait to Last-ACK closure of the connection, transitioning from CLOSE-WAIT to Last-ACK
and finally freeing the resources after reception of the ACK of the and finally freeing the resources after reception of the ACK of the
FIN. An application running on top of a Multipath TCP enabled Linux FIN. An application running on top of an MPTCP-enabled Linux kernel
kernel might also use this approach. The difference here is that the might also use this approach. The difference here is that the
close() of the connection (Step 1 in Figure 5) only triggers the close() of the connection (step 1 in Figure 5) only triggers the
sending of a DATA_FIN. Nothing guarantees that the kernel is ready sending of a DATA_FIN. Nothing guarantees that the kernel is ready
to combine the DATA_FIN with a subflow-FIN. The reception of the to combine the DATA_FIN with a subflow-FIN. The reception of the
DATA_FIN will make the application trigger the closure of the DATA_FIN will make the application trigger the closure of the
connection (step 2), trying to avoid Time-Wait state with this late connection (step 2), trying to avoid TIME-WAIT state with this late
closure. This time, the kernel might decide to combine the DATA_FIN closure. This time, the kernel might decide to combine the DATA_FIN
with a subflow-FIN. This decision will be fatal, as the subflow's with a subflow-FIN. This decision will be fatal, as the subflow's
state machine will not transition from Close-Wait to Last-Ack, but state machine will not transition from CLOSE_WAIT to Last-ACK, but
rather go through Fin-Wait-2 into Time-Wait state. The Time-Wait rather go through FIN_WAIT-2 into TIME-WAIT state. The TIME-WAIT
state will consume resources on the host for at least 2 MSL (Maximum state will consume resources on the host for at least 2 MSL (Maximum
Segment Lifetime). Thus, a smart application that tries to avoid Segment Lifetime). Thus, a smart application that tries to avoid
Time-Wait state by doing late closure of the connection actually ends TIME-WAIT state by doing late closure of the connection actually ends
up with one of its subflows in Time-Wait state. A high-performance up with one of its subflows in TIME-WAIT state. A high-performance
Multipath TCP kernel implementation should honor the desire of the Multipath TCP kernel implementation should honor the desire of the
application to do passive closure of the connection and successfully application to do passive closure of the connection and successfully
avoid Time-Wait state - even on the subflows. avoid TIME-WAIT state -- even on the subflows.
The solution to this problem lies in an optimistic assumption that a The solution to this problem lies in an optimistic assumption that a
host doing active-closure of a Multipath TCP connection by sending a host doing active-closure of a Multipath TCP connection by sending a
DATA_FIN will soon also send a FIN on all its subflows. Thus, the DATA_FIN will soon also send a FIN on all its subflows. Thus, the
passive closer of the connection can simply wait for the peer to send passive closer of the connection can simply wait for the peer to send
exactly this FIN - enforcing passive closure even on the subflows. exactly this FIN -- enforcing passive closure even on the subflows.
Of course, to avoid consuming resources indefinitely, a timer must Of course, to avoid consuming resources indefinitely, a timer must
limit the time our implementation waits for the FIN. limit the time our implementation waits for the FIN.
3.7. Packet schedulers 3.7. Packet Schedulers
In a Multipath TCP implementation, the packet scheduler is the In a Multipath TCP implementation, the packet scheduler is the
algorithm that is executed when transmitting each packet to decide on algorithm that is executed when transmitting each packet to decide on
which subflow it needs to be transmitted. The packet scheduler which subflow it needs to be transmitted. The packet scheduler
itself does not have any impact on the interoperability of Multipath itself does not have any impact on the interoperability of Multipath
TCP implementations. However, it may clearly impact the performance TCP implementations. However, it may clearly impact the performance
of Multipath TCP sessions. The Multipath TCP implementation in the of Multipath TCP sessions. The Multipath TCP implementation in the
Linux kernel supports a pluggable architecture for the packet Linux kernel supports a pluggable architecture for the packet
scheduler [PaaschPhD]. As of this writing, two schedulers have been scheduler [PaaschPhD]. As of this writing, two schedulers have been
implemented: round-robin and lowest-rtt-first. The second scheduler implemented: round-robin and lowest-rtt-first. The second scheduler
relies on the round-trip-time (rtt) measured on each TCP subflow and relies on the round-trip time (rtt) measured on each TCP subflow and
sends first segments over the subflow having the lowest round-trip- sends first segments over the subflow having the lowest round-trip
time. They are compared in [CSWS14]. The experiments and time. They are compared in [CSWS14]. The experiments and
measurements described in [CSWS14] show that the lowest-rtt-first measurements described in [CSWS14] show that the lowest-rtt-first
scheduler appears to be the best compromise from a performance scheduler appears to be the best compromise from a performance
viewpoint. Another study of the packet schedulers is presented in viewpoint. Another study of the packet schedulers is presented in
[PAMS2014]. This study relies on simulations with the Multipath TCP [PAMS2014]. This study relies on simulations with the Multipath TCP
implementation in the Linux kernel. They compare the lowest-rtt- implementation in the Linux kernel. They compare the lowest-rtt-
first with the round-robin and a random scheduler. They show some first with the round-robin and a random scheduler. They show some
situations where the lowest-rtt-first scheduler does not perform as situations where the lowest-rtt-first scheduler does not perform as
well as the other schedulers, but there are many scenarios where the well as the other schedulers, but there are many scenarios where the
opposite is true. [PAMS2014] notes that "it is highly likely that opposite is true. [PAMS2014] notes that "it is highly likely that
the optimal scheduling strategy depends on the characteristics of the the optimal scheduling strategy depends on the characteristics of the
paths being used." paths being used."
3.8. Segment size selection 3.8. Segment Size Selection
When an application performs a write/send system call, the kernel When an application performs a write/send system call, the kernel
allocates a packet buffer (sk_buff in Linux) to store the data the allocates a packet buffer (sk_buff in Linux) to store the data the
application wants to send. The kernel will store at most one MSS application wants to send. The kernel will store at most one MSS
(Maximum Segment Size) of data per buffer. As the MSS can differ (Maximum Segment Size) of data per buffer. As the MSS can differ
amongst subflows, an MPTCP implementation must select carefully the amongst subflows, an MPTCP implementation must select carefully the
MSS used to generate application data. The Linux kernel MSS used to generate application data. The Linux kernel
implementation had various ways of selecting the MSS: minimum or implementation had various ways of selecting the MSS: minimum or
maximum amongst the different subflows. However, these heuristics of maximum amongst the different subflows. However, these heuristics of
MSS selection can cause significant performance issues in some MSS selection can cause significant performance issues in some
environment. Consider the following example. An MPTCP connection environments. Consider the following example. An MPTCP connection
has two established subflows that respectively use a MSS of 1420 and has two established subflows that respectively use an MSS of 1420 and
1428 bytes. If MPTCP selects the maximum, then the application will 1428 bytes. If MPTCP selects the maximum, then the application will
generate segments of 1428 bytes of data. An MPTCP implementation generate segments of 1428 bytes of data. An MPTCP implementation
will have to split the segment in two ( 1420-byte and 8-byte) will have to split the segment in two (1420-byte and 8-byte) segments
segments when pushing on the subflow with the smallest MSS. The when pushing on the subflow with the smallest MSS. The latter
latter segment will introduce a large overhead as for a single data segment will introduce a large overhead as this single data segment
segment 2 slots will be used in the congestion window (in packets) will use 2 slots in the congestion window (in packets) therefore
therefore reducing by roughly twice the potential throughput (in reducing by roughly twice the potential throughput (in bytes/s) of
bytes/s) of this subflow. Taking the smallest MSS does not solve the this subflow. Taking the smallest MSS does not solve the issue as
issue as there might be a case where the subflow with the smallest there might be a case where the subflow with the smallest MSS only
MSS only sends a few packets therefore reducing the potential sends a few packets, therefore reducing the potential throughput of
throughput of the other subflows. the other subflows.
The Linux implementation recently took another approach [DetalMSS]. The Linux implementation recently took another approach [DetalMSS].
Instead of selecting the minimum and maximum values, it now Instead of selecting the minimum and maximum values, it now
dynamically adapts the MSS based on the contribution of all the dynamically adapts the MSS based on the contribution of all the
subflows to the connection's throughput. For this it computes, for subflows to the connection's throughput. For each subflow, it
each subflow, the potential throughput achieved by selecting each MSS computes the potential throughput achieved by selecting each MSS
value and by taking into account the lost space in the congestion value and by taking into account the lost space in the congestion
window. It then selects the MSS that allows to achieve the highest window. It then selects the MSS that allows to achieve the highest
potential throughput. potential throughput.
Given the prevalence of middleboxes that clamp the MSS, Multipath TCP Given the prevalence of middleboxes that clamp the MSS, Multipath TCP
implementations must be able to efficiently support subflows with implementations must be able to efficiently support subflows with
different MSS values. The strategy described above is a possible different MSS values. The strategy described above is a possible
solution to this problem. solution to this problem.
3.9. Interactions with the Domain Name System 3.9. Interactions with the Domain Name System
skipping to change at page 20, line 13 skipping to change at page 19, line 21
DNS query may vary with the interface over which it has been sent. DNS query may vary with the interface over which it has been sent.
cdn1 cdn1
| |
client -- cellular -- internet -- cdn3 client -- cellular -- internet -- cdn3
| | | |
+----- wifi --------+ +----- wifi --------+
| |
cdn2 cdn2
Figure 6: Simple network topology Figure 6: Simple Network Topology
If the client sends a DNS query over the WiFi interface, the answer If the client sends a DNS query over the WiFi interface, the answer
will point to the cdn2 server while the same request sent over the will point to the cdn2 server while the same request sent over the
cellular interface will point to the cdn1 server. This might cause cellular interface will point to the cdn1 server. This might cause
problems for CDN providers that locate their servers inside ISP problems for CDN providers that locate their servers inside ISP
networks and have contracts that specify that the CDN server will networks and have contracts that specify that the CDN server will
only be accessed from within this particular ISP. Assume now that only be accessed from within this particular ISP. Assume now that
both the client and the CDN servers support Multipath TCP. In this both the client and the CDN servers support Multipath TCP. In this
case, a Multipath TCP session from cdn1 or cdn2 would potentially use case, a Multipath TCP session from cdn1 or cdn2 would potentially use
both the cellular network and the WiFi network. Serving the client both the cellular network and the WiFi network. Serving the client
from cdn2 over the cellular interface could violate the contract from cdn2 over the cellular interface could violate the contract
between the CDN provider and the network operators. A similar between the CDN provider and the network operators. A similar
problem occurs with regular TCP if the client caches DNS replies. problem occurs with regular TCP if the client caches DNS replies.
For example the client obtains a DNS answer over the cellular For example, the client obtains a DNS answer over the cellular
interface and then stops this interface and starts to use its WiFi interface and then stops this interface and starts to use its WiFi
interface. If the client retrieves data from cdn1 over its WiFi interface. If the client retrieves data from cdn1 over its WiFi
interface, this may also violate the contract between the CDN and the interface, this may also violate the contract between the CDN and the
network operators. network operators.
A possible solution to prevent this problem would be to modify the A possible solution to prevent this problem would be to modify the
DNS resolution on the client. The client subnet EDNS extension DNS resolution on the client. The client subnet Extension Mechanisms
defined in [RFC7871] could be used for this purpose. When the client for DNS (EDNS) defined in [RFC7871] could be used for this purpose.
sends a DNS query from its WiFi interface, it should also send the When the client sends a DNS query from its WiFi interface, it should
client subnet corresponding to the cellular interface in this also send the client subnet corresponding to the cellular interface
request. This would indicate to the resolver that the answer should in this request. This would indicate to the resolver that the answer
be valid for both the WiFi and the cellular interfaces (e.g., the should be valid for both the WiFi and the cellular interfaces (e.g.,
cdn3 server). the cdn3 server).
3.10. Captive portals 3.10. Captive Portals
Multipath TCP enables a host to use different interfaces to reach a Multipath TCP enables a host to use different interfaces to reach a
server. In theory, this should ensure connectivity when at least one server. In theory, this should ensure connectivity when at least one
of the interfaces is active. In practice however, there are some of the interfaces is active. However, in practice, there are some
particular scenarios with captive portals that may cause operational particular scenarios with captive portals that may cause operational
problems. The reference environment is shown in Figure 7. problems. The reference environment is shown in Figure 7.
client ----- network1 client ----- network1
| |
+------- internet ------------- server +------- internet ------------- server
Figure 7: Issue with captive portal Figure 7: Issue with Captive Portal
The client is attached to two networks : network1 that provides The client is attached to two networks: network1 that provides
limited connectivity and the entire Internet through the second limited connectivity and the entire Internet through the second
network interface. In practice, this scenario corresponds to an open network interface. In practice, this scenario corresponds to an open
WiFi network with a captive portal for network1 and a cellular WiFi network with a captive portal for network1 and a cellular
service for the second interface. On many smartphones, the WiFi service for the second interface. On many smartphones, the WiFi
interface is preferred over the cellular interface. If the interface is preferred over the cellular interface. If the
smartphone learns a default route via both interfaces, it will smartphone learns a default route via both interfaces, it will
typically prefer to use the WiFi interface to send its DNS request typically prefer to use the WiFi interface to send its DNS request
and create the first subflow. This is not optimal with Multipath and create the first subflow. This is not optimal with Multipath
TCP. A better approach would probably be to try a few attempts on TCP. A better approach would probably be to try a few attempts on
the WiFi interface and then, upon failure of these attempts, try to the WiFi interface and then, upon failure of these attempts, try to
use the second interface for the initial subflow as well. use the second interface for the initial subflow as well.
3.11. Stateless webservers 3.11. Stateless Webservers
MPTCP has been designed to interoperate with webservers that benefit MPTCP has been designed to interoperate with webservers that benefit
from SYN-cookies to protect against SYN-flooding attacks [RFC4987]. from SYN-cookies to protect against SYN-flooding attacks [RFC4987].
MPTCP achieves this by echoing the keys negotiated during the MPTCP achieves this by echoing the keys negotiated during the
MP_CAPABLE handshake in the third ACK of the 3-way handshake. MP_CAPABLE handshake in the third ACK of the three-way handshake.
Reception of this third ACK then allows the server to reconstruct the Reception of this third ACK then allows the server to reconstruct the
state specific to MPTCP. state specific to MPTCP.
However, one caveat to this mechanism is the non-reliable nature of However, one caveat to this mechanism is the unreliable nature of the
the third ACK. Indeed, when the third ACK gets lost, the server will third ACK. Indeed, when the third ACK gets lost, the server will not
not be able to reconstruct the MPTCP-state. MPTCP will fallback to be able to reconstruct the MPTCP state. MPTCP will fall back to
regular TCP in this case. This is in contrast to regular TCP. When regular TCP in this case. This is in contrast to regular TCP. When
the client starts sending data, the first data segment also includes the client starts sending data, the first data segment also includes
the SYN-cookie, which allows the server to reconstruct the TCP-state. the SYN-cookie, which allows the server to reconstruct the TCP-state.
Further, this data segment will be retransmitted by the client in Further, this data segment will be retransmitted by the client in
case it gets lost and thus is resilient against loss. MPTCP does not case it gets lost and thus is resilient against loss. MPTCP does not
include the keys in this data segment and thus the server cannot include the keys in this data segment and thus the server cannot
reconstruct the MPTCP state. reconstruct the MPTCP state.
This issue might be considered as a minor one for MPTCP. Losing the This issue might be considered as a minor one for MPTCP. Losing the
third ACK should only happen when packet loss is high. However, when third ACK should only happen when packet loss is high; in this case,
packet-loss is high MPTCP provides a lot of benefits as it can move MPTCP provides a lot of benefits as it can move traffic away from the
traffic away from the lossy link. It is undesirable that MPTCP has a lossy link. It is undesirable that MPTCP has a higher chance to fall
higher chance to fall back to regular TCP in those lossy back to regular TCP in those lossy environments.
environments.
[I-D.paasch-mptcp-syncookies] discusses this issue and suggests a [MPTCP-DEPLOY] discusses this issue and suggests a modified handshake
modified handshake mechanism that ensures reliable delivery of the mechanism that ensures reliable delivery of the MP_CAPABLE, following
MP_CAPABLE, following the 3-way handshake. This modification will the three-way handshake. This modification will make MPTCP reliable,
make MPTCP reliable, even in lossy environments when servers need to even in lossy environments when servers need to use SYN-cookies to
use SYN-cookies to protect against SYN-flooding attacks. protect against SYN-flooding attacks.
3.12. Loadbalanced server farms 3.12. Load-Balanced Server Farms
Large-scale server farms typically deploy thousands of servers behind Large-scale server farms typically deploy thousands of servers behind
a single virtual IP (VIP). Steering traffic to these servers is done a single virtual IP (VIP). Steering traffic to these servers is done
through layer-4 load balancers that ensure that a TCP-flow will through Layer 4 load-balancers that ensure that a TCP-flow will
always be routed to the same server [Presto08]. always be routed to the same server [Presto08].
As Multipath TCP uses multiple different TCP subflows to steer the As Multipath TCP uses multiple different TCP subflows to steer the
traffic across the different paths, load balancers need to ensure traffic across the different paths, load-balancers need to ensure
that all these subflows are routed to the same server. This implies that all these subflows are routed to the same server. This implies
that the load balancers need to track the MPTCP-related state, that the load-balancers need to track the MPTCP-related state,
allowing them to parse the token in the MP_JOIN and assign those allowing them to parse the token in the MP_JOIN and assign those
subflows to the appropriate server. However, server farms typically subflows to the appropriate server. However, server farms typically
deploy several load balancers for reliability and capacity reasons. deploy several load-balancers for reliability and capacity reasons.
As a TCP subflow might get routed to any of these load balancers, As a TCP subflow might get routed to any of these load-balancers,
they would need to synchronize the MPTCP-related state - a solution they would need to synchronize the MPTCP-related state -- a solution
that is not feasible at large scale. that is not feasible on a large scale.
The token (carried in the MP_JOIN) contains the information The token (carried in the MP_JOIN) contains the information
indicating which MPTCP-session the subflow belongs to. As the token indicating to which MPTCP-session the subflow belongs. As the token
is a hash of the key, servers are not able to generate the token in is a hash of the key, servers are not able to generate the token in
such a way that the token can provide the necessary information to such a way that the token can provide the necessary information to
the load balancers, which would allow them to route TCP subflows to the load-balancers, which would allow them to route TCP subflows to
the appropriate server. [I-D.paasch-mptcp-loadbalancer] discusses the appropriate server. [MPTCP-LOAD] discusses this issue in detail
this issue in detail and suggests two alternative MP_CAPABLE and suggests two alternative MP_CAPABLE handshakes to overcome these.
handshakes to overcome these.
4. IANA Considerations
There are no IANA considerations in this informational document.
5. Security Considerations 4. Security Considerations
This informational document discusses use-cases and operational This informational document discusses use cases and operational
experience with Multipath TCP. An extensive analysis of the experience with Multipath TCP. An extensive analysis of the
remaining security issues in the Multipath TCP specification has been remaining security issues in the Multipath TCP specification has been
published in [RFC7430], together with suggestions for possible published in [RFC7430], together with suggestions for possible
solutions. solutions.
From a security viewpoint, it is important to note that Multipath From a security viewpoint, it is important to note that Multipath
TCP, like other multipath solutions such as SCTP, has the ability to TCP, like other multipath solutions such as SCTP, has the ability to
send packets belonging to a single connection over different paths. send packets belonging to a single connection over different paths.
This design feature of Multipath TCP implies that middleboxes that This design feature of Multipath TCP implies that middleboxes that
have been deployed on-path assuming that they would observe all the have been deployed on-path assuming that they would observe all the
packets exchanged for a given connection in both directions may not packets exchanged for a given connection in both directions may not
function correctly anymore. A typical example are firewalls, IDS or function correctly anymore. A typical example are firewalls,
DPIs deployed in enterprise networks. Those devices expect to Intrusion Detection System (IDS) or deep packet inspections (DPIs)
observe all the packets from all TCP connections. With Multipath deployed in enterprise networks. Those devices expect to observe all
TCP, those middleboxes may not observe anymore all packets since some the packets from all TCP connections. With Multipath TCP, those
of them may follow a different path. The two examples below middleboxes may not observe anymore all packets since some of them
illustrate typical deployments of such middleboxes. The first may follow a different path. The two examples below illustrate
example, Figure 8, shows a Multipath TCP enabled smartphone attached typical deployments of such middleboxes. The first example,
to both an enterprise and a cellular network. If a Multipath TCP Figure 8, shows an MPTCP-enabled smartphone attached to both an
connection is established by the smartphone towards a server, some of enterprise and a cellular network. If a Multipath TCP connection is
the packets sent by the smartphone or the server may be transmitted established by the smartphone towards a server, some of the packets
over the cellular network and thus be invisible for the enterprise sent by the smartphone or the server may be transmitted over the
middlebox. cellular network and thus be invisible for the enterprise middlebox.
smartphone +----- entreprise net --- MBox----+------ server smartphone +----- enterprise net --- MBox----+------ server
| | | |
+----- cellular net -------------+ +----- cellular net -------------+
Figure 8: Enterprise Middlebox may not observe all packets from Figure 8: Enterprise Middlebox May Not Observe
multihomed host All Packets from Multihomed Host
The second example, Figure 9, shows a possible issue when multiple The second example, Figure 9, shows a possible issue when multiple
middleboxes are deployed inside a network. For simplicity, we assume middleboxes are deployed inside a network. For simplicity, we assume
that network1 is the default IPv4 path while network2 is the default that network1 is the default IPv4 path while network2 is the default
IPv6 path. A similar issue could occur with per-flow load balancing IPv6 path. A similar issue could occur with per-flow load-balancing
such as ECMP [RFC2992]. With regular TCP, all packets from each such as ECMP [RFC2992]. With regular TCP, all packets from each
connection would either pass through Mbox1 or Mbox2. With Multipath connection would either pass through Mbox1 or Mbox2. With Multipath
TCP, the client can easily establish a subflow over network1 and TCP, the client can easily establish a subflow over network1 and
another over network2 and each middlebox would only observe a part of another over network2 and each middlebox would only observe a part of
the traffic of the end-to-end Multipath TCP connection. the traffic of the end-to-end Multipath TCP connection.
client ----R-- network1 --- MBox1 -----R------------- server client ----R-- network1 --- MBox1 -----R------------- server
| | | |
+-- network2 --- MBox2 -----+ +-- network2 --- MBox2 -----+
Figure 9: Interactions between load balancing and security Figure 9: Interactions between
Middleboxes Load-Balancing and Security Middleboxes
In these two cases, it is possible for an attacker to evade some In these two cases, it is possible for an attacker to evade some
security measures operating on the TCP byte stream and implemented on security measures operating on the TCP byte stream and implemented on
the middleboxes by controlling the bytes that are actually sent over the middleboxes by controlling the bytes that are actually sent over
each subflow and there are tools that ease those kinds of evasion each subflow and there are tools that ease those kinds of evasion
[PZ15] [PT14]. This is not a security issue for Multipath TCP itself [PZ15] [PT14]. This is not a security issue for Multipath TCP itself
since Multipath TCP behaves correctly. However, this demonstrates since Multipath TCP behaves correctly. However, this demonstrates
the difficulty of enforcing security policies by relying only on on- the difficulty of enforcing security policies by relying only on
path middleboxes instead of enforcing them directly on the endpoints. on-path middleboxes instead of enforcing them directly on the
endpoints.
6. Acknowledgements
This work was partially supported by the FP7-Trilogy2 project. We
would like to thank all the implementers and users of the Multipath
TCP implementation in the Linux kernel. This document has benefited
from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley,
Jaehyun Hwang, Mirja Kuehlewind, Benoit Claise, Jari Arkko, Qin Wu,
Spencer Dawkins and Ben Campbell.
7. References 5. References
7.1. Normative References 5.1. Normative References
[RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J. [RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J.
Iyengar, "Architectural Guidelines for Multipath TCP Iyengar, "Architectural Guidelines for Multipath TCP
Development", RFC 6182, DOI 10.17487/RFC6182, March 2011, Development", RFC 6182, DOI 10.17487/RFC6182, March 2011,
<http://www.rfc-editor.org/info/rfc6182>. <http://www.rfc-editor.org/info/rfc6182>.
[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, [RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure,
"TCP Extensions for Multipath Operation with Multiple "TCP Extensions for Multipath Operation with Multiple
Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013, Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013,
<http://www.rfc-editor.org/info/rfc6824>. <http://www.rfc-editor.org/info/rfc6824>.
7.2. Informative References 5.2. Informative References
[Apple-MPTCP]
Apple, Inc, ., "iOS - Multipath TCP Support in iOS 7",
n.d., <https://support.apple.com/en-us/HT201373>.
[BALIA] Peng, Q., Walid, A., Hwang, J., and S. Low, "Multipath TCP [BALIA] Peng, Q., Walid, A., Hwang, J., and S. Low, "Multipath
Analysis, Design, and Implementation", IEEE/ACM Trans. on TCP: analysis, design, and implementation", IEEE/ACM
Networking, Vol. 24, No. 1 , 2016. Trans. on Networking (TON), Volume 24, Issue 1, February
2016.
[CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP", [CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP",
Communications of the ACM, 57(4):51-57 , April 2014, Communications of the ACM, 57(4):51-57, April 2014,
<http://inl.info.ucl.ac.be/publications/multipath-tcp>. <http://inl.info.ucl.ac.be/publications/multipath-tcp>.
[Cellnet12]
Paasch, C., Detal, G., Duchene, F., Raiciu, C., and O.
Bonaventure, "Exploring Mobile/WiFi Handover with
Multipath TCP", ACM SIGCOMM workshop on Cellular
Networks (Cellnet12), August 2012,
<http://inl.info.ucl.ac.be/publications/
exploring-mobilewifi-handover-multipath-tcp>.
[COMCOM2016] [COMCOM2016]
Tran, V., De Coninck, Q., Hesmans, B., Sadre, R., and O. Tran, V., De Coninck, Q., Hesmans, B., Sadre, R., and O.
Bonaventure, "Observing real Multipath TCP traffic", Bonaventure, "Observing real Multipath TCP traffic",
Computer Communications , April 2016, <http:// Computer Communications, DOI 10.1016/j.comcom.2016.01.014,
inl.info.ucl.ac.be/publications/ April 2016, <http://inl.info.ucl.ac.be/publications/
observing-real-multipath-tcp-traffic>. observing-real-multipath-tcp-traffic>.
[COMMAG2016] [COMMAG2016]
De Coninck, Q., Baerts, M., Hesmans, B., and O. De Coninck, Q., Baerts, M., Hesmans, B., and O.
Bonaventure, "Observing Real Smartphone Applications over Bonaventure, "Observing Real Smartphone Applications over
Multipath TCP", IEEE Communications Magazine , March 2016, Multipath TCP", IEEE Communications Magazine Network
<http://inl.info.ucl.ac.be/publications/ Testing Series, 54(3), March 2016,
observing-real-smartphone-applications-over-multipath- <http://inl.info.ucl.ac.be/publications/observing-real-
tcp>. smartphone-applications-over-multipath-tcp>.
[CONEXT12] [CONEXT12] Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J.
Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. Leboudec, "MPTCP is not Pareto-Optimal: Performance Issues
Leboudec, "MPTCP is not pareto-optimal performance issues and a Possible Solution", CoNEXT '12: Proceedings of the
and a possible solution", Proceedings of the 8th 8th international conference on Emerging networking
international conference on Emerging networking experiments and technologies, DOI 10.1145/2413176.2413178,
experiments and technologies (CoNEXT12) , 2012. December 2012.
[CONEXT13] [CONEXT13] Paasch, C., Khalili, R., and O. Bonaventure, "On the
Paasch, C., Khalili, R., and O. Bonaventure, "On the
Benefits of Applying Experimental Design to Improve Benefits of Applying Experimental Design to Improve
Multipath TCP", Conference on emerging Networking Multipath TCP", Conference on emerging Networking
EXperiments and Technologies (CoNEXT) , December 2013, <ht EXperiments and Technologies (CoNEXT),
tp://inl.info.ucl.ac.be/publications/ DOI 10.1145/2535372.2535403, December 2013,
benefits-applying-experimental-design-improve-multipath- <http://inl.info.ucl.ac.be/publications/benefits-applying-
tcp>. experimental-design-improve-multipath-tcp>.
[CONEXT15] [CONEXT15] Hesmans, B., Detal, G., Barre, S., Bauduin, R., and O.
Hesmans, B., Detal, G., Barre, S., Bauduin, R., and O. Bonaventure, "SMAPP: Towards Smart Multipath TCP-enabled
Bonaventure, "SMAPP - Towards Smart Multipath TCP-enabled APPlications", Proc. Conext 2015, Heidelberg, Germany,
APPlications", Proc. Conext 2015, Heidelberg, Germany ,
December 2015, <http://inl.info.ucl.ac.be/publications/ December 2015, <http://inl.info.ucl.ac.be/publications/
smapp-towards-smart-multipath-tcp-enabled-applications>. smapp-towards-smart-multipath-tcp-enabled-applications>.
[CSWS14] Paasch, C., Ferlin, S., Alay, O., and O. Bonaventure, [CSWS14] Paasch, C., Ferlin, S., Alay, O., and O. Bonaventure,
"Experimental Evaluation of Multipath TCP Schedulers", "Experimental evaluation of multipath TCP schedulers",
SIGCOMM CSWS2014 workshop , August 2014. CSWS '14: Proceedings of the 2014 ACM SIGCOMM workshop on
Capacity sharing workshop, DOI 10.1145/2630088.2631977,
[Cellnet12] August 2014.
Paasch, C., Detal, G., Duchene, F., Raiciu, C., and O.
Bonaventure, "Exploring Mobile/WiFi Handover with
Multipath TCP", ACM SIGCOMM workshop on Cellular Networks
(Cellnet12) , 2012, <http://inl.info.ucl.ac.be/
publications/exploring-mobilewifi-handover-multipath-tcp>.
[DetalMSS] [DetalMSS] Detal, G., "dynamically adapt mss value", Post on the
Detal, G., "Adaptive MSS value", Post on the mptcp-dev mptcp-dev mailing list, September 2014,
mailing list , September 2014, <https:// <https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev/
listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev/2014-09/ 2014-09/msg00130.html>.
msg00130.html>.
[FreeBSD-MPTCP] [FreeBSD-MPTCP]
Williams, N., "Multipath TCP For FreeBSD Kernel Patch Williams, N., "Multipath TCP For FreeBSD Kernel Patch
v0.5", n.d., <http://caia.swin.edu.au/urp/newtcp/mptcp>. v0.5", <http://caia.swin.edu.au/urp/newtcp/mptcp>.
[GRE-NOTIFY]
Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M.
Zhang, "GRE Notifications for Hybrid Access", Work in
Progress, draft-lhwxz-gre-notifications-hybrid-access-01,
January 2015.
[HAMPEL] Hampel, G., Rana, A., and T. Klein, "Seamless TCP mobility [HAMPEL] Hampel, G., Rana, A., and T. Klein, "Seamless TCP mobility
using lightweight MPTCP proxy", Proceedings of the 11th using lightweight MPTCP proxy", MobiWac '13: Proceedings
ACM international symposium on Mobility management and of the 11th ACM international symposium on Mobility
wireless access (MobiWac '13). , 2013. management and wireless access,
DOI 10.1145/2508222.2508226, November 2013.
[HotMiddlebox13] [HotMiddlebox13]
Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O. Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O.
Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT
workshop HotMiddlebox , December 2013, <http:// workshop Hot Middlebox, December 2013,
inl.info.ucl.ac.be/publications/ <http://inl.info.ucl.ac.be/publications/
are-tcp-extensions-middlebox-proof>. are-tcp-extensions-middlebox-proof>.
[HotMiddlebox13b] [HotMiddlebox13b]
Detal, G., Paasch, C., and O. Bonaventure, "Multipath in Detal, G., Paasch, C., and O. Bonaventure, "Multipath in
the Middle(Box)", HotMiddlebox'13 , December 2013, <http:/ the Middle(Box)", HotMiddlebox '13, December 2013,
/inl.info.ucl.ac.be/publications/multipath-middlebox>. <http://inl.info.ucl.ac.be/publications/
multipath-middlebox>.
[HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A., [HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A.,
Wischik, D., and M. Handley, "Data center networking with Wischik, D., and M. Handley, "Data center networking with
multipath TCP", Proceedings of the 9th ACM SIGCOMM multipath TCP", Hotnetx-IX: Proceedings of the 9th ACM
Workshop on Hot Topics in Networks (Hotnets-IX) , 2010, SIGCOMM Workshop on Hot Topics in Networks Article No. 10,
DOI 10.1145/1868447.1868457, October 2010,
<http://doi.acm.org/10.1145/1868447.1868457>. <http://doi.acm.org/10.1145/1868447.1868457>.
[I-D.boucadair-mptcp-max-subflow] [HYA-ARCH] Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M.
Boucadair, M. and C. Jacquenet, "Negotiating the Maximum Zhang, "Hybrid Access Network Architecture", Work in
Number of Multipath TCP (MPTCP) Subflows", Progress, draft-lhwxz-hybrid-access-network-
draft-boucadair-mptcp-max-subflow-02 (work in progress), architecture-02, January 2015.
May 2016.
[I-D.lhwxz-gre-notifications-hybrid-access]
Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M.
Zhang, "GRE Notifications for Hybrid Access",
draft-lhwxz-gre-notifications-hybrid-access-01 (work in
progress), January 2015.
[I-D.lhwxz-hybrid-access-network-architecture]
Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M.
Zhang, "Hybrid Access Network Architecture",
draft-lhwxz-hybrid-access-network-architecture-02 (work in
progress), January 2015.
[I-D.paasch-mptcp-loadbalancer]
Paasch, C., Greenway, G., and A. Ford, "Multipath TCP
behind Layer-4 loadbalancers",
draft-paasch-mptcp-loadbalancer-00 (work in progress),
September 2015.
[I-D.paasch-mptcp-syncookies]
Paasch, C., Biswas, A., and D. Haas, "Making Multipath TCP
robust for stateless webservers",
draft-paasch-mptcp-syncookies-02 (work in progress),
October 2015.
[ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion [ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion
control for multipath TCP", 20th IEEE International control for multipath TCP", 20th IEEE International
Conference on Network Protocols (ICNP) , 2012. Conference on Network Protocols (INCP),
DOI 10.1109/ICNP.2012.6459978, October 2012.
[IETFJ] Bonaventure, O. and S. Seo, "Multipath TCP Deployments,", [IETF88] Stewart, L., "IETF 88 Meeting minutes of the MPTCP working
IETF Journal, Vol. 12, Issue 2 , November 2016. group", November 2013, <https://www.ietf.org/proceedings/
88/minutes/minutes-88-mptcp>.
[IETFJ] Bonaventure, O. and S. Seo, "Multipath TCP Deployments",
IETF Journal, Vol. 12, Issue 2, November 2016.
[IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., [IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A.,
Handley, M., and H. Tokuda, "Is it still possible to Handley, M., and H. Tokuda, "Is it still possible to
extend TCP?", Proceedings of the 2011 ACM SIGCOMM extend TCP?", IMC '11: Proceedings of the 2011 ACM SIGCOMM
conference on Internet measurement conference (IMC '11) , conference on Internet measurement conference,
2011, <http://doi.acm.org/10.1145/2068816.2068834>. DOI 10.1145/2068816.2068834, November 2011,
<http://doi.acm.org/10.1145/2068816.2068834>.
[IMC13a] Detal, G., Hesmans, B., Bonaventure, O., Vanaubel, Y., and [IMC13a] Detal, G., Hesmans, B., Bonaventure, O., Vanaubel, Y., and
B. Donnet, "Revealing Middlebox Interference with B. Donnet, "Revealing Middlebox Interference with
Tracebox", Proceedings of the 2013 ACM SIGCOMM conference Tracebox", Proceedings of the 2013 ACM SIGCOMM conference
on Internet measurement conference , 2013, <http:// on Internet measurement conference,
inl.info.ucl.ac.be/publications/ DOI 10.1145/2504730.2504757, October 2013,
<http://inl.info.ucl.ac.be/publications/
revealing-middlebox-interference-tracebox>. revealing-middlebox-interference-tracebox>.
[IMC13b] Chen, Y., Lim, Y., Gibbens, R., Nahum, E., Khalili, R., [IMC13b] Chen, Y., Lim, Y., Gibbens, R., Nahum, E., Khalili, R.,
and D. Towsley, "A measurement-based study of MultiPath and D. Towsley, "A measurement-based study of MultiPath
TCP performance over wireless network", Proceedings of the TCP performance over wireless network", ICM '13:
2013 conference on Internet measurement conference (IMC Proceedings of the 2013 conference on Internet
'13) , n.d., <http://doi.acm.org/10.1145/2504730.2504751>. measurement conference, DOI 10.1145/2504730.2504751,
October 2013,
<http://doi.acm.org/10.1145/2504730.2504751>.
[IMC13c] Pelsser, C., Cittadini, L., Vissicchio, S., and R. Bush, [IMC13c] Pelsser, C., Cittadini, L., Vissicchio, S., and R. Bush,
"From Paris to Tokyo on the suitability of ping to "From Paris to Tokyo: on the suitability of ping to
measure latency", Proceedings of the 2013 conference on measure latency", IMC '13: Proceedings of the 2013
Internet measurement conference (IMC '13) , 2013, conference on Internet measurement Conference,
DOI 10.1145/2504730.2504765, October 2013,
<http://doi.acm.org/10.1145/2504730.2504765>. <http://doi.acm.org/10.1145/2504730.2504765>.
[INFOCOM14] [INFOCOM14]
Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee, Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee,
"Cross-Layer Path Management in Multi-path Transport "Cross-layer path management in multi-path transport
Protocol for Mobile Devices", IEEE INFOCOM'14 , 2014. protocol for mobile devices", IEEE INFOCOM'14,
DOI 10.1109/INFOCOM.2014.6848120, April 2014.
[IOS7] Apple, ., "Multipath TCP Support in iOS 7", January 2014,
<http://support.apple.com/kb/HT5977>.
[KT] Seo, S., "KT's GiGA LTE", July 2015, <https:// [KT] Seo, S., "KT's GiGA LTE", July 2015,
www.ietf.org/proceedings/93/slides/slides-93-mptcp-3.pdf>. <https://www.ietf.org/proceedings/93/slides/
slides-93-mptcp-3.pdf>.
[MBTest] Hesmans, B., "MBTest", 2013, [MBTest] Hesmans, B., "MBTest", October 2013,
<https://bitbucket.org/bhesmans/mbtest>. <https://bitbucket.org/bhesmans/mbtest>.
[MPTCPBIB]
Bonaventure, O., "Multipath TCP - An annotated
bibliography", Technical report , April 2015,
<https://github.com/obonaventure/mptcp-bib>.
[Mobicom15] [Mobicom15]
De Coninck, Q., Baerts, M., Hesmans, B., and O. De Coninck, Q., Baerts, M., Hesmans, B., and O.
Bonaventure, "Poster - Evaluating Android Applications Bonaventure, "Poster - Evaluating Android Applications
with Multipath TCP", Mobicom 2015 (Poster) , with Multipath TCP", Mobicom 2015 (Poster),
September 2015. DOI 10.1145/2789168.2795165, September 2015.
[MPTCP-DEPLOY]
Paasch, C., Biswas, A., and D. Haas, "Making Multipath TCP
robust for stateless webservers", Work in Progress,
draft-paasch-mptcp-syncookies-02, October 2015.
[MPTCP-LOAD]
Paasch, C., Greenway, G., and A. Ford, "Multipath TCP
behind Layer-4 loadbalancers", Work in Progress,
draft-paasch-mptcp-loadbalancer-00, September 2015.
[MPTCP-MAX-SUB]
Boucadair, M. and C. Jacquenet, "Negotiating the Maximum
Number of Multipath TCP (MPTCP) Subflows", Work in
Progress draft-boucadair-mptcp-max-subflow-02, May 2016.
[MPTCPBIB] Bonaventure, O., "Multipath TCP - Annotated bibliography",
Technical report, April 2015,
<https://github.com/obonaventure/mptcp-bib>.
[MultipathTCP-Linux] [MultipathTCP-Linux]
Paasch, C., Barre, S., and . et al, "Multipath TCP Paasch, C., Barre, S., and . et al, "Multipath TCP - Linux
implementation in the Linux kernel", n.d., Kernel implementation", <http://www.multipath-tcp.org>.
<http://www.multipath-tcp.org>.
[NSDI11] Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley, [NSDI11] Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley,
"Design, implementation and evaluation of congestion "Design, implementation and evaluation of congestion
control for Multipath TCP", In Proceedings of the 8th control for multipath TCP", NSDI11: In Proceedings of the
USENIX conference on Networked systems design and 8th USENIX conference on Networked systems design
implementation (NSDI11) , 2011. and implementation, 2011.
[NSDI12] Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M., [NSDI12] Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M.,
Duchene, F., Bonaventure, O., and M. Handley, "How Hard Duchene, F., Bonaventure, O., and M. Handley, "How Hard
Can It Be? Designing and Implementing a Deployable Can It Be? Designing and Implementing a Deployable
Multipath TCP", USENIX Symposium of Networked Systems Multipath TCP", NSDI '12: USENIX Symposium of Networked
Design and Implementation (NSDI12) , April 2012, <http:// Systems Design and implementation, April 2012,
inl.info.ucl.ac.be/publications/ <http://inl.info.ucl.ac.be/publications/how-hard-can-it-
how-hard-can-it-be-designing-and-implementing-deployable- be-designing-and-implementing-deployable-multipath-tcp>.
multipath-tcp>.
[PaaschPhD]
Paasch, C., "Improving Multipath TCP", Ph.D. Thesis ,
November 2014, <http://inl.info.ucl.ac.be/publications/
improving-multipath-tcp>.
[PAM2016] De Coninck, Q., Baerts, M., Hesmans, B., and O. [PAM2016] De Coninck, Q., Baerts, M., Hesmans, B., and O.
Bonaventure, "A First Analysis of Multipath TCP on Bonaventure, "A First Analysis of Multipath TCP on
Smartphones", 17th International Passive and Active Smartphones", 17th International Passive and Active
Measurements Conference (PAM2016) , March 2016, <http:// Measurements Conference (PAM2016) volume 17, March 2016,
inl.info.ucl.ac.be/publications/ <http://inl.info.ucl.ac.be/publications/
first-analysis-multipath-tcp-smartphones>. first-analysis-multipath-tcp-smartphones>.
[PAMS2014] [PAMS2014] Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo,
Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo,
"Impact of Path Selection and Scheduling Policies on MPTCP "Impact of Path Selection and Scheduling Policies on MPTCP
Performance", PAMS2014 , 2014. Performance", PAMS2014, DOI 10.1109/WAINA.2014.121, May
2014.
[Presto08] Greenberg, A., Lahiri, P., Maltz, D., Patel, P., and S.
Sengupta, "Towards a next generation data center
architecture: scalability and commoditization", ACM
PRESTO 2008, DOI 10.1145/1397718.1397732, August 2008,
<http://dl.acm.org/citation.cfm?id=1397732>.
[PT14] Pearce, C. and P. Thomas, "Multipath TCP Breaking Today's [PT14] Pearce, C. and P. Thomas, "Multipath TCP Breaking Today's
Networks with Tomorrow's Protocols", Proc. Blackhat Networks with Tomorrow's Protocols", Proc.
Briefings , 2014, <http://www.blackhat.com/docs/us-14/ Blackhat Briefings, 2014, <http://www.blackhat.com/docs/
materials/ us-14/materials/us-14-Pearce-Multipath-TCP-Breaking-
us-14-Pearce-Multipath-TCP-Breaking-Todays-Networks-With- Todays-Networks-With-Tomorrows-Protocols-WP.pdf>.
Tomorrows-Protocols-WP.pdf>.
[PZ15] Pearce, C. and S. Zeadally, "Ancillary Impacts of [PZ15] Pearce, C. and S. Zeadally, "Ancillary Impacts of
Multipath TCP on Current and Future Network Security", Multipath TCP on Current and Future Network Security",
IEEE Internet Computing, vol. 19, no. 5, pp. 58-65 , 2015. IEEE Internet Computing, vol. 19, no. 5, pp. 58-65,
DOI 10.1109/MIC.2015.70, September 2015.
[PaaschPhD]
Paasch, C., "Improving Multipath TCP", Ph.D. Thesis ,
November 2014, <http://inl.info.ucl.ac.be/publications/
improving-multipath-tcp>.
[Presto08]
Greenberg, A., Lahiri, P., Maltz, D., Parveen, P., and S.
Sengupta, "Towards a Next Generation Data Center
Architecture - Scalability and Commoditization", ACM
PRESTO 2008 , August 2008,
<http://dl.acm.org/citation.cfm?id=1397732>.
[RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", [RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers",
RFC 1812, DOI 10.17487/RFC1812, June 1995, RFC 1812, DOI 10.17487/RFC1812, June 1995,
<http://www.rfc-editor.org/info/rfc1812>. <http://www.rfc-editor.org/info/rfc1812>.
[RFC1928] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and [RFC1928] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and
L. Jones, "SOCKS Protocol Version 5", RFC 1928, L. Jones, "SOCKS Protocol Version 5", RFC 1928,
DOI 10.17487/RFC1928, March 1996, DOI 10.17487/RFC1928, March 1996,
<http://www.rfc-editor.org/info/rfc1928>. <http://www.rfc-editor.org/info/rfc1928>.
skipping to change at page 32, line 41 skipping to change at page 29, line 12
Mitigations", RFC 4987, DOI 10.17487/RFC4987, August 2007, Mitigations", RFC 4987, DOI 10.17487/RFC4987, August 2007,
<http://www.rfc-editor.org/info/rfc4987>. <http://www.rfc-editor.org/info/rfc4987>.
[RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled [RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled
Congestion Control for Multipath Transport Protocols", Congestion Control for Multipath Transport Protocols",
RFC 6356, DOI 10.17487/RFC6356, October 2011, RFC 6356, DOI 10.17487/RFC6356, October 2011,
<http://www.rfc-editor.org/info/rfc6356>. <http://www.rfc-editor.org/info/rfc6356>.
[RFC7430] Bagnulo, M., Paasch, C., Gont, F., Bonaventure, O., and C. [RFC7430] Bagnulo, M., Paasch, C., Gont, F., Bonaventure, O., and C.
Raiciu, "Analysis of Residual Threats and Possible Fixes Raiciu, "Analysis of Residual Threats and Possible Fixes
for Multipath TCP (MPTCP)", RFC 7430, DOI 10.17487/ for Multipath TCP (MPTCP)", RFC 7430,
RFC7430, July 2015, DOI 10.17487/RFC7430, July 2015,
<http://www.rfc-editor.org/info/rfc7430>. <http://www.rfc-editor.org/info/rfc7430>.
[RFC7871] Contavalli, C., van der Gaast, W., Lawrence, D., and W. [RFC7871] Contavalli, C., van der Gaast, W., Lawrence, D., and W.
Kumari, "Client Subnet in DNS Queries", RFC 7871, Kumari, "Client Subnet in DNS Queries", RFC 7871,
DOI 10.17487/RFC7871, May 2016, DOI 10.17487/RFC7871, May 2016,
<http://www.rfc-editor.org/info/rfc7871>. <http://www.rfc-editor.org/info/rfc7871>.
[SIGCOMM11] [SIGCOMM11]
Raiciu, C., Barre, S., Pluntke, C., Greenhalgh, A., Raiciu, C., Barre, S., Pluntke, C., Greenhalgh, A.,
Wischik, D., and M. Handley, "Improving datacenter Wischik, D., and M. Handley, "Improving datacenter
performance and robustness with multipath TCP", performance and robustness with multipath TCP", SIGCOMM
Proceedings of the ACM SIGCOMM 2011 conference , n.d., '11: Proceedings of the ACM SIGCOMM 2011 conference,
DOI 10.1145/2018436.2018467, August 2011,
<http://doi.acm.org/10.1145/2018436.2018467>. <http://doi.acm.org/10.1145/2018436.2018467>.
[SOCKET] Hesmans, B. and O. Bonaventure, "An Enhanced Socket API [SOCKET] Hesmans, B. and O. Bonaventure, "An enhanced socket API
for Multipath TCP", Proceedings of the 2016 Applied for Multipath TCP", Proceedings of the 2016 Applied
Networking Research Workshop , July 2016, Networking Research Workshop, DOI 10.1145/2959424.2959433,
<http://doi.acm.org/10.1145/2959424.2959433>. July 2016, <http://doi.acm.org/10.1145/2959424.2959433>.
[StrangeMbox] [StrangeMbox]
Bonaventure, O., "Multipath TCP through a strange Bonaventure, O., "Multipath TCP through a strange
middlebox", Blog post , January 2015, <http:// middlebox", Blog post, January 2015,
blog.multipath-tcp.org/blog/html/2015/01/30/ <http://blog.multipath-tcp.org/blog/html/2015/01/30/
multipath_tcp_through_a_strange_middlebox.html>. multipath_tcp_through_a_strange_middlebox.html>.
[TMA2015] Hesmans, B., Tran Viet, H., Sadre, R., and O. Bonaventure, [TMA2015] Hesmans, B., Tran Viet, H., Sadre, R., and O. Bonaventure,
"A First Look at Real Multipath TCP Traffic", Traffic "A First Look at Real Multipath TCP Traffic", Traffic
Monitoring and Analysis , 2015, <http:// Monitoring and Analysis, 2015,
inl.info.ucl.ac.be/publications/ <http://inl.info.ucl.ac.be/publications/
first-look-real-multipath-tcp-traffic>. first-look-real-multipath-tcp-traffic>.
[TR-348] Broadband Forum, ., "TR 348 - Hybrid Access Broadband [TR-348] Broadband Forum, ., "TR 348 - Hybrid Access Broadband
Network Architecture", July 2016, <https:// Network Architecture", Issue: 1, July 2016,
www.broadband-forum.org/technical/download/TR-348.pdf>. <https://www.broadband-forum.org/technical/download/
TR-348.pdf>.
[ietf88] Stewart, L., "IETF'88 Meeting minutes of the MPTCP working
group", n.d., <http://tools.ietf.org/wg/mptcp/
minutes?item=minutes-88-mptcp.html>.
[tracebox]
Detal, G. and O. Tilmans, "tracebox", 2013,
<http://www.tracebox.org>.
Appendix A. Changelog
This section should be removed before final publication
o initial version : September 16th, 2014 : Added section Section 3.8
that discusses some performance problems that appeared with the
Linux implementation when using subflows having different MSS
values
o update with a description of the middlebox that replaces an
unknown TCP option with EOL [StrangeMbox]
o version ietf-02 : July 2015, answer to last call comments
* Reorganised text to better separate use cases and operational
experience
* New use case on Multipath TCP proxies in Section 2.3
* Added some text on middleboxes in Section 3.1
* Removed the discussion on SDN
* Restructured text and improved writing in some parts
o version ietf-03 : September 2015, answer to comments from Phil
Eardley
* Improved introduction
* Added details about using SOCKS and Korea Telecom's use-case in
Section 2.3.
* Added issue around clients caching DNS-results in Section 3.9
* Explained issue of MPTCP with stateless webservers Section 3.11
* Added description of MPTCP's use behind layer-4 load balancers
Section 3.12
* Restructured text and improved writing in some parts
o version ietf-04 : April 2016, answer to last comments
* Updated text on measurements with smartphones
* Updated conclusion
o version ietf-06 : August 2016, answer to AD's review
* removed some expired drafts
* removed conclusion
o version ietf-07 : September 2016, answer to IESG comments [tracebox] Detal, G. and O. Tilmans, "Tracebox: A Middlebox Detection
Tool", 2013, <http://www.tracebox.org>.
* small nits Acknowledgements
* added more information in the security considerations as This work was partially supported by the FP7-Trilogy2 project. We
suggested by Stephen Farrel would like to thank all the implementers and users of the Multipath
TCP implementation in the Linux kernel. This document has benefited
from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley,
Jaehyun Hwang, Mirja Kuehlewind, Benoit Claise, Jari Arkko, Qin Wu,
Spencer Dawkins, and Ben Campbell.
Authors' Addresses Authors' Addresses
Olivier Bonaventure Olivier Bonaventure
UCLouvain UCLouvain
Email: Olivier.Bonaventure@uclouvain.be Email: Olivier.Bonaventure@uclouvain.be
Christoph Paasch Christoph Paasch
Apple, Inc. Apple, Inc.
 End of changes. 160 change blocks. 
560 lines changed or deleted 478 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/