draft-ietf-tcpm-rto-consider-17.txt   rfc8961.txt 
Internet Engineering Task Force M. Allman Internet Engineering Task Force (IETF) M. Allman
INTERNET-DRAFT ICSI Request for Comments: 8961 ICSI
File: draft-ietf-tcpm-rto-consider-17.txt July 27, 2020 BCP: 233 November 2020
Intended Status: Best Current Practice Category: Best Current Practice
Expires: January 27, 2021 ISSN: 2070-1721
Requirements for Time-Based Loss Detection Requirements for Time-Based Loss Detection
Status of this Memo Abstract
This Internet-Draft is submitted in full conformance with the Many protocols must detect packet loss for various reasons (e.g., to
provisions of BCP 78 and BCP 79. Internet-Drafts are working ensure reliability using retransmissions or to understand the level
documents of the Internet Engineering Task Force (IETF), its areas, of congestion along a network path). While many mechanisms have been
and its working groups. Note that other groups may also distribute designed to detect loss, ultimately, protocols can only count on the
working documents as Internet-Drafts. passage of time without delivery confirmation to declare a packet
"lost". Each implementation of a time-based loss detection mechanism
represents a balance between correctness and timeliness; therefore,
no implementation suits all situations. This document provides high-
level requirements for time-based loss detectors appropriate for
general use in unicast communication across the Internet. Within the
requirements, implementations have latitude to define particulars
that best address each situation.
Internet-Drafts are draft documents valid for a maximum of six Status of This Memo
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at This memo documents an Internet Best Current Practice.
http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be accessed at This document is a product of the Internet Engineering Task Force
http://www.ietf.org/shadow.html (IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Further information on
BCPs is available in Section 2 of RFC 7841.
This Internet-Draft will expire on January 27, 2021. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8961.
Copyright Notice Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with carefully, as they describe your rights and restrictions with respect
respect to this document. Code Components extracted from this to this document. Code Components extracted from this document must
document must include Simplified BSD License text as described in include Simplified BSD License text as described in Section 4.e of
Section 4.e of the Trust Legal Provisions and are provided without the Trust Legal Provisions and are provided without warranty as
warranty as described in the Simplified BSD License. described in the Simplified BSD License.
Abstract Table of Contents
Many protocols must detect packet loss for various reasons (e.g., to 1. Introduction
ensure reliability using retransmissions or to understand the level 1.1. Terminology
of congestion along a network path). While many mechanisms have 2. Context
been designed to detect loss, ultimately, protocols can only count 3. Scope
on the passage of time without delivery confirmation to declare a 4. Requirements
packet "lost". Each implementation of a time-based loss detection 5. Discussion
mechanism represents a balance between correctness and timeliness 6. Security Considerations
and therefore no implementation suits all situations. This document 7. IANA Considerations
provides high-level requirements for time-based loss detectors 8. References
appropriate for general use in unicast communication across the 8.1. Normative References
Internet. Within the requirements, implementations have latitude to 8.2. Informative References
define particulars that best address each situation. Acknowledgments
Author's Address
Terminology 1. Introduction
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", As a network of networks, the Internet consists of a large variety of
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and links and systems that support a wide variety of tasks and workloads.
"OPTIONAL" in this document are to be interpreted as described in The service provided by the network varies from best-effort delivery
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all among loosely connected components to highly predictable delivery
capitals, as shown here. within controlled environments (e.g., between physically connected
nodes, within a tightly controlled data center). Each path through
the network has a set of path properties, e.g., available capacity,
delay, and packet loss. Given the range of networks that make up the
Internet, these properties range from largely static to highly
dynamic.
1 Introduction This document provides guidelines for developing an understanding of
one path property: packet loss. In particular, we offer guidelines
for developing and implementing time-based loss detectors that have
been gradually learned over the last several decades. We focus on
the general case where the loss properties of a path are (a) unknown
a priori and (b) dynamically varying over time. Further, while there
are numerous root causes of packet loss, we leverage the conservative
notion that loss is an implicit indication of congestion [RFC5681].
While this stance is not always correct, as a general assumption it
has historically served us well [Jac88]. As we discuss further in
Section 2, the guidelines in this document should be viewed as a
general default for unicast communication across best-effort networks
and not as optimal -- or even applicable -- for all situations.
As a network of networks, the Internet consists of a large variety Given that packet loss is routine in best-effort networks, loss
of links and systems that support a wide variety of tasks and detection is a crucial activity for many protocols and applications
workloads. The service provided by the network varies from and is generally undertaken for two major reasons:
best-effort delivery among loosely connected components to highly
predictable delivery within controlled environments (e.g., between
physically connected nodes, within a tightly controlled data
center). Each path through the network has a set of path
properties---e.g., available capacity, delay, packet loss. Given
the range of networks that make up the Internet, these properties
range from largely static to highly dynamic.
This document provides guidelines for developing an understanding of (1) Ensuring reliable data delivery
one path property: packet loss. In particular, we offer guidelines
for developing and implementing time-based loss detectors that have
been gradually learned over the last several decades. We focus on
the general case where the loss properties of a path are (a) unknown
a priori and (b) dynamically vary over time. Further, while there
are numerous root causes of packet loss, we leverage the
conservative notion that loss is an implicit indication of
congestion [RFC5681]. While this stance is not always correct, as a
general assumption it has historically served us well [Jac88]. As
we discuss further in section 2, the guidelines in this document
should be viewed as a general default for unicast communication
across best-effort networks and not as optimal---or even
applicable---for all situations.
Given that packet loss is routine in best-effort networks, loss This requires a data sender to develop an understanding of which
detection is a crucial activity for many protocols and applications transmitted packets have not arrived at the receiver. This
and is generally undertaken for two major reasons: knowledge allows the sender to retransmit missing data.
(1) Ensuring reliable data delivery. (2) Congestion control
This requires a data sender to develop an understanding of As we mention above, packet loss is often taken as an implicit
which transmitted packets have not arrived at the receiver. indication that the sender is transmitting too fast and is
This knowledge allows the sender to retransmit missing data. overwhelming some portion of the network path. Data senders can
therefore use loss to trigger transmission rate reductions.
(2) Congestion control. Various mechanisms are used to detect losses in a packet stream.
Often, we use continuous or periodic acknowledgments from the
recipient to inform the sender's notion of which pieces of data are
missing. However, despite our best intentions and most robust
mechanisms, we cannot place ultimate faith in receiving such
acknowledgments but can only truly depend on the passage of time.
Therefore, our ultimate backstop to ensuring that we detect all loss
is a timeout. That is, the sender sets some expectation for how long
to wait for confirmation of delivery for a given piece of data. When
this time period passes without delivery confirmation, the sender
concludes the data was lost in transit.
As we mention above, packet loss is often taken as an The specifics of time-based loss detection schemes represent a
implicit indication that the sender is transmitting too fast tradeoff between correctness and responsiveness. In other words, we
and is overwhelming some portion of the network path. Data wish to simultaneously:
senders can therefore use loss to trigger transmission rate
reductions.
Various mechanisms are used to detect losses in a packet stream. * wait long enough to ensure the detection of loss is correct, and
Often we use continuous or periodic acknowledgments from the
recipient to inform the sender's notion of which pieces of data are
missing. However, despite our best intentions and most robust
mechanisms we cannot place ultimate faith in receiving such
acknowledgments, but can only truly depend on the passage of time.
Therefore, our ultimate backstop to ensuring that we detect all loss
is a timeout. That is, the sender sets some expectation for how
long to wait for confirmation of delivery for a given piece of data.
When this time period passes without delivery confirmation the
sender concludes the data was lost in transit.
The specifics of time-based loss detection schemes represent a * minimize the amount of delay we impose on applications (before
tradeoff between correctness and responsiveness. In other words we repairing loss) and the network (before we reduce the congestion).
wish to simultaneously:
- wait long enough to ensure the detection of loss is correct, and Serving both of these goals is difficult, as they pull in opposite
directions [AP99]. By not waiting long enough to accurately
determine a packet has been lost, we may provide a needed
retransmission in a timely manner but risk both sending unnecessary
("spurious") retransmissions and needlessly lowering the transmission
rate. By waiting long enough that we are unambiguously certain a
packet has been lost, we cannot repair losses in a timely manner and
we risk prolonging network congestion.
- minimize the amount of delay we impose on applications (before Many protocols and applications -- such as TCP [RFC6298], SCTP
repairing loss) and the network (before we reduce the [RFC4960], and SIP [RFC3261] -- use their own time-based loss
congestion). detection mechanisms. At this point, our experience leads to a
recognition that often specific tweaks that deviate from standardized
time-based loss detectors do not materially impact network safety
with respect to congestion control [AP99]. Therefore, in this
document we outline a set of high-level, protocol-agnostic
requirements for time-based loss detection. The intent is to provide
a safe foundation on which implementations have the flexibility to
instantiate mechanisms that best realize their specific goals.
Serving both of these goals is difficult as they pull in opposite 1.1. Terminology
directions [AP99]. By not waiting long enough to accurately
determine a packet has been lost we may provide a needed
retransmission in a timely manner, but risk sending unnecessary
("spurious") retransmissions and needlessly lowering the
transmission rate. By waiting long enough that we are unambiguously
certain a packet has been lost we cannot repair losses in a timely
manner and we risk prolonging network congestion.
Many protocols and applications---such as TCP [RFC6298], SCTP The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
[RFC4960], SIP [RFC3261]---use their own time-based loss detection "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
mechanisms. At this point, our experience leads to a recognition "OPTIONAL" in this document are to be interpreted as described in
that often specific tweaks that deviate from standardized time-based BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
loss detectors do not materially impact network safety with respect capitals, as shown here.
to congestion control [AP99]. Therefore, in this document we
outline a set of high-level protocol-agnostic requirements for
time-based loss detection. The intent is to provide a safe
foundation on which implementations have the flexibility to
instantiate mechanisms that best realize their specific goals.
2 Context 2. Context
This document is different from the way we ideally like to engineer This document is different from the way we ideally like to engineer
systems. Usually, we strive to understand high-level requirements systems. Usually, we strive to understand high-level requirements as
as a starting point. We then methodically engineer specific a starting point. We then methodically engineer specific protocols,
protocols, algorithms and systems that meet these requirements. algorithms, and systems that meet these requirements. Within the
Within the IETF standards process we have derived many time-based IETF standards process, we have derived many time-based loss
loss detection schemes without benefit from some over-arching detection schemes without the benefit of some over-arching
requirements document---because we had no idea how to write such a requirements document -- because we had no idea how to write such a
document! Therefore, we made the best specific decisions we could document! Therefore, we made the best specific decisions we could in
in response to specific needs. response to specific needs.
At this point, however, the community's experience has matured to At this point, however, the community's experience has matured to the
the point where we can define a set of general, high-level point where we can define a set of general, high-level requirements
requirements for time-based loss detection schemes. We now for time-based loss detection schemes. We now understand how to
understand how to separate the strategies these mechanisms use that separate the strategies these mechanisms use that are crucial for
are crucial for network safety from those small details that do not network safety from those small details that do not materially impact
materially impact network safety. The requirements in this document network safety. The requirements in this document may not be
may not be appropriate in all cases. In particular, the guidelines appropriate in all cases. In particular, the guidelines in Section 4
in section 4 are concerned with the general case, but specific are concerned with the general case, but specific situations may
situations may allow for more flexibility in terms of loss detection allow for more flexibility in terms of loss detection because
because specific facets of the environment are known (e.g., when specific facets of the environment are known (e.g., when operating
operating over a single physical link or within a tightly controlled over a single physical link or within a tightly controlled data
data center). Therefore, variants, deviations or wholly different center). Therefore, variants, deviations, or wholly different time-
time-based loss detectors may be necessary or useful in some cases. based loss detectors may be necessary or useful in some cases. The
The correct way to view this document is as the default case and not correct way to view this document is as the default case and not as
as a one-size-fits-all that is optimal in all cases. one-size-fits-all guidance that is optimal in all cases.
Adding a requirements umbrella to a body of existing specifications Adding a requirements umbrella to a body of existing specifications
is inherently messy and we run the risk of creating inconsistencies is inherently messy and we run the risk of creating inconsistencies
with both past and future mechanisms. Therefore, we make the with both past and future mechanisms. Therefore, we make the
following statements about the relationship of this document to past following statements about the relationship of this document to past
and future specifications: and future specifications:
- This document does not update or obsolete any existing RFC. * This document does not update or obsolete any existing RFC. These
These previous specifications---while generally consistent with previous specifications -- while generally consistent with the
the requirements in this document---reflect community consensus requirements in this document -- reflect community consensus, and
and this document does not change that consensus. this document does not change that consensus.
- The requirements in this document are meant to provide for * The requirements in this document are meant to provide for network
network safety and, as such, SHOULD be used by all future safety and, as such, SHOULD be used by all future time-based loss
time-based loss detection mechanisms. detection mechanisms.
- The requirements in this document may not be appropriate in all * The requirements in this document may not be appropriate in all
cases and, therefore, deviations and variants may be necessary cases; therefore, deviations and variants may be necessary in the
in the future (hence the "SHOULD" in the last bullet). However, future (hence the "SHOULD" in the last bullet). However,
inconsistencies MUST be (a) explained and (b) gather consensus. inconsistencies MUST be (a) explained and (b) gather consensus.
3 Scope 3. Scope
The principles we outline in this document are protocol-agnostic and The principles we outline in this document are protocol-agnostic and
widely applicable. We make the following scope statements about widely applicable. We make the following scope statements about the
the application of the requirements discussed in Section 4: application of the requirements discussed in Section 4:
(S.1) While there are a bevy of uses for timers in protocols---from (S.1) While there are a bevy of uses for timers in protocols -- from
rate-based pacing to connection failure detection and rate-based pacing to connection failure detection and beyond --
beyond---this document is focused only on loss detection. this document is focused only on loss detection.
(S.2) The requirements for time-based loss detection mechanisms in (S.2) The requirements for time-based loss detection mechanisms in
this document are for the primary or "last resort" loss this document are for the primary or "last resort" loss
detection mechanism whether the mechanism is the sole loss detection mechanism, whether the mechanism is the sole loss
repair strategy or works in concert with other mechanisms. repair strategy or works in concert with other mechanisms.
While a straightforward time-based loss detector is sufficient While a straightforward time-based loss detector is sufficient
for simple protocols like DNS [RFC1034,RFC1035], more complex for simple protocols like DNS [RFC1034] [RFC1035], more complex
protocols often use more advanced loss detectors to aid protocols often use more advanced loss detectors to aid
performance. For instance, TCP and SCTP have methods to performance. For instance, TCP and SCTP have methods to detect
detect (and repair) loss based on explicit endpoint state (and repair) loss based on explicit endpoint state sharing
sharing [RFC2018,RFC4960,RFC6675]. Such mechanisms often [RFC2018] [RFC4960] [RFC6675]. Such mechanisms often provide
provide more timely and precise loss detection than time-based more timely and precise loss detection than time-based loss
loss detectors. However, these mechanisms do not obviate the detectors. However, these mechanisms do not obviate the need
need for a "retransmission timeout" or "RTO" because---as we for a "retransmission timeout" or "RTO" because, as we discuss
discuss in Section 1---only the passage of time can ultimately in Section 1, only the passage of time can ultimately be relied
be relied upon to detect loss. In other words, ultimately we upon to detect loss. In other words, we ultimately cannot
cannot count on acknowledgments to arrive at the data sender count on acknowledgments to arrive at the data sender to
to indicate which packets never arrived at the receiver. In indicate which packets never arrived at the receiver. In cases
cases such as these we need a time-based loss detector to such as these, we need a time-based loss detector to function
functions as a "last resort". as a "last resort".
Also, note, that some recent proposals have incorporated time Also, note that some recent proposals have incorporated time as
as a component of advanced loss detection methods---either as a component of advanced loss detection methods either as an
an aggressive first loss detector in certain situations or in aggressive first loss detector in certain situations or in
conjunction with endpoint state sharing [DCCM13,CCDJ20,IS20]. conjunction with endpoint state sharing [DCCM13] [CCDJ20]
While these mechanisms can aid timely loss recovery, the [IS20]. While these mechanisms can aid timely loss recovery,
protocol ultimately leans on another more conservative timer the protocol ultimately leans on another more conservative
to ensure reliability when these mechanisms break down. The timer to ensure reliability when these mechanisms break down.
requirements in this document are only directly applicable to The requirements in this document are only directly applicable
last resort loss detection. However, we expect that many of to last-resort loss detection. However, we expect that many of
the requirements can serve as useful guidelines for more the requirements can serve as useful guidelines for more
aggressive non-last resort timers, as well. aggressive non-last-resort timers as well.
(S.3) The requirements in this document apply only to endpoint-to- (S.3) The requirements in this document apply only to endpoint-to-
endpoint unicast communication. Reliable multicast (e.g., endpoint unicast communication. Reliable multicast (e.g.,
[RFC5740]) protocols are explicitly outside the scope of this [RFC5740]) protocols are explicitly outside the scope of this
document. document.
Protocols such as SCTP [RFC4960] and MP-TCP [RFC6182] that Protocols such as SCTP [RFC4960] and Multipath TCP (MP-TCP)
communicate in a unicast fashion with multiple specific [RFC6182] that communicate in a unicast fashion with multiple
endpoints can leverage the requirements in this document specific endpoints can leverage the requirements in this
provided they track state and follow the requirements for each document provided they track state and follow the requirements
endpoint independently. I.e., if host A communicates with for each endpoint independently. That is, if host A
addresses B and C, A needs to use independent time-based loss communicates with addresses B and C, A needs to use independent
detector instances for traffic sent to B and C. time-based loss detector instances for traffic sent to B and C.
(S.4) There are cases where state is shared across connections (S.4) There are cases where state is shared across connections or
or flows (e.g., [RFC2140], [RFC3124]). State pertaining to flows (e.g., [RFC2140] and [RFC3124]). State pertaining to
time-based loss detection is often discussed as sharable. time-based loss detection is often discussed as sharable.
These situations raise issues that the simple flow-oriented These situations raise issues that the simple flow-oriented
time-based loss detection mechanism discussed in this document time-based loss detection mechanism discussed in this document
does not consider (e.g., how long to preserve state between does not consider (e.g., how long to preserve state between
connections). Therefore, while the general principles given connections). Therefore, while the general principles given in
in Section 4 are likely applicable, sharing time-based loss Section 4 are likely applicable, sharing time-based loss
detection information across flows is outside the scope of detection information across flows is outside the scope of this
this document. document.
4 Requirements 4. Requirements
We now list the requirements that apply when designing primary or We now list the requirements that apply when designing primary or
last resort time-based loss detection mechanisms. For historical last-resort time-based loss detection mechanisms. For historical
reasons and ease of exposition, we refer to the time between sending reasons and ease of exposition, we refer to the time between sending
a packet and determining the packet has been lost due to lack of a packet and determining the packet has been lost due to lack of
delivery confirmation as the "retransmission timeout" or "RTO". delivery confirmation as the "retransmission timeout" or "RTO".
After the RTO passes without delivery confirmation, the sender may After the RTO passes without delivery confirmation, the sender may
safely assume the packet is lost. However, as discussed above, the safely assume the packet is lost. However, as discussed above, the
detected loss need not be repaired (i.e., the loss could be detected detected loss need not be repaired (i.e., the loss could be detected
only for congestion control and not reliability purposes). only for congestion control and not reliability purposes).
(1) As we note above, loss detection happens when a sender does not (1) As we note above, loss detection happens when a sender does not
receive delivery confirmation within some expected period of receive delivery confirmation within some expected period of
time. In the absence of any knowledge about the latency of a time. In the absence of any knowledge about the latency of a
path, the initial RTO MUST be conservatively set to no less than path, the initial RTO MUST be conservatively set to no less than
1 second. 1 second.
Correctness is of the utmost importance when transmitting into a Correctness is of the utmost importance when transmitting into a
network with unknown properties because: network with unknown properties because:
- Premature loss detection can trigger spurious retransmits that * Premature loss detection can trigger spurious retransmits
could cause issues when a network is already congested. that could cause issues when a network is already congested.
- Premature loss detection can needlessly cause congestion * Premature loss detection can needlessly cause congestion
control to dramatically lower the sender's allowed control to dramatically lower the sender's allowed
transmission rate---especially since the rate is already transmission rate, especially since the rate is already
likely low at this stage of the communication. Recovering likely low at this stage of the communication. Recovering
from such a rate change can taken a relatively long time. from such a rate change can take a relatively long time.
- Finally, as discussed below, sometimes using time-based * Finally, as discussed below, sometimes using time-based loss
loss detection and retransmissions can cause ambiguities in detection and retransmissions can cause ambiguities in
assessing the latency of a network path. Therefore, it is assessing the latency of a network path. Therefore, it is
especially important for the first latency sample to be free especially important for the first latency sample to be free
of ambiguities such that there is a baseline for the remainder of ambiguities such that there is a baseline for the
of the communication. remainder of the communication.
The specific constant (1 second) comes from the analysis of The specific constant (1 second) comes from the analysis of
Internet RTTs found in Appendix A of [RFC6298]. Internet round-trip times (RTTs) found in Appendix A of
[RFC6298].
(2) We now specify four requirements that pertain to setting (2) We now specify four requirements that pertain to setting an
an expected time interval for delivery confirmation. expected time interval for delivery confirmation.
Often measuring the time required for delivery confirmation is Often, measuring the time required for delivery confirmation is
framed as assessing the "round-trip time (RTT)" of the network framed as assessing the RTT of the network path. The RTT is the
path. The RTT is the minimum amount of time required to receive minimum amount of time required to receive delivery confirmation
delivery confirmation and also often follows protocol behavior and also often follows protocol behavior whereby acknowledgments
whereby acknowledgments are generated quickly after data are generated quickly after data arrives. For instance, this is
arrives. For instance, this is the case for the RTO used by TCP the case for the RTO used by TCP [RFC6298] and SCTP [RFC4960].
[RFC6298] and SCTP [RFC4960]. However, this is somewhat However, this is somewhat misleading, and the expected latency
mis-leading and the expected latency is better framed as the is better framed as the "feedback time" (FT). In other words,
"feedback time" (FT). In other words, the expectation is not the expectation is not always simply a network property; it can
always simply a network property, but can include additional include additional time before a sender should reasonably expect
time before a sender should reasonably expect a response. a response.
For instance, consider a UDP-based DNS request from a client to For instance, consider a UDP-based DNS request from a client to
a recursive resolver [RFC1035]. When the request can be served a recursive resolver [RFC1035]. When the request can be served
from the resolver's cache the FT likely well approximates the from the resolver's cache, the feedback time (FT) likely well
network RTT between the client and resolver. However, on a approximates the network RTT between the client and resolver.
cache miss the resolver will request the needed information from However, on a cache miss, the resolver will request the needed
one or more authoritative DNS servers, which will non-trivially information from one or more authoritative DNS servers, which
increase the FT compared to the network RTT between the client will non-trivially increase the FT compared to the network RTT
and resolver. between the client and resolver.
Therefore, we express the requirements in terms of FT. Again, Therefore, we express the requirements in terms of FT. Again,
for ease of exposition we use "RTO" to indicate the interval for ease of exposition, we use "RTO" to indicate the interval
between a packet transmission and the decision the packet has between a packet transmission and the decision that the packet
been lost---regardless of whether the packet will be has been lost, regardless of whether the packet will be
retransmitted. retransmitted.
(a) The RTO SHOULD be set based on multiple observations of the (a) The RTO SHOULD be set based on multiple observations of the
FT when available. FT when available.
In other words, the RTO should represent an empirically- In other words, the RTO should represent an empirically
derived reasonable amount of time that the sender should derived reasonable amount of time that the sender should
wait for delivery confirmation before deciding the given wait for delivery confirmation before deciding the given
data is lost. Network paths are inherently dynamic and data is lost. Network paths are inherently dynamic;
therefore it is crucial to incorporate multiple recent FT therefore, it is crucial to incorporate multiple recent FT
samples in the RTO to take into account the delay variation samples in the RTO to take into account the delay variation
across time. across time.
For example, TCP's RTO [RFC6298] would satisfy this For example, TCP's RTO [RFC6298] would satisfy this
requirement due to its use of an exponentially-weighted requirement due to its use of an exponentially weighted
moving average (EWMA) to combine multiple FT samples into a moving average (EWMA) to combine multiple FT samples into a
"smoothed RTT". In the name of conservativeness, TCP goes "smoothed RTT". In the name of conservativeness, TCP goes
further to also include an explicit variance term when further to also include an explicit variance term when
computing the RTO. computing the RTO.
While multiple FT samples are crucial for capturing the While multiple FT samples are crucial for capturing the
delay dynamics of a path, we explicitly do not tightly delay dynamics of a path, we explicitly do not tightly
specify the process---including the number of FT samples to specify the process -- including the number of FT samples
use and how/when to age samples out of the RTO to use and how/when to age samples out of the RTO
calculation---as the particulars could depend on the calculation -- as the particulars could depend on the
situation and/or goals of each specific loss detector. situation and/or goals of each specific loss detector.
Finally, FT samples come from packet exchanges between Finally, FT samples come from packet exchanges between
peers. We encourage protocol designers---especially for new peers. We encourage protocol designers -- especially for
protocols---to strive to ensure the feedback is not easily new protocols -- to strive to ensure the feedback is not
spoofable by on- or off-path attackers such that they can easily spoofable by on- or off-path attackers such that
perturb a host's notion of the FT. Ideally, all messages they can perturb a host's notion of the FT. Ideally, all
would be cryptographically secure, but given that this is messages would be cryptographically secure, but given that
not always possible---especially in legacy protocols---using this is not always possible -- especially in legacy
a healthy amount of randomness in the packets is encouraged. protocols -- using a healthy amount of randomness in the
packets is encouraged.
(b) FT observations SHOULD be taken and incorporated into the (b) FT observations SHOULD be taken and incorporated into the
RTO at least once per RTT or as frequently as data is RTO at least once per RTT or as frequently as data is
exchanged in cases where that happens less frequently than exchanged in cases where that happens less frequently than
once per RTT. once per RTT.
Internet measurements show that taking only a single FT Internet measurements show that taking only a single FT
sample per TCP connection results in a relatively poorly sample per TCP connection results in a relatively poorly
performing RTO mechanism [AP99], hence this requirement that performing RTO mechanism [AP99], hence this requirement
the FT be sampled continuously throughout the lifetime of that the FT be sampled continuously throughout the lifetime
communication. of communication.
As an example, TCP takes an FT sample roughly once per RTT, As an example, TCP takes an FT sample roughly once per RTT,
or if using the timestamp option [RFC7323] on each or, if using the timestamp option [RFC7323], on each
acknowledgment arrival. [AP99] shows that both these acknowledgment arrival. [AP99] shows that both these
approaches result in roughly equivalent performance for the approaches result in roughly equivalent performance for the
RTO estimator. RTO estimator.
(c) FT observations MAY be taken from non-data exchanges. (c) FT observations MAY be taken from non-data exchanges.
Some protocols use non-data exchanges for various Some protocols use non-data exchanges for various reasons,
reasons---e.g., keepalives, heartbeats, control messages. e.g., keepalives, heartbeats, and control messages. To the
To the extent that the latency of these exchanges mirrors extent that the latency of these exchanges mirrors data
data exchange, they can be leveraged to take FT samples exchange, they can be leveraged to take FT samples within
within the RTO mechanism. Such samples can help protocols the RTO mechanism. Such samples can help protocols keep
keep their RTO accurate during lulls in data transmission. their RTO accurate during lulls in data transmission.
However, given that these messages may not be subject to the However, given that these messages may not be subject to
same delays as data transmission, we do not take a general the same delays as data transmission, we do not take a
view on whether this is useful or not. general view on whether this is useful or not.
(d) An RTO mechanism MUST NOT use ambiguous FT samples. (d) An RTO mechanism MUST NOT use ambiguous FT samples.
Assume two copies of some packet X are transmitted at times Assume two copies of some packet X are transmitted at times
t0 and t1 and then at time t2 the sender receives t0 and t1. Then, at time t2, the sender receives
confirmation that X in fact arrived. In some cases, it is confirmation that X in fact arrived. In some cases, it is
not clear which copy of X triggered the confirmation and not clear which copy of X triggered the confirmation;
hence the actual FT is either t2-t1 or t2-t0, but which is a hence, the actual FT is either t2-t1 or t2-t0, but which is
mystery. Therefore, in this situation an implementation a mystery. Therefore, in this situation, an implementation
MUST NOT use either version of the FT sample and hence not MUST NOT use either version of the FT sample and hence not
update the RTO (as discussed in [KP87,RFC6298]). update the RTO (as discussed in [KP87] and [RFC6298]).
There are cases where two copies of some data are There are cases where two copies of some data are
transmitted in a way whereby the sender can tell which is transmitted in a way whereby the sender can tell which is
being acknowledged by an incoming ACK. E.g., TCP's being acknowledged by an incoming ACK. For example, TCP's
timestamp option [RFC7323] allows for packets to be timestamp option [RFC7323] allows for packets to be
uniquely identified and hence avoid the ambiguity. In such uniquely identified and hence avoid the ambiguity. In such
cases there is no ambiguity and the resulting samples can cases, there is no ambiguity and the resulting samples can
update the RTO. update the RTO.
(3) Loss detected by the RTO mechanism MUST be taken as an (3) Loss detected by the RTO mechanism MUST be taken as an
indication of network congestion and the sending rate adapted indication of network congestion and the sending rate adapted
using a standard mechanism (e.g., TCP collapses the congestion using a standard mechanism (e.g., TCP collapses the congestion
window to one packet [RFC5681]). window to one packet [RFC5681]).
This ensures network safety. This ensures network safety.
An exception to this rule is if an IETF standardized mechanism An exception to this rule is if an IETF standardized mechanism
determines that a particular loss is due to a non-congestion determines that a particular loss is due to a non-congestion
event (e.g., packet corruption). In such a case a congestion event (e.g., packet corruption). In such a case, a congestion
control action is not required. Additionally, congestion control action is not required. Additionally, congestion
control actions taken based on time-based loss detection could control actions taken based on time-based loss detection could
be reversed when a standard mechanism post-facto determines that be reversed when a standard mechanism post facto determines that
the cause of the loss was not congestion (e.g., [RFC5682]). the cause of the loss was not congestion (e.g., [RFC5682]).
(4) Each time the RTO is used to detect a loss, the value of the RTO (4) Each time the RTO is used to detect a loss, the value of the RTO
MUST be exponentially backed off such that the next firing MUST be exponentially backed off such that the next firing
requires a longer interval. The backoff SHOULD be removed after requires a longer interval. The backoff SHOULD be removed after
either (a) the subsequent successful transmission of either (a) the subsequent successful transmission of non-
non-retransmitted data, or (b) an RTO passes without detecting retransmitted data, or (b) an RTO passes without detecting
additional losses. The former will generally be quicker. The additional losses. The former will generally be quicker. The
latter covers cases where loss is detected, but not repaired. latter covers cases where loss is detected but not repaired.
A maximum value MAY be placed on the RTO. The maximum RTO MUST A maximum value MAY be placed on the RTO. The maximum RTO MUST
NOT be less than 60 seconds (as specified in [RFC6298]). NOT be less than 60 seconds (as specified in [RFC6298]).
This ensures network safety. This ensures network safety.
As with guideline (3), an exception to this rule exists if an As with guideline (3), an exception to this rule exists if an
IETF standardized mechanism determines that a particular loss is IETF standardized mechanism determines that a particular loss is
not due to congestion. not due to congestion.
5 Discussion 5. Discussion
We note that research has shown the tension between the We note that research has shown the tension between the
responsiveness and correctness of time-based loss detection seems to responsiveness and correctness of time-based loss detection seems to
be a fundamental tradeoff in the context of TCP [AP99]. That is, be a fundamental tradeoff in the context of TCP [AP99]. That is,
making the RTO more aggressive (e.g., via changing TCP's making the RTO more aggressive (e.g., via changing TCP's
exponentially weighted moving average (EWMA) gains, lowering the exponentially weighted moving average (EWMA) gains, lowering the
minimum RTO, etc.) can reduce the time required to detect actual minimum RTO, etc.) can reduce the time required to detect actual
loss. However, at the same time, such aggressiveness leads to more loss. However, at the same time, such aggressiveness leads to more
cases of mistakenly declaring packets lost that ultimately arrived cases of mistakenly declaring packets lost that ultimately arrived at
at the receiver. Therefore, being as aggressive as the requirements the receiver. Therefore, being as aggressive as the requirements
given in the previous section allow in any particular situation may given in the previous section allow in any particular situation may
not be the best course of action because detecting loss---even if not be the best course of action because detecting loss, even if
falsely---carries a requirement to invoke a congestion response falsely, carries a requirement to invoke a congestion response that
which will ultimately reduce the transmission rate. will ultimately reduce the transmission rate.
While the tradeoff between responsiveness and correctness seems While the tradeoff between responsiveness and correctness seems
fundamental, the tradeoff can be made less relevant if the sender fundamental, the tradeoff can be made less relevant if the sender can
can detect and recover from mistaken loss detection. Several detect and recover from mistaken loss detection. Several mechanisms
mechanisms have been proposed for this purpose, such as Eifel have been proposed for this purpose, such as Eifel [RFC3522], Forward
[RFC3522], F-RTO [RFC5682] and DSACK [RFC2883,RFC3708]. Using such RTO-Recovery (F-RTO) [RFC5682], and Duplicate Selective
mechanisms may allow a data originator to tip towards being more Acknowledgement (DSACK) [RFC2883] [RFC3708]. Using such mechanisms
responsive without incurring (as much of) the attendant costs of may allow a data originator to tip towards being more responsive
mistakenly declaring packets to be lost. without incurring (as much of) the attendant costs of mistakenly
declaring packets to be lost.
Also, note that, in addition to the experiments discussed in [AP99], Also, note that, in addition to the experiments discussed in [AP99],
the Linux TCP implementation has been using various non-standard RTO the Linux TCP implementation has been using various non-standard RTO
mechanisms for many years seemingly without large-scale problems mechanisms for many years seemingly without large-scale problems
(e.g., using different EWMA gains than specified in [RFC6298]). (e.g., using different EWMA gains than specified in [RFC6298]).
Further, a number of TCP implementations use a steady-state minimum Further, a number of TCP implementations use a steady-state minimum
RTO that is less than the 1 second specified in [RFC6298]. While RTO that is less than the 1 second specified in [RFC6298]. While the
the implication of these deviations from the standard may be more implication of these deviations from the standard may be more
spurious retransmits (per [AP99]), we are aware of no large-scale spurious retransmits (per [AP99]), we are aware of no large-scale
network safety issues caused by this change to the minimum RTO. network safety issues caused by this change to the minimum RTO. This
This informs the guidelines in the last section (e.g., there is no informs the guidelines in the last section (e.g., there is no minimum
minimum RTO specified). RTO specified).
Finally, we note that while allowing implementations to be more Finally, we note that while allowing implementations to be more
aggressive could in fact increase the number of needless aggressive could in fact increase the number of needless
retransmissions, the above requirements fail safe in that they retransmissions, the above requirements fail safely in that they
insist on exponential backoff and a transmission rate reduction. insist on exponential backoff and a transmission rate reduction.
Therefore, providing implementers more latitude than they have Therefore, providing implementers more latitude than they have
traditionally been given in IETF specifications of RTO mechanisms traditionally been given in IETF specifications of RTO mechanisms
does not somehow open the flood gates to aggressive behavior. Since does not somehow open the flood gates to aggressive behavior. Since
there is a downside to being aggressive, the incentives for proper there is a downside to being aggressive, the incentives for proper
behavior are retained in the mechanism. behavior are retained in the mechanism.
6 Security Considerations 6. Security Considerations
This document does not alter the security properties of time-based This document does not alter the security properties of time-based
loss detection mechanisms. See [RFC6298] for a discussion of these loss detection mechanisms. See [RFC6298] for a discussion of these
within the context of TCP. within the context of TCP.
7 IANA Considerations 7. IANA Considerations
This document has no IANA considerations. This document has no IANA actions.
Acknowledgments 8. References
This document benefits from years of discussions with Ethan Blanton, 8.1. Normative References
Sally Floyd, Jana Iyengar, Shawn Ostermann, Vern Paxson, and the
members of the TCPM and TCP-IMPL working groups. Ran Atkinson,
Yuchung Cheng, David Black, Stewart Bryant, Martin Duke, Wesley
Eddy, Gorry Fairhurst, Rahul Arvind Jadhav, Benjamin Kaduk, Mirja
Kuhlewind, Nicolas Kuhn, Jonathan Looney and Michael Scharf provided
useful comments on previous versions of this document.
Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
Requirement Levels", BCP 14, RFC 2119, March 1997. 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 8.2. Informative References
2119 Key Words", RFC 8174, May 2017.
Informative References [AP99] Allman, M. and V. Paxson, "On Estimating End-to-End
Network Path Properties", Proceedings of the ACM SIGCOMM
Technical Symposium, September 1999.
[AP99] Allman, M., V. Paxson, "On Estimating End-to-End Network Path [CCDJ20] Cheng, Y., Cardwell, N., Dukkipati, N., and P. Jha, "The
Properties", Proceedings of the ACM SIGCOMM Technical Symposium, RACK-TLP loss detection algorithm for TCP", Work in
September 1999. Progress, Internet-Draft, draft-ietf-tcpm-rack-13, 2
November 2020,
<https://tools.ietf.org/html/draft-ietf-tcpm-rack-13>.
[CCDJ20] Cheng, Y., N. Cardwell, N. Dukkipati, P. Jha, "RACK: a [DCCM13] Dukkipati, N., Cardwell, N., Cheng, Y., and M. Mathis,
time-based fast loss detection algorithm for TCP", "Tail Loss Probe (TLP): An Algorithm for Fast Recovery of
Internet-Draft draft-ietf-tcpm-rack-08.txt (work in progress), Tail Losses", Work in Progress, Internet-Draft, draft-
March 2020. dukkipati-tcpm-tcp-loss-probe-01, 25 February 2013,
<https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-
loss-probe-01>.
[DCCM13] Dukkipati, N., N. Cardwell, Y. Cheng, M. Mathis, "Tail Loss [IS20] Iyengar, J., Ed. and I. Swett, Ed., "QUIC Loss Detection
Probe (TLP): An Algorithm for Fast Recovery of Tail Losses", and Congestion Control", Work in Progress, Internet-Draft,
Internet-Draft draft-dukkipati-tcpm-tcp-loss-probe-01.txt (work draft-ietf-quic-recovery-32, 20 October 2020,
in progress), February 2013. <https://tools.ietf.org/html/draft-ietf-quic-recovery-32>.
[IS20] Iyengar, I., I. Swett, "QUIC Loss Detection and Congestion [Jac88] Jacobson, V., "Congestion avoidance and control", ACM
Control", Internet-Draft SIGCOMM, DOI 10.1145/52325.52356, August 1988,
draft-ietf-quic-recovery-27.txt (work in progress), March 2020. <https://doi.org/10.1145/52325.52356>.
[Jac88] Jacobson, V., "Congestion Avoidance and Control", ACM [KP87] Karn, P. and C. Partridge, "Improving Round-Trip Time
SIGCOMM, August 1988. Estimates in Reliable Transport Protocols", SIGCOMM 87.
[KP87] Karn, P. and C. Partridge, "Improving Round-Trip Time [RFC1034] Mockapetris, P., "Domain names - concepts and facilities",
Estimates in Reliable Transport Protocols", SIGCOMM 87. STD 13, RFC 1034, DOI 10.17487/RFC1034, November 1987,
<https://www.rfc-editor.org/info/rfc1034>.
[RFC1034] Mockapetris, P. "Domain Names - Concepts and Facilities", [RFC1035] Mockapetris, P., "Domain names - implementation and
RFC 1034, November 1987. specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
November 1987, <https://www.rfc-editor.org/info/rfc1035>.
[RFC1035] Mockapetris, P. "Domain Names - Implementation and [RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP
Specification", RFC 1035, November 1987. Selective Acknowledgment Options", RFC 2018,
DOI 10.17487/RFC2018, October 1996,
<https://www.rfc-editor.org/info/rfc2018>.
[RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP [RFC2140] Touch, J., "TCP Control Block Interdependence", RFC 2140,
Selective Acknowledgment Options", RFC 2018, October 1996. DOI 10.17487/RFC2140, April 1997,
<https://www.rfc-editor.org/info/rfc2140>.
[RFC2140] Touch, J., "TCP Control Block Interdependence", RFC 2140, [RFC2883] Floyd, S., Mahdavi, J., Mathis, M., and M. Podolsky, "An
April 1997. Extension to the Selective Acknowledgement (SACK) Option
for TCP", RFC 2883, DOI 10.17487/RFC2883, July 2000,
<https://www.rfc-editor.org/info/rfc2883>.
[RFC2883] Floyd, S., Mahdavi, J., Mathis, M., and M. Podolsky, "An [RFC3124] Balakrishnan, H. and S. Seshan, "The Congestion Manager",
Extension to the Selective Acknowledgement (SACK) Option for RFC 3124, DOI 10.17487/RFC3124, June 2001,
TCP", RFC 2883, July 2000. <https://www.rfc-editor.org/info/rfc3124>.
[RFC3124] Balakrishnan, H., S. Seshan, "The Congestion Manager", RFC [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
3124, June 2001. A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261,
DOI 10.17487/RFC3261, June 2002,
<https://www.rfc-editor.org/info/rfc3261>.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, [RFC3522] Ludwig, R. and M. Meyer, "The Eifel Detection Algorithm
A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, for TCP", RFC 3522, DOI 10.17487/RFC3522, April 2003,
"SIP: Session Initiation Protocol", RFC 3261, June 2002. <https://www.rfc-editor.org/info/rfc3522>.
[RFC3522] Ludwig, R., M. Meyer, "The Eifel Detection Algorithm for [RFC3708] Blanton, E. and M. Allman, "Using TCP Duplicate Selective
TCP", RFC 3522, april 2003. Acknowledgement (DSACKs) and Stream Control Transmission
Protocol (SCTP) Duplicate Transmission Sequence Numbers
(TSNs) to Detect Spurious Retransmissions", RFC 3708,
DOI 10.17487/RFC3708, February 2004,
<https://www.rfc-editor.org/info/rfc3708>.
[RFC3708] Blanton, E., M. Allman, "Using TCP Duplicate Selective [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol",
Acknowledgement (DSACKs) and Stream Control Transmission RFC 4960, DOI 10.17487/RFC4960, September 2007,
Protocol (SCTP) Duplicate Transmission Sequence Numbers (TSNs) <https://www.rfc-editor.org/info/rfc4960>.
to Detect Spurious Retransmissions", RFC 3708, February 2004.
[RFC4960] Stweart, R., "Stream Control Transmission Protocol", RFC [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
4960, September 2007. Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<https://www.rfc-editor.org/info/rfc5681>.
[RFC5681] Allman, M., V. Paxson, E. Blanton, "TCP Congestion [RFC5682] Sarolahti, P., Kojo, M., Yamamoto, K., and M. Hata,
Control", RFC 5681, September 2009. "Forward RTO-Recovery (F-RTO): An Algorithm for Detecting
Spurious Retransmission Timeouts with TCP", RFC 5682,
DOI 10.17487/RFC5682, September 2009,
<https://www.rfc-editor.org/info/rfc5682>.
[RFC5682] Sarolahti, P., M. Kojo, K. Yamamoto, M. Hata, "Forward [RFC5740] Adamson, B., Bormann, C., Handley, M., and J. Macker,
RTO-Recovery (F-RTO): An Algorithm for Detecting Spurious "NACK-Oriented Reliable Multicast (NORM) Transport
Retransmission Timeouts with TCP", RFC 5682, September 2009. Protocol", RFC 5740, DOI 10.17487/RFC5740, November 2009,
<https://www.rfc-editor.org/info/rfc5740>.
[RFC5740] Adamson, B., C. Bormann, M. Handley, J. Macker, [RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J.
"NACK-Oriented Reliable Multicast (NORM) Transport Protocol", Iyengar, "Architectural Guidelines for Multipath TCP
RFC 5740, November 2009. Development", RFC 6182, DOI 10.17487/RFC6182, March 2011,
<https://www.rfc-editor.org/info/rfc6182>.
[RFC6182] Ford, A., C. Raiciu, M. Handley, S. Barre, J. Iyengar, [RFC6298] Paxson, V., Allman, M., Chu, J., and M. Sargent,
"Architectural Guidelines for Multipath TCP Development", March "Computing TCP's Retransmission Timer", RFC 6298,
2011, RFC 6182. DOI 10.17487/RFC6298, June 2011,
<https://www.rfc-editor.org/info/rfc6298>.
[RFC6298] Paxson, V., M. Allman, H.K. Chu, M. Sargent, "Computing [RFC6675] Blanton, E., Allman, M., Wang, L., Jarvinen, I., Kojo, M.,
TCP's Retransmission Timer", June 2011, RFC 6298. and Y. Nishida, "A Conservative Loss Recovery Algorithm
Based on Selective Acknowledgment (SACK) for TCP",
RFC 6675, DOI 10.17487/RFC6675, August 2012,
<https://www.rfc-editor.org/info/rfc6675>.
[RFC6675] Blanton, E., M. Allman, L. Wang, I. Jarvinen, M. Kojo, [RFC7323] Borman, D., Braden, B., Jacobson, V., and R.
Y. Nishida, "A Conservative Loss Recovery Algorithm Based on Scheffenegger, Ed., "TCP Extensions for High Performance",
Selective Acknowledgment (SACK) for TCP", August 2012, RFC 6675. RFC 7323, DOI 10.17487/RFC7323, September 2014,
<https://www.rfc-editor.org/info/rfc7323>.
[RFC7323] Borman D., B. Braden, V. Jacobson, R. Scheffenegger, "TCP Acknowledgments
Extensions for High Performance", September 2014, RFC 7323.
Authors' Addresses This document benefits from years of discussions with Ethan Blanton,
Sally Floyd, Jana Iyengar, Shawn Ostermann, Vern Paxson, and the
members of the TCPM and TCPIMPL Working Groups. Ran Atkinson,
Yuchung Cheng, David Black, Stewart Bryant, Martin Duke, Wesley Eddy,
Gorry Fairhurst, Rahul Arvind Jadhav, Benjamin Kaduk, Mirja
K├╝hlewind, Nicolas Kuhn, Jonathan Looney, and Michael Scharf provided
useful comments on previous draft versions of this document.
Mark Allman Author's Address
International Computer Science Institute
1947 Center St. Suite 600
Berkeley, CA 94704
EMail: mallman@icir.org Mark Allman
http://www.icir.org/mallman International Computer Science Institute
2150 Shattuck Ave., Suite 1100
Berkeley, CA 94704
United States of America
Email: mallman@icir.org
URI: https://www.icir.org/mallman
 End of changes. 114 change blocks. 
473 lines changed or deleted 511 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/