draft-ietf-rmcat-eval-criteria-07.txt   draft-ietf-rmcat-eval-criteria-08.txt 
RMCAT WG V. Singh RMCAT WG V. Singh
Internet-Draft callstats.io Internet-Draft callstats.io
Intended status: Informational J. Ott Intended status: Informational J. Ott
Expires: November 2, 2018 Technical University of Munich Expires: May 9, 2019 Technical University of Munich
S. Holmer S. Holmer
Google Google
May 1, 2018 November 5, 2018
Evaluating Congestion Control for Interactive Real-time Media Evaluating Congestion Control for Interactive Real-time Media
draft-ietf-rmcat-eval-criteria-07 draft-ietf-rmcat-eval-criteria-08
Abstract Abstract
The Real-time Transport Protocol (RTP) is used to transmit media in The Real-time Transport Protocol (RTP) is used to transmit media in
telephony and video conferencing applications. This document telephony and video conferencing applications. This document
describes the guidelines to evaluate new congestion control describes the guidelines to evaluate new congestion control
algorithms for interactive point-to-point real-time media. algorithms for interactive point-to-point real-time media.
Status of This Memo Status of This Memo
skipping to change at page 1, line 36 skipping to change at page 1, line 36
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on November 2, 2018. This Internet-Draft will expire on May 9, 2019.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 15 skipping to change at page 2, line 15
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 5 3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 5
4. List of Network Parameters . . . . . . . . . . . . . . . . . 5 4. List of Network Parameters . . . . . . . . . . . . . . . . . 5
4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 5 4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 5
4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 5 4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 6
4.3. DropTail Router Queue Length . . . . . . . . . . . . . . 6 4.3. Drop Tail Router Queue Length . . . . . . . . . . . . . . 6
4.4. Loss generation model . . . . . . . . . . . . . . . . . . 6 4.4. Loss generation model . . . . . . . . . . . . . . . . . . 7
4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 6 4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 7
4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 7 4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 8
4.5.2. Approximately Random Subject to No-Reordering Bounded 4.5.2. Approximately Random Subject to No-Reordering Bounded
PDV (NR-RPVD) . . . . . . . . . . . . . . . . 8 PDV (NR-RPVD) . . . . . . . . . . . . . . . . 9
4.5.3. Recommended distribution . . . . . . . . . . . . . . 9 4.5.3. Recommended distribution . . . . . . . . . . . . . . 9
5. WiFi or Cellular Links . . . . . . . . . . . . . . . . . . . 9 5. WiFi or Cellular Links . . . . . . . . . . . . . . . . . . . 10
6. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 9 6. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 10
6.1. TCP taffic model . . . . . . . . . . . . . . . . . . . . 9 6.1. TCP traffic model . . . . . . . . . . . . . . . . . . . . 10
6.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 10 6.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 10
6.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 10 6.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 11
7. Security Considerations . . . . . . . . . . . . . . . . . . . 10 7. Security Considerations . . . . . . . . . . . . . . . . . . . 11
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11
9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 10 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 11
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 10 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 11
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 11 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 12
11.1. Normative References . . . . . . . . . . . . . . . . . . 11 11.1. Normative References . . . . . . . . . . . . . . . . . . 12
11.2. Informative References . . . . . . . . . . . . . . . . . 12 11.2. Informative References . . . . . . . . . . . . . . . . . 13
Appendix A. Application Trade-off . . . . . . . . . . . . . . . 13 Appendix A. Application Trade-off . . . . . . . . . . . . . . . 14
A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 13 A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 14
Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 13 Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 14
B.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 13 B.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 14
B.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 13 B.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 14
B.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 14 B.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 14
B.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 14 B.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 14
B.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 14 B.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 15
B.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 14 B.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 15
B.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 14 B.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 15
B.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 14 B.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 15
B.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 14 B.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 15
B.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 15 B.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 15
B.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 15 B.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 16
B.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 15 B.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 16
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 15 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 16
1. Introduction 1. Introduction
This memo describes the guidelines to help with evaluating new This memo describes the guidelines to help with evaluating new
congestion control algorithms for interactive point-to-point real congestion control algorithms for interactive point-to-point real
time media. The requirements for the congestion control algorithm time media. The requirements for the congestion control algorithm
are outlined in [I-D.ietf-rmcat-cc-requirements]). This document are outlined in [I-D.ietf-rmcat-cc-requirements]). This document
builds upon previous work at the IETF: Specifying New Congestion builds upon previous work at the IETF: Specifying New Congestion
Control Algorithms [RFC5033] and Metrics for the Evaluation of Control Algorithms [RFC5033] and Metrics for the Evaluation of
Congestion Control Algorithms [RFC5166]. Congestion Control Algorithms [RFC5166].
The guidelines proposed in the document are intended to help prevent The guidelines proposed in the document are intended to help prevent
a congestion collapse, promote fair capacity usage and optimize the a congestion collapse, promote fair capacity usage and optimize the
media flow's throughput. Furthermore, the proposed algorithms are media flow's throughput. Furthermore, the proposed algorithms are
expected to operate within the envelope of the circuit breakers expected to operate within the envelope of the circuit breakers
defined in RFC8083 [RFC8083]. defined in RFC8083 [RFC8083].
This document only provides broad-level criteria for evaluating a new This document only provides broad-level criteria for evaluating a new
congestion control algorithm. The minimal requirement for RMCAT congestion control algorithm. The minimal requirement for congestion
proposals is to produce or present results for the test scenarios control proposals is to produce or present results for the test
described in [I-D.ietf-rmcat-eval-test] (Basic Test Cases). scenarios described in [I-D.ietf-rmcat-eval-test] (Basic Test Cases).
Additionally, proponents may produce evaluation results for the Additionally, proponents may produce evaluation results for the
wireless test scenarios [I-D.ietf-rmcat-wireless-tests]. wireless test scenarios [I-D.ietf-rmcat-wireless-tests].
2. Terminology 2. Terminology
The terminology defined in RTP [RFC3550], RTP Profile for Audio and The terminology defined in RTP [RFC3550], RTP Profile for Audio and
Video Conferences with Minimal Control [RFC3551], RTCP Extended Video Conferences with Minimal Control [RFC3551], RTCP Extended
Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback
(RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506] (RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506]
apply. apply.
3. Metrics 3. Metrics
This document specifies testing criteria for evaluating congestion
control algorithms for RTP media flows. Proposed algorithms are to
prove their performance by means of simulation and/or emulation
experiments for all the cases described.
Each experiment is expected to log every incoming and outgoing packet Each experiment is expected to log every incoming and outgoing packet
(the RTP logging format is described in Section 3.1). The logging (the RTP logging format is described in Section 3.1). The logging
can be done inside the application or at the endpoints using PCAP can be done inside the application or at the endpoints using PCAP
(packet capture, e.g., tcpdump, wireshark). The following are (packet capture, e.g., tcpdump, wireshark). The following are
calculated based on the information in the packet logs: calculated based on the information in the packet logs:
1. Sending rate, Receiver rate, Goodput (measured at 200ms 1. Sending rate, Receiver rate, Goodput (measured at 200ms
intervals) intervals)
2. Packets sent, Packets received 2. Packets sent, Packets received
skipping to change at page 3, line 48 skipping to change at page 4, line 4
Each experiment is expected to log every incoming and outgoing packet Each experiment is expected to log every incoming and outgoing packet
(the RTP logging format is described in Section 3.1). The logging (the RTP logging format is described in Section 3.1). The logging
can be done inside the application or at the endpoints using PCAP can be done inside the application or at the endpoints using PCAP
(packet capture, e.g., tcpdump, wireshark). The following are (packet capture, e.g., tcpdump, wireshark). The following are
calculated based on the information in the packet logs: calculated based on the information in the packet logs:
1. Sending rate, Receiver rate, Goodput (measured at 200ms 1. Sending rate, Receiver rate, Goodput (measured at 200ms
intervals) intervals)
2. Packets sent, Packets received 2. Packets sent, Packets received
3. Bytes sent, bytes received 3. Bytes sent, bytes received
4. Packet delay 4. Packet delay
5. Packets lost, Packets discarded (from the playout or de-jitter 5. Packets lost, Packets discarded (from the playout or de-jitter
buffer) buffer)
6. If using, retransmission or FEC: post-repair loss 6. If using, retransmission or FEC: post-repair loss
7. Self-Fairness and Fairness with respect to cross traffic: 7. Self-Fairness and Fairness with respect to cross traffic:
Experiments testing a given RMCAT proposal must report on Experiments testing a given congestion control proposal must
relative ratios of the average throughput (measured at coarser report on relative ratios of the average throughput (measured at
time intervals) obtained by each RMCAT stream. In the presence coarser time intervals) obtained by each RTP media stream. In
of background cross-traffic such as TCP, the report must also the presence of background cross-traffic such as TCP, the report
include the relative ratio between average throughput of RMCAT must also include the relative ratio between average throughput
streams and cross-traffic streams. of RTP media streams and cross-traffic streams.
During static periods of a test (i.e., when bottleneck bandwidth During static periods of a test (i.e., when bottleneck bandwidth
is constant and no arrival/departure of streams), these report is constant and no arrival/departure of streams), these report
on relative ratios serve as an indicator of how fair the RMCAT on relative ratios serve as an indicator of how fair the RTP
streams share bandwidth amongst themselves and against cross- streams share bandwidth amongst themselves and against cross-
traffic streams. The throughput measurement interval should be traffic streams. The throughput measurement interval should be
set at a few values (for example, at 1s, 5s, and 20s) in order set at a few values (for example, at 1s, 5s, and 20s) in order
to measure fairness across different time scales. to measure fairness across different time scales.
As a general guideline, the relative ratio between RMCAT flows As a general guideline, the relative ratio between congestion
with the same priority level and similar path RTT should be controlled RTP flows with the same priority level and similar
bounded between (0.333 and 3.) For example, see the test path RTT should be bounded between (0.333 and 3.) For example,
scenarios described in [I-D.ietf-rmcat-eval-test]. see the test scenarios described in [I-D.ietf-rmcat-eval-test].
8. Convergence time: The time taken to reach a stable rate at 8. Convergence time: The time taken to reach a stable rate at
startup, after the available link capacity changes, or when new startup, after the available link capacity changes, or when new
flows get added to the bottleneck link. flows get added to the bottleneck link.
9. Instability or oscillation in the sending rate: The frequency or 9. Instability or oscillation in the sending rate: The frequency or
number of instances when the sending rate oscillates between an number of instances when the sending rate oscillates between an
high watermark level and a low watermark level, or vice-versa in high watermark level and a low watermark level, or vice-versa in
a defined time window. For example, the watermarks can be set a defined time window. For example, the watermarks can be set
at 4x interval: 500 Kbps, 2 Mbps, and a time window of 500ms. at 4x interval: 500 Kbps, 2 Mbps, and a time window of 500ms.
10. [Editor's note: Section 3, in [I-D.ietf-netvc-testing] contains 10. Bandwidth Utilization, defined as ratio of the instantaneous
objective Metrics for evaluating codecs.]
11. Bandwidth Utilization, defined as ratio of the instantaneous
sending rate to the instantaneous bottleneck capacity. This sending rate to the instantaneous bottleneck capacity. This
metric is useful only when an RMCAT flow is by itself or metric is useful only when a congestion controlled RTP flow is
competing with similar cross-traffic. by itself or competing with similar cross-traffic.
Note that the above metrics are all objective application-independent
metrics. Refer to Section 3, in [I-D.ietf-netvc-testing] for
objective metrics for evaluating codecs.
From the logs the statistical measures (min, max, mean, standard From the logs the statistical measures (min, max, mean, standard
deviation and variance) for the whole duration or any specific part deviation and variance) for the whole duration or any specific part
of the session can be calculated. Also the metrics (sending rate, of the session can be calculated. Also the metrics (sending rate,
receiver rate, goodput, latency) can be visualized in graphs as receiver rate, goodput, latency) can be visualized in graphs as
variation over time, the measurements in the plot are at 1 second variation over time, the measurements in the plot are at 1 second
intervals. Additionally, from the logs it is possible to plot the intervals. Additionally, from the logs it is possible to plot the
histogram or CDF of packet delay. histogram or CDF of packet delay.
3.1. RTP Log Format 3.1. RTP Log Format
The log file is tab or comma separated containing the following Having a common log format simplifies running analyses across and
details: comparing different measurements. The log file SHOULD be tab or
comma separated containing the following details:
Send or receive timestamp (unix) Send or receive timestamp (unix)
RTP payload type RTP payload type
SSRC SSRC
RTP sequence no RTP sequence no
RTP timestamp RTP timestamp
marker bit marker bit
payload size payload size
If the congestion control implements, retransmissions or FEC, the If the congestion control implements, retransmissions or FEC, the
skipping to change at page 5, line 33 skipping to change at page 5, line 40
resilience). resilience).
4. List of Network Parameters 4. List of Network Parameters
The implementors initially are encouraged to choose evaluation The implementors initially are encouraged to choose evaluation
settings from the following values: settings from the following values:
4.1. One-way Propagation Delay 4.1. One-way Propagation Delay
Experiments are expected to verify that the congestion control is Experiments are expected to verify that the congestion control is
able to work in challenging situations, for example over trans- able to work across a broad range of path characteristics, also
continental and/or satellite links. Typical values are: including challenging situations, for example over trans-continental
and/or satellite links. Tests thus account for the following
different latencies:
1. Very low latency: 0-1ms 1. Very low latency: 0-1ms
2. Low latency: 50ms 2. Low latency: 50ms
3. High latency: 150ms 3. High latency: 150ms
4. Extreme latency: 300ms 4. Extreme latency: 300ms
4.2. End-to-end Loss 4.2. End-to-end Loss
To model lossy links, the experiments can choose one of the following Many paths in the Internet today are largely lossless but, with
loss rates, the fractional loss is the ratio of packets lost and wireless networks and interference, towards remote regions, or in
packets sent. scenarios featuring high/fast mobility, media flows may exhibit
substantial packet loss. This variety needs to be reflected
appropriately by the tests.
To model a wide range of lossy links, the experiments can choose one
of the following loss rates, the fractional loss is the ratio of
packets lost and packets sent.
1. no loss: 0% 1. no loss: 0%
2. 1% 2. 1%
3. 5% 3. 5%
4. 10% 4. 10%
5. 20% 5. 20%
4.3. DropTail Router Queue Length 4.3. Drop Tail Router Queue Length
Routers SHOULD be configured to use Drop Trail queues in the
experiments due to their (still) prevalent nature. Experimentation
with AQM schemes is encouraged but not mandatory.
The router queue length is measured as the time taken to drain the The router queue length is measured as the time taken to drain the
FIFO queue. It has been noted in various discussions that the queue FIFO queue. It has been noted in various discussions that the queue
length in the current deployed Internet varies significantly. While length in the current deployed Internet varies significantly. While
the core backbone network has very short queue length, the home the core backbone network has very short queue length, the home
gateways usually have larger queue length. Those various queue gateways usually have larger queue length. Those various queue
lengths can be categorized in the following way: lengths can be categorized in the following way:
1. QoS-aware (or short): 70ms 1. QoS-aware (or short): 70ms
skipping to change at page 6, line 50 skipping to change at page 7, line 22
It is known that independent loss models may reflect reality poorly It is known that independent loss models may reflect reality poorly
and hence more sophisticated loss models could be considered. and hence more sophisticated loss models could be considered.
Suitable models for correlated losses includes the Gilbert-Elliot Suitable models for correlated losses includes the Gilbert-Elliot
model and losses generated by modeling a queue including its model and losses generated by modeling a queue including its
(different) drop behaviors. (different) drop behaviors.
4.5. Jitter models 4.5. Jitter models
This section defines jitter models for the purposes of this document. This section defines jitter models for the purposes of this document.
When jitter is to be applied to both the RMCAT flow and any competing When jitter is to be applied to both the congestion controlled RTP
flow (such as a TCP competing flow), the competing flow will use the flow and any competing flow (such as a TCP competing flow), the
jitter definition below that does not allow for re-ordering of competing flow will use the jitter definition below that does not
packets on the competing flow (see NR-RBPDV definition below). allow for re-ordering of packets on the competing flow (see NR-RBPDV
definition below).
Jitter is an overloaded term in communications. Its meaning is Jitter is an overloaded term in communications. Its meaning is
typically associated with the variation of a metric (e.g., delay) typically associated with the variation of a metric (e.g., delay)
with respect to some reference metric (e.g., average delay or minimum with respect to some reference metric (e.g., average delay or minimum
delay). For example, RFC 3550 jitter is a smoothed estimate of delay). For example, RFC 3550 jitter is a smoothed estimate of
jitter which is particularly meaningful if the underlying packet jitter which is particularly meaningful if the underlying packet
delay variation was caused by a Gaussian random process. delay variation was caused by a Gaussian random process.
Because jitter is an overloaded term, we instead use the term Packet Because jitter is an overloaded term, we instead use the term Packet
Delay Variation (PDV) to describe the variation of delay of Delay Variation (PDV) to describe the variation of delay of
skipping to change at page 7, line 31 skipping to change at page 7, line 52
Most PDV distributions in packet network systems are one-sided Most PDV distributions in packet network systems are one-sided
distributions (the measurement of which with a finite number of distributions (the measurement of which with a finite number of
measurement samples result in one-sided histograms). In the usual measurement samples result in one-sided histograms). In the usual
packet network transport case there is typically one packet that packet network transport case there is typically one packet that
transited the network with the minimum delay, then a majority of transited the network with the minimum delay, then a majority of
packets also transit the system within some variation from this packets also transit the system within some variation from this
minimum delay, and then a minority of the packets transit the network minimum delay, and then a minority of the packets transit the network
with delays higher than the median or average transit time (these are with delays higher than the median or average transit time (these are
outliers). Although infrequent, outliers can cause significant outliers). Although infrequent, outliers can cause significant
deleterious operation in adaptive systems and should be considered in deleterious operation in adaptive systems and should be considered in
RMCAT adaptation designs. rate adaptation designs for RTP congestion control.
In this section we define two different bounded PDV characteristics, In this section we define two different bounded PDV characteristics,
1) Random Bounded PDV and 2) Approximately Random Subject to No- 1) Random Bounded PDV and 2) Approximately Random Subject to No-
Reordering Bounded PDV. Reordering Bounded PDV.
The former, 1) Random Bounded PDV is presented for information only, The former, 1) Random Bounded PDV is presented for information only,
while the latte, 2) Approximately Random Subject to No-Reordering while the latter, 2) Approximately Random Subject to No-Reordering
Bounded PDV, must be used in the evaluation. Bounded PDV, must be used in the evaluation.
4.5.1. Random Bounded PDV (RBPDV) 4.5.1. Random Bounded PDV (RBPDV)
The RBPDV probability distribution function (pdf) is specified to be The RBPDV probability distribution function (PDF) is specified to be
of some mathematically describable function which includes some of some mathematically describable function which includes some
practical minimum and maximum discrete values suitable for testing. practical minimum and maximum discrete values suitable for testing.
For example, the minimum value, x_min, might be specified as the For example, the minimum value, x_min, might be specified as the
minimum transit time packet and the maximum value, x_max, might be minimum transit time packet and the maximum value, x_max, might be
idefined to be two standard deviations higher than the mean. defined to be two standard deviations higher than the mean.
Since we are typically interested in the distribution relative to the Since we are typically interested in the distribution relative to the
mean delay packet, we define the zero mean PDV sample, z(n), to be mean delay packet, we define the zero mean PDV sample, z(n), to be
z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random
variable x and x_mean is the mean of x. variable x and x_mean is the mean of x.
We assume here that s(n) is the original source time of packet n and We assume here that s(n) is the original source time of packet n and
the post-jitter induced emmission time, j(n), for packet n is j(n) = the post-jitter induced emission time, j(n), for packet n is j(n) =
{[z(n) + x_mean] + s(n)}. It follows that the separation in the post- {[z(n) + x_mean] + s(n)}. It follows that the separation in the post-
jitter time of packets n and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}. jitter time of packets n and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}.
Since the first term is always a positive quantity, we note that Since the first term is always a positive quantity, we note that
packet reordering at the receiver is possible whenever the second packet reordering at the receiver is possible whenever the second
term is greater than the first. Said another way, whenever the term is greater than the first. Said another way, whenever the
difference in possible zero mean PDV sample delays (i.e., [x_max- difference in possible zero mean PDV sample delays (i.e., [x_max-
x_min]) exceeds the inter-departure time of any two sent packets, we x_min]) exceeds the inter-departure time of any two sent packets, we
have the possibility of packet re-ordering. have the possibility of packet re-ordering.
There are important use cases in real networks where packets can There are important use cases in real networks where packets can
skipping to change at page 9, line 10 skipping to change at page 9, line 34
We note that this assumption holds for some important exception We note that this assumption holds for some important exception
cases, such as packets immediately following outliers. There are a cases, such as packets immediately following outliers. There are a
multitude of software controlled elements common on end-to-end multitude of software controlled elements common on end-to-end
Internet paths (such as firewalls, ALGs and other middleboxes) which Internet paths (such as firewalls, ALGs and other middleboxes) which
stop processing packets while servicing other functions (e.g., stop processing packets while servicing other functions (e.g.,
garbage collection). Often these devices do not drop packets, but garbage collection). Often these devices do not drop packets, but
rather queue them for later processing and cause many of the rather queue them for later processing and cause many of the
outliers. Thus NR-RPVD models this particular use case (assuming outliers. Thus NR-RPVD models this particular use case (assuming
serial(n+1) is defined appropriately for the device causing the serial(n+1) is defined appropriately for the device causing the
outlier) and thus is believed to be important for adaptation outlier) and thus is believed to be important for adaptation
development for RMCAT. development for congestion controlled RTP streams.
4.5.3. Recommended distribution 4.5.3. Recommended distribution
It is recommended that z(n) is distributed according to a truncated Whether Random Bounded PDV or Approximately Random Subject to No-
Gaussian: Reordering Bounded PDV, it is recommended that z(n) is distributed
according to a truncated Gaussian for the above jitter models:
z(n) ~ |max(min(N(0, std^2), N_STD * std), -N_STD * std)| z(n) ~ |max(min(N(0, std^2), N_STD * std), -N_STD * std)|
where N(0, std^2) is the Gaussian distribution with zero mean and where N(0, std^2) is the Gaussian distribution with zero mean and
standard deviation std. Recommended values: standard deviation std. Recommended values:
o std = 5 ms o std = 5 ms
o N_STD = 3 o N_STD = 3
5. WiFi or Cellular Links 5. WiFi or Cellular Links
[I-D.ietf-rmcat-wireless-tests] describes the test cases to simulate [I-D.ietf-rmcat-wireless-tests] describes the test cases to simulate
networks with wireless links. The document describes mechanism to networks with wireless links. The document describes mechanism to
simulate both cellular and WiFi networks. simulate both cellular and WiFi networks.
6. Traffic Models 6. Traffic Models
6.1. TCP taffic model 6.1. TCP traffic model
Long-lived TCP flows will download data throughout the session and Long-lived TCP flows will download data throughout the session and
are expected to have infinite amount of data to send or receive. For are expected to have infinite amount of data to send or receive.
example, to This roughly applies, for example, when downloading software
distributions.
Each short TCP flow is modeled as a sequence of file downloads Each short TCP flow is modeled as a sequence of file downloads
interleaved with idle periods. Not all short TCPs start at the same interleaved with idle periods. Not all short TCP flows start at the
time, i.e., some start in the ON state while others start in the OFF same time, i.e., some start in the ON state while others start in the
state. OFF state.
The short TCP flows can be modelled as follows: 30 connections start The short TCP flows can be modeled as follows: 30 connections start
simultaneously fetching small (30-50 KB) amounts of data. This simultaneously fetching small (30-50 KB) amounts of data. This
covers the case where the short TCP flows are not fetching a video covers the case where the short TCP flows are not fetching a video
file. file.
The idle period between bursts of starting a group of TCP flows is The idle period between bursts of starting a group of TCP flows is
typically derived from an exponential distribution with the mean typically derived from an exponential distribution with the mean
value of 10 seconds. value of 10 seconds.
[These values were picked based on the data available at [These values were picked based on the data available at
http://httparchive.org/interesting.php as of October 2015]. http://httparchive.org/interesting.php as of October 2015].
Many different TCP congestion control schemes are deployed today.
Therefore, experimentation with a range of different schemes,
especially including CUBIC, is encouraged. Experiments MUST document
in detail which congestion control schemes they tested against and
which parameters were used.
6.2. RTP Video model 6.2. RTP Video model
[I-D.ietf-rmcat-video-traffic-model] describes two types of video [I-D.ietf-rmcat-video-traffic-model] describes two types of video
traffic models for evaluating RMCAT candidate algorithms. The first traffic models for evaluating candidate algorithms for RTP congestion
model statistically characterizes the behavior of a video encoder. control. The first model statistically characterizes the behavior of
Whereas the second model uses video traces. a video encoder. Whereas the second model uses video traces.
For example, test sequences are available at: [xiph-seq] and For example, test sequences are available at: [xiph-seq] and
[HEVC-seq]. The currently chosen video streams are: Foreman and [HEVC-seq]. The currently chosen video streams are: Foreman and
FourPeople. FourPeople.
6.3. Background UDP 6.3. Background UDP
Background UDP flow is modeled as a constant bit rate (CBR) flow. It Background UDP flow is modeled as a constant bit rate (CBR) flow. It
will download data at a particular CBR rate for the complete session, will download data at a particular CBR rate for the complete session,
or will change to particular CBR rate at predefined intervals. The or will change to particular CBR rate at predefined intervals. The
inter packet interval is calculated based on the CBR and the packet inter packet interval is calculated based on the CBR and the packet
size (is typically set to the path MTU size, the default value can be size (is typically set to the path MTU size, the default value can be
1500 bytes). 1500 bytes).
Note that new transport protocols such as QUIC may use UDP but, due
to their congestion control algorithms, will exhibit behavior
conceptually similar in nature to TCP flows above and can thus be
subsumed by the above, including the division into short- and long-
lived flows. As QUIC evolves independently of TCP congestion control
algorithms, its future congestion control SHOULD be considered as
competing traffic as appropriate.
7. Security Considerations 7. Security Considerations
Security issues have not been discussed in this memo. Security issues have not been discussed in this memo.
8. IANA Considerations 8. IANA Considerations
There are no IANA impacts in this memo. There are no IANA impacts in this memo.
9. Contributors 9. Contributors
The content and concepts within this document are a product of the The content and concepts within this document are a product of the
discussion carried out in the Design Team. discussion carried out in the Design Team.
Michael Ramalho provided the text for the Jitter model. Michael Ramalho provided the text for the Jitter model.
10. Acknowledgements 10. Acknowledgments
Much of this document is derived from previous work on congestion Much of this document is derived from previous work on congestion
control at the IETF. control at the IETF.
The authors would like to thank Harald Alvestrand, Anna Brunstrom, The authors would like to thank Harald Alvestrand, Anna Brunstrom,
Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde, Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde,
Stefan Holmer, Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers O'Hanlon, Colin
O'Hanlon, Colin Perkins, Michael Ramalho, Zaheduzzaman Sarker, Perkins, Michael Ramalho, Zaheduzzaman Sarker, Timothy B.
Timothy B. Terriberry, Michael Welzl, Mo Zanaty, and Xiaoqing Zhu Terriberry, Michael Welzl, Mo Zanaty, and Xiaoqing Zhu for providing
for providing valuable feedback on earlier versions of this draft. valuable feedback on earlier versions of this draft. Additionally,
Additionally, also thank the participants of the design team for also thank the participants of the design team for their comments and
their comments and discussion related to the evaluation criteria. discussion related to the evaluation criteria.
11. References 11. References
11.1. Normative References 11.1. Normative References
[I-D.ietf-rmcat-cc-requirements] [I-D.ietf-rmcat-cc-requirements]
Jesup, R. and Z. Sarker, "Congestion Control Requirements Jesup, R. and Z. Sarker, "Congestion Control Requirements
for Interactive Real-Time Media", draft-ietf-rmcat-cc- for Interactive Real-Time Media", draft-ietf-rmcat-cc-
requirements-09 (work in progress), December 2014. requirements-09 (work in progress), December 2014.
[I-D.ietf-rmcat-wireless-tests] [I-D.ietf-rmcat-wireless-tests]
Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and
M. Ramalho, "Evaluation Test Cases for Interactive Real- M. Ramalho, "Evaluation Test Cases for Interactive Real-
Time Media over Wireless Networks", draft-ietf-rmcat- Time Media over Wireless Networks", draft-ietf-rmcat-
wireless-tests-04 (work in progress), May 2017. wireless-tests-05 (work in progress), June 2018.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/info/rfc3550>. July 2003, <https://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551, Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003, <https://www.rfc- DOI 10.17487/RFC3551, July 2003, <https://www.rfc-
editor.org/info/rfc3551>. editor.org/info/rfc3551>.
skipping to change at page 12, line 24 skipping to change at page 13, line 14
11.2. Informative References 11.2. Informative References
[HEVC-seq] [HEVC-seq]
HEVC, "Test Sequences", HEVC, "Test Sequences",
http://www.netlab.tkk.fi/~varun/test_sequences/ . http://www.netlab.tkk.fi/~varun/test_sequences/ .
[I-D.ietf-netvc-testing] [I-D.ietf-netvc-testing]
Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec
Testing and Quality Measurement", draft-ietf-netvc- Testing and Quality Measurement", draft-ietf-netvc-
testing-06 (work in progress), October 2017. testing-07 (work in progress), July 2018.
[I-D.ietf-rmcat-eval-test] [I-D.ietf-rmcat-eval-test]
Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test
Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat-
eval-test-05 (work in progress), April 2017. eval-test-07 (work in progress), October 2018.
[I-D.ietf-rmcat-video-traffic-model] [I-D.ietf-rmcat-video-traffic-model]
Zhu, X., Cruz, S., and Z. Sarker, "Modeling Video Traffic Zhu, X., Cruz, S., and Z. Sarker, "Video Traffic Models
Sources for RMCAT Evaluations", draft-ietf-rmcat-video- for RTP Congestion Control Evaluations", draft-ietf-rmcat-
traffic-model-04 (work in progress), January 2018. video-traffic-model-06 (work in progress), November 2018.
[RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion
Control Algorithms", BCP 133, RFC 5033, Control Algorithms", BCP 133, RFC 5033,
DOI 10.17487/RFC5033, August 2007, <https://www.rfc- DOI 10.17487/RFC5033, August 2007, <https://www.rfc-
editor.org/info/rfc5033>. editor.org/info/rfc5033>.
[RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion [RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion
Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March
2008, <https://www.rfc-editor.org/info/rfc5166>. 2008, <https://www.rfc-editor.org/info/rfc5166>.
 End of changes. 45 change blocks. 
98 lines changed or deleted 135 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/