draft-ietf-rmcat-eval-criteria-09.txt   draft-ietf-rmcat-eval-criteria-10.txt 
RMCAT WG V. Singh RMCAT WG V. Singh
Internet-Draft callstats.io Internet-Draft callstats.io
Intended status: Informational J. Ott Intended status: Informational J. Ott
Expires: January 3, 2020 Technical University of Munich Expires: May 7, 2020 Technical University of Munich
S. Holmer S. Holmer
Google Google
July 2, 2019 November 4, 2019
Evaluating Congestion Control for Interactive Real-time Media Evaluating Congestion Control for Interactive Real-time Media
draft-ietf-rmcat-eval-criteria-09 draft-ietf-rmcat-eval-criteria-10
Abstract Abstract
The Real-time Transport Protocol (RTP) is used to transmit media in The Real-time Transport Protocol (RTP) is used to transmit media in
telephony and video conferencing applications. This document telephony and video conferencing applications. This document
describes the guidelines to evaluate new congestion control describes the guidelines to evaluate new congestion control
algorithms for interactive point-to-point real-time media. algorithms for interactive point-to-point real-time media.
Status of This Memo Status of This Memo
skipping to change at page 1, line 36 skipping to change at page 1, line 36
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 3, 2020. This Internet-Draft will expire on May 7, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 38 skipping to change at page 2, line 38
7. Security Considerations . . . . . . . . . . . . . . . . . . . 11 7. Security Considerations . . . . . . . . . . . . . . . . . . . 11
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12
9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12
10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 12
11.1. Normative References . . . . . . . . . . . . . . . . . . 12 11.1. Normative References . . . . . . . . . . . . . . . . . . 12
11.2. Informative References . . . . . . . . . . . . . . . . . 13 11.2. Informative References . . . . . . . . . . . . . . . . . 13
Appendix A. Application Trade-off . . . . . . . . . . . . . . . 14 Appendix A. Application Trade-off . . . . . . . . . . . . . . . 14
A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 14 A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 14
Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 14 Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 14
B.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 15 B.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 14
B.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 15 B.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 15
B.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 15 B.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 15
B.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 15 B.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 15
B.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 15 B.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 15
B.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 15 B.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 15
B.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 15 B.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 15
B.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 16 B.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 15
B.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 16 B.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 15
B.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 16 B.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 16
B.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 16 B.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 16
B.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 16 B.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 16
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 17 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 16
1. Introduction 1. Introduction
This memo describes the guidelines to help with evaluating new This memo describes the guidelines to help with evaluating new
congestion control algorithms for interactive point-to-point real congestion control algorithms for interactive point-to-point real
time media. The requirements for the congestion control algorithm time media. The requirements for the congestion control algorithm
are outlined in [I-D.ietf-rmcat-cc-requirements]). This document are outlined in [I-D.ietf-rmcat-cc-requirements]). This document
builds upon previous work at the IETF: Specifying New Congestion builds upon previous work at the IETF: Specifying New Congestion
Control Algorithms [RFC5033] and Metrics for the Evaluation of Control Algorithms [RFC5033] and Metrics for the Evaluation of
Congestion Control Algorithms [RFC5166]. Congestion Control Algorithms [RFC5166].
skipping to change at page 5, line 16 skipping to change at page 5, line 16
deviation and variance) for the whole duration or any specific part deviation and variance) for the whole duration or any specific part
of the session can be calculated. Also the metrics (sending rate, of the session can be calculated. Also the metrics (sending rate,
receiver rate, goodput, latency) can be visualized in graphs as receiver rate, goodput, latency) can be visualized in graphs as
variation over time, the measurements in the plot are at 1 second variation over time, the measurements in the plot are at 1 second
intervals. Additionally, from the logs it is possible to plot the intervals. Additionally, from the logs it is possible to plot the
histogram or CDF of packet delay. histogram or CDF of packet delay.
3.1. RTP Log Format 3.1. RTP Log Format
Having a common log format simplifies running analyses across and Having a common log format simplifies running analyses across and
comparing different measurements. The log file SHOULD be tab or comparing different measurements. The log file should be tab or
comma separated containing the following details: comma separated containing the following details:
Send or receive timestamp (unix) Send or receive timestamp (unix)
RTP payload type RTP payload type
SSRC SSRC
RTP sequence no RTP sequence no
RTP timestamp RTP timestamp
marker bit marker bit
payload size payload size
skipping to change at page 6, line 29 skipping to change at page 6, line 29
2. 1% 2. 1%
3. 5% 3. 5%
4. 10% 4. 10%
5. 20% 5. 20%
4.3. Drop Tail Router Queue Length 4.3. Drop Tail Router Queue Length
Routers SHOULD be configured to use Drop Trail queues in the Routers should be configured to use Drop Trail queues in the
experiments due to their (still) prevalent nature. Experimentation experiments due to their (still) prevalent nature. Experimentation
with AQM schemes is encouraged but not mandatory. with AQM schemes is encouraged but not mandatory.
The router queue length is measured as the time taken to drain the The router queue length is measured as the time taken to drain the
FIFO queue. It has been noted in various discussions that the queue FIFO queue. It has been noted in various discussions that the queue
length in the current deployed Internet varies significantly. While length in the current deployed Internet varies significantly. While
the core backbone network has very short queue length, the home the core backbone network has very short queue length, the home
gateways usually have larger queue length. Those various queue gateways usually have larger queue length. Those various queue
lengths can be categorized in the following way: lengths can be categorized in the following way:
skipping to change at page 10, line 39 skipping to change at page 10, line 39
The idle period between bursts of starting a group of TCP flows is The idle period between bursts of starting a group of TCP flows is
typically derived from an exponential distribution with the mean typically derived from an exponential distribution with the mean
value of 10 seconds. value of 10 seconds.
[These values were picked based on the data available at [These values were picked based on the data available at
http://httparchive.org/interesting.php as of October 2015]. http://httparchive.org/interesting.php as of October 2015].
Many different TCP congestion control schemes are deployed today. Many different TCP congestion control schemes are deployed today.
Therefore, experimentation with a range of different schemes, Therefore, experimentation with a range of different schemes,
especially including CUBIC, is encouraged. Experiments MUST document especially including CUBIC, is encouraged. Experiments must document
in detail which congestion control schemes they tested against and in detail which congestion control schemes they tested against and
which parameters were used. which parameters were used.
6.2. RTP Video model 6.2. RTP Video model
[I-D.ietf-rmcat-video-traffic-model] describes two types of video [RFC8593] describes two types of video traffic models for evaluating
traffic models for evaluating candidate algorithms for RTP congestion candidate algorithms for RTP congestion control. The first model
control. The first model statistically characterizes the behavior of statistically characterizes the behavior of a video encoder. Whereas
a video encoder. Whereas the second model uses video traces. the second model uses video traces.
For example, test sequences are available at: [xiph-seq] and For example, test sequences are available at: [xiph-seq] and
[HEVC-seq]. The currently chosen video streams are: Foreman and [HEVC-seq]. The currently chosen video streams are: Foreman and
FourPeople. FourPeople.
6.3. Background UDP 6.3. Background UDP
Background UDP flow is modeled as a constant bit rate (CBR) flow. It Background UDP flow is modeled as a constant bit rate (CBR) flow. It
will download data at a particular CBR rate for the complete session, will download data at a particular CBR rate for the complete session,
or will change to particular CBR rate at predefined intervals. The or will change to particular CBR rate at predefined intervals. The
inter packet interval is calculated based on the CBR and the packet inter packet interval is calculated based on the CBR and the packet
size (is typically set to the path MTU size, the default value can be size (is typically set to the path MTU size, the default value can be
1500 bytes). 1500 bytes).
Note that new transport protocols such as QUIC may use UDP but, due Note that new transport protocols such as QUIC may use UDP but, due
to their congestion control algorithms, will exhibit behavior to their congestion control algorithms, will exhibit behavior
conceptually similar in nature to TCP flows above and can thus be conceptually similar in nature to TCP flows above and can thus be
subsumed by the above, including the division into short- and long- subsumed by the above, including the division into short- and long-
lived flows. As QUIC evolves independently of TCP congestion control lived flows. As QUIC evolves independently of TCP congestion control
algorithms, its future congestion control SHOULD be considered as algorithms, its future congestion control should be considered as
competing traffic as appropriate. competing traffic as appropriate.
7. Security Considerations 7. Security Considerations
This document specifies evaluation criteria and parameters for This document specifies evaluation criteria and parameters for
assessing and comparing the performance of congestion control assessing and comparing the performance of congestion control
protocola and algorithm for real-time communication. This memo protocola and algorithm for real-time communication. This memo
itself is thus not subject to security considerations but the itself is thus not subject to security considerations but the
protocols and algorithms evaluated may be. In particular, successful protocols and algorithms evaluated may be. In particular, successful
operation under all tests defined in this document may suffice for a operation under all tests defined in this document may suffice for a
skipping to change at page 12, line 43 skipping to change at page 12, line 43
[I-D.ietf-rmcat-cc-requirements] [I-D.ietf-rmcat-cc-requirements]
Jesup, R. and Z. Sarker, "Congestion Control Requirements Jesup, R. and Z. Sarker, "Congestion Control Requirements
for Interactive Real-Time Media", draft-ietf-rmcat-cc- for Interactive Real-Time Media", draft-ietf-rmcat-cc-
requirements-09 (work in progress), December 2014. requirements-09 (work in progress), December 2014.
[I-D.ietf-rmcat-wireless-tests] [I-D.ietf-rmcat-wireless-tests]
Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and
M. Ramalho, "Evaluation Test Cases for Interactive Real- M. Ramalho, "Evaluation Test Cases for Interactive Real-
Time Media over Wireless Networks", draft-ietf-rmcat- Time Media over Wireless Networks", draft-ietf-rmcat-
wireless-tests-07 (work in progress), July 2019. wireless-tests-08 (work in progress), July 2019.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/info/rfc3550>. July 2003, <https://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551, Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003, DOI 10.17487/RFC3551, July 2003,
<https://www.rfc-editor.org/info/rfc3551>. <https://www.rfc-editor.org/info/rfc3551>.
skipping to change at page 13, line 47 skipping to change at page 13, line 47
[I-D.ietf-netvc-testing] [I-D.ietf-netvc-testing]
Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec
Testing and Quality Measurement", draft-ietf-netvc- Testing and Quality Measurement", draft-ietf-netvc-
testing-08 (work in progress), January 2019. testing-08 (work in progress), January 2019.
[I-D.ietf-rmcat-eval-test] [I-D.ietf-rmcat-eval-test]
Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test
Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat-
eval-test-10 (work in progress), May 2019. eval-test-10 (work in progress), May 2019.
[I-D.ietf-rmcat-video-traffic-model]
Zhu, X., Cruz, S., and Z. Sarker, "Video Traffic Models
for RTP Congestion Control Evaluations", draft-ietf-rmcat-
video-traffic-model-07 (work in progress), February 2019.
[RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion
Control Algorithms", BCP 133, RFC 5033, Control Algorithms", BCP 133, RFC 5033,
DOI 10.17487/RFC5033, August 2007, DOI 10.17487/RFC5033, August 2007,
<https://www.rfc-editor.org/info/rfc5033>. <https://www.rfc-editor.org/info/rfc5033>.
[RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion [RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion
Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March
2008, <https://www.rfc-editor.org/info/rfc5166>. 2008, <https://www.rfc-editor.org/info/rfc5166>.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion [RFC8593] Zhu, X., Mena, S., and Z. Sarker, "Video Traffic Models
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, for RTP Congestion Control Evaluations", RFC 8593,
<https://www.rfc-editor.org/info/rfc5681>. DOI 10.17487/RFC8593, May 2019,
<https://www.rfc-editor.org/info/rfc8593>.
[SA4-LR] S4-050560, 3GPP., "Error Patterns for MBMS Streaming over
UTRAN and GERAN", 3GPP S4-050560, 5 2008.
[TCP-eval-suite]
Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier,
R., Gang, W., Eggert, L., Ha, S., and I. Rhee, "Towards a
Common TCP Evaluation Suite", Proc. PFLDnet. 2008, August
2008.
[xiph-seq] [xiph-seq]
Daede, T., "Video Test Media Set", Daede, T., "Video Test Media Set",
https://people.xiph.org/~tdaede/sets/ . https://people.xiph.org/~tdaede/sets/ .
Appendix A. Application Trade-off Appendix A. Application Trade-off
Application trade-off is yet to be defined. see RMCAT requirements Application trade-off is yet to be defined. see RMCAT requirements
[I-D.ietf-rmcat-cc-requirements] document. Perhaps each experiment [I-D.ietf-rmcat-cc-requirements] document. Perhaps each experiment
should define the application's expectation or trade-off. should define the application's expectation or trade-off.
 End of changes. 15 change blocks. 
34 lines changed or deleted 21 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/