--- 1/draft-ietf-rmcat-eval-criteria-10.txt 2020-02-12 09:13:55.741028723 -0800 +++ 2/draft-ietf-rmcat-eval-criteria-11.txt 2020-02-12 09:13:55.777029633 -0800 @@ -1,21 +1,21 @@ RMCAT WG V. Singh Internet-Draft callstats.io Intended status: Informational J. Ott -Expires: May 7, 2020 Technical University of Munich +Expires: August 15, 2020 Technical University of Munich S. Holmer Google - November 4, 2019 + February 12, 2020 Evaluating Congestion Control for Interactive Real-time Media - draft-ietf-rmcat-eval-criteria-10 + draft-ietf-rmcat-eval-criteria-11 Abstract The Real-time Transport Protocol (RTP) is used to transmit media in telephony and video conferencing applications. This document describes the guidelines to evaluate new congestion control algorithms for interactive point-to-point real-time media. Status of This Memo @@ -25,25 +25,25 @@ Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on May 7, 2020. + This Internet-Draft will expire on August 15, 2020. Copyright Notice - Copyright (c) 2019 IETF Trust and the persons identified as the + Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as @@ -58,37 +58,36 @@ 4. List of Network Parameters . . . . . . . . . . . . . . . . . 5 4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 5 4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 6 4.3. Drop Tail Router Queue Length . . . . . . . . . . . . . . 6 4.4. Loss generation model . . . . . . . . . . . . . . . . . . 7 4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 7 4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 8 4.5.2. Approximately Random Subject to No-Reordering Bounded PDV (NR-RPVD) . . . . . . . . . . . . . . . . 9 4.5.3. Recommended distribution . . . . . . . . . . . . . . 9 - 5. WiFi or Cellular Links . . . . . . . . . . . . . . . . . . . 10 - 6. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 10 - 6.1. TCP traffic model . . . . . . . . . . . . . . . . . . . . 10 - 6.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 10 - 6.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 11 - 7. Security Considerations . . . . . . . . . . . . . . . . . . . 11 - 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 - 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12 - 10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12 - 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 - 11.1. Normative References . . . . . . . . . . . . . . . . . . 12 - 11.2. Informative References . . . . . . . . . . . . . . . . . 13 + 5. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 10 + 5.1. TCP traffic model . . . . . . . . . . . . . . . . . . . . 10 + 5.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 10 + 5.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 10 + 6. Security Considerations . . . . . . . . . . . . . . . . . . . 11 + 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 + 8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12 + 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12 + 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 + 10.1. Normative References . . . . . . . . . . . . . . . . . . 12 + 10.2. Informative References . . . . . . . . . . . . . . . . . 13 Appendix A. Application Trade-off . . . . . . . . . . . . . . . 14 A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 14 Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 14 B.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 14 - B.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 15 + B.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 14 B.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 15 B.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 15 B.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 15 B.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 15 B.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 15 B.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 15 B.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 15 B.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 16 B.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 16 B.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 16 @@ -103,24 +102,25 @@ builds upon previous work at the IETF: Specifying New Congestion Control Algorithms [RFC5033] and Metrics for the Evaluation of Congestion Control Algorithms [RFC5166]. The guidelines proposed in the document are intended to help prevent a congestion collapse, promote fair capacity usage and optimize the media flow's throughput. Furthermore, the proposed algorithms are expected to operate within the envelope of the circuit breakers defined in RFC8083 [RFC8083]. - This document only provides broad-level criteria for evaluating a new - congestion control algorithm. The minimal requirement for congestion - control proposals is to produce or present results for the test - scenarios described in [I-D.ietf-rmcat-eval-test] (Basic Test Cases). + This document only provides the broad set of network parameters and + and traffic models for evaluating a new congestion control algorithm. + The minimal requirements for congestion control proposals is to + produce or present results for the test scenarios described in + [I-D.ietf-rmcat-eval-test] (Basic Test Cases), which also defines . Additionally, proponents may produce evaluation results for the wireless test scenarios [I-D.ietf-rmcat-wireless-tests]. 2. Terminology The terminology defined in RTP [RFC3550], RTP Profile for Audio and Video Conferences with Minimal Control [RFC3551], RTCP Extended Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback (RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506] apply. @@ -128,27 +128,28 @@ 3. Metrics This document specifies testing criteria for evaluating congestion control algorithms for RTP media flows. Proposed algorithms are to prove their performance by means of simulation and/or emulation experiments for all the cases described. Each experiment is expected to log every incoming and outgoing packet (the RTP logging format is described in Section 3.1). The logging can be done inside the application or at the endpoints using PCAP - (packet capture, e.g., tcpdump, wireshark). The following are - calculated based on the information in the packet logs: + (packet capture, e.g., tcpdump, wireshark). The following metrics + are calculated based on the information in the packet logs: 1. Sending rate, Receiver rate, Goodput (measured at 200ms intervals) 2. Packets sent, Packets received + 3. Bytes sent, bytes received 4. Packet delay 5. Packets lost, Packets discarded (from the playout or de-jitter buffer) 6. If using, retransmission or FEC: post-repair loss 7. Self-Fairness and Fairness with respect to cross traffic: @@ -416,91 +417,85 @@ z(n) ~ |max(min(N(0, std^2), N_STD * std), -N_STD * std)| where N(0, std^2) is the Gaussian distribution with zero mean and standard deviation std. Recommended values: o std = 5 ms o N_STD = 3 -5. WiFi or Cellular Links - - [I-D.ietf-rmcat-wireless-tests] describes the test cases to simulate - networks with wireless links. The document describes mechanism to - simulate both cellular and WiFi networks. - -6. Traffic Models +5. Traffic Models -6.1. TCP traffic model +5.1. TCP traffic model Long-lived TCP flows will download data throughout the session and are expected to have infinite amount of data to send or receive. This roughly applies, for example, when downloading software distributions. Each short TCP flow is modeled as a sequence of file downloads interleaved with idle periods. Not all short TCP flows start at the same time, i.e., some start in the ON state while others start in the OFF state. The short TCP flows can be modeled as follows: 30 connections start - simultaneously fetching small (30-50 KB) amounts of data. This - covers the case where the short TCP flows are not fetching a video - file. + simultaneously fetching small (30-50 KB) amounts of data, evenly + distributed. This covers the case where the short TCP flows are + fetching web page resources rather than video files. The idle period between bursts of starting a group of TCP flows is typically derived from an exponential distribution with the mean value of 10 seconds. [These values were picked based on the data available at http://httparchive.org/interesting.php as of October 2015]. Many different TCP congestion control schemes are deployed today. Therefore, experimentation with a range of different schemes, especially including CUBIC, is encouraged. Experiments must document in detail which congestion control schemes they tested against and which parameters were used. -6.2. RTP Video model +5.2. RTP Video model [RFC8593] describes two types of video traffic models for evaluating candidate algorithms for RTP congestion control. The first model - statistically characterizes the behavior of a video encoder. Whereas + statistically characterizes the behavior of a video encoder, whereas the second model uses video traces. - For example, test sequences are available at: [xiph-seq] and - [HEVC-seq]. The currently chosen video streams are: Foreman and - FourPeople. + Sample video test sequences are available at: [xiph-seq] and + [HEVC-seq]. The following two video streams are the recommended + minimum for testing: Foreman and FourPeople. -6.3. Background UDP +5.3. Background UDP Background UDP flow is modeled as a constant bit rate (CBR) flow. It will download data at a particular CBR rate for the complete session, or will change to particular CBR rate at predefined intervals. The inter packet interval is calculated based on the CBR and the packet size (is typically set to the path MTU size, the default value can be 1500 bytes). Note that new transport protocols such as QUIC may use UDP but, due to their congestion control algorithms, will exhibit behavior conceptually similar in nature to TCP flows above and can thus be subsumed by the above, including the division into short- and long- lived flows. As QUIC evolves independently of TCP congestion control algorithms, its future congestion control should be considered as competing traffic as appropriate. -7. Security Considerations +6. Security Considerations This document specifies evaluation criteria and parameters for assessing and comparing the performance of congestion control - protocola and algorithm for real-time communication. This memo + protocols and algorithms for real-time communication. This memo itself is thus not subject to security considerations but the protocols and algorithms evaluated may be. In particular, successful operation under all tests defined in this document may suffice for a comparative evaluation but must not be interpreted that the protocol is free of risks when deployed on the Internet as briefly described in the following by example. Such evaluations are expected to be carried out in controlled environments for limited numbers of parallel flows. As such, these evaluations are by definition limited and will not be able to @@ -512,48 +507,48 @@ This specification focuses on assessing the regular operation of the protocols and algorithms under considerations. It does not suggest checks against malicious use of the protocols -- by the sender, the receiver, or intermediate parties, e.g., through faked, dropped, replicated, or modified congestion signals. It is up to the protocol specifications themselves to ensure that authenticity, integrity, and/or plausibility of received signals are checked and the appropriate actions (or non-actions) are taken. -8. IANA Considerations +7. IANA Considerations There are no IANA impacts in this memo. -9. Contributors +8. Contributors The content and concepts within this document are a product of the discussion carried out in the Design Team. Michael Ramalho provided the text for the Jitter model. -10. Acknowledgments +9. Acknowledgments Much of this document is derived from previous work on congestion control at the IETF. The authors would like to thank Harald Alvestrand, Anna Brunstrom, Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde, Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers O'Hanlon, Colin Perkins, Michael Ramalho, Zaheduzzaman Sarker, Timothy B. Terriberry, Michael Welzl, Mo Zanaty, and Xiaoqing Zhu for providing valuable feedback on earlier versions of this draft. Additionally, also thank the participants of the design team for their comments and discussion related to the evaluation criteria. -11. References +10. References -11.1. Normative References +10.1. Normative References [I-D.ietf-rmcat-cc-requirements] Jesup, R. and Z. Sarker, "Congestion Control Requirements for Interactive Real-Time Media", draft-ietf-rmcat-cc- requirements-09 (work in progress), December 2014. [I-D.ietf-rmcat-wireless-tests] Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and M. Ramalho, "Evaluation Test Cases for Interactive Real- Time Media over Wireless Networks", draft-ietf-rmcat- @@ -583,30 +578,30 @@ [RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size Real-Time Transport Control Protocol (RTCP): Opportunities and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 2009, . [RFC8083] Perkins, C. and V. Singh, "Multimedia Congestion Control: Circuit Breakers for Unicast RTP Sessions", RFC 8083, DOI 10.17487/RFC8083, March 2017, . -11.2. Informative References +10.2. Informative References [HEVC-seq] HEVC, "Test Sequences", http://www.netlab.tkk.fi/~varun/test_sequences/ . [I-D.ietf-netvc-testing] Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec Testing and Quality Measurement", draft-ietf-netvc- - testing-08 (work in progress), January 2019. + testing-09 (work in progress), January 2020. [I-D.ietf-rmcat-eval-test] Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- eval-test-10 (work in progress), May 2019. [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion Control Algorithms", BCP 133, RFC 5033, DOI 10.17487/RFC5033, August 2007, . @@ -743,21 +738,21 @@ scenarios. Authors' Addresses Varun Singh CALLSTATS I/O Oy Runeberginkatu 4c A 4 Helsinki 00100 Finland - Email: varun@callstats.io + Email: varun.singh@iki.fi URI: https://www.callstats.io/about Joerg Ott Technical University of Munich Faculty of Informatics Boltzmannstrasse 3 Garching bei Muenchen, DE 85748 Germany Email: ott@in.tum.de