draft-ietf-rmcat-eval-criteria-12.txt   draft-ietf-rmcat-eval-criteria-13.txt 
RMCAT WG V. Singh RMCAT WG V. Singh
Internet-Draft callstats.io Internet-Draft callstats.io
Intended status: Informational J. Ott Intended status: Informational J. Ott
Expires: August 30, 2020 Technical University of Munich Expires: September 10, 2020 Technical University of Munich
S. Holmer S. Holmer
Google Google
February 27, 2020 March 9, 2020
Evaluating Congestion Control for Interactive Real-time Media Evaluating Congestion Control for Interactive Real-time Media
draft-ietf-rmcat-eval-criteria-12 draft-ietf-rmcat-eval-criteria-13
Abstract Abstract
The Real-time Transport Protocol (RTP) is used to transmit media in The Real-time Transport Protocol (RTP) is used to transmit media in
telephony and video conferencing applications. This document telephony and video conferencing applications. This document
describes the guidelines to evaluate new congestion control describes the guidelines to evaluate new congestion control
algorithms for interactive point-to-point real-time media. algorithms for interactive point-to-point real-time media.
Status of This Memo Status of This Memo
skipping to change at page 1, line 36 skipping to change at page 1, line 36
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 30, 2020. This Internet-Draft will expire on September 10, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 16 skipping to change at page 2, line 16
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 5 3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 5
4. List of Network Parameters . . . . . . . . . . . . . . . . . 6 4. List of Network Parameters . . . . . . . . . . . . . . . . . 6
4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 6 4.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 6
4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 6 4.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 6
4.3. Drop Tail Router Queue Length . . . . . . . . . . . . . . 6 4.3. Drop Tail Router Queue Length . . . . . . . . . . . . . . 7
4.4. Loss generation model . . . . . . . . . . . . . . . . . . 7 4.4. Loss generation model . . . . . . . . . . . . . . . . . . 7
4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 7 4.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 7
4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 8 4.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 8
4.5.2. Approximately Random Subject to No-Reordering Bounded 4.5.2. Approximately Random Subject to No-Reordering Bounded
PDV (NR-RPVD) . . . . . . . . . . . . . . . . 9 PDV (NR-RPVD) . . . . . . . . . . . . . . . . 9
4.5.3. Recommended distribution . . . . . . . . . . . . . . 10 4.5.3. Recommended distribution . . . . . . . . . . . . . . 10
5. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 10 5. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 10
5.1. TCP traffic model . . . . . . . . . . . . . . . . . . . . 10 5.1. TCP traffic model . . . . . . . . . . . . . . . . . . . . 10
5.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 11 5.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 11
5.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 11 5.3. Background UDP . . . . . . . . . . . . . . . . . . . . . 11
6. Security Considerations . . . . . . . . . . . . . . . . . . . 11 6. Security Considerations . . . . . . . . . . . . . . . . . . . 11
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12
8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12 8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12
9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 12 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 12
10.1. Normative References . . . . . . . . . . . . . . . . . . 12 10.1. Normative References . . . . . . . . . . . . . . . . . . 13
10.2. Informative References . . . . . . . . . . . . . . . . . 13 10.2. Informative References . . . . . . . . . . . . . . . . . 13
Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 14 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 14
A.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 14 A.1. Changes in draft-ietf-rmcat-eval-criteria-07 . . . . . . 15
A.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 14 A.2. Changes in draft-ietf-rmcat-eval-criteria-06 . . . . . . 15
A.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 15 A.3. Changes in draft-ietf-rmcat-eval-criteria-05 . . . . . . 15
A.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 15 A.4. Changes in draft-ietf-rmcat-eval-criteria-04 . . . . . . 15
A.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 15 A.5. Changes in draft-ietf-rmcat-eval-criteria-03 . . . . . . 15
A.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 15 A.6. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 15
A.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 15 A.7. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 15
A.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 15 A.8. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 16
A.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 15 A.9. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 16
A.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 16 A.10. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 16
A.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 16 A.11. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 16
A.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 16 A.12. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 16
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 16 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 17
1. Introduction 1. Introduction
This memo describes the guidelines to help with evaluating new This memo describes the guidelines to help with evaluating new
congestion control algorithms for interactive point-to-point real congestion control algorithms for interactive point-to-point real
time media. The requirements for the congestion control algorithm time media. The requirements for the congestion control algorithm
are outlined in [I-D.ietf-rmcat-cc-requirements]). This document are outlined in [I-D.ietf-rmcat-cc-requirements]). This document
builds upon previous work at the IETF: Specifying New Congestion builds upon previous work at the IETF: Specifying New Congestion
Control Algorithms [RFC5033] and Metrics for the Evaluation of Control Algorithms [RFC5033] and Metrics for the Evaluation of
Congestion Control Algorithms [RFC5166]. Congestion Control Algorithms [RFC5166].
skipping to change at page 4, line 17 skipping to change at page 4, line 17
3. Metrics 3. Metrics
This document specifies testing criteria for evaluating congestion This document specifies testing criteria for evaluating congestion
control algorithms for RTP media flows. Proposed algorithms are to control algorithms for RTP media flows. Proposed algorithms are to
prove their performance by means of simulation and/or emulation prove their performance by means of simulation and/or emulation
experiments for all the cases described. experiments for all the cases described.
Each experiment is expected to log every incoming and outgoing packet Each experiment is expected to log every incoming and outgoing packet
(the RTP logging format is described in Section 3.1). The logging (the RTP logging format is described in Section 3.1). The logging
can be done inside the application or at the endpoints using PCAP can be done inside the application or at the endpoints using PCAP
(packet capture, e.g., tcpdump, wireshark). The following metrics (packet capture, e.g., tcpdump [tcpdump], wireshark [wireshark]).
are calculated based on the information in the packet logs: The following metrics are calculated based on the information in the
packet logs:
1. Sending rate, Receiver rate, Goodput (measured at 200ms 1. Sending rate, Receiver rate, Goodput (measured at 200ms
intervals) intervals)
2. Packets sent, Packets received 2. Packets sent, Packets received
3. Bytes sent, bytes received 3. Bytes sent, bytes received
4. Packet delay 4. Packet delay
skipping to change at page 6, line 5 skipping to change at page 6, line 5
RTP sequence no RTP sequence no
RTP timestamp RTP timestamp
marker bit marker bit
payload size payload size
If the congestion control implements, retransmissions or FEC, the If the congestion control implements, retransmissions or FEC, the
evaluation should report both packet loss (before applying error- evaluation should report both packet loss (before applying error-
resilience) and residual packet loss (after applying error- resilience) and residual packet loss (after applying error-
resilience). resilience).
These data should suffice to compute the media-encoding independent
metrics described above. Use of a common log will allow simplified
post-processing and analysis across different implementations.
4. List of Network Parameters 4. List of Network Parameters
The implementors initially are encouraged to choose evaluation The implementors initially are encouraged to choose evaluation
settings from the following values: settings from the following values:
4.1. One-way Propagation Delay 4.1. One-way Propagation Delay
Experiments are expected to verify that the congestion control is Experiments are expected to verify that the congestion control is
able to work across a broad range of path characteristics, also able to work across a broad range of path characteristics, also
including challenging situations, for example over trans-continental including challenging situations, for example over trans-continental
skipping to change at page 7, line 35 skipping to change at page 7, line 41
Many models for generating packet loss are available, some yield Many models for generating packet loss are available, some yield
correlated, others independent losses; losses can also be extracted correlated, others independent losses; losses can also be extracted
from packet traces. As a (simple) minimum loss model with minimal from packet traces. As a (simple) minimum loss model with minimal
parameterization (i.e., the loss rate), independent random losses parameterization (i.e., the loss rate), independent random losses
must be used in the evaluation. must be used in the evaluation.
It is known that independent loss models may reflect reality poorly It is known that independent loss models may reflect reality poorly
and hence more sophisticated loss models could be considered. and hence more sophisticated loss models could be considered.
Suitable models for correlated losses includes the Gilbert-Elliot Suitable models for correlated losses includes the Gilbert-Elliot
model and losses generated by modeling a queue including its model [gilbert-elliott] and losses generated by modeling a queue
(different) drop behaviors. including its (different) drop behaviors.
4.5. Jitter models 4.5. Jitter models
This section defines jitter models for the purposes of this document. This section defines jitter models for the purposes of this document.
When jitter is to be applied to both the congestion controlled RTP When jitter is to be applied to both the congestion controlled RTP
flow and any competing flow (such as a TCP competing flow), the flow and any competing flow (such as a TCP competing flow), the
competing flow will use the jitter definition below that does not competing flow will use the jitter definition below that does not
allow for re-ordering of packets on the competing flow (see NR-RBPDV allow for re-ordering of packets on the competing flow (see NR-RBPDV
definition below). definition below).
Jitter is an overloaded term in communications. It is is typically Jitter is an overloaded term in communications. It is typically used
used to refer to the variation of a metric (e.g., delay) with respect to refer to the variation of a metric (e.g., delay) with respect to
to some reference metric (e.g., average delay or minimum delay). For some reference metric (e.g., average delay or minimum delay). For
example, RFC 3550 jitter is computed as the smoothed difference in example, RFC 3550 jitter is computed as the smoothed difference in
packet arrival times relative to their respective expected arrival packet arrival times relative to their respective expected arrival
times, which is particularly meaningful if the underlying packet times, which is particularly meaningful if the underlying packet
delay variation was caused by a Gaussian random process. delay variation was caused by a Gaussian random process.
Because jitter is an overloaded term, we use the term Packet Delay Because jitter is an overloaded term, we use the term Packet Delay
Variation (PDV) instead to describe the variation of delay of Variation (PDV) instead to describe the variation of delay of
individual packets in the same sense as the IETF IPPM WG has defined individual packets in the same sense as the IETF IPPM WG has defined
PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has
defined IP Packet Delay Variation (IPDV) in their documents (e.g., defined IP Packet Delay Variation (IPDV) in their documents (e.g.,
skipping to change at page 11, line 14 skipping to change at page 11, line 18
in detail which congestion control schemes they tested against and in detail which congestion control schemes they tested against and
which parameters were used. which parameters were used.
5.2. RTP Video model 5.2. RTP Video model
[RFC8593] describes two types of video traffic models for evaluating [RFC8593] describes two types of video traffic models for evaluating
candidate algorithms for RTP congestion control. The first model candidate algorithms for RTP congestion control. The first model
statistically characterizes the behavior of a video encoder, whereas statistically characterizes the behavior of a video encoder, whereas
the second model uses video traces. the second model uses video traces.
Sample video test sequences are available at: [xiph-seq] and Sample video test sequences are available at [xiph-seq]. The
[HEVC-seq]. The following two video streams are the recommended following two video streams are the recommended minimum for testing:
minimum for testing: Foreman and FourPeople. Foreman (CIF sequence) and FourPeople (720p); both come as raw video
data to be encoded dynamically. As these video sequences are short
(300 and 600 frames, respectively, they shall be stitched together
repeatedly until the desired length is reached.
5.3. Background UDP 5.3. Background UDP
Background UDP flow is modeled as a constant bit rate (CBR) flow. It Background UDP flow is modeled as a constant bit rate (CBR) flow. It
will download data at a particular CBR rate for the complete session, will download data at a particular CBR rate for the complete session,
or will change to particular CBR rate at predefined intervals. The or will change to particular CBR rate at predefined intervals. The
inter packet interval is calculated based on the CBR and the packet inter packet interval is calculated based on the CBR and the packet
size (is typically set to the path MTU size, the default value can be size (is typically set to the path MTU size, the default value can be
1500 bytes). 1500 bytes).
skipping to change at page 13, line 43 skipping to change at page 14, line 5
DOI 10.17487/RFC8083, March 2017, DOI 10.17487/RFC8083, March 2017,
<https://www.rfc-editor.org/info/rfc8083>. <https://www.rfc-editor.org/info/rfc8083>.
[RFC8593] Zhu, X., Mena, S., and Z. Sarker, "Video Traffic Models [RFC8593] Zhu, X., Mena, S., and Z. Sarker, "Video Traffic Models
for RTP Congestion Control Evaluations", RFC 8593, for RTP Congestion Control Evaluations", RFC 8593,
DOI 10.17487/RFC8593, May 2019, DOI 10.17487/RFC8593, May 2019,
<https://www.rfc-editor.org/info/rfc8593>. <https://www.rfc-editor.org/info/rfc8593>.
10.2. Informative References 10.2. Informative References
[HEVC-seq] [gilbert-elliott]
HEVC, "Test Sequences", Hasslinger, G. and O. Hohlfeld, "The Gilbert-Elliott Model
http://www.netlab.tkk.fi/~varun/test_sequences/ . for Packet Loss in Real Time Services on the Internet",
14th GI/ITG Conference - Measurement, Modelling and
Evalutation of Computer and Communication Systems , 3
2008.
[I-D.ietf-netvc-testing] [I-D.ietf-netvc-testing]
Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec
Testing and Quality Measurement", draft-ietf-netvc- Testing and Quality Measurement", draft-ietf-netvc-
testing-09 (work in progress), January 2020. testing-09 (work in progress), January 2020.
[I-D.ietf-rmcat-eval-test] [I-D.ietf-rmcat-eval-test]
Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test
Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat-
eval-test-10 (work in progress), May 2019. eval-test-10 (work in progress), May 2019.
[I-D.ietf-rmcat-wireless-tests] [I-D.ietf-rmcat-wireless-tests]
Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W., and
M. Ramalho, "Evaluation Test Cases for Interactive Real- M. Ramalho, "Evaluation Test Cases for Interactive Real-
Time Media over Wireless Networks", draft-ietf-rmcat- Time Media over Wireless Networks", draft-ietf-rmcat-
wireless-tests-08 (work in progress), July 2019. wireless-tests-09 (work in progress), February 2020.
[RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion [RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion
Control Algorithms", BCP 133, RFC 5033, Control Algorithms", BCP 133, RFC 5033,
DOI 10.17487/RFC5033, August 2007, DOI 10.17487/RFC5033, August 2007,
<https://www.rfc-editor.org/info/rfc5033>. <https://www.rfc-editor.org/info/rfc5033>.
[RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion [RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion
Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March
2008, <https://www.rfc-editor.org/info/rfc5166>. 2008, <https://www.rfc-editor.org/info/rfc5166>.
[tcpdump] "Homepage of tcpdump and libpcap",
https://www.tcpdump.org/index.html .
[wireshark]
"Homepage of Wireshark", https://www.wireshark.org .
[xiph-seq] [xiph-seq]
Daede, T., "Video Test Media Set", Daede, T., "Video Test Media Set",
https://people.xiph.org/~tdaede/sets/ . https://media.xiph.org/video/derf/ .
Appendix A. Change Log Appendix A. Change Log
Note to the RFC-Editor: please remove this section prior to Note to the RFC-Editor: please remove this section prior to
publication as an RFC. publication as an RFC.
A.1. Changes in draft-ietf-rmcat-eval-criteria-07 A.1. Changes in draft-ietf-rmcat-eval-criteria-07
Updated the draft according to the discussion at IETF-101. Updated the draft according to the discussion at IETF-101.
 End of changes. 18 change blocks. 
26 lines changed or deleted 43 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/