draft-ietf-rmcat-eval-criteria-01.txt   draft-ietf-rmcat-eval-criteria-02.txt 
RMCAT WG V. Singh RMCAT WG V. Singh
Internet-Draft J. Ott Internet-Draft J. Ott
Intended status: Informational Aalto University Intended status: Informational Aalto University
Expires: September 11, 2014 March 10, 2014 Expires: January 26, 2015 July 25, 2014
Evaluating Congestion Control for Interactive Real-time Media Evaluating Congestion Control for Interactive Real-time Media
draft-ietf-rmcat-eval-criteria-01 draft-ietf-rmcat-eval-criteria-02
Abstract Abstract
The Real-time Transport Protocol (RTP) is used to transmit media in The Real-time Transport Protocol (RTP) is used to transmit media in
telephony and video conferencing applications. This document telephony and video conferencing applications. This document
describes the guidelines to evaluate new congestion control describes the guidelines to evaluate new congestion control
algorithms for interactive point-to-point real-time media. algorithms for interactive point-to-point real-time media.
Status of This Memo Status of This Memo
skipping to change at page 1, line 33 skipping to change at page 1, line 33
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 11, 2014. This Internet-Draft will expire on January 26, 2015.
Copyright Notice Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 15 skipping to change at page 2, line 15
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 4 3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 4
4. Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . 5 4. Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . 5
4.1. Avoiding Congestion Collapse . . . . . . . . . . . . . . 5 4.1. Avoiding Congestion Collapse . . . . . . . . . . . . . . 5
4.2. Stability . . . . . . . . . . . . . . . . . . . . . . . . 5 4.2. Stability . . . . . . . . . . . . . . . . . . . . . . . . 5
4.3. Media Traffic . . . . . . . . . . . . . . . . . . . . . . 5 4.3. Media Traffic . . . . . . . . . . . . . . . . . . . . . . 5
4.4. Start-up Behaviour . . . . . . . . . . . . . . . . . . . 5 4.4. Start-up Behaviour . . . . . . . . . . . . . . . . . . . 6
4.5. Diverse Environments . . . . . . . . . . . . . . . . . . 6 4.5. Diverse Environments . . . . . . . . . . . . . . . . . . 6
4.6. Varying Path Characteristics . . . . . . . . . . . . . . 6 4.6. Varying Path Characteristics . . . . . . . . . . . . . . 6
4.7. Reacting to Transient Events or Interruptions . . . . . . 6 4.7. Reacting to Transient Events or Interruptions . . . . . . 6
4.8. Fairness With Similar Cross-Traffic . . . . . . . . . . . 7 4.8. Fairness With Similar Cross-Traffic . . . . . . . . . . . 7
4.9. Impact on Cross-Traffic . . . . . . . . . . . . . . . . . 7 4.9. Impact on Cross-Traffic . . . . . . . . . . . . . . . . . 7
4.10. Extensions to RTP/RTCP . . . . . . . . . . . . . . . . . 7 4.10. Extensions to RTP/RTCP . . . . . . . . . . . . . . . . . 7
5. Minimum Requirements for Evaluation . . . . . . . . . . . . . 7 5. Minimum Requirements for Evaluation . . . . . . . . . . . . . 7
6. Status of Proposals . . . . . . . . . . . . . . . . . . . . . 7 6. Status of Proposals . . . . . . . . . . . . . . . . . . . . . 7
7. Security Considerations . . . . . . . . . . . . . . . . . . . 8 7. Security Considerations . . . . . . . . . . . . . . . . . . . 8
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8
9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 8 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 8
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 8 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 8
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 8 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 8
11.1. Normative References . . . . . . . . . . . . . . . . . . 8 11.1. Normative References . . . . . . . . . . . . . . . . . . 8
11.2. Informative References . . . . . . . . . . . . . . . . . 9 11.2. Informative References . . . . . . . . . . . . . . . . . 9
Appendix A. Application Trade-off . . . . . . . . . . . . . . . 10 Appendix A. Application Trade-off . . . . . . . . . . . . . . . 10
A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 10 A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 10
Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 10 Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 10
B.1. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 10 B.1. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 10
B.2. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 10 B.2. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 10
B.3. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 10 B.3. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 10
B.4. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 11 B.4. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 10
B.5. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 11 B.5. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 11
B.6. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 11 B.6. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 11
B.7. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 11
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11
1. Introduction 1. Introduction
This memo describes the guidelines to help with evaluating new This memo describes the guidelines to help with evaluating new
congestion control algorithms for interactive point-to-point real congestion control algorithms for interactive point-to-point real
time media. The requirements for the congestion control algorithm time media. The requirements for the congestion control algorithm
are outlined in [I-D.ietf-rmcat-cc-requirements]). This document are outlined in [I-D.ietf-rmcat-cc-requirements]). This document
builds upon previous work at the IETF: Specifying New Congestion builds upon previous work at the IETF: Specifying New Congestion
Control Algorithms [RFC5033] and Metrics for the Evaluation of Control Algorithms [RFC5033] and Metrics for the Evaluation of
skipping to change at page 4, line 10 skipping to change at page 4, line 10
1. Sending rate, Receiver rate, Goodput 1. Sending rate, Receiver rate, Goodput
2. Packet delay 2. Packet delay
3. Packet loss 3. Packet loss
4. If using, retransmission or FEC: residual loss 4. If using, retransmission or FEC: residual loss
5. Packets discarded from the playout or de-jitter buffer 5. Packets discarded from the playout or de-jitter buffer
[Open issue (1): The "unfairness" test is (measured at 1s intervals): 6. Fairness or Unfairness: Experiments testing the performance of an
1. Does not trigger the circuit breaker. RMCAT proposal against any cross-traffic must define its expected
2. Over 3 times or less than 1/3 times the throughput for an RMCAT criteria for fairness. The "unfairness" test guideline (measured
media stream compared to identical RMCAT streams competing on a at 1s intervals) is:
bottleneck, for a case when the competing streams have similar RTTs. 1. Does not trigger the circuit breaker.
3. Over 3 times delay compared to RTT measurements performed before 2. No RMCAT stream achieves more than 3 times the average
starting the RMCAT flow or for the case when competing with identical throughput of the RMCAT stream with the lowest average
RMCAT streams having similar RTTs. throughput, for a case when the competing streams have similar
] RTTs.
3. RTT should not grow by a factor of 3 for the existing flows
[Open issue (2): Possibly using Jain-fairness index.] when a new flow is added.
For example, see the test scenarios described in
[I-D.sarker-rmcat-eval-test].
Convergence time: the time taken to reach a stable rate at startup, 7. Convergence time: The time taken to reach a stable rate at
after the available link capacity changes, or when new flows get startup, after the available link capacity changes, or when new
added to the bottleneck link. flows get added to the bottleneck link.
Bandwidth Utilization, defined as ratio of the instantaneous sending 8. Bandwidth Utilization, defined as ratio of the instantaneous
rate to the instantaneous bottleneck capacity. This metric is useful sending rate to the instantaneous bottleneck capacity. This
when an RMCAT flow is by itself or competing with similar cross- metric is useful when an RMCAT flow is by itself or competing
traffic. with similar cross-traffic.
From the logs the statistical measures (min, max, mean, standard From the logs the statistical measures (min, max, mean, standard
deviation and variance) for the whole duration or any specific part deviation and variance) for the whole duration or any specific part
of the session can be calculated. Also the metrics (sending rate, of the session can be calculated. Also the metrics (sending rate,
receiver rate, goodput, latency) can be visualized in graphs as receiver rate, goodput, latency) can be visualized in graphs as
variation over time, the measurements in the plot are at 1 second variation over time, the measurements in the plot are at 1 second
intervals. Additionally, from the logs it is possible to plot the intervals. Additionally, from the logs it is possible to plot the
histogram or CDF of packet delay. histogram or CDF of packet delay.
[Open issue (1): Using Jain-fairness index (JFI) for measuring self-
fairness between RTP flows? measured at what intervals? visualized as
a CDF or a timeseries? Additionally: Use JFI for comparing fairness
between RTP and long TCP flows? ]
3.1. RTP Log Format 3.1. RTP Log Format
The log file is tab or comma separated containing the following The log file is tab or comma separated containing the following
details: details:
Send or receive timestamp (unix) Send or receive timestamp (unix)
RTP payload type RTP payload type
SSRC SSRC
RTP sequence no RTP sequence no
RTP timestamp RTP timestamp
skipping to change at page 7, line 32 skipping to change at page 7, line 35
various amounts of TCP and similar cross-traffic. various amounts of TCP and similar cross-traffic.
4.10. Extensions to RTP/RTCP 4.10. Extensions to RTP/RTCP
The congestion control algorithm should indicate if any protocol The congestion control algorithm should indicate if any protocol
extensions are required to implement it and should carefully describe extensions are required to implement it and should carefully describe
the impact of the extension. the impact of the extension.
5. Minimum Requirements for Evaluation 5. Minimum Requirements for Evaluation
[Editor's Note: If needed, a minimum evaluation criteria can be based The minimal requirements for RMCAT proposals is to produce or present
on the above guidelines or defined tests/scenarios.] results for the test scenarios described in Section 5 of
[I-D.sarker-rmcat-eval-test] (Basic Test Cases).
6. Status of Proposals 6. Status of Proposals
Congestion control algorithms are expected to be published as Congestion control algorithms are expected to be published as
"Experimental" documents until they are shown to be safe to deploy. "Experimental" documents until they are shown to be safe to deploy.
An algorithm published as a draft should be experimented in An algorithm published as a draft should be experimented in
simulation, or a controlled environment (testbed) to show its simulation, or a controlled environment (testbed) to show its
applicability. Every congestion control algorithm should include a applicability. Every congestion control algorithm should include a
note describing the environments in which the algorithm is tested and note describing the environments in which the algorithm is tested and
safe to deploy. It is possible that an algorithm is not recommended safe to deploy. It is possible that an algorithm is not recommended
skipping to change at page 8, line 26 skipping to change at page 8, line 32
discussion carried out in the Design Team. discussion carried out in the Design Team.
Michael Ramalho provided the text for a specific scenario, which is Michael Ramalho provided the text for a specific scenario, which is
now covered in [I-D.sarker-rmcat-eval-test]. now covered in [I-D.sarker-rmcat-eval-test].
10. Acknowledgements 10. Acknowledgements
Much of this document is derived from previous work on congestion Much of this document is derived from previous work on congestion
control at the IETF. control at the IETF.
The authors would like to thank Harald Alvestrand, Luca De Cicco, The authors would like to thank Harald Alvestrand, Anna Brunstrom,
Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde, Stefan Holmer, Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde,
Randell Jesup, Piers O'Hanlon, Colin Perkins, Michael Ramalho, Stefan Holmer, Randell Jesup, Karen Nielsen, Piers O'Hanlon, Colin
Zaheduzzaman Sarker, Timothy B. Terriberry, Michael Welzl, and Mo Perkins, Michael Ramalho, Zaheduzzaman Sarker, Timothy B. Terriberry,
Zanaty for providing valuable feedback on earlier versions of this Michael Welzl, and Mo Zanaty for providing valuable feedback on
draft. Additionally, also thank the participants of the design team earlier versions of this draft. Additionally, also thank the
for their comments and discussion related to the evaluation criteria. participants of the design team for their comments and discussion
related to the evaluation criteria.
11. References 11. References
11.1. Normative References 11.1. Normative References
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, July 2003. Applications", STD 64, RFC 3550, July 2003.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
skipping to change at page 10, line 24 skipping to change at page 10, line 30
currently an open issue. However, there is consensus that congestion currently an open issue. However, there is consensus that congestion
control algorithm should be able to show that it is useful for control algorithm should be able to show that it is useful for
interactive video by performing analysis using a real codec and video interactive video by performing analysis using a real codec and video
sequences. sequences.
Appendix B. Change Log Appendix B. Change Log
Note to the RFC-Editor: please remove this section prior to Note to the RFC-Editor: please remove this section prior to
publication as an RFC. publication as an RFC.
B.1. Changes in draft-ietf-rmcat-eval-criteria-01 B.1. Changes in draft-ietf-rmcat-eval-criteria-02
o Incorporated fairness test as a working test.
o Updated text on mimimum evaluation requirements.
B.2. Changes in draft-ietf-rmcat-eval-criteria-01
o Removed Appendix B. o Removed Appendix B.
o Removed Section on Evaluation Parameters. o Removed Section on Evaluation Parameters.
B.2. Changes in draft-ietf-rmcat-eval-criteria-00 B.3. Changes in draft-ietf-rmcat-eval-criteria-00
o Updated references. o Updated references.
o Resubmitted as WG draft. o Resubmitted as WG draft.
B.3. Changes in draft-singh-rmcat-cc-eval-04 B.4. Changes in draft-singh-rmcat-cc-eval-04
o Incorporate feedback from IETF 87, Berlin. o Incorporate feedback from IETF 87, Berlin.
o Clarified metrics: convergence time, bandwidth utilization. o Clarified metrics: convergence time, bandwidth utilization.
o Changed fairness criteria to fairness test. o Changed fairness criteria to fairness test.
o Added measuring pre- and post-repair loss. o Added measuring pre- and post-repair loss.
o Added open issue of measuring video quality to appendix. o Added open issue of measuring video quality to appendix.
o clarified use of DropTail and AQM. o clarified use of DropTail and AQM.
o Updated text in "Minimum Requirements for Evaluation" o Updated text in "Minimum Requirements for Evaluation"
B.4. Changes in draft-singh-rmcat-cc-eval-03 B.5. Changes in draft-singh-rmcat-cc-eval-03
o Incorporate the discussion within the design team. o Incorporate the discussion within the design team.
o Added a section on evaluation parameters, it describes the flow o Added a section on evaluation parameters, it describes the flow
and network characteristics. and network characteristics.
o Added Appendix with self-fairness experiment. o Added Appendix with self-fairness experiment.
o Changed bottleneck parameters from a proposal to an example set. o Changed bottleneck parameters from a proposal to an example set.
o o
B.5. Changes in draft-singh-rmcat-cc-eval-02 B.6. Changes in draft-singh-rmcat-cc-eval-02
o Added scenario descriptions. o Added scenario descriptions.
B.6. Changes in draft-singh-rmcat-cc-eval-01 B.7. Changes in draft-singh-rmcat-cc-eval-01
o Removed QoE metrics. o Removed QoE metrics.
o Changed stability to steady-state. o Changed stability to steady-state.
o Added measuring impact against few and many flows. o Added measuring impact against few and many flows.
o Added guideline for idle and data-limited periods. o Added guideline for idle and data-limited periods.
o Added reference to TCP evaluation suite in example evaluation o Added reference to TCP evaluation suite in example evaluation
 End of changes. 17 change blocks. 
43 lines changed or deleted 59 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/