draft-ietf-rmcat-eval-criteria-02.txt		draft-ietf-rmcat-eval-criteria-03.txt
	RMCAT WG V. Singh		RMCAT WG V. Singh
	Internet-Draft J. Ott		Internet-Draft J. Ott
	Intended status: Informational Aalto University		Intended status: Informational Aalto University
	Expires: January 26, 2015 July 25, 2014		Expires: September 11, 2015 March 10, 2015
	Evaluating Congestion Control for Interactive Real-time Media		Evaluating Congestion Control for Interactive Real-time Media
	draft-ietf-rmcat-eval-criteria-02		draft-ietf-rmcat-eval-criteria-03
	Abstract		Abstract
	The Real-time Transport Protocol (RTP) is used to transmit media in		The Real-time Transport Protocol (RTP) is used to transmit media in
	telephony and video conferencing applications. This document		telephony and video conferencing applications. This document
	describes the guidelines to evaluate new congestion control		describes the guidelines to evaluate new congestion control
	algorithms for interactive point-to-point real-time media.		algorithms for interactive point-to-point real-time media.
	Status of This Memo		Status of This Memo
	skipping to change at page 1, line 33		skipping to change at page 1, line 33
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF). Note that other groups may also distribute		Task Force (IETF). Note that other groups may also distribute
	working documents as Internet-Drafts. The list of current Internet-		working documents as Internet-Drafts. The list of current Internet-
	Drafts is at http://datatracker.ietf.org/drafts/current/.		Drafts is at http://datatracker.ietf.org/drafts/current/.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
	This Internet-Draft will expire on January 26, 2015.		This Internet-Draft will expire on September 11, 2015.
	Copyright Notice		Copyright Notice
	Copyright (c) 2014 IETF Trust and the persons identified as the		Copyright (c) 2015 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents		Provisions Relating to IETF Documents
	(http://trustee.ietf.org/license-info) in effect on the date of		(http://trustee.ietf.org/license-info) in effect on the date of
	publication of this document. Please review these documents		publication of this document. Please review these documents
	carefully, as they describe your rights and restrictions with respect		carefully, as they describe your rights and restrictions with respect
	to this document. Code Components extracted from this document must		to this document. Code Components extracted from this document must
	include Simplified BSD License text as described in Section 4.e of		include Simplified BSD License text as described in Section 4.e of
	the Trust Legal Provisions and are provided without warranty as		the Trust Legal Provisions and are provided without warranty as
	described in the Simplified BSD License.		described in the Simplified BSD License.
	Table of Contents		Table of Contents
	1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2		1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
	2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3		2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
	3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3		3. Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
	3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 4		3.1. RTP Log Format . . . . . . . . . . . . . . . . . . . . . 5
	4. Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . 5		4. Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . 5
	4.1. Avoiding Congestion Collapse . . . . . . . . . . . . . . 5		4.1. Avoiding Congestion Collapse . . . . . . . . . . . . . . 5
	4.2. Stability . . . . . . . . . . . . . . . . . . . . . . . . 5		4.2. Stability . . . . . . . . . . . . . . . . . . . . . . . . 5
	4.3. Media Traffic . . . . . . . . . . . . . . . . . . . . . . 5		4.3. Media Traffic . . . . . . . . . . . . . . . . . . . . . . 5
	4.4. Start-up Behaviour . . . . . . . . . . . . . . . . . . . 6		4.4. Start-up Behaviour . . . . . . . . . . . . . . . . . . . 6
	4.5. Diverse Environments . . . . . . . . . . . . . . . . . . 6		4.5. Diverse Environments . . . . . . . . . . . . . . . . . . 6
	4.6. Varying Path Characteristics . . . . . . . . . . . . . . 6		4.6. Varying Path Characteristics . . . . . . . . . . . . . . 6
	4.7. Reacting to Transient Events or Interruptions . . . . . . 6		4.7. Reacting to Transient Events or Interruptions . . . . . . 7
	4.8. Fairness With Similar Cross-Traffic . . . . . . . . . . . 7		4.8. Fairness With Similar Cross-Traffic . . . . . . . . . . . 7
	4.9. Impact on Cross-Traffic . . . . . . . . . . . . . . . . . 7		4.9. Impact on Cross-Traffic . . . . . . . . . . . . . . . . . 7
	4.10. Extensions to RTP/RTCP . . . . . . . . . . . . . . . . . 7		4.10. Extensions to RTP/RTCP . . . . . . . . . . . . . . . . . 7
	5. Minimum Requirements for Evaluation . . . . . . . . . . . . . 7		5. List of Network Parameters . . . . . . . . . . . . . . . . . 7
	6. Status of Proposals . . . . . . . . . . . . . . . . . . . . . 7		5.1. One-way Propagation Delay . . . . . . . . . . . . . . . . 7
	7. Security Considerations . . . . . . . . . . . . . . . . . . . 8		5.2. End-to-end Loss . . . . . . . . . . . . . . . . . . . . . 8
	8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8		5.3. DropTail Router Queue Length . . . . . . . . . . . . . . 8
	9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 8		5.4. Loss generation model . . . . . . . . . . . . . . . . . . 8
	10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 8		5.5. Jitter models . . . . . . . . . . . . . . . . . . . . . . 9
	11. References . . . . . . . . . . . . . . . . . . . . . . . . . 8		5.5.1. Random Bounded PDV (RBPDV) . . . . . . . . . . . . . 9
	11.1. Normative References . . . . . . . . . . . . . . . . . . 8		5.5.2. Approximately Random Subject to No-Reordering Bounded
	11.2. Informative References . . . . . . . . . . . . . . . . . 9		PDV (NR-RPVD) . . . . . . . . . . . . . . . . 10
	Appendix A. Application Trade-off . . . . . . . . . . . . . . . 10		6. Traffic Models . . . . . . . . . . . . . . . . . . . . . . . 11
	A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 10		6.1. TCP taffic model . . . . . . . . . . . . . . . . . . . . 11
	Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 10		6.2. RTP Video model . . . . . . . . . . . . . . . . . . . . . 12
	B.1. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 10		7. Security Considerations . . . . . . . . . . . . . . . . . . . 12
	B.2. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 10		8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12
	B.3. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 10		9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12
	B.4. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 10		10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12
	B.5. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 11		11. References . . . . . . . . . . . . . . . . . . . . . . . . . 12
	B.6. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 11		11.1. Normative References . . . . . . . . . . . . . . . . . . 12
	B.7. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 11		11.2. Informative References . . . . . . . . . . . . . . . . . 13
	Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11		Appendix A. Application Trade-off . . . . . . . . . . . . . . . 14
			A.1. Measuring Quality . . . . . . . . . . . . . . . . . . . . 14
			Appendix B. Change Log . . . . . . . . . . . . . . . . . . . . . 14
			B.1. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 14
			B.2. Changes in draft-ietf-rmcat-eval-criteria-02 . . . . . . 14
			B.3. Changes in draft-ietf-rmcat-eval-criteria-01 . . . . . . 14
			B.4. Changes in draft-ietf-rmcat-eval-criteria-00 . . . . . . 15
			B.5. Changes in draft-singh-rmcat-cc-eval-04 . . . . . . . . . 15
			B.6. Changes in draft-singh-rmcat-cc-eval-03 . . . . . . . . . 15
			B.7. Changes in draft-singh-rmcat-cc-eval-02 . . . . . . . . . 15
			B.8. Changes in draft-singh-rmcat-cc-eval-01 . . . . . . . . . 15
			Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 16
	1. Introduction		1. Introduction
	This memo describes the guidelines to help with evaluating new		This memo describes the guidelines to help with evaluating new
	congestion control algorithms for interactive point-to-point real		congestion control algorithms for interactive point-to-point real
	time media. The requirements for the congestion control algorithm		time media. The requirements for the congestion control algorithm
	are outlined in [I-D.ietf-rmcat-cc-requirements]). This document		are outlined in [I-D.ietf-rmcat-cc-requirements]). This document
	builds upon previous work at the IETF: Specifying New Congestion		builds upon previous work at the IETF: Specifying New Congestion
	Control Algorithms [RFC5033] and Metrics for the Evaluation of		Control Algorithms [RFC5033] and Metrics for the Evaluation of
	Congestion Control Algorithms [RFC5166].		Congestion Control Algorithms [RFC5166].
	The guidelines proposed in the document are intended to help prevent		The guidelines proposed in the document are intended to help prevent
	a congestion collapse, promote fair capacity usage and optimize the		a congestion collapse, promote fair capacity usage and optimize the
	media flow's throughput. Furthermore, the proposed algorithms are		media flow's throughput. Furthermore, the proposed algorithms are
	expected to operate within the envelope of the circuit breakers		expected to operate within the envelope of the circuit breakers
	defined in [I-D.ietf-avtcore-rtp-circuit-breakers].		defined in [I-D.ietf-avtcore-rtp-circuit-breakers].
	This document only provides broad-level criteria for evaluating a new		This document only provides broad-level criteria for evaluating a new
	congestion control algorithm and the working group should expect a		congestion control algorithm. The minimal requirement for RMCAT
	thorough scientific study to make its decision. The results of the		proposals is to produce or present results for the test scenarios
	evaluation are not expected to be included within the internet-draft		described in [I-D.ietf-rmcat-eval-test] (Basic Test Cases). The
	but should be cited in the document.		results of the evaluation are not expected to be included within the
			internet-draft but should be cited in the document.
	2. Terminology		2. Terminology
	The terminology defined in RTP [RFC3550], RTP Profile for Audio and		The terminology defined in RTP [RFC3550], RTP Profile for Audio and
	Video Conferences with Minimal Control [RFC3551], RTCP Extended		Video Conferences with Minimal Control [RFC3551], RTCP Extended
	Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback		Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback
	(RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506]		(RTP/AVPF) [RFC4585] and Support for Reduced-Size RTCP [RFC5506]
	apply.		apply.
	3. Metrics		3. Metrics
	[RFC5166] describes the basic metrics for congestion control.
	Metrics that are of interest for interactive multimedia are:
	o Throughput.
	o Minimizing oscillations in the transmission rate (stability) when
	the end-to-end capacity varies slowly.
	o Delay.
	o Reactivity to transient events.
	o Packet losses and discards.
	o Section 2.1 of [RFC5166] discusses the tradeoff between
	throughput, delay and loss.
	Each experiment is expected to log every incoming and outgoing packet		Each experiment is expected to log every incoming and outgoing packet
	(the RTP logging format is described in Section 3.1). The logging		(the RTP logging format is described in Section 3.1). The logging
	can be done inside the application or at the endpoints using pcap		can be done inside the application or at the endpoints using PCAP
	(packet capture, e.g., tcpdump, wireshark). The following are		(packet capture, e.g., tcpdump, wireshark). The following are
	calculated based on the information in the packet logs:		calculated based on the information in the packet logs:
	1. Sending rate, Receiver rate, Goodput		1. Sending rate, Receiver rate, Goodput (measured at 200ms
			intervals)
	2. Packet delay		2. Packets sent, Packets received
	3. Packet loss
	4. If using, retransmission or FEC: residual loss		3. Bytes sent, bytes received
	5. Packets discarded from the playout or de-jitter buffer		4. Packet delay
			5. Packets lost, Packets discarded (from the playout or de-jitter
			buffer)
	6. Fairness or Unfairness: Experiments testing the performance of an		6. If using, retransmission or FEC: post-repair loss
	RMCAT proposal against any cross-traffic must define its expected
	criteria for fairness. The "unfairness" test guideline (measured
	at 1s intervals) is:
	1. Does not trigger the circuit breaker.
	2. No RMCAT stream achieves more than 3 times the average
	throughput of the RMCAT stream with the lowest average
	throughput, for a case when the competing streams have similar
	RTTs.
	3. RTT should not grow by a factor of 3 for the existing flows
	when a new flow is added.
	For example, see the test scenarios described in
	[I-D.sarker-rmcat-eval-test].
	7. Convergence time: The time taken to reach a stable rate at		7. Fairness or Unfairness: Experiments testing the performance of
	startup, after the available link capacity changes, or when new		an RMCAT proposal against any cross-traffic must define its
	flows get added to the bottleneck link.		expected criteria for fairness. The "unfairness" test guideline
			(measured at 1s intervals) is:
			1. Does not trigger the circuit breaker.
			2. No RMCAT stream achieves more than 3 times the average
			throughput of the RMCAT stream with the lowest average
			throughput, for a case when the competing streams have similar
			RTTs.
			3. RTT should not grow by a factor of 3 for the existing flows
			when a new flow is added.
			For example, see the test scenarios described in
			[I-D.ietf-rmcat-eval-test].
	8. Bandwidth Utilization, defined as ratio of the instantaneous		8. Convergence time: The time taken to reach a stable rate at
	sending rate to the instantaneous bottleneck capacity. This		startup, after the available link capacity changes, or when new
	metric is useful when an RMCAT flow is by itself or competing		flows get added to the bottleneck link.
	with similar cross-traffic.
			9. Instability or oscillation in the sending rate: The frequency or
			number of instances when the sending rate oscillates between an
			high watermark level and a low watermark level, or vice-versa in
			a defined time window. For example, the watermarks can be set
			at 4x interval: 500 Kbps, 2 Mbps, and a time window of 500ms.
			10. Bandwidth Utilization, defined as ratio of the instantaneous
			sending rate to the instantaneous bottleneck capacity. This
			metric is useful only when an RMCAT flow is by itself or
			competing with similar cross-traffic.
	From the logs the statistical measures (min, max, mean, standard		From the logs the statistical measures (min, max, mean, standard
	deviation and variance) for the whole duration or any specific part		deviation and variance) for the whole duration or any specific part
	of the session can be calculated. Also the metrics (sending rate,		of the session can be calculated. Also the metrics (sending rate,
	receiver rate, goodput, latency) can be visualized in graphs as		receiver rate, goodput, latency) can be visualized in graphs as
	variation over time, the measurements in the plot are at 1 second		variation over time, the measurements in the plot are at 1 second
	intervals. Additionally, from the logs it is possible to plot the		intervals. Additionally, from the logs it is possible to plot the
	histogram or CDF of packet delay.		histogram or CDF of packet delay.
	[Open issue (1): Using Jain-fairness index (JFI) for measuring self-		[Open issue (1): Using Jain-fairness index (JFI) for measuring self-
	skipping to change at page 7, line 33		skipping to change at page 7, line 39
	The proposal should also measure the impact on varied number of		The proposal should also measure the impact on varied number of
	cross-traffic sources, i.e., few and many competing flows, or mixing		cross-traffic sources, i.e., few and many competing flows, or mixing
	various amounts of TCP and similar cross-traffic.		various amounts of TCP and similar cross-traffic.
	4.10. Extensions to RTP/RTCP		4.10. Extensions to RTP/RTCP
	The congestion control algorithm should indicate if any protocol		The congestion control algorithm should indicate if any protocol
	extensions are required to implement it and should carefully describe		extensions are required to implement it and should carefully describe
	the impact of the extension.		the impact of the extension.
	5. Minimum Requirements for Evaluation		5. List of Network Parameters
	The minimal requirements for RMCAT proposals is to produce or present		The implementors initially are encouraged to choose evaluation
	results for the test scenarios described in Section 5 of		settings from the following values:
	[I-D.sarker-rmcat-eval-test] (Basic Test Cases).
	6. Status of Proposals		5.1. One-way Propagation Delay
	Congestion control algorithms are expected to be published as		Experiments are expected to verify that the congestion control is
	"Experimental" documents until they are shown to be safe to deploy.		able to work in challenging situations, for example over trans-
	An algorithm published as a draft should be experimented in		continental and/or satellite links. Typical values are:
	simulation, or a controlled environment (testbed) to show its
	applicability. Every congestion control algorithm should include a
	note describing the environments in which the algorithm is tested and
	safe to deploy. It is possible that an algorithm is not recommended
	for certain environments or perform sub-optimally for the user.
	[Editor's Note: Should there be a distinction between "Informational"		1. Very low latency: 0-1ms
	and "Experimental" drafts for congestion control algorithms in RMCAT.
	[RFC5033] describes Informational proposals as algorithms that are		2. Low latency: 50ms
	not safe for deployment but are proposals to experiment with in		3. High latency: 150ms
	simulation/testbeds. While Experimental algorithms are ones that are
	deemed safe in some environments but require a more thorough		4. Extreme latency: 300ms
	evaluation (from the community).]
			5.2. End-to-end Loss
			To model lossy links, the experiments can choose one of the following
			loss rates, the fractional loss is the ratio of packets lost and
			packets sent.
			1. no loss: 0%
			2. 1%
			3. 5%
			4. 10%
			5. 20%
			5.3. DropTail Router Queue Length
			The router queue length is measured as the time taken to drain the
			FIFO queue. It has been noted in various discussions that the queue
			length in the current deployed Internet varies significantly. While
			the core backbone network has very short queue length, the home
			gateways usually have larger queue length. Those various queue
			lengths can be categorized in the following way:
			1. QoS-aware (or short): 70ms
			2. Nominal: 300-500ms
			3. Buffer-bloated: 1000-2000ms
			Here the size of the queue is measured in bytes or packets and to
			convert the queue length measured in seconds to queue length in
			bytes:
			QueueSize (in bytes) = QueueSize (in sec) x Throughput (in bps)/8
			5.4. Loss generation model
			[Editor's note : Describes the model for generating packet losses,
			for example, losses can be generated using traces, or using the
			Gilbert-Elliot model, or randomly (uncorrelated loss).]
			5.5. Jitter models
			This section defines jitter model for the purposes of this document.
			When jitter is to be applied to both the RMCAT flow and any competing
			flow (such as a TCP competing flow), the competing flow will use the
			jitter definition below that does not allow for re-ordering of
			packets on the competing flow (see NR-RBPDV definition below).
			Jitter is an overloaded term in communications. Its meaning is
			typically associated with the variation of a metric (e.g., delay)
			with respect to some reference metric (e.g., average delay or minimum
			delay). For example, RFC 3550 jitter is a smoothed estimate of
			jitter which is particularly meaningful if the underlying packet
			delay variation was caused by a Gaussian random process.
			Because jitter is an overloaded term, we instead use the term Packet
			Delay Variation (PDV) to describe the variation of delay of
			individual packets in the same sense as the IETF IPPM WG has defined
			PDV in their documents (e.g., RFC 3393) and as the ITU-T SG16 has
			defined IP Packet Delay Variation (IPDV) in their documents (e.g.,
			Y.1540).
			Most PDV distributions in packet network systems are one-sided
			distributions (the measurement of which with a finite number of
			measurement samples result in one-sided histograms). In the usual
			packet network transport case there is typically one packet that
			transited the network with the minimum delay, then a majority of
			packets also transit the system within some variation from this
			minimum delay, and then a minority of the packets transits the
			network with delays higher than the median or average transit time
			(these are outliers). Although infrequent, outliers can cause
			significant deleterious operation in adaptive systems and should be
			considered in RMCAT adaptation designs.
			In this section we define two different bounded PDV characteristics,
			1) Random Bounded PDV and 2) Approximately Random Subject to No-
			Reordering Bounded PDV.
			5.5.1. Random Bounded PDV (RBPDV)
			The RBPDV probability distribution function (pdf) is specified to be
			of some mathematically describable function which includes some
			practical minimum and maximum discrete values suitable for testing.
			For example, the minimum value, x_min, might be specified as the
			minimum transit time packet and the maximum value, x_max, might be
			idefined to be two standard deviations higher than the mean.
			Since we are typically interested in the distribution relative to the
			mean delay packet, we define the zero mean PVD sample, z(n), to be
			z(n) = x(n) - x_mean, where x(n) is a sample of the RBPDV random
			variable x and x_mean is the mean of x.
			We assume here that s(n) is the original source time of packet n and
			the post-jitter induced emmission time, j(n), for packet n is j(n) =
			{[z(n) + x_mean] + s(n)}. It follows that the separation in the post-
			jitter time of packets n and n+1 is {[s(n+1)-s(n)] - [z(n)-z(n+1)]}.
			Since the first term is always a positive quantity, we note that
			packet reordering at the receiver is possible whenever the second
			term is greater than the first. Said another way, whenever the
			difference in possible zero mean PDV sample delays (i.e., [x_max-
			x_min]) exceeds the inter-departure time of any two sent packets, we
			have the possibility of packet re-ordering.
			There are important use cases in real networks where packets can
			become re-ordered such as in load balancing topologies and during
			route changes. However, for the vast majority of cases there is no
			packet re-ordering because most of the time packets follow the same
			path. Due to this, if a packet becomes overly delayed, the packets
			after it on that flow are also delayed. This is especially true for
			mobile wireless links where there are per-flow queues prior to base
			station scheduling. Owing to this important use case, we define
			another PDV profile similar to the above, but one that does not allow
			for re-ordering within a flow.
			5.5.2. Approximately Random Subject to No-Reordering Bounded PDV (NR-
			RPVD)
			No Reordering RPDV, NR-RPVD, is defined similarly to the above with
			one important exception. Let serial(n) be defined as the
			serialization delay of packet n at the lowest bottleneck link rate
			(or other appropriate rate) in a given test. Then we produce all the
			post-jitter values for j(n) for n = 1, 2, ... N, where N is the
			length of the source sequence s to be offset-ed. The exception can
			be stated as follows: We revisit all j(n) beginning from index n=2,
			and if j(n) is determined to be less than [j(n-1)+serial(n-1)], we
			redefine j(n) to be equal to [j(n-1)+serial(n-1)] and continue for
			all remaining n (i.e., n = 3, 4, .. N). This models the case where
			the packet n is sent immediately after packet (n-1) at the bottleneck
			link rate. Although this is generally the theoretical minimum in
			that it assumes that no other packets from other flows are in-between
			packet n and n+1 at the bottleneck link, it is a reasonable
			assumption for per flow queuing.
			We note that this assumption holds for some important exception
			cases, such as packets immediately following outliers. There are a
			multitude of software controlled elements common on end-to-end
			Internet paths (such as firewalls, ALGs and other middleboxes) which
			stop processing packets while servicing other functions (e.g.,
			garbage collection). Often these devices do not drop packets, but
			rather queue them for later processing and cause many of the
			outliers. Thus NR-RPVD models this particular use case (assuming
			serial(n+1) is defined appropriately for the device causing the
			outlier) and thus is believed to be important for adaptation
			development for RMCAT.
			[Editor's Note: It may require to define test distributions as well.
			Example test distribution may include-
			1 - Two-sided: Uniform PDV Distribution. Two quantities to define:
			x_min and x_max.
			2 - Two-sided: Truncated Gaussian PDV Distribution. Four quantities
			to define: the appropriate x_min and x_max for test (e.g., +/- two
			sigma values), the standard deviation and the mean.
			3 - One Sided: TBD]
			6. Traffic Models
			6.1. TCP taffic model
			Long-lived TCP flows will download data throughout the session and
			are expected to have infinite amount of data to send or receive.
			Each short TCP flow is modeled as a sequence of file downloads
			interleaved with idle periods. Not all short TCPs start at the same
			time, i.e., some start in the ON state while others start in the OFF
			state.
			The short TCP flows can be modelled in two ways, 1) 100s of flows
			fetching small (5-20 KB) amounts of data, or 2) 10s of flows fetching
			slightly larger (100-1000KB) amounts of data.
			The idle period is typically derived from an exponential distribution
			with the mean value of 10 seconds.
			[Open issue: short-lived/bursty TCP cross-traffic parameters are
			still to be agreed upon].
			6.2. RTP Video model
			[I-D.zhu-rmcat-video-traffic-source] describes two types of video
			traffic models for evaluating RMCAT candidate algorithms. The first
			model statistically characterizes the behavior of a video encoder.
			Whereas the second model uses video traces.
	7. Security Considerations		7. Security Considerations
	Security issues have not been discussed in this memo.		Security issues have not been discussed in this memo.
	8. IANA Considerations		8. IANA Considerations
	There are no IANA impacts in this memo.		There are no IANA impacts in this memo.
	9. Contributors		9. Contributors
	The content and concepts within this document are a product of the		The content and concepts within this document are a product of the
	discussion carried out in the Design Team.		discussion carried out in the Design Team.
	Michael Ramalho provided the text for a specific scenario, which is		Michael Ramalho provided the text for the Jitter model.
	now covered in [I-D.sarker-rmcat-eval-test].
	10. Acknowledgements		10. Acknowledgements
	Much of this document is derived from previous work on congestion		Much of this document is derived from previous work on congestion
	control at the IETF.		control at the IETF.
	The authors would like to thank Harald Alvestrand, Anna Brunstrom,		The authors would like to thank Harald Alvestrand, Anna Brunstrom,
	Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde,		Luca De Cicco, Wesley Eddy, Lars Eggert, Kevin Gross, Vinayak Hegde,
	Stefan Holmer, Randell Jesup, Karen Nielsen, Piers O'Hanlon, Colin		Stefan Holmer, Randell Jesup, Mirja Kuehlewind, Karen Nielsen, Piers
	Perkins, Michael Ramalho, Zaheduzzaman Sarker, Timothy B. Terriberry,		O'Hanlon, Colin Perkins, Michael Ramalho, Zaheduzzaman Sarker,
	Michael Welzl, and Mo Zanaty for providing valuable feedback on		Timothy B. Terriberry, Michael Welzl, and Mo Zanaty for providing
	earlier versions of this draft. Additionally, also thank the		valuable feedback on earlier versions of this draft. Additionally,
	participants of the design team for their comments and discussion		also thank the participants of the design team for their comments and
	related to the evaluation criteria.		discussion related to the evaluation criteria.
	11. References		11. References
	11.1. Normative References		11.1. Normative References
	[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.		[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
	Jacobson, "RTP: A Transport Protocol for Real-Time		Jacobson, "RTP: A Transport Protocol for Real-Time
	Applications", STD 64, RFC 3550, July 2003.		Applications", STD 64, RFC 3550, July 2003.
	[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and		[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
	skipping to change at page 9, line 19		skipping to change at page 13, line 19
	[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,		[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,
	"Extended RTP Profile for Real-time Transport Control		"Extended RTP Profile for Real-time Transport Control
	Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July		Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July
	2006.		2006.
	[RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size		[RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size
	Real-Time Transport Control Protocol (RTCP): Opportunities		Real-Time Transport Control Protocol (RTCP): Opportunities
	and Consequences", RFC 5506, April 2009.		and Consequences", RFC 5506, April 2009.
	[I-D.ietf-rmcat-cc-requirements]		[I-D.ietf-rmcat-cc-requirements]
	Jesup, R., "Congestion Control Requirements For RMCAT",		Jesup, R. and Z. Sarker, "Congestion Control Requirements
	draft-ietf-rmcat-cc-requirements-02 (work in progress),		for Interactive Real-Time Media", draft-ietf-rmcat-cc-
	February 2014.		requirements-09 (work in progress), December 2014.
	[I-D.ietf-avtcore-rtp-circuit-breakers]		[I-D.ietf-avtcore-rtp-circuit-breakers]
	Perkins, C. and V. Singh, "Multimedia Congestion Control:		Perkins, C. and V. Singh, "Multimedia Congestion Control:
	Circuit Breakers for Unicast RTP Sessions", draft-ietf-		Circuit Breakers for Unicast RTP Sessions", draft-ietf-
	avtcore-rtp-circuit-breakers-05 (work in progress),		avtcore-rtp-circuit-breakers-09 (work in progress), March
	February 2014.		2015.
	11.2. Informative References		11.2. Informative References
	[RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion		[RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion
	Control Algorithms", BCP 133, RFC 5033, August 2007.		Control Algorithms", BCP 133, RFC 5033, August 2007.
	[RFC5166] Floyd, S., "Metrics for the Evaluation of Congestion		[RFC5166] Floyd, S., "Metrics for the Evaluation of Congestion
	Control Mechanisms", RFC 5166, March 2008.		Control Mechanisms", RFC 5166, March 2008.
	[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion		[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
	Control", RFC 5681, September 2009.		Control", RFC 5681, September 2009.
	[I-D.sarker-rmcat-eval-test]		[I-D.ietf-rmcat-eval-test]
	Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test		Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test
	Cases for Evaluating RMCAT Proposals", draft-sarker-rmcat-		Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat-
	eval-test-00 (work in progress), February 2014.		eval-test-00 (work in progress), August 2014.
			[I-D.zhu-rmcat-video-traffic-source]
			Zhu, X., Cruz, S., and Z. Sarker, "Modeling Video Traffic
			Sources for RMCAT Evaluations", draft-zhu-rmcat-video-
			traffic-source-00 (work in progress), October 2014.
	[SA4-EVAL]		[SA4-EVAL]
	R1-081955, 3GPP., "LTE Link Level Throughput Data for SA4		R1-081955, 3GPP., "LTE Link Level Throughput Data for SA4
	Evaluation Framework", 3GPP R1-081955, 5 2008.		Evaluation Framework", 3GPP R1-081955, 5 2008.
	[SA4-LR] S4-050560, 3GPP., "Error Patterns for MBMS Streaming over		[SA4-LR] S4-050560, 3GPP., "Error Patterns for MBMS Streaming over
	UTRAN and GERAN", 3GPP S4-050560, 5 2008.		UTRAN and GERAN", 3GPP S4-050560, 5 2008.
	[TCP-eval-suite]		[TCP-eval-suite]
	Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier,		Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier,
	skipping to change at page 10, line 32		skipping to change at page 14, line 35
	interactive video by performing analysis using a real codec and video		interactive video by performing analysis using a real codec and video
	sequences.		sequences.
	Appendix B. Change Log		Appendix B. Change Log
	Note to the RFC-Editor: please remove this section prior to		Note to the RFC-Editor: please remove this section prior to
	publication as an RFC.		publication as an RFC.
	B.1. Changes in draft-ietf-rmcat-eval-criteria-02		B.1. Changes in draft-ietf-rmcat-eval-criteria-02
			o Keep-alive version.
			o Moved link parameters and traffic models from eval-test
			B.2. Changes in draft-ietf-rmcat-eval-criteria-02
	o Incorporated fairness test as a working test.		o Incorporated fairness test as a working test.
	o Updated text on mimimum evaluation requirements.		o Updated text on mimimum evaluation requirements.
	B.2. Changes in draft-ietf-rmcat-eval-criteria-01		B.3. Changes in draft-ietf-rmcat-eval-criteria-01
	o Removed Appendix B.		o Removed Appendix B.
	o Removed Section on Evaluation Parameters.		o Removed Section on Evaluation Parameters.
	B.3. Changes in draft-ietf-rmcat-eval-criteria-00		B.4. Changes in draft-ietf-rmcat-eval-criteria-00
	o Updated references.		o Updated references.
	o Resubmitted as WG draft.		o Resubmitted as WG draft.
	B.4. Changes in draft-singh-rmcat-cc-eval-04		B.5. Changes in draft-singh-rmcat-cc-eval-04
	o Incorporate feedback from IETF 87, Berlin.		o Incorporate feedback from IETF 87, Berlin.
	o Clarified metrics: convergence time, bandwidth utilization.		o Clarified metrics: convergence time, bandwidth utilization.
	o Changed fairness criteria to fairness test.		o Changed fairness criteria to fairness test.
	o Added measuring pre- and post-repair loss.		o Added measuring pre- and post-repair loss.
	o Added open issue of measuring video quality to appendix.		o Added open issue of measuring video quality to appendix.
	o clarified use of DropTail and AQM.		o clarified use of DropTail and AQM.
	o Updated text in "Minimum Requirements for Evaluation"		o Updated text in "Minimum Requirements for Evaluation"
	B.5. Changes in draft-singh-rmcat-cc-eval-03		B.6. Changes in draft-singh-rmcat-cc-eval-03
	o Incorporate the discussion within the design team.		o Incorporate the discussion within the design team.
	o Added a section on evaluation parameters, it describes the flow		o Added a section on evaluation parameters, it describes the flow
	and network characteristics.		and network characteristics.
	o Added Appendix with self-fairness experiment.		o Added Appendix with self-fairness experiment.
	o Changed bottleneck parameters from a proposal to an example set.		o Changed bottleneck parameters from a proposal to an example set.
	o		o
	B.6. Changes in draft-singh-rmcat-cc-eval-02		B.7. Changes in draft-singh-rmcat-cc-eval-02
	o Added scenario descriptions.		o Added scenario descriptions.
	B.7. Changes in draft-singh-rmcat-cc-eval-01		B.8. Changes in draft-singh-rmcat-cc-eval-01
	o Removed QoE metrics.		o Removed QoE metrics.
	o Changed stability to steady-state.		o Changed stability to steady-state.
	o Added measuring impact against few and many flows.		o Added measuring impact against few and many flows.
	o Added guideline for idle and data-limited periods.		o Added guideline for idle and data-limited periods.
	o Added reference to TCP evaluation suite in example evaluation		o Added reference to TCP evaluation suite in example evaluation
End of changes. 37 change blocks.
	116 lines changed or deleted		312 lines changed or added
This html diff was produced by rfcdiff 1.42. The latest version is available from http://tools.ietf.org/tools/rfcdiff/