draft-ietf-rmcat-wireless-tests-07.txt   draft-ietf-rmcat-wireless-tests-08.txt 
Network Working Group Z. Sarker Network Working Group Z. Sarker
Internet-Draft I. Johansson Internet-Draft I. Johansson
Intended status: Informational Ericsson AB Intended status: Informational Ericsson AB
Expires: January 2, 2020 X. Zhu Expires: January 6, 2020 X. Zhu
J. Fu J. Fu
W. Tan W. Tan
M. Ramalho M. Ramalho
Cisco Systems Cisco Systems
July 1, 2019 July 5, 2019
Evaluation Test Cases for Interactive Real-Time Media over Wireless Evaluation Test Cases for Interactive Real-Time Media over Wireless
Networks Networks
draft-ietf-rmcat-wireless-tests-07 draft-ietf-rmcat-wireless-tests-08
Abstract Abstract
The Real-time Transport Protocol (RTP) is used for interactive The Real-time Transport Protocol (RTP) is a common transport choice
multimedia communication applications. A congestion control for interactive multimedia communication applications. The
algorithm is typically required by these applications. To ensure performance of such applications typically depends on a well-
seamless and robust user experience, a well-designed RTP-based functioning congestion control algorithm. To ensure seamless and
congestion control algorithm should work well across all access robust user experience, a well-designed RTP-based congestion control
network types. This document describes test cases for evaluating algorithm should work well across all access network types. This
performances of such congestion control algorithms over LTE and Wi-Fi document describes test cases for evaluating performances of such
networks. congestion control algorithms over LTE and Wi-Fi networks.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 2, 2020. This Internet-Draft will expire on January 6, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 23 skipping to change at page 2, line 23
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminologies . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminologies . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Cellular Network Specific Test Cases . . . . . . . . . . . . 3 3. Cellular Network Specific Test Cases . . . . . . . . . . . . 3
3.1. Varying Network Load . . . . . . . . . . . . . . . . . . 6 3.1. Varying Network Load . . . . . . . . . . . . . . . . . . 6
3.1.1. Network Connection . . . . . . . . . . . . . . . . . 6 3.1.1. Network Connection . . . . . . . . . . . . . . . . . 6
3.1.2. Simulation Setup . . . . . . . . . . . . . . . . . . 7 3.1.2. Simulation Setup . . . . . . . . . . . . . . . . . . 7
3.2. Bad Radio Coverage . . . . . . . . . . . . . . . . . . . 8 3.2. Bad Radio Coverage . . . . . . . . . . . . . . . . . . . 9
3.2.1. Network connection . . . . . . . . . . . . . . . . . 9 3.2.1. Network connection . . . . . . . . . . . . . . . . . 9
3.2.2. Simulation Setup . . . . . . . . . . . . . . . . . . 9 3.2.2. Simulation Setup . . . . . . . . . . . . . . . . . . 9
3.3. Desired Evaluation Metrics for cellular test cases . . . 10 3.3. Desired Evaluation Metrics for cellular test cases . . . 10
4. Wi-Fi Networks Specific Test Cases . . . . . . . . . . . . . 10 4. Wi-Fi Networks Specific Test Cases . . . . . . . . . . . . . 10
4.1. Bottleneck in Wired Network . . . . . . . . . . . . . . . 12 4.1. Bottleneck in Wired Network . . . . . . . . . . . . . . . 12
4.1.1. Network topology . . . . . . . . . . . . . . . . . . 12 4.1.1. Network topology . . . . . . . . . . . . . . . . . . 12
4.1.2. Test setup . . . . . . . . . . . . . . . . . . . . . 13 4.1.2. Test setup . . . . . . . . . . . . . . . . . . . . . 13
4.1.3. Typical test scenarios . . . . . . . . . . . . . . . 14 4.1.3. Typical test scenarios . . . . . . . . . . . . . . . 14
4.1.4. Expected behavior . . . . . . . . . . . . . . . . . . 15 4.1.4. Expected behavior . . . . . . . . . . . . . . . . . . 15
4.2. Bottleneck in Wi-Fi Network . . . . . . . . . . . . . . . 15 4.2. Bottleneck in Wi-Fi Network . . . . . . . . . . . . . . . 15
4.2.1. Network topology . . . . . . . . . . . . . . . . . . 15 4.2.1. Network topology . . . . . . . . . . . . . . . . . . 15
4.2.2. Test setup . . . . . . . . . . . . . . . . . . . . . 15 4.2.2. Test setup . . . . . . . . . . . . . . . . . . . . . 15
4.2.3. Typical test scenarios . . . . . . . . . . . . . . . 16 4.2.3. Typical test scenarios . . . . . . . . . . . . . . . 17
4.2.4. Expected behavior . . . . . . . . . . . . . . . . . . 18 4.2.4. Expected behavior . . . . . . . . . . . . . . . . . . 18
4.3. Other Potential Test Cases . . . . . . . . . . . . . . . 19 4.3. Other Potential Test Cases . . . . . . . . . . . . . . . 19
4.3.1. EDCA/WMM usage . . . . . . . . . . . . . . . . . . . 19 4.3.1. EDCA/WMM usage . . . . . . . . . . . . . . . . . . . 19
4.3.2. Effects of Legacy 802.11b Devices . . . . . . . . . . 19 4.3.2. Effects of Legacy 802.11b Devices . . . . . . . . . . 19
5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 19 5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 19
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20
7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20
8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 20 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 20
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 20
9.1. Normative References . . . . . . . . . . . . . . . . . . 20 9.1. Normative References . . . . . . . . . . . . . . . . . . 20
9.2. Informative References . . . . . . . . . . . . . . . . . 21 9.2. Informative References . . . . . . . . . . . . . . . . . 21
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22
1. Introduction 1. Introduction
Wireless networks (both cellular and Wi-Fi [IEEE802.11]) are an Wireless networks (both cellular and Wi-Fi [IEEE802.11]) are an
integral part of the Internet. Mobile devices connected to the integral part of the Internet. Mobile devices connected to the
wireless networks account for an increasingly more significant wireless networks account for an increasingly more significant
portion of the media traffic over the Internet. Application portion of the media traffic over the Internet. Application
scenarios range from video conferencing calls in a bus or train to scenarios range from video conferencing calls in a bus or train to
media consumption by someone sitting on a living room couch. It is media consumption by someone on a living room couch. It is well
well known that the characteristics and technical challenges for known that the characteristics and technical challenges for
supporting multimedia services over wireless are very different from supporting multimedia services over wireless are very different from
those of providing the same service over a wired network. Even those of providing the same service over a wired network. Even
though basic test cases for evaluating RTP-based congestion control though basic test cases for evaluating RTP-based congestion control
schemes as defined in [I-D.ietf-rmcat-eval-test] have covered many schemes as defined in [I-D.ietf-rmcat-eval-test] have covered many
effects of the impairments common to both wired and wireless effects of the impairments common to both wired and wireless
networks, there remain characteristics and dynamics unique to a given networks, there remain characteristics and dynamics unique to a given
wireless environment. For example, in LTE networks, the base station wireless environment. For example, in LTE networks, the base station
maintains individual queues per radio bearer per user hence it leads maintains individual queues per radio bearer per user hence it leads
to a different nature of interaction between traffic flows of to a different nature of interactions between traffic flows of
different users. This contrasts with wired network, where traffic different users. This contrasts with wired networks, where traffic
from all users share the same queue. Furthermore, user mobility flows from all users share the same queue. Furthermore, user
patterns in a cellular network differs from those in a Wi-Fi network. mobility patterns in a cellular network differ from those in a Wi-Fi
Therefore, it is important to evaluate the performance of proposed network. Therefore, it is important to evaluate the performance of
candidate RTP-based congestion control solutions over cellular mobile proposed candidate RTP-based congestion control solutions over
networks and over Wi-Fi networks respectively. cellular mobile networks and over Wi-Fi networks respectively.
RMCAT evaluation criteria [I-D.ietf-rmcat-eval-criteria] document RMCAT evaluation criteria document [I-D.ietf-rmcat-eval-criteria]
provides the guideline for evaluating candidate algorithms and provides the guideline for evaluating candidate algorithms and
recognizes the importance of testing over wireless access networks. recognizes the importance of testing over wireless access networks.
However, it does not describe any specific test cases for performance However, it does not describe any specific test cases for performance
evaluation of candidate algorithms. This document describes test evaluation of candidate algorithms. This document describes test
cases specifically targeting cellular networks such as LTE networks cases specifically targeting cellular networks such as LTE networks
and Wi-Fi networks. and Wi-Fi networks.
2. Terminologies 2. Terminologies
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP "OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
3. Cellular Network Specific Test Cases 3. Cellular Network Specific Test Cases
A cellular environment is more complicated than a wireline ditto A cellular environment is more complicated than its wireline
since it seeks to provide services in the context of variable counterpart since it seeks to provide services in the context of
available bandwidth, location dependencies and user mobilities at variable available bandwidth, location dependencies and user
different speeds. In a cellular network the user may reach the cell mobilities at different speeds. In a cellular network, the user may
edge which may lead to a significant amount of retransmissions to reach the cell edge which may lead to a significant amount of
deliver the data from the base station to the destination and vice retransmissions to deliver the data from the base station to the
versa. These network links or radio links will often act as a destination and vice versa. These network links or radio links will
bottleneck for the rest of the network which will eventually lead to often act as a bottleneck for the rest of the network and will
excessive delays or packet drops. An efficient retransmission or eventually lead to excessive delays or packet drops. An efficient
link adaptation mechanism can reduce the packet loss probability but retransmission or link adaptation mechanism can reduce the packet
there will still be some packet losses and delay variations. loss probability but there will still be some packet losses and delay
Moreover, with increased cell load or handover to a congested cell, variations. Moreover, with increased cell load or handover to a
congestion in transport network will become even worse. Besides, congested cell, congestion in the transport network will become even
there are certain characteristics which make the cellular network worse. Besides, there are certain characteristics which make the
different from and more challenging than other types of access cellular network different from and more challenging than other types
networks such as Wi-Fi and wired network. In a cellular network - of access networks such as Wi-Fi and wired network. In a cellular
network --
o The bottleneck is often a shared link with relatively few users. o The bottleneck is often a shared link with relatively few users.
* The cost per bit over the shared link varies over time and is * The cost per bit over the shared link varies over time and is
different for different users. different for different users.
* Left over/ unused resource can be grabbed by other greedy * Leftover/unused resource can be consumed by other greedy users.
users.
o Queues are always per radio bearer hence each user can have many o Queues are always per radio bearer hence each user can have many
of such queues. of such queues.
o Users can experience both Inter and Intra Radio Access Technology o Users can experience both Inter and Intra Radio Access Technology
(RAT) handovers ("handover" definition in [HO-def-3GPP] ). (RAT) handovers (see [HO-def-3GPP] for the definition of
"handover").
o Handover between cells, or change of serving cells (see in o Handover between cells or change of serving cells (as described in
[HO-LTE-3GPP] and [HO-UMTS-3GPP] ) might cause user plane [HO-LTE-3GPP] and [HO-UMTS-3GPP]) might cause user plane
interruptions which can lead to bursts of packet losses, delay interruptions which can lead to bursts of packet losses, delay
and/or jitter. The exact behavior depends on the type of radio and/or jitter. The exact behavior depends on the type of radio
bearer. Typically, the default best effort bearers do not bearer. Typically, the default best-effort bearers do not
generate packet loss, instead packets are queued up and generate packet loss, instead, packets are queued up and
transmitted once the handover is completed. transmitted once the handover is completed.
o The network part decides how much the user can transmit. o The network part decides how much the user can transmit.
o The cellular network has variable link capacity per user o The cellular network has variable link capacity per user
* Can vary as fast as a period of milliseconds. * Can vary as fast as a period of milliseconds.
* Depends on lots of facts (such as distance, speed, * Depends on many factors (such as distance, speed, interference,
interference, different flows). different flows).
* Uses complex and smart link adaptation which makes the link * Uses complex and smart link adaptation which makes the link
behavior ever more dynamic. behavior ever more dynamic.
* The scheduling priority depends on the estimated throughput. * The scheduling priority depends on the estimated throughput.
o Both Quality of Service (QoS) and non-QoS radio bearers can be o Both Quality of Service (QoS) and non-QoS radio bearers can be
used. used.
Hence, a real-time communication application operating in such a Hence, a real-time communication application operating in such a
cellular network need to cope with shared bottleneck link and cellular network needs to cope with a shared bottleneck link and
variable link capacity, event likes handover, non-congestion related variable link capacity, events like handover, non-congestion related
loss, abrupt change in bandwidth (both short term and long term) due loss, abrupt changes in bandwidth (both short term and long term) due
to handover, network load and bad radio coverage. Even though 3GPP to handover, network load and bad radio coverage. Even though 3GPP
define QoS bearers [QoS-3GPP] to ensure high quality user experience, define QoS bearers [QoS-3GPP] to ensure high-quality user experience,
adaptive real-time applications are desired. adaptive real-time applications are desired.
Different mobile operators deploy their own cellular network with Different mobile operators deploy their own cellular network with
their own set of network functionalities and policies. Usually, a their own set of network functionalities and policies. Usually, a
mobile operator network includes 2G, EDGE, 3G and 4G radio access mobile operator network includes 2G, EDGE, 3G and 4G radio access
technologies. Looking at the specifications of such radio technologies. Looking at the specifications of such radio
technologies it is evident that only 3G and 4G radio technologies can technologies it is evident that only 3G and 4G radio technologies can
support the high bandwidth requirements from real-time interactive support the high bandwidth requirements from real-time interactive
video applications. The future real-time interactive application video applications. The future real-time interactive application
will impose even greater demand on cellular network performance which will impose even greater demand on cellular network performance which
skipping to change at page 5, line 35 skipping to change at page 5, line 35
technology for such genre of application. technology for such genre of application.
The key factors to define test cases for cellular networks are The key factors to define test cases for cellular networks are
o Shared and varying link capacity o Shared and varying link capacity
o Mobility o Mobility
o Handover o Handover
However, for cellular network it is very hard to separate such events However, for cellular networks, it is very hard to separate such
from one another as these events are heavily related. Hence instead events from one another as these events are heavily related. Hence
of devising separate test cases for all those important events we instead of devising separate test cases for all those important
have divided the test case in two categories. It should be noted events, we have divided the test case into two categories. It should
that in the following test cases the goal is to evaluate the be noted that the goal of the following test cases is to evaluate the
performance of candidate algorithms over radio interface of the performance of candidate algorithms over the radio interface of the
cellular network. Hence it is assumed that the radio interface is cellular network. Hence it is assumed that the radio interface is
the bottleneck link between the communicating peers and that the core the bottleneck link between the communicating peers and that the core
network does not add any extra congestion in the path. Also the network does not add any extra congestion in the path. Also, the
combination of multiple access technologies such as one user has LTE combination of multiple access technologies such as one user has LTE
connection and another has Wi-Fi connection is kept out of the scope connection and another has Wi-Fi connection is kept out of the scope
of this document. However, later those additional scenarios can also of this document. However, later those additional scenarios can also
be added in this list of test cases. While defining the test cases be added in this list of test cases. While defining the test cases
we assumed a typical real-time telephony scenario over cellular we assumed a typical real-time telephony scenario over cellular
networks where one real-time session consists of one voice stream and networks where one real-time session consists of one voice stream and
one video stream. We recommend that an LTE network simulator is used one video stream.
for the test cases defined in this document, for example-NS-3 LTE
simulator [LTE-simulator]. Even though it is possible to carry out tests over operational LTE
(and 5G) networks, and actually such tests are already available
today, these tests cannot in the general case be carried out in a
deterministic fashion or to ensure repeatability. The main reason is
that these networks are in the control of cellular operators and
there exist various amounts of competing traffic in the same cell(s).
In practice, it is only in underground mines that one can carry out
near deterministic testing. Even there, it is not guaranteed either
as workers in the mines may carry with them their personal mobile
phones. Furthermore, the underground mining setting may not reflect
typical usage patterns in an urban setting. We, therefore, recommend
that an LTE network simulator is used for the test cases defined in
this document, for example --- NS-3 LTE simulator [LTE-simulator].
3.1. Varying Network Load 3.1. Varying Network Load
The goal of this test is to evaluate the performance of the candidate The goal of this test is to evaluate the performance of the candidate
congestion control algorithm under varying network load. The network congestion control algorithm under varying network load. The network
load variation is created by adding and removing network users a.k.a. load variation is created by adding and removing network users a.k.a.
User Equipments (UEs) during the simulation. In this test case, each User Equipments (UEs) during the simulation. In this test case, each
of the user/UE in the media session is an RMCAT compliant endpoint. of the user/UE in the media session is an RMCAT compliant endpoint.
The arrival of users follows a Poisson distribution, which is The arrival of users follows a Poisson distribution proportional to
proportional to the length of the call, so that the number of users the length of the call so as to keep the number of users per cell
per cell is kept fairly constant during the evaluation period. At fairly constant during the evaluation period. At the beginning of
the beginning of the simulation there should be enough time to warm- the simulation, there should be enough time to warm-up the network.
up the network. This is to avoid running the evaluation in an empty This is to avoid running the evaluation in an empty network where
network where network nodes are having empty buffers, low network nodes are having empty buffers, low interference at the
interference at the beginning of the simulation. This network beginning of the simulation. This network initialization period is
initialization period is therefore excluded from the evaluation therefore excluded from the evaluation period.
period.
This test case also includes user mobility and competing traffic. This test case also includes user mobility and some competing
The competing traffic includes both same kind of flows (with same traffic. The latter includes both same kind of flows (with same
adaptation algorithms) and different kind of flows (with different adaptation algorithms) and different kind of flows (with different
service and congestion control). The investigated congestion control services and congestion control schemes). The investigated
algorithms should show maximum possible network utilization and congestion control algorithms should show maximum possible network
stability in terms of rate variations, lowest possible end to end utilization and stability in terms of rate variations, lowest
frame latency, network latency and Packet Loss Rate (PLR) at possible end to end frame latency, network latency and Packet Loss
different cell load level. Rate (PLR) at different cell load level.
3.1.1. Network Connection 3.1.1. Network Connection
Each mobile user is connected to a fixed user. The connection Each mobile user is connected to a fixed user. The connection
between the mobile user and fixed user consists of a LTE radio between the mobile user and fixed user consists of an LTE radio
access, an Evolved Packet Core (EPC) and an Internet connection. The access, an Evolved Packet Core (EPC) and an Internet connection. The
mobile user is connected to the EPC using LTE radio access technology mobile user is connected to the EPC using LTE radio access technology
which is further connected to the Internet. The fixed user is which is further connected to the Internet. The fixed user is
connected to the Internet via wired connection with sufficiently high connected to the Internet via wired connection with sufficiently high
bandwidth, for instance, 10 Gbps, so that the system is resource- bandwidth, for instance, 10 Gbps, so that the system is resource-
limited on the wireless interface. The Internet and wired connection limited on the wireless interface. The wired connection to the
in this setup does not introduce any network impairments to the test; Internet in this setup does not introduce any network impairments to
it only adds 10ms of one-way propagation delay. the test; it only adds 10 ms of one-way propagation delay.
The path from the fixed user to mobile user is defines as "Downlink" The path from the fixed user to the mobile users is defined as
and the path from mobile user to the fixed user is defined as "Downlink" and the path from the mobile users to the fixed user is
"Uplink". We assume that only uplink or downlink is congested for defined as "Uplink". We assume that only uplink or downlink is
the mobile users. Hence, we recommend that the uplink and downlink congested for mobile users. Hence, we recommend that the uplink and
simulations are run separately. downlink simulations are run separately.
uplink uplink
++))) +--------------------------> ++))) +-------------------------->
++-+ ((o)) ++-+ ((o))
| | / \ +-------+ +------+ +---+ | | / \ +-------+ +------+ +---+
+--+ / \----+ +-----+ +----+ | +--+ / \----+ +-----+ +----+ |
/ \ +-------+ +------+ +---+ / \ +-------+ +------+ +---+
UE BS EPC Internet fixed UE BS EPC Internet fixed
<--------------------------+ <--------------------------+
downlink downlink
Figure 1: Simulation Topology Figure 1: Simulation Topology
3.1.2. Simulation Setup 3.1.2. Simulation Setup
The values enclosed within " [ ] " for the following simulation The values enclosed within "[ ]" for the following simulation
attributes follow the notion set in [I-D.ietf-rmcat-eval-test]. The attributes follow the same notion as in [I-D.ietf-rmcat-eval-test].
desired simulation setup as follows- The desired simulation setup is as follows --
1. Radio environment 1. Radio environment:
A. Deployment and propagation model : 3GPP case 1[Deployment] A. Deployment and propagation model: 3GPP case 1 [Deployment]
B. Antenna: Multiple-Input and Multiple-Output (MIMO), [2D, 3D] B. Antenna: Multiple-Input and Multiple-Output (MIMO), [2D, 3D]
C. Mobility: [3km/h, 30km/h] C. Mobility: [3km/h, 30km/h]
D. Transmission bandwidth: 10Mhz D. Transmission bandwidth: 10Mhz
E. Number of cells: multi cell deployment (3 Cells per Base E. Number of cells: multi-cell deployment (3 Cells per Base
Station (BS) * 7 BS) = 21 cells Station (BS) * 7 BS) = 21 cells
F. Cell radius: 166.666 Meters F. Cell radius: 166.666 Meters
G. Scheduler: Proportional fair with no priority G. Scheduler: Proportional fair with no priority
H. Bearer: Default bearer for all traffic. H. Bearer: Default bearer for all traffic.
I. Active Queue Management (AQM) settings: AQM [on,off] I. Active Queue Management (AQM) settings: AQM [on,off]
2. End to end Round Trip Time (RTT): [ 40, 150] 2. End to end Round Trip Time (RTT): [40, 150]
3. User arrival model: Poisson arrival model 3. User arrival model: Poisson arrival model
4. User intensity: 4. User intensity:
* Downlink user intensity: {0.7, 1.4, 2.1, 2.8, 3.5, 4.2, 4.9, * Downlink user intensity: {0.7, 1.4, 2.1, 2.8, 3.5, 4.2, 4.9,
5.6, 6.3, 7.0, 7.7, 8.4, 9,1, 9.8, 10.5} 5.6, 6.3, 7.0, 7.7, 8.4, 9,1, 9.8, 10.5}
* Uplink user intensity : {0.7, 1.4, 2.1, 2.8, 3.5, 4.2, 4.9, * Uplink user intensity : {0.7, 1.4, 2.1, 2.8, 3.5, 4.2, 4.9,
5.6, 6.3, 7.0} 5.6, 6.3, 7.0}
5. Simulation duration: 91s 5. Simulation duration: 91s
6. Evaluation period : 30s-60s 6. Evaluation period: 30s-60s
7. Media traffic 7. Media traffic:
1. Media type: Video 1. Media type: Video
a. Media direction: [Uplink, Downlink] a. Media direction: [Uplink, Downlink]
b. Number of Media source per user: One (1) b. Number of Media source per user: One (1)
c. Media duration per user: 30s c. Media duration per user: 30s
d. Media source: same as define in section 4.3 of d. Media source: same as defined in Section 4.3 of
[I-D.ietf-rmcat-eval-test] [I-D.ietf-rmcat-eval-test]
2. Media Type : Audio 2. Media Type: Audio
a. Media direction: Uplink and Downlink a. Media direction: Uplink and Downlink
b. Number of Media source per user: One (1) b. Number of Media source per user: One (1)
c. Media duration per user: 30s c. Media duration per user: 30s
d. Media codec: Constant BitRate (CBR) d. Media codec: Constant Bit Rate (CBR)
e. Media bitrate : 20 Kbps e. Media bitrate: 20 Kbps
f. Adaptation: off f. Adaptation: off
8. Other traffic model: 8. Other traffic models:
* Downlink simulation: Maximum of 4Mbps/cell (web browsing or * Downlink simulation: Maximum of 4Mbps/cell (web browsing or
FTP traffic following default TCP congestion control FTP traffic following default TCP congestion control
[RFC5681]) [RFC5681])
* Unlink simulation: Maximum of 2Mbps/cell (web browsing or FTP * Unlink simulation: Maximum of 2Mbps/cell (web browsing or FTP
traffic following default TCP congestion control [RFC5681]) traffic following default TCP congestion control [RFC5681])
3.2. Bad Radio Coverage 3.2. Bad Radio Coverage
The goal of this test is to evaluate the performance of candidate The goal of this test is to evaluate the performance of candidate
congestion control algorithm when users visit part of the network congestion control algorithm when users visit part of the network
with bad radio coverage. The scenario is created by using larger with bad radio coverage. The scenario is created by using a larger
cell radius than previous test case. In this test case each of the cell radius than that in the previous test case. In this test case,
user/UE in the media session is an RMCAT compliant endpoint. The each of the user/UE in the media session is an RMCAT compliant
arrival of users follows a Poisson distribution, which is endpoint. The arrival of users follows a Poisson distribution
proportional to the length of the call, so that the number of users proportional to the length of the call, so as to keep the number of
per cell is kept fairly constant during the evaluation period. At users per cell fairly constant during the evaluation period. At the
the beginning of the simulation there should be enough amount of time beginning of the simulation, there should be enough amount of time to
to warm-up the network. This is to avoid running the evaluation in warm-up the network. This is to avoid running the evaluation in an
an empty network where network nodes are having empty buffers, low empty network where network nodes are having empty buffers, low
interference at the beginning of the simulation. This network interference at the beginning of the simulation. This network
initialization period is therefore excluded from the evaluation initialization period is therefore excluded from the evaluation
period. period.
This test case also includes user mobility and competing traffic. This test case also includes user mobility and some competing
The competing traffic includes same kind of flows (with same traffic. The latter includes the same kind of flows (with same
adaptation algorithms) . The investigated congestion control adaptation algorithms). The investigated congestion control
algorithms should show maximum possible network utilization and algorithms should result in maximum possible network utilization and
stability in terms of rate variations, lowest possible end to end stability in terms of rate variations, lowest possible end to end
frame latency, network latency and Packet Loss Rate (PLR) at frame latency, network latency and Packet Loss Rate (PLR) at
different cell load level. different cell load levels.
3.2.1. Network connection 3.2.1. Network connection
Same as defined in Section 3.1.1 Same as defined in Section 3.1.1
3.2.2. Simulation Setup 3.2.2. Simulation Setup
The desired simulation setup is same as Varying Network Load test The desired simulation setup is the same as the Varying Network Load
case defined in Section 3.1 except following changes: test case defined in Section 3.1 except the following changes:
1. Radio environment: Same as defined in Section 3.1.2 except the 1. Radio environment: Same as defined in Section 3.1.2 except the
following: following:
A. Deployment and propagation model : 3GPP case 3 [Deployment] A. Deployment and propagation model: 3GPP case 3 [Deployment]
B. Cell radius: 577.3333 Meters B. Cell radius: 577.3333 Meters
C. Mobility: 3km/h C. Mobility: 3km/h
2. User intensity = {0.7, 1.4, 2.1, 2.8, 3.5, 4.2, 4.9, 5.6, 6.3, 2. User intensity = {0.7, 1.4, 2.1, 2.8, 3.5, 4.2, 4.9, 5.6, 6.3,
7.0} 7.0}
3. Media traffic model: Same as defined in Section 3.1.2 3. Media traffic model: Same as defined in Section 3.1.2
4. Other traffic model: 4. Other traffic models:
* Downlink simulation: Maximum of 2Mbps/cell (web browsing or * Downlink simulation: Maximum of 2Mbps/cell (web browsing or
FTP traffic following default TCP congestion control FTP traffic following default TCP congestion control
[RFC5681]) [RFC5681])
* Unlink simulation: Maximum of 1Mbps/cell (web browsing or FTP * Unlink simulation: Maximum of 1Mbps/cell (web browsing or FTP
traffic following default TCP congestion control [RFC5681]) traffic following default TCP congestion control [RFC5681])
3.3. Desired Evaluation Metrics for cellular test cases 3.3. Desired Evaluation Metrics for cellular test cases
RMCAT evaluation criteria document [I-D.ietf-rmcat-eval-criteria] RMCAT evaluation criteria document [I-D.ietf-rmcat-eval-criteria]
defines metrics to be used to evaluate candidate algorithms. defines the metrics to be used to evaluate candidate algorithms.
However, looking at the nature and distinction of cellular networks However, looking at the nature and distinction of cellular networks
we recommend at minimum following metrics to be used to evaluate the we recommend that at least the following metrics be used to evaluate
performance of the candidate algorithms for the test cases defined in the performance of the candidate algorithms for the test cases
this document. defined in this document.
The desired metrics are- The desired metrics are --
o Average cell throughput (for all cells), shows cell utilizations. o Average cell throughput (for all cells), shows cell utilizations.
o Application sending and receiving bitrate, goodput. o Application sending and receiving bitrate, goodput.
o Packet Loss Rate (PLR). o Packet Loss Rate (PLR).
o End to end Media frame delay. For video, this means the delay o End to end Media frame delay. For video, this means the delay
from capture to display. from capture to display.
o Transport delay. o Transport delay.
o Algorithm stability in terms of rate variation. o Algorithm stability in terms of rate variation.
4. Wi-Fi Networks Specific Test Cases 4. Wi-Fi Networks Specific Test Cases
Given the prevalence of Internet access links over Wi-Fi, it is Given the prevalence of Internet access links over Wi-Fi, it is
important to evaluate candidate RMCAT congestion control solutions important to evaluate candidate RMCAT congestion control solutions
over test cases that include Wi-Fi access lines. Such evaluations over test cases that include Wi-Fi access lines. Such evaluations
should also highlight the inherent different characteristics of Wi-Fi should also highlight the inherently different characteristics of Wi-
networks in contrast to wired networks: Fi networks in contrast to wired networks:
o The wireless radio channel is subject to interference from nearby o The wireless radio channel is subject to interference from nearby
transmitters, multipath fading, and shadowing, causing transmitters, multipath fading, and shadowing, causing
fluctuations in link throughput and sometimes an error-prone fluctuations in link throughput and sometimes an error-prone
communication environment communication environment
o Available network bandwidth is not only shared over the air o Available network bandwidth is not only shared over the air
between cocurrent users, but also between uplink and downlink between concurrent users but also between uplink and downlink
traffic due to the half duplex nature of wireless transmission traffic due to the half-duplex nature of wireless transmission
medium. medium.
o Packet transmissions over Wi-Fi are susceptible to contentions and o Packet transmissions over Wi-Fi are susceptible to contentions and
collisions over the air. Consequently, traffic load beyond a collisions over the air. Consequently, traffic load beyond a
certain utilization level over a Wi-Fi network can introduce certain utilization level over a Wi-Fi network can introduce
frequent collisions over the air and significant network overhead, frequent collisions over the air and significant network overhead,
as well as packet drops due to buffer overflow at the as well as packet drops due to buffer overflow at the
transmitters. This, in turn, leads to excessive delay, transmitters. This, in turn, leads to excessive delay,
retransmissions, packet losses and lower effective bandwidth for retransmissions, packet losses and lower effective bandwidth for
applications. Note, however, that the consequent delay and loss applications. Note, however, that the consequent delay and loss
patterns caused by collisions are qualitatively different from patterns caused by collisions are qualitatively different from
those induced by congestion over a wired connection. those induced by congestion over a wired connection.
o The IEEE 802.11 standard (i.e., Wi-Fi) supports multi-rate o The IEEE 802.11 standard (i.e., Wi-Fi) supports multi-rate
transmission capabilities by dynamically choosing the most transmission capabilities by dynamically choosing the most
appropriate modulation scheme for a given received signal appropriate modulation scheme for the given received signal
strength. A different choice of physical-layer rate leads to strength. A different choice of physical-layer rate leads to
different application-layer throughput. different application-layer throughput.
o Presence of legancy 802.11b networks can significantly slow down o Presence of legacy 802.11b networks can significantly slow down
the the rest of a modern Wi-Fi Network. As discussed in the rest of a modern Wi-Fi network. As discussed in [Heusse2003],
[Heusse2003]since it takes longer to transmit the same packet over the main reason for such abnomaly is that it takes longer to
a slower link than over a faster link. transmit the same packet over a slower link than over a faster
link.
o Handover from one Wi-Fi Access Point (AP) to another may lead to o Handover from one Wi-Fi Access Point (AP) to another may lead to
packet delay and losses during the process. packet delay and losses during the process.
o IEEE 802.11e defined EDCA/WMM (Enhanced DCF Channel Access/Wi-Fi o IEEE 802.11e defined EDCA/WMM (Enhanced DCF Channel Access/Wi-Fi
Multi-Media) to give voice and video streams higher priority over Multi-Media) to give voice and video streams higher priority over
pure data applications (e.g., file transfers). pure data applications (e.g., file transfers).
In summary, presence of Wi-Fi access links in different network In summary, the presence of Wi-Fi access links in different network
topologies can exert different impact on the network performance in topologies can exert different impact on the network performance in
terms of application-layer effective throughput, packet loss rate, terms of application-layer effective throughput, packet loss rate,
and packet delivery delay. These, in turn, influence the behavior of and packet delivery delay. These, in turn, influence the behavior of
end-to-end real-time multimedia congestion control. end-to-end real-time multimedia congestion control.
Unless otherwise mentioned, test cases in this section are described Unless otherwise mentioned, test cases in this section are described
using the underlying PHY- and MAC-layer parameters based on the IEEE using the underlying PHY- and MAC-layer parameters based on the IEEE
802.11n Standard. Statistics collected from enterprise Wi-Fi 802.11n Standard. Statistics collected from enterprise Wi-Fi
networks show that the two dominant physical modes are 802.11n and networks show that the two dominant physical modes are 802.11n and
802.11ac, accounting for 41% and 58% of connected devices. As Wi-Fi 802.11ac, accounting for 41% and 58% of connected devices. As Wi-Fi
standards evolve over time, for instance, with the introduction of standards evolve over time -- for instance, with the introduction of
the emerging Wi-Fi 6 (802.11ax) products, the PHY- and MAC-layer test the emerging Wi-Fi 6 (802.11ax) products -- the PHY- and MAC-layer
case specifications need to be updated accordingly to reflect such test case specifications need to be updated accordingly to reflect
changes. such changes.
Typically, a Wi-Fi access network connects to a wired infrastructure. Typically, a Wi-Fi access network connects to a wired infrastructure.
Either the wired or the Wi-Fi segment of the network could be the Either the wired or the Wi-Fi segment of the network could be the
bottleneck. In the following sections, we describe basic test cases bottleneck. In the following sections, we describe basic test cases
for both scenarios separately. The same set of performance metrics for both scenarios separately. The same set of performance metrics
as in [I-D.ietf-rmcat-eval-test]) should be collected for each test as in [I-D.ietf-rmcat-eval-test]) should be collected for each test
case. case.
All test cases described below can be carried out using simulations, All test cases described below can be carried out using simulations,
e.g. based on [ns-2] or [ns-3]. When feasible, it is also encouraged e.g. based on [ns-2] or [ns-3]. When feasible, it is also encouraged
skipping to change at page 12, line 20 skipping to change at page 12, line 34
schemes. schemes.
4.1. Bottleneck in Wired Network 4.1. Bottleneck in Wired Network
The test scenarios below are intended to mimic the setup of video The test scenarios below are intended to mimic the setup of video
conferencing over Wi-Fi connections from the home. Typically, the conferencing over Wi-Fi connections from the home. Typically, the
Wi-Fi home network is not congested and the bottleneck is present Wi-Fi home network is not congested and the bottleneck is present
over the wired home access link. Although it is expected that test over the wired home access link. Although it is expected that test
evaluation results from this section are similar to those from test evaluation results from this section are similar to those from test
cases defined for wired networks (see [I-D.ietf-rmcat-eval-test]), it cases defined for wired networks (see [I-D.ietf-rmcat-eval-test]), it
is worthwhile to run through these tests as sanity checks. is still worthwhile to run through these tests as sanity checks.
4.1.1. Network topology 4.1.1. Network topology
Figure 2 shows topology of the network for Wi-Fi test cases. The Figure 2 shows the network topology of Wi-Fi test cases. The test
test contains multiple mobile nodes (MNs) connected to a common Wi-Fi contains multiple mobile nodes (MNs) connected to a common Wi-Fi
access point (AP) and their corresponding wired clients on fixed access point (AP) and their corresponding wired clients on fixed
nodes (FNs). Each connection carries either RMCAT or TCP traffic nodes (FNs). Each connection carries either a RMCAT or a TCP traffic
flow. Directions of the flows can be uplink, downlink, or bi- flow. Directions of the flows can be uplink, downlink, or bi-
directional. directional.
uplink Uplink
+----------------->+ +----------------->+
+------+ +------+ +------+ +------+
| MN_1 |)))) /=====| FN_1 | | MN_1 |)))) /=====| FN_1 |
+------+ )) // +------+ +------+ )) // +------+
. )) // . . )) // .
. )) // . . )) // .
. )) // . . )) // .
+------+ +----+ +-----+ +------+ +------+ +----+ +-----+ +------+
| MN_N | ))))))) | | | |========| FN_N | | MN_N | ))))))) | | | |========| FN_N |
+------+ | | | | +------+ +------+ | | | | +------+
skipping to change at page 13, line 27 skipping to change at page 13, line 27
+----------+ | | | | +----------+ +----------+ | | | | +----------+
| MN_tcp_1 | )))) | | | |======| MN_tcp_1 | | MN_tcp_1 | )))) | | | |======| MN_tcp_1 |
+----------+ +----+ +-----+ +----------+ +----------+ +----+ +-----+ +----------+
. )) \\ . . )) \\ .
. )) \\ . . )) \\ .
. )) \\ . . )) \\ .
+----------+ )) \\ +----------+ +----------+ )) \\ +----------+
| MN_tcp_M |))) \=====| MN_tcp_M | | MN_tcp_M |))) \=====| MN_tcp_M |
+----------+ +----------+ +----------+ +----------+
+<-----------------+ +<-----------------+
downlink Downlink
Figure 2: Network topology for Wi-Fi test cases Figure 2: Network topology for Wi-Fi test cases
4.1.2. Test setup 4.1.2. Test setup
o Test duration: 120s o Test duration: 120s
o Wi-Fi network characteristics: o Wi-Fi network characteristics:
* Radio propagation model: Log-distance path loss propagation * Radio propagation model: Log-distance path loss propagation
skipping to change at page 15, line 15 skipping to change at page 15, line 15
o One RMCAT flow competing against one long-live TCP flow over o One RMCAT flow competing against one long-live TCP flow over
uplink: N=1 (uplink) and M = 1(uplink), TCP start time at 0s and uplink: N=1 (uplink) and M = 1(uplink), TCP start time at 0s and
end time at 119s. end time at 119s.
4.1.4. Expected behavior 4.1.4. Expected behavior
o Single uplink RMCAT flow: the candidate algorithm is expected to o Single uplink RMCAT flow: the candidate algorithm is expected to
detect the path capacity constraint, to converge to bottleneck detect the path capacity constraint, to converge to bottleneck
link capacity and to adapt the flow to avoid unwanted oscillation link capacity and to adapt the flow to avoid unwanted oscillation
when the sending bit rate is approaching the bottleneck link when the sending bit rate is approaching the bottleneck link
capacity. No excessive rate oscillations should be present. capacity. No excessive oscillations in the media rate should be
present.
o Bi-directional RMCAT flows: It is expected that the candidate o Bi-directional RMCAT flows: It is expected that the candidate
algorithm is able to converge to the bottleneck capacity of the algorithm is able to converge to the bottleneck capacity of the
wired path on both directions despite presence of measurment noise wired path on both directions despite the presence of measurement
over the Wi-Fi connection. In the presence of background TCP or noise over the Wi-Fi connection. In the presence of background
CBR over UDP traffic, the rate of RMCAT flows should adapt in a TCP or CBR over UDP traffic, the rate of RMCAT flows should adapt
timely manner to changes in the available bottleneck bandwidth. in a timely manner to changes in the available bottleneck
bandwidth.
o One RMCAT flow competing with long-live TCP flow over uplink: the o One RMCAT flow competing with long-live TCP flow over uplink: the
candidate algorithm should be able to avoid congestion collapse, candidate algorithm should be able to avoid congestion collapse,
and to stablize at a fair share of the bottleneck link capacity. and to stabilize at a fair share of the bottleneck link capacity.
4.2. Bottleneck in Wi-Fi Network 4.2. Bottleneck in Wi-Fi Network
These test cases assume that the wired portion along the media path These test cases assume that the wired portion along the media path
is well-provisioned whereas the bottleneck exists over the Wi-Fi is well-provisioned whereas the bottleneck exists over the Wi-Fi
access network. This is to mimic the application scenarios typically access network. This is to mimic the application scenarios typically
encountered by users in an enterprise environment or at a coffee encountered by users in an enterprise environment or at a coffee
house. house.
4.2.1. Network topology 4.2.1. Network topology
skipping to change at page 15, line 50 skipping to change at page 16, line 4
4.2.2. Test setup 4.2.2. Test setup
o Test duration: 120s o Test duration: 120s
o Wi-Fi network characteristics: o Wi-Fi network characteristics:
* Radio propagation model: Log-distance path loss propagation * Radio propagation model: Log-distance path loss propagation
model [NS3WiFi] model [NS3WiFi]
* PHY- and MAC-layer configuration: IEEE 802.11n * PHY- and MAC-layer configuration: IEEE 802.11n
* MCS Index at 11: 16-QAM 1/2, Raw Data Rate at 52Mbps * MCS Index at 11: 16-QAM 1/2, Raw Data Rate at 52Mbps
o Wired path characteristics: o Wired path characteristics:
* Path capacity: 100Mbps * Path capacity: 100Mbps.
* One-Way propagation delay: 50ms. * One-Way propagation delay: 50ms.
* Maximum end-to-end jitter: 30ms * Maximum end-to-end jitter: 30ms.
* Bottleneck queue type: Drop tail. * Bottleneck queue type: Drop tail.
* Bottleneck queue size: 300ms. * Bottleneck queue size: 300ms.
* Path loss ratio: 0%. * Path loss ratio: 0%.
o Application characteristics: o Application characteristics:
* Media Traffic: * Media Traffic:
+ Media type: Video + Media type: Video
+ Media direction: See Section 4.2.3 + Media direction: See Section 4.2.3.
+ Number of media sources (N): See Section 4.2.3 + Number of media sources (N): See Section 4.2.3.
+ Media timeline: + Media timeline:
- Start time: 0s. - Start time: 0s.
- End time: 119s. - End time: 119s.
* Competing traffic: * Competing traffic:
+ Type of sources: long-lived TCP or CBR over UDP + Type of sources: long-lived TCP or CBR over UDP.
+ Number of sources (M): See Section 4.2.3 + Number of sources (M): See Section 4.2.3.
+ Traffic direction: See Section 4.2.3 + Traffic direction: See Section 4.2.3.
+ Congestion control: Default TCP congestion control [RFC5681] + Congestion control: Default TCP congestion control [RFC5681]
or constant-bit-rate (CBR) traffic over UDP or constant-bit-rate (CBR) traffic over UDP.
+ Traffic timeline: See Section 4.2.3 + Traffic timeline: See Section 4.2.3.
4.2.3. Typical test scenarios 4.2.3. Typical test scenarios
This section describes a few test scenarios that are deemed as This section describes a few test scenarios that are deemed as
important for understanding the behavior of a RMCAT candidate important for understanding the behavior of a candidate RMCAT
solution over a Wi-Fi network. solution over a Wi-Fi network.
o Multiple RMCAT Flows Sharing the Wireless Downlink: N=16 (all a. Multiple RMCAT Flows Sharing the Wireless Downlink: N=16 (all
downlink); M = 0. This test case is for studying the impact of downlink); M = 0. This test case is for studying the impact of
contention on competing RMCAT flows. For an 802.11n network, contention on the multiple concurrent RMCAT flows. For an
given the MCS Index of 11 and the corresponding raw data rate of 802.11n network, given the MCS Index of 11 and the corresponding
52Mbps, the total application-layer throughput (assuming raw data rate of 52Mbps, the total application-layer throughput
reasonable distance, low interference and infrequent contentions (assuming reasonable distance, low interference and infrequent
caused by competing streams) is around 20Mbps. Consequently, a contentions caused by competing streams) is around 20Mbps.
total of N=16 RMCAT flows are needed to saturate the wireless Consequently, a total of N=16 RMCAT flows are needed to saturate
interface in this experiment. Evaluation of a given candidate the wireless interface in this experiment. Evaluation of a given
solution should focus on whether downlink RMCAT flows can stablize candidate solution should focus on whether downlink RMCAT flows
at a fair share of total application-layer throughput. can stabilize at a fair share of total application-layer
throughput.
o Multiple RMCAT Flows Sharing the Wireless Uplink: N = 16 (all b. Multiple RMCAT Flows Sharing the Wireless Uplink: N = 16 (all
downlink); M = 0. When multiple clients attempt to transmit video downlink); M = 0. When multiple clients attempt to transmit
packets uplink over the wireless interface, they introduce more video packets uplink over the wireless interface, they introduce
frequent contentions and potential collisions. Per-flow more frequent contentions and potential collisions. Per-flow
throughput is expected to be lower than that in the previous throughput is expected to be lower than that in the previous
downlink-only scenario. Evaluation of a given candidate solution downlink-only scenario. Evaluation of a given candidate solution
should focus on whether uplink flows can stablize at a fair share should focus on whether uplink flows can stabilize at a fair
of application-layer throughput. share of application-layer throughput.
o Multiple Bi-directional RMCAT Flows: N = 16 (8 uplink and 8 c. Multiple Bi-directional RMCAT Flows: N = 16 (8 uplink and 8
downlink); M = 0. The goal of this test is to evaluate downlink); M = 0. The goal of this test is to evaluate the
performance of the candidate solution in terms of bandwidth performance of the candidate solution in terms of bandwidth
fairness between uplink and downlink flows. fairness between uplink and downlink flows.
o Multiple Bi-directional RMCAT Flows with on-off CBR traffic: N = d. Multiple Bi-directional RMCAT Flows with on-off CBR traffic: N =
16 (8 uplink and 8 downlink); M = 5(uplink). The goal of this 16 (8 uplink and 8 downlink); M = 5(uplink). The goal of this
test is to evaluate adaptation behavior of the candidate solution test is to evaluate the adaptation behavior of the candidate
when its available bandwidth changes due to departure of solution when its available bandwidth changes due to the
background traffic. The background traffic consists of several departure of background traffic. The background traffic consists
(e.g., M=5) CBR flows transported over UDP, which are ON at times of several (e.g., M=5) CBR flows transported over UDP. These
t=0-60s and are OFF at times t=61-120s. background flows are ON at times t=0-60s and are OFF at times
t=61-120s.
o Multiple Bi-directional RMCAT Flows with off-on CBR traffic: N = e. Multiple Bi-directional RMCAT Flows with off-on CBR traffic: N =
16 (8 uplink and 8 downlink); M = 5(uplink). The goal of this 16 (8 uplink and 8 downlink); M = 5(uplink). The goal of this
test is to evaluate adaptation behavior of the candidate solution test is to evaluate the adaptation behavior of the candidate
when its available bandwidth changes due to arrival of background solution when its available bandwidth changes due to the arrival
traffic. The background traffic consists of several (e.g., M=5) of background traffic. The background traffic consists of
parallel CBR flows transported over UDP, which are OFF at times several (e.g., M=5) parallel CBR flows transported over UDP.
t=0-60s and are ON at times t=61-120s.
o Multiple Bi-directional RMCAT flows in the presence of background These background flows are OFF at times t=0-60s and are ON at
TCP traffic: N=16 (8 uplink and 8 downlink); M = 5 (uplink). The times t=61-120s.
goal of this test is to evaluate how RMCAT flows compete against
TCP over a congested Wi-Fi network for a given candidate solution.
TCP start time: 40s, end time: 80s.
o Varying number of RMCAT flows. A series of tests can be carried f. Multiple Bi-directional RMCAT flows in the presence of background
out for the above test cases with different values of N, e.g., N = TCP traffic: N=16 (8 uplink and 8 downlink); M = 5 (uplink). The
[4, 8, 12, 16, 20]. The goal of this test is to evaluate how a goal of this test is to evaluate how RMCAT flows compete against
candidate RMCAT solution responds to varying traffic load/demand TCP over a congested Wi-Fi network for a given candidate
over a congested Wi-Fi network. The start time of these RMCAT solution. TCP start time: 40s, end time: 80s.
flows is randomly distributed within a window of t=0-10s, whereas
their end times are randomly distributed within a window of g. Varying number of RMCAT flows: A series of tests can be carried
t=110-120s. out for the above test cases with different values of N, e.g., N
= [4, 8, 12, 16, 20]. The goal of this test is to evaluate how a
candidate RMCAT solution responds to varying traffic load/demand
over a congested Wi-Fi network. The start time of these RMCAT
flows is randomly distributed within a window of t=0-10s, whereas
their end times are randomly distributed within a window of
t=110-120s.
4.2.4. Expected behavior 4.2.4. Expected behavior
o Multiple downlink RMCAT flows: each RMCAT flow should get its fair o Multiple downlink RMCAT flows: each RMCAT flow should get its fair
share of the total bottleneck link bandwidth. Overall bandwidth share of the total bottleneck link bandwidth. Overall bandwidth
usage should not be significantly lower than that experienced by usage should not be significantly lower than that experienced by
the same number of concurrent downlink TCP flows. In other words, the same number of concurrent downlink TCP flows. In other words,
the performance of multiple concurrent TCP flows will be used as a the performance of multiple concurrent TCP flows will be used as a
performance benchmark for this test scenario. The end-to-end performance benchmark for this test scenario. The end-to-end
delay and packet loss ratio experienced by each flow should be delay and packet loss ratio experienced by each flow should be
within acceptable range for real-time multimedia applications. within an acceptable range for real-time multimedia applications.
o Multiple uplink RMCAT flows: overall bandwidth usage shared by all o Multiple uplink RMCAT flows: overall bandwidth usage shared by all
RMCAT flows should not be significantly lower than that RMCAT flows should not be significantly lower than that
experienced by the same number of concurrent uplink TCP flows. In experienced by the same number of concurrent uplink TCP flows. In
other words, the performance of multiple concurrent TCP flows will other words, the performance of multiple concurrent TCP flows will
be used as a performance benchmark for this test scenario. be used as a performance benchmark for this test scenario.
o Multiple bi-directional RMCAT flows with dynamic background o Multiple bi-directional RMCAT flows with dynamic background
traffic carrying CBR flows over UDP: RMCAT flows should adapt in a traffic carrying CBR flows over UDP: RMCAT flows should adapt in a
timely fashion to the resulting changes in available bandwidth. timely fashion to the resulting changes in available bandwidth.
skipping to change at page 18, line 48 skipping to change at page 19, line 8
significantly lower than those achieved by the same number of bi- significantly lower than those achieved by the same number of bi-
directional TCP flows. In other words, the performance of directional TCP flows. In other words, the performance of
multiple concurrent TCP flows will be used as a performance multiple concurrent TCP flows will be used as a performance
benchmark for this test scenario. All downlink RMCAT flows are benchmark for this test scenario. All downlink RMCAT flows are
expected to obtain similar bandwidth with respect to each other. expected to obtain similar bandwidth with respect to each other.
The throughput of RMCAT flows should decrease upon the arrival of The throughput of RMCAT flows should decrease upon the arrival of
TCP background traffic and increase upon their departure, both TCP background traffic and increase upon their departure, both
reactions should occur in a timely fashion (e.g., within 10s of reactions should occur in a timely fashion (e.g., within 10s of
seconds). seconds).
o Varying number of RMCAT flows: the test results for varying values o Varying number of bi-directional RMCAT flows: the test results for
of N -- while keeping all other parameters constant -- is expected varying values of N -- while keeping all other parameters constant
to show steady and stable per-flow throughtput for each value of -- is expected to show steady and stable per-flow throughput for
N. The average throughput of all RMCAT flows is expected to stay each value of N. The average throughput of all RMCAT flows is
constant around the maximum rate when N is small, then gradually expected to stay constant around the maximum rate when N is small,
decrease with increasing number of RMCAT flows till it reaches the then gradually decrease with increasing number of RMCAT flows till
minimum allowed rate, beyond which the offered load to the Wi-Fi it reaches the minimum allowed rate, beyond which the offered load
network (with a large value of N) is exceeding its capacity. to the Wi-Fi network (with a large value of N) is exceeding its
capacity.
4.3. Other Potential Test Cases 4.3. Other Potential Test Cases
4.3.1. EDCA/WMM usage 4.3.1. EDCA/WMM usage
EDCA/WMM is prioritized QoS with four traffic classes (or Access EDCA/WMM is prioritized QoS with four traffic classes (or Access
Categories) with differing priorities. RMCAT flows should achieve Categories) with differing priorities. RMCAT flows should achieve
better performance (i.e., lower delay, fewer packet losses) with better performance (i.e., lower delay, fewer packet losses) with
EDCA/WMM enabled when competing against non-interactive background EDCA/WMM enabled when competing against non-interactive background
traffic (e.g., file transfers). When most of the traffic over Wi-Fi traffic (e.g., file transfers). When most of the traffic over Wi-Fi
is dominated by media, however, turning on WMM may actually degrade is dominated by media, however, turning on WMM may actually degrade
performance since all media flows now attempt to access the wireless performance since all media flows now attempt to access the wireless
transmission medium more aggressively, thereby causing more frequent transmission medium more aggressively, thereby causing more frequent
collisions and collision-induced losses. This is a topic worthy of collisions and collision-induced losses. This is a topic worthy of
further investigation. further investigation.
4.3.2. Effects of Legacy 802.11b Devices 4.3.2. Effects of Legacy 802.11b Devices
When there is 802.11b devices connected to modern 802.11 network, it When there exist 802.11b devices connected to a modern 802.11
may affect the performance of the whole network. Additional test network, they may affect the performance of the whole network.
cases can be added to evaluate the affects of legancy devices on the Additional test cases can be added to evaluate the impacts of legacy
performance of RMCAT congestion control algorithm. devices on the performance of the candidate congestion control
algorithm.
5. Conclusion 5. Conclusion
This document defines a collection of test cases that are considered This document defines a collection of test cases that are considered
important for cellular and Wi-Fi networks. Moreover, this document important for cellular and Wi-Fi networks. Moreover, this document
also provides a framework for defining additional test cases over also provides a framework for defining additional test cases over
wireless cellular/Wi-Fi networks. wireless cellular/Wi-Fi networks.
6. IANA Considerations 6. IANA Considerations
This memo includes no request to IANA. This memo includes no request to IANA.
7. Security Considerations 7. Security Considerations
The security considerations in [I-D.ietf-rmcat-eval-criteria] and the The security considerations in [I-D.ietf-rmcat-eval-criteria] and the
relevant congestion control algorithms apply. The principles for relevant congestion control algorithms apply. The principles for
congestion control are described in [RFC2914], and in particular any congestion control are described in [RFC2914], and in particular, any
new method MUST implement safeguards to avoid congestion collapse of new method MUST implement safeguards to avoid congestion collapse of
the Internet. the Internet.
The evaluation of the test cases are intended to be run in a The evaluations of the test cases are intended to carry out in a
controlled lab environment. Hence, the applications, simulators and controlled lab environment. Hence, the applications, simulators and
network nodes ought to be well-behaved and should not impact the network nodes ought to be well-behaved and should not impact the
desired results. It is important to take appropriate caution to desired results. It is important to take appropriate caution to
avoid leaking non-responsive traffic from unproven congestion avoid leaking non-responsive traffic from unproven congestion
avoidance techniques onto the open Internet. avoidance techniques onto the open Internet.
8. Acknowledgments 8. Acknowledgments
We would like to thank Tomas Frankkila, Magnus Westerlund, Kristofer The authors would like to thank Tomas Frankkila, Magnus Westerlund,
Sandlund, and Sergio Mena de la Cruz for their valuable input and Kristofer Sandlund, and Sergio Mena de la Cruz for their valuable
review comments regarding this draft. input and review comments regarding this draft.
9. References 9. References
9.1. Normative References 9.1. Normative References
[Deployment] [Deployment]
TS 25.814, 3GPP., "Physical layer aspects for evolved TS 25.814, 3GPP., "Physical layer aspects for evolved
Universal Terrestrial Radio Access (UTRA)", October 2006, Universal Terrestrial Radio Access (UTRA)", October 2006,
<http://www.3gpp.org/ftp/specs/ <http://www.3gpp.org/ftp/specs/
archive/25_series/25.814/25814-710.zip>. archive/25_series/25.814/25814-710.zip>.
 End of changes. 88 change blocks. 
233 lines changed or deleted 252 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/