--- 1/draft-ietf-rmcat-wireless-tests-10.txt 2020-03-13 12:13:12.622673377 -0700 +++ 2/draft-ietf-rmcat-wireless-tests-11.txt 2020-03-13 12:13:12.674674698 -0700 @@ -1,24 +1,21 @@ Network Working Group Z. Sarker Internet-Draft Ericsson AB Intended status: Informational X. Zhu -Expires: September 10, 2020 J. Fu - W. Tan +Expires: September 14, 2020 J. Fu Cisco Systems - M. Ramalho - AcousticComms - March 9, 2020 + March 13, 2020 Evaluation Test Cases for Interactive Real-Time Media over Wireless Networks - draft-ietf-rmcat-wireless-tests-10 + draft-ietf-rmcat-wireless-tests-11 Abstract The Real-time Transport Protocol (RTP) is a common transport choice for interactive multimedia communication applications. The performance of these applications typically depends on a well- functioning congestion control algorithm. To ensure a seamless and robust user experience, a well-designed RTP-based congestion control algorithm should work well across all access network types. This document describes test cases for evaluating performances of @@ -33,21 +30,21 @@ Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on September 10, 2020. + This Internet-Draft will expire on September 14, 2020. Copyright Notice Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents @@ -57,44 +54,47 @@ the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Cellular Network Specific Test Cases . . . . . . . . . . . . 3 2.1. Varying Network Load . . . . . . . . . . . . . . . . . . 6 2.1.1. Network Connection . . . . . . . . . . . . . . . . . 6 2.1.2. Simulation Setup . . . . . . . . . . . . . . . . . . 7 + 2.1.3. Expected behavior . . . . . . . . . . . . . . . . . . 9 2.2. Bad Radio Coverage . . . . . . . . . . . . . . . . . . . 9 2.2.1. Network connection . . . . . . . . . . . . . . . . . 9 2.2.2. Simulation Setup . . . . . . . . . . . . . . . . . . 9 + 2.2.3. Expected behavior . . . . . . . . . . . . . . . . . . 10 2.3. Desired Evaluation Metrics for cellular test cases . . . 10 3. Wi-Fi Networks Specific Test Cases . . . . . . . . . . . . . 10 3.1. Bottleneck in Wired Network . . . . . . . . . . . . . . . 12 3.1.1. Network topology . . . . . . . . . . . . . . . . . . 12 - 3.1.2. Test setup . . . . . . . . . . . . . . . . . . . . . 13 + 3.1.2. Test/simulation setup . . . . . . . . . . . . . . . . 13 3.1.3. Typical test scenarios . . . . . . . . . . . . . . . 14 3.1.4. Expected behavior . . . . . . . . . . . . . . . . . . 15 3.2. Bottleneck in Wi-Fi Network . . . . . . . . . . . . . . . 15 3.2.1. Network topology . . . . . . . . . . . . . . . . . . 15 - 3.2.2. Test setup . . . . . . . . . . . . . . . . . . . . . 15 + 3.2.2. Test/simulation setup . . . . . . . . . . . . . . . . 16 3.2.3. Typical test scenarios . . . . . . . . . . . . . . . 17 3.2.4. Expected behavior . . . . . . . . . . . . . . . . . . 18 3.3. Other Potential Test Cases . . . . . . . . . . . . . . . 19 3.3.1. EDCA/WMM usage . . . . . . . . . . . . . . . . . . . 19 3.3.2. Effect of heterogeneous link rates . . . . . . . . . 19 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 5. Security Considerations . . . . . . . . . . . . . . . . . . . 20 - 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 20 - 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 - 7.1. Normative References . . . . . . . . . . . . . . . . . . 20 - 7.2. Informative References . . . . . . . . . . . . . . . . . 21 + 6. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 20 + 7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 21 + 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 + 8.1. Normative References . . . . . . . . . . . . . . . . . . 21 + 8.2. Informative References . . . . . . . . . . . . . . . . . 22 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 1. Introduction Wireless networks (both cellular and Wi-Fi [IEEE802.11]) are an integral and increasingly more significant part of the Internet. Typical application scenarios for interactive multimedia communication over wireless include from video conferencing calls in a bus or train as well as live media streaming at home. It is well known that the characteristics and technical challenges for @@ -252,25 +252,21 @@ the network. This is to avoid running the evaluation in an empty network where network nodes are having empty buffers, low interference at the beginning of the simulation. This network initialization period should be excluded from the evaluation period. Typically, the evaluation period starts 30 seconds after test initialization. This test case also includes user mobility and some competing traffic. The latter includes both the same types of flows (with same adaptation algorithms) and different types of flows (with different - services and congestion control schemes). The investigated - congestion control algorithms should show maximum possible network - utilization and stability in terms of rate variations, lowest - possible end to end frame latency, network latency and Packet Loss - Rate (PLR) at different cell load level. + services and congestion control schemes). 2.1.1. Network Connection Each mobile user is connected to a fixed user. The connection between the mobile user and fixed user consists of a cellular radio access, an Evolved Packet Core (EPC) and an Internet connection. The mobile user is connected to the EPC using cellular radio access technology which is further connected to the Internet. At the other end, the fixed user is connected to the Internet via wired connection with sufficiently high bandwidth, for instance, 10 Gbps, so that the @@ -370,45 +367,48 @@ 8. Other traffic models: * Downlink simulation: Maximum of 4Mbps/cell (web browsing or FTP traffic following default TCP congestion control [RFC5681]) * Unlink simulation: Maximum of 2Mbps/cell (web browsing or FTP traffic following default TCP congestion control [RFC5681]) +2.1.3. Expected behavior + + The investigated congestion control algorithms should result in + maximum possible network utilization and stability in terms of rate + variations, lowest possible end to end frame latency, network latency + and Packet Loss Rate (PLR) at different cell load levels. + 2.2. Bad Radio Coverage The goal of this test is to evaluate the performance of candidate congestion control algorithm when users visit part of the network with bad radio coverage. The scenario is created by using a larger cell radius than that in the previous test case. In this test case, each user/UE in the media session is an endpoint following RTP-based congestion control. User arrivals follow a Poisson distribution proportional to the length of the call, to keep the number of users per cell fairly constant during the evaluation period. At the beginning of the simulation, there should be enough amount of time to warm-up the network. This is to avoid running the evaluation in an empty network where network nodes are having empty buffers, low interference at the beginning of the simulation. This network initialization period should be excluded from the evaluation period. Typically, the evaluation period starts 30 seconds after test initialization. This test case also includes user mobility and some competing traffic. The latter includes the same kind of flows (with same - adaptation algorithms). The investigated congestion control - algorithms should result in maximum possible network utilization and - stability in terms of rate variations, lowest possible end to end - frame latency, network latency and Packet Loss Rate (PLR) at - different cell load levels. + adaptation algorithms). 2.2.1. Network connection Same as defined in Section 2.1.1 2.2.2. Simulation Setup The desired simulation setup is the same as the Varying Network Load test case defined in Section 2.1 except the following changes: @@ -429,20 +429,27 @@ 4. Other traffic models: * Downlink simulation: Maximum of 2Mbps/cell (web browsing or FTP traffic following default TCP congestion control [RFC5681]) * Unlink simulation: Maximum of 1Mbps/cell (web browsing or FTP traffic following default TCP congestion control [RFC5681]) +2.2.3. Expected behavior + + The investigated congestion control algorithms should result in + maximum possible network utilization and stability in terms of rate + variations, lowest possible end to end frame latency, network latency + and Packet Loss Rate (PLR) at different cell load levels. + 2.3. Desired Evaluation Metrics for cellular test cases The evaluation criteria document [I-D.ietf-rmcat-eval-criteria] defines the metrics to be used to evaluate candidate algorithms. Considering the nature and distinction of cellular networks we recommend that at least the following metrics be used to evaluate the performance of the candidate algorithms: o Average cell throughput (for all cells), shows cell utilizations. @@ -583,21 +590,21 @@ . )) \\ . . )) \\ . +----------+ )) \\ +----------+ | MN_tcp_M |))) \=====| FN_tcp_M | +----------+ +----------+ +<-----------------+ Downlink Figure 2: Network topology for Wi-Fi test cases -3.1.2. Test setup +3.1.2. Test/simulation setup o Test duration: 120s o Wi-Fi network characteristics: * Radio propagation model: Log-distance path loss propagation model (see [NS3WiFi]) * PHY- and MAC-layer configuration: IEEE 802.11n @@ -692,30 +699,31 @@ The test cases in this section assume that the wired segment along the media path is well-provisioned whereas the bottleneck exists over the Wi-Fi access network. This is to mimic the application scenarios typically encountered by users in an enterprise environment or at a coffee house. 3.2.1. Network topology Same as defined in Section 3.1.1 -3.2.2. Test setup +3.2.2. Test/simulation setup o Test duration: 120s o Wi-Fi network characteristics: * Radio propagation model: Log-distance path loss propagation model (see [NS3WiFi]) * PHY- and MAC-layer configuration: IEEE 802.11n + * MCS Index at 11: 16-QAM 1/2, Raw Data Rate at 52Mbps o Wired path characteristics: * Path capacity: 100Mbps. * One-Way propagation delay: 50ms. * Maximum end-to-end jitter: 30ms. @@ -912,43 +920,62 @@ Given the difficulty of deterministic wireless testing, it is recommended and expected that the tests described in this document would be done via simulations. However, in the case where these test cases are carried out in a testbed setting, the evaluation should take place in a controlled lab environment. In the testbed, the applications, simulators and network nodes ought to be well-behaved and should not impact the desired results. It is important to take appropriate caution to avoid leaking non-responsive traffic with unproven congestion avoidance behavior onto the open Internet. -6. Acknowledgments +6. Contributors - The authors would like to thank Ingemar Johansson for contributing to - the cellular test cases during the earlier stage of this draft. + The following individuals contributed to the design, implementation, + and verification of the proposed test cases during earlier stages of + this work. They have helped to validate and substantially improve + this specification. + + Ingemar Johansson, of Ericsson AB + contributing to the description and validation of cellular test cases + during the earlier stage of this draft. + + Wei-Tian Tan, , of Cisco Systems designed and set up + a Wi-Fi testbed for evaluating parallel video conferencing streams, + based upon which proposed Wi-Fi test cases are described. He also + recommended additional test cases to consider, such as the impact of + EDCA/WMM usage. + + Michael A. Ramalho, of AcousticComms Consulting + (previously at Cisco Systems) applied learnings from Cisco's internal + experimentation to the early versions of the draft. He also worked + on validating the proposed test cases in a VM-based lab setting. + +7. Acknowledgments The authors would like to thank Tomas Frankkila, Magnus Westerlund, Kristofer Sandlund, Sergio Mena de la Cruz, and Mirja Kuehlewind for their valuable inputs and review comments regarding this draft. -7. References +8. References -7.1. Normative References +8.1. Normative References [HO-deploy-3GPP] TS 25.814, 3GPP., "Physical layer aspects for evolved Universal Terrestrial Radio Access (UTRA)", October 2006, . [I-D.ietf-rmcat-eval-criteria] Singh, V., Ott, J., and S. Holmer, "Evaluating Congestion Control for Interactive Real-time Media", draft-ietf- - rmcat-eval-criteria-11 (work in progress), February 2020. + rmcat-eval-criteria-13 (work in progress), March 2020. [I-D.ietf-rmcat-eval-test] Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test Cases for Evaluating RMCAT Proposals", draft-ietf-rmcat- eval-test-10 (work in progress), May 2019. [IEEE802.11] IEEE, "Standard for Information technology-- Telecommunications and information exchange between systems Local and metropolitan area networks--Specific @@ -960,21 +987,21 @@ classns3_1_1_yans_wifi_channel.html>. [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . -7.2. Informative References +8.2. Informative References [Heusse2003] Heusse, M., Rousseau, F., Berger-Sabbatel, G., and A. Duda, "Performance anomaly of 802.11b", in Proc. 23th Annual Joint Conference of the IEEE Computer and Communications Societies, (INFOCOM'03), March 2003. [HO-def-3GPP] TR 21.905, 3GPP., "Vocabulary for 3GPP Specifications", December 2009,