draft-ietf-netvc-testing-04.txt   draft-ietf-netvc-testing-05.txt 
Network Working Group T. Daede Network Working Group T. Daede
Internet-Draft Mozilla Internet-Draft Mozilla
Intended status: Informational A. Norkin Intended status: Informational A. Norkin
Expires: May 4, 2017 Netflix Expires: September 28, 2017 Netflix
I. Brailovskiy I. Brailovskiy
Amazon Lab126 Amazon Lab126
October 31, 2016 March 27, 2017
Video Codec Testing and Quality Measurement Video Codec Testing and Quality Measurement
draft-ietf-netvc-testing-04 draft-ietf-netvc-testing-05
Abstract Abstract
This document describes guidelines and procedures for evaluating a This document describes guidelines and procedures for evaluating a
video codec. This covers subjective and objective tests, test video codec. This covers subjective and objective tests, test
conditions, and materials used for the test. conditions, and materials used for the test.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
skipping to change at page 1, line 35 skipping to change at page 1, line 35
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 4, 2017. This Internet-Draft will expire on September 28, 2017.
Copyright Notice Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
skipping to change at page 2, line 30 skipping to change at page 2, line 30
3.6. CIEDE2000 . . . . . . . . . . . . . . . . . . . . . . . . 5 3.6. CIEDE2000 . . . . . . . . . . . . . . . . . . . . . . . . 5
3.7. VMAF . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.7. VMAF . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4. Comparing and Interpreting Results . . . . . . . . . . . . . 6 4. Comparing and Interpreting Results . . . . . . . . . . . . . 6
4.1. Graphing . . . . . . . . . . . . . . . . . . . . . . . . 6 4.1. Graphing . . . . . . . . . . . . . . . . . . . . . . . . 6
4.2. BD-Rate . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.2. BD-Rate . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.3. Ranges . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.3. Ranges . . . . . . . . . . . . . . . . . . . . . . . . . 7
5. Test Sequences . . . . . . . . . . . . . . . . . . . . . . . 7 5. Test Sequences . . . . . . . . . . . . . . . . . . . . . . . 7
5.1. Sources . . . . . . . . . . . . . . . . . . . . . . . . . 7 5.1. Sources . . . . . . . . . . . . . . . . . . . . . . . . . 7
5.2. Test Sets . . . . . . . . . . . . . . . . . . . . . . . . 8 5.2. Test Sets . . . . . . . . . . . . . . . . . . . . . . . . 8
5.2.1. regression-1 . . . . . . . . . . . . . . . . . . . . 8 5.2.1. regression-1 . . . . . . . . . . . . . . . . . . . . 8
5.2.2. objective-1 . . . . . . . . . . . . . . . . . . . . . 8 5.2.2. objective-2-slow . . . . . . . . . . . . . . . . . . 8
5.2.3. objective-1-fast . . . . . . . . . . . . . . . . . . 11 5.2.3. objective-2-fast . . . . . . . . . . . . . . . . . . 12
5.3. Operating Points . . . . . . . . . . . . . . . . . . . . 13 5.2.4. objective-1.1 . . . . . . . . . . . . . . . . . . . . 14
5.3.1. Common settings . . . . . . . . . . . . . . . . . . . 13 5.2.5. objective-1-fast . . . . . . . . . . . . . . . . . . 17
5.3.2. High Latency CQP . . . . . . . . . . . . . . . . . . 13 5.3. Operating Points . . . . . . . . . . . . . . . . . . . . 18
5.3.3. Low Latency CQP . . . . . . . . . . . . . . . . . . . 13 5.3.1. Common settings . . . . . . . . . . . . . . . . . . . 18
5.3.4. Unconstrained High Latency . . . . . . . . . . . . . 14 5.3.2. High Latency CQP . . . . . . . . . . . . . . . . . . 19
5.3.5. Unconstrained Low Latency . . . . . . . . . . . . . . 14 5.3.3. Low Latency CQP . . . . . . . . . . . . . . . . . . . 19
6. Automation . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.3.4. Unconstrained High Latency . . . . . . . . . . . . . 19
6.1. Regression tests . . . . . . . . . . . . . . . . . . . . 15 5.3.5. Unconstrained Low Latency . . . . . . . . . . . . . . 19
6.2. Objective performance tests . . . . . . . . . . . . . . . 15 6. Automation . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.3. Periodic tests . . . . . . . . . . . . . . . . . . . . . 15 6.1. Regression tests . . . . . . . . . . . . . . . . . . . . 20
7. Informative References . . . . . . . . . . . . . . . . . . . 16 6.2. Objective performance tests . . . . . . . . . . . . . . . 20
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 17 6.3. Periodic tests . . . . . . . . . . . . . . . . . . . . . 21
7. Informative References . . . . . . . . . . . . . . . . . . . 21
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23
1. Introduction 1. Introduction
When developing a video codec, changes and additions to the codec When developing a video codec, changes and additions to the codec
need to be decided based on their performance tradeoffs. In need to be decided based on their performance tradeoffs. In
addition, measurements are needed to determine when the codec has met addition, measurements are needed to determine when the codec has met
its performance goals. This document specifies how the tests are to its performance goals. This document specifies how the tests are to
be carried about to ensure valid comparisons when evaluating changes be carried about to ensure valid comparisons when evaluating changes
under consideration. Authors of features or changes should provide under consideration. Authors of features or changes should provide
the results of the appropriate test when proposing codec the results of the appropriate test when proposing codec
skipping to change at page 8, line 31 skipping to change at page 8, line 31
very small number of clips. very small number of clips.
o kirlandvga (640x360, 8bit, 4:2:0, 300 frames) o kirlandvga (640x360, 8bit, 4:2:0, 300 frames)
o FourPeople (1280x720, 8bit, 4:2:0, 60 frames) o FourPeople (1280x720, 8bit, 4:2:0, 60 frames)
o Narrarator (4096x2160, 10bit, 4:2:0, 15 frames) o Narrarator (4096x2160, 10bit, 4:2:0, 15 frames)
o CSGO (1920x1080, 8bit, 4:4:4 60 frames) o CSGO (1920x1080, 8bit, 4:4:4 60 frames)
5.2.2. objective-1 5.2.2. objective-2-slow
This test set is a comprehensive test set, grouped by resolution. This test set is a comprehensive test set, grouped by resolution.
These test clips were created from originals at [TESTSEQUENCES]. These test clips were created from originals at [TESTSEQUENCES].
They have been scaled and cropped to match the resolution of their They have been scaled and cropped to match the resolution of their
category. Other deviations are noted in parenthesis. category. This test set requires compiling with high bit depth
support.
4096x2160, 4:2:0, 60 frames:
o Netflix_BarScene_4096x2160_60fps_10bit_420_60f
o Netflix_BoxingPractice_4096x2160_60fps_10bit_420_60f
o Netflix_Dancers_4096x2160_60fps_10bit_420_60f
o Netflix_Narrator_4096x2160_60fps_10bit_420_60f
o Netflix_RitualDance_4096x2160_60fps_10bit_420_60f
o Netflix_ToddlerFountain_4096x2160_60fps_10bit_420_60f
o Netflix_WindAndNature_4096x2160_60fps_10bit_420_60f
o street_hdr_amazon_2160p
1920x1080, 4:2:0, 60 frames:
o aspen_1080p_60f
o crowd_run_1080p50_60f
o ducks_take_off_1080p50_60f
o guitar_hdr_amazon_1080p
o life_1080p30_60f
o Netflix_Aerial_1920x1080_60fps_8bit_420_60f
o Netflix_Boat_1920x1080_60fps_8bit_420_60f
o Netflix_Crosswalk_1920x1080_60fps_8bit_420_60f
o Netflix_FoodMarket_1920x1080_60fps_8bit_420_60f
o Netflix_PierSeaside_1920x1080_60fps_8bit_420_60f
o Netflix_SquareAndTimelapse_1920x1080_60fps_8bit_420_60f
o Netflix_TunnelFlag_1920x1080_60fps_8bit_420_60f
o old_town_cross_1080p50_60f
o pan_hdr_amazon_1080p
o park_joy_1080p50_60f
o pedestrian_area_1080p25_60f
o rush_field_cuts_1080p_60f
o rush_hour_1080p25_60f
o seaplane_hdr_amazon_1080p
o station2_1080p25_60f
o touchdown_pass_1080p_60f
1280x720, 4:2:0, 120 frames:
o boat_hdr_amazon_720p
o dark720p_120f
o FourPeople_1280x720_60_120f
o gipsrestat720p_120f
o Johnny_1280x720_60_120f
o KristenAndSara_1280x720_60_120f
o Netflix_DinnerScene_1280x720_60fps_8bit_420_120f
o Netflix_DrivingPOV_1280x720_60fps_8bit_420_120f
o Netflix_FoodMarket2_1280x720_60fps_8bit_420_120f
o Netflix_RollerCoaster_1280x720_60fps_8bit_420_120f
o Netflix_Tango_1280x720_60fps_8bit_420_120f
o rain_hdr_amazon_720p
o vidyo1_720p_60fps_120f
o vidyo3_720p_60fps_120f
o vidyo4_720p_60fps_120f
640x360, 4:2:0, 120 frames:
o blue_sky_360p_120f
o controlled_burn_640x360_120f
o desktop2360p_120f
o kirland360p_120f
o mmstationary360p_120f
o niklas360p_120f
o rain2_hdr_amazon_360p
o red_kayak_360p_120f
o riverbed_360p25_120f
o shields2_640x360_120f
o snow_mnt_640x360_120f
o speed_bag_640x360_120f
o stockholm_640x360_120f
o tacomanarrows360p_120f
o thaloundeskmtg360p_120f
o water_hdr_amazon_360p
426x240, 4:2:0, 120 frames:
o bqfree_240p_120f
o bqhighway_240p_120f
o bqzoom_240p_120f
o chairlift_240p_120f
o dirtbike_240p_120f
o mozzoom_240p_120f
1920x1080, 4:4:4 or 4:2:0, 60 frames:
o CSGO_60f.y4m
o DOTA2_60f_420.y4m
o MINECRAFT_60f_420.y4m
o STARCRAFT_60f_420.y4m
o EuroTruckSimulator2_60f.y4m
o Hearthstone_60f.y4m
o wikipedia_420.y4m
o pvq_slideshow.y4m
5.2.3. objective-2-fast
This test set is a strict subset of objective-2-slow. It is designed
for faster runtime. This test set requires compiling with high bit
depth support.
1920x1080, 4:2:0, 60 frames:
o aspen_1080p_60f
o ducks_take_off_1080p50_60f
o life_1080p30_60f
o Netflix_Aerial_1920x1080_60fps_8bit_420_60f
o Netflix_Boat_1920x1080_60fps_8bit_420_60f
o Netflix_FoodMarket_1920x1080_60fps_8bit_420_60f
o Netflix_PierSeaside_1920x1080_60fps_8bit_420_60f
o Netflix_SquareAndTimelapse_1920x1080_60fps_8bit_420_60f
o Netflix_TunnelFlag_1920x1080_60fps_8bit_420_60f
o rush_hour_1080p25_60f
o seaplane_hdr_amazon_1080p
o touchdown_pass_1080p_60f
1280x720, 4:2:0, 120 frames:
o boat_hdr_amazon_720p
o dark720p_120f
o gipsrestat720p_120f
o KristenAndSara_1280x720_60_120f
o Netflix_DrivingPOV_1280x720_60fps_8bit_420_60f
o Netflix_RollerCoaster_1280x720_60fps_8bit_420_60f
o vidyo1_720p_60fps_120f
o vidyo4_720p_60fps_120f
640x360, 4:2:0, 120 frames:
o blue_sky_360p_120f
o controlled_burn_640x360_120f
o kirland360p_120f
o niklas360p_120f
o rain2_hdr_amazon_360p
o red_kayak_360p_120f
o riverbed_360p25_120f
o shields2_640x360_120f
o speed_bag_640x360_120f
o thaloundeskmtg360p_120f
426x240, 4:2:0, 120 frames:
o bqfree_240p_120f
o bqzoom_240p_120f
o dirtbike_240p_120f
1290x1080, 4:2:0, 60 frames:
o DOTA2_60f_420.y4m
o MINECRAFT_60f_420.y4m
o STARCRAFT_60f_420.y4m
o wikipedia_420.y4m
5.2.4. objective-1.1
This test set is an old version of objective-2-slow.
4096x2160, 10bit, 4:2:0, 60 frames: 4096x2160, 10bit, 4:2:0, 60 frames:
o Aerial (start frame 600) o Aerial (start frame 600)
o BarScene (start frame 120) o BarScene (start frame 120)
o Boat (start frame 0) o Boat (start frame 0)
o BoxingPractice (start frame 0) o BoxingPractice (start frame 0)
skipping to change at page 11, line 28 skipping to change at page 17, line 5
o tacomascmvvga o tacomascmvvga
o desktop2360p o desktop2360p
o mmmovingvga o mmmovingvga
o mmstationaryvga o mmstationaryvga
o niklasvga o niklasvga
5.2.3. objective-1-fast 5.2.5. objective-1-fast
This test set is based on objective-1, but requires much less This is an old version of objective-2-fast.
computation. It is intended to be a predictor for the results from
objective-1.
2048x1080, 8bit, 4:2:0, 60 frames: 1920x1080, 8bit, 4:2:0, 60 frames:
o Aerial (start frame 600) o Aerial (start frame 600)
o Boat (start frame 0) o Boat (start frame 0)
o Crosswalk (start frame 0) o Crosswalk (start frame 0)
o FoodMarket o FoodMarket
o PierSeaside o PierSeaside
 End of changes. 11 change blocks. 
26 lines changed or deleted 276 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/