draft-ietf-netvc-testing-05.txt   draft-ietf-netvc-testing-06.txt 
Network Working Group T. Daede Network Working Group T. Daede
Internet-Draft Mozilla Internet-Draft Mozilla
Intended status: Informational A. Norkin Intended status: Informational A. Norkin
Expires: September 28, 2017 Netflix Expires: May 3, 2018 Netflix
I. Brailovskiy I. Brailovskiy
Amazon Lab126 Amazon Lab126
March 27, 2017 October 30, 2017
Video Codec Testing and Quality Measurement Video Codec Testing and Quality Measurement
draft-ietf-netvc-testing-05 draft-ietf-netvc-testing-06
Abstract Abstract
This document describes guidelines and procedures for evaluating a This document describes guidelines and procedures for evaluating a
video codec. This covers subjective and objective tests, test video codec. This covers subjective and objective tests, test
conditions, and materials used for the test. conditions, and materials used for the test.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
skipping to change at page 1, line 35 skipping to change at page 1, line 35
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 28, 2017. This Internet-Draft will expire on May 3, 2018.
Copyright Notice Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 45 skipping to change at page 2, line 45
5.3.1. Common settings . . . . . . . . . . . . . . . . . . . 18 5.3.1. Common settings . . . . . . . . . . . . . . . . . . . 18
5.3.2. High Latency CQP . . . . . . . . . . . . . . . . . . 19 5.3.2. High Latency CQP . . . . . . . . . . . . . . . . . . 19
5.3.3. Low Latency CQP . . . . . . . . . . . . . . . . . . . 19 5.3.3. Low Latency CQP . . . . . . . . . . . . . . . . . . . 19
5.3.4. Unconstrained High Latency . . . . . . . . . . . . . 19 5.3.4. Unconstrained High Latency . . . . . . . . . . . . . 19
5.3.5. Unconstrained Low Latency . . . . . . . . . . . . . . 19 5.3.5. Unconstrained Low Latency . . . . . . . . . . . . . . 19
6. Automation . . . . . . . . . . . . . . . . . . . . . . . . . 20 6. Automation . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.1. Regression tests . . . . . . . . . . . . . . . . . . . . 20 6.1. Regression tests . . . . . . . . . . . . . . . . . . . . 20
6.2. Objective performance tests . . . . . . . . . . . . . . . 20 6.2. Objective performance tests . . . . . . . . . . . . . . . 20
6.3. Periodic tests . . . . . . . . . . . . . . . . . . . . . 21 6.3. Periodic tests . . . . . . . . . . . . . . . . . . . . . 21
7. Informative References . . . . . . . . . . . . . . . . . . . 21 7. Informative References . . . . . . . . . . . . . . . . . . . 21
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22
1. Introduction 1. Introduction
When developing a video codec, changes and additions to the codec When developing a video codec, changes and additions to the codec
need to be decided based on their performance tradeoffs. In need to be decided based on their performance tradeoffs. In
addition, measurements are needed to determine when the codec has met addition, measurements are needed to determine when the codec has met
its performance goals. This document specifies how the tests are to its performance goals. This document specifies how the tests are to
be carried about to ensure valid comparisons when evaluating changes be carried about to ensure valid comparisons when evaluating changes
under consideration. Authors of features or changes should provide under consideration. Authors of features or changes should provide
the results of the appropriate test when proposing codec the results of the appropriate test when proposing codec
skipping to change at page 19, line 15 skipping to change at page 19, line 15
5.3.2. High Latency CQP 5.3.2. High Latency CQP
High Latency CQP is used for evaluating incremental changes to a High Latency CQP is used for evaluating incremental changes to a
codec. This method is well suited to compare codecs with similar codec. This method is well suited to compare codecs with similar
coding tools. It allows codec features with intrinsic frame delay. coding tools. It allows codec features with intrinsic frame delay.
o daala: -v=x -b 2 o daala: -v=x -b 2
o vp9: -end-usage=q -cq-level=x -lag-in-frames=25 -auto-alt-ref=2 o vp9: -end-usage=q -cq-level=x -lag-in-frames=25 -auto-alt-ref=2
o av1: -end-usage=q -cq-level=x -lag-in-frames=25 -auto-alt-ref=2 o av1: -end-usage=q -cq-level=x -auto-alt-ref=2
5.3.3. Low Latency CQP 5.3.3. Low Latency CQP
Low Latency CQP is used for evaluating incremental changes to a Low Latency CQP is used for evaluating incremental changes to a
codec. This method is well suited to compare codecs with similar codec. This method is well suited to compare codecs with similar
coding tools. It requires the codec to be set for zero intrinsic coding tools. It requires the codec to be set for zero intrinsic
frame delay. frame delay.
o daala: -v=x o daala: -v=x
skipping to change at page 20, line 49 skipping to change at page 20, line 49
bitstream should run an objective performance test. The performance bitstream should run an objective performance test. The performance
tests should be run on a wider number of sequences. The following tests should be run on a wider number of sequences. The following
data should be reported: data should be reported:
o Identifying information for the encoder used, such as the git o Identifying information for the encoder used, such as the git
commit hash. commit hash.
o Command line options to the encoder, configure script, and o Command line options to the encoder, configure script, and
anything else necessary to replicate the experiment. anything else necessary to replicate the experiment.
o The name of the test set run (objective-1) o The name of the test set run (objective-1-fast)
o For both high and low latency CQP modes, and for each objective o For both high and low latency CQP modes, and for each objective
metric: metric:
* The BD-Rate score, in percent, for each clip. * The BD-Rate score, in percent, for each clip.
* The average of all BD-Rate scores, equally weighted, for each * The average of all BD-Rate scores, equally weighted, for each
resolution category in the test set. resolution category in the test set.
* The average of all BD-Rate scores for all videos in all * The average of all BD-Rate scores for all videos in all
categories. categories.
For non-tool contributions, the test set objective-1-fast can be Normally, the encoder should always be run at the slowest, highest
substituted. quality speed setting (cpu-used=0 in the case of AV1 and VP9).
However, in the case of computation time, both the reference and
changed encoder can be built with some options disabled. For AV1, -
disable-ext_partition and -disable-ext_partition_types can be passed
to the configure script to substantially speed up encoding, but the
usage of these options must be reported in the test results.
6.3. Periodic tests 6.3. Periodic tests
Periodic tests are run on a wide range of bitrates in order to gauge Periodic tests are run on a wide range of bitrates in order to gauge
progress over time, as well as detect potential regressions missed by progress over time, as well as detect potential regressions missed by
other tests. other tests.
7. Informative References 7. Informative References
[AWCY] Xiph.Org, "Are We Compressed Yet?", 2016, [AWCY] Xiph.Org, "Are We Compressed Yet?", 2016,
skipping to change at page 22, line 5 skipping to change at page 22, line 9
<http://compare-codecs.appspot.com/>. <http://compare-codecs.appspot.com/>.
[DAALA-GIT] [DAALA-GIT]
Xiph.Org, "Daala Git Repository", 2015, Xiph.Org, "Daala Git Repository", 2015,
<http://git.xiph.org/?p=daala.git;a=summary>. <http://git.xiph.org/?p=daala.git;a=summary>.
[DERFVIDEO] [DERFVIDEO]
Terriberry, T., "Xiph.org Video Test Media", n.d., Terriberry, T., "Xiph.org Video Test Media", n.d.,
<https://media.xiph.org/video/derf/>. <https://media.xiph.org/video/derf/>.
[FASTSSIM]
Chen, M. and A. Bovik, "Fast structural similarity index
algorithm", 2010,
<http://live.ece.utexas.edu/publications/2011/
chen_rtip_2011.pdf>.
[I-D.ietf-netvc-requirements] [I-D.ietf-netvc-requirements]
Filippov, A., Norkin, A., and j. Filippov, A., Norkin, A., and j.
jose.roberto.alvarez@huawei.com, "<Video Codec jose.roberto.alvarez@huawei.com, "<Video Codec
Requirements and Evaluation Methodology>", draft-ietf- Requirements and Evaluation Methodology>", draft-ietf-
netvc-requirements-02 (work in progress), June 2016. netvc-requirements-02 (work in progress), June 2016.
[L1100] Bossen, F., "Common test conditions and software reference [L1100] Bossen, F., "Common test conditions and software reference
configurations", JCTVC L1100, 2013, configurations", JCTVC L1100, 2013,
<http://phenix.int-evry.fr/jct/>. <http://phenix.int-evry.fr/jct/>.
 End of changes. 9 change blocks. 
15 lines changed or deleted 14 lines changed or added

This html diff was produced by rfcdiff 1.46. The latest version is available from http://tools.ietf.org/tools/rfcdiff/