draft-ietf-netvc-testing-07.txt   draft-ietf-netvc-testing-08.txt 
Network Working Group T. Daede Network Working Group T. Daede
Internet-Draft Mozilla Internet-Draft Mozilla
Intended status: Informational A. Norkin Intended status: Informational A. Norkin
Expires: January 3, 2019 Netflix Expires: July 29, 2019 Netflix
I. Brailovskiy I. Brailovskiy
Amazon Lab126 Amazon Lab126
July 02, 2018 January 25, 2019
Video Codec Testing and Quality Measurement Video Codec Testing and Quality Measurement
draft-ietf-netvc-testing-07 draft-ietf-netvc-testing-08
Abstract Abstract
This document describes guidelines and procedures for evaluating a This document describes guidelines and procedures for evaluating a
video codec. This covers subjective and objective tests, test video codec. This covers subjective and objective tests, test
conditions, and materials used for the test. conditions, and materials used for the test.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
skipping to change at page 1, line 35 skipping to change at page 1, line 35
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 3, 2019. This Internet-Draft will expire on July 29, 2019.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
skipping to change at page 3, line 28 skipping to change at page 3, line 28
metrics disagree. metrics disagree.
Selection of a testing methodology depends on the feature being Selection of a testing methodology depends on the feature being
tested and the resources available. Test methodologies are presented tested and the resources available. Test methodologies are presented
in order of increasing accuracy and cost. in order of increasing accuracy and cost.
Testing relies on the resources of participants. For this reason, Testing relies on the resources of participants. For this reason,
even if the group agrees that a particular test is important, if no even if the group agrees that a particular test is important, if no
one volunteers to do it, or if volunteers do not complete it in a one volunteers to do it, or if volunteers do not complete it in a
timely fashion, then that test should be discarded. This ensures timely fashion, then that test should be discarded. This ensures
that only important tests be done in particular, the tests that are that only important tests be done; in particular, the tests that are
important to participants. important to participants.
Subjective tests should use the same operating points as the Subjective tests should use the same operating points as the
objective tests. objective tests.
2.1. Still Image Pair Comparison 2.1. Still Image Pair Comparison
A simple way to determine superiority of one compressed image is to A simple way to determine superiority of one compressed image is to
visually compare two compressed images, and have the viewer judge visually compare two compressed images, and have the viewer judge
which one has a higher quality. For example, this test may be which one has a higher quality. For example, this test may be
skipping to change at page 6, line 4 skipping to change at page 6, line 4
3.2. Frame-averaged PSNR 3.2. Frame-averaged PSNR
PSNR can also be calculated per-frame, and then the values averaged PSNR can also be calculated per-frame, and then the values averaged
together. This is reported in the same way as overall PSNR. together. This is reported in the same way as overall PSNR.
3.3. PSNR-HVS-M 3.3. PSNR-HVS-M
The PSNR-HVS metric performs a DCT transform of 8x8 blocks of the The PSNR-HVS metric performs a DCT transform of 8x8 blocks of the
image, weights the coefficients, and then calculates the PSNR of image, weights the coefficients, and then calculates the PSNR of
those coefficients. Several different sets of weights have been those coefficients. Several different sets of weights have been
considered. [PSNRHVS] The weights used by the dump_pnsrhvs.c tool in considered [PSNRHVS]. The weights used by the dump_pnsrhvs.c tool in
the Daala repository have been found to be the best match to real MOS the Daala repository have been found to be the best match to real MOS
scores. scores.
3.4. SSIM 3.4. SSIM
SSIM (Structural Similarity Image Metric) is a still image quality SSIM (Structural Similarity Image Metric) is a still image quality
metric introduced in 2004 [SSIM]. It computes a score for each metric introduced in 2004 [SSIM]. It computes a score for each
individual pixel, using a window of neighboring pixels. These scores individual pixel, using a window of neighboring pixels. These scores
can then be averaged to produce a global score for the entire image. can then be averaged to produce a global score for the entire image.
The original paper produces scores ranging between 0 and 1. The original paper produces scores ranging between 0 and 1.
skipping to change at page 22, line 33 skipping to change at page 22, line 33
<http://dx.doi.org/10.1155/2012/273723>. <http://dx.doi.org/10.1155/2012/273723>.
[COMPARECODECS] [COMPARECODECS]
Alvestrand, H., "Compare Codecs", 2015, Alvestrand, H., "Compare Codecs", 2015,
<http://compare-codecs.appspot.com/>. <http://compare-codecs.appspot.com/>.
[DAALA-GIT] [DAALA-GIT]
Xiph.Org, "Daala Git Repository", 2015, Xiph.Org, "Daala Git Repository", 2015,
<http://git.xiph.org/?p=daala.git;a=summary>. <http://git.xiph.org/?p=daala.git;a=summary>.
[DERFVIDEO]
Terriberry, T., "Xiph.org Video Test Media", n.d.,
<https://media.xiph.org/video/derf/>.
[I-D.ietf-netvc-requirements] [I-D.ietf-netvc-requirements]
Filippov, A., Norkin, A., and j. Filippov, A. and A. Norkin, "<Video Codec Requirements and
jose.roberto.alvarez@huawei.com, "<Video Codec Evaluation Methodology>", draft-ietf-netvc-requirements-08
Requirements and Evaluation Methodology>", draft-ietf- (work in progress), May 2018.
netvc-requirements-02 (work in progress), June 2016.
[L1100] Bossen, F., "Common test conditions and software reference
configurations", JCTVC L1100, 2013,
<http://phenix.int-evry.fr/jct/>.
[MSSSIM] Wang, Z., Simoncelli, E., and A. Bovik, "Multi-Scale [MSSSIM] Wang, Z., Simoncelli, E., and A. Bovik, "Multi-Scale
Structural Similarity for Image Quality Assessment", n.d., Structural Similarity for Image Quality Assessment", n.d.,
<http://www.cns.nyu.edu/~zwang/files/papers/msssim.pdf>. <http://www.cns.nyu.edu/~zwang/files/papers/msssim.pdf>.
[PSNRHVS] Egiazarian, K., Astola, J., Ponomarenko, N., Lukin, V., [PSNRHVS] Egiazarian, K., Astola, J., Ponomarenko, N., Lukin, V.,
Battisti, F., and M. Carli, "A New Full-Reference Quality Battisti, F., and M. Carli, "A New Full-Reference Quality
Metrics Based on HVS", 2002. Metrics Based on HVS", 2002.
[RD_TOOL] Xiph.Org, "rd_tool", 2016, <https://github.com/tdaede/ [RD_TOOL] Xiph.Org, "rd_tool", 2016, <https://github.com/tdaede/
rd_tool>. rd_tool>.
[SSIM] Wang, Z., Bovik, A., Sheikh, H., and E. Simoncelli, "Image [SSIM] Wang, Z., Bovik, A., Sheikh, H., and E. Simoncelli, "Image
Quality Assessment: From Error Visibility to Structural Quality Assessment: From Error Visibility to Structural
Similarity", 2004, Similarity", 2004,
<http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf>. <http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf>.
[STEAM] Valve Corporation, "Steam Hardware & Software Survey: June
2015", June 2015,
<http://store.steampowered.com/hwsurvey>.
[TESTSEQUENCES] [TESTSEQUENCES]
Daede, T., "Test Sets", n.d., Daede, T., "Test Sets", n.d.,
<https://people.xiph.org/~tdaede/sets/>. <https://people.xiph.org/~tdaede/sets/>.
[VMAF] Aaron, A., Li, Z., Manohara, M., Lin, J., Wu, E., and C. [VMAF] Aaron, A., Li, Z., Manohara, M., Lin, J., Wu, E., and C.
Kuo, "VMAF - Video Multi-Method Assessment Fusion", 2015, Kuo, "VMAF - Video Multi-Method Assessment Fusion", 2015,
<https://github.com/Netflix/vmaf>. <https://github.com/Netflix/vmaf>.
Authors' Addresses Authors' Addresses
 End of changes. 10 change blocks. 
23 lines changed or deleted 10 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/