draft-ietf-netvc-testing-08.txt   draft-ietf-netvc-testing-09.txt 
Network Working Group T. Daede Network Working Group T. Daede
Internet-Draft Mozilla Internet-Draft Mozilla
Intended status: Informational A. Norkin Intended status: Informational A. Norkin
Expires: July 29, 2019 Netflix Expires: August 3, 2020 Netflix
I. Brailovskiy I. Brailovskiy
Amazon Lab126 Amazon Lab126
January 25, 2019 January 31, 2020
Video Codec Testing and Quality Measurement Video Codec Testing and Quality Measurement
draft-ietf-netvc-testing-08 draft-ietf-netvc-testing-09
Abstract Abstract
This document describes guidelines and procedures for evaluating a This document describes guidelines and procedures for evaluating a
video codec. This covers subjective and objective tests, test video codec. This covers subjective and objective tests, test
conditions, and materials used for the test. conditions, and materials used for the test.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on July 29, 2019. This Internet-Draft will expire on August 3, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Subjective quality tests . . . . . . . . . . . . . . . . . . 3 2. Subjective quality tests . . . . . . . . . . . . . . . . . . 3
2.1. Still Image Pair Comparison . . . . . . . . . . . . . . . 3 2.1. Still Image Pair Comparison . . . . . . . . . . . . . . . 3
2.2. Video Pair Comparison . . . . . . . . . . . . . . . . . . 4 2.2. Video Pair Comparison . . . . . . . . . . . . . . . . . . 4
2.3. Mean Opinion Score . . . . . . . . . . . . . . . . . . . 4 2.3. Mean Opinion Score . . . . . . . . . . . . . . . . . . . 4
3. Objective Metrics . . . . . . . . . . . . . . . . . . . . . . 5 3. Objective Metrics . . . . . . . . . . . . . . . . . . . . . . 5
3.1. Overall PSNR . . . . . . . . . . . . . . . . . . . . . . 5 3.1. Overall PSNR . . . . . . . . . . . . . . . . . . . . . . 5
3.2. Frame-averaged PSNR . . . . . . . . . . . . . . . . . . . 5 3.2. Frame-averaged PSNR . . . . . . . . . . . . . . . . . . . 5
3.3. PSNR-HVS-M . . . . . . . . . . . . . . . . . . . . . . . 5 3.3. PSNR-HVS-M . . . . . . . . . . . . . . . . . . . . . . . 6
3.4. SSIM . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.4. SSIM . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.5. Multi-Scale SSIM . . . . . . . . . . . . . . . . . . . . 6 3.5. Multi-Scale SSIM . . . . . . . . . . . . . . . . . . . . 6
3.6. CIEDE2000 . . . . . . . . . . . . . . . . . . . . . . . . 6 3.6. CIEDE2000 . . . . . . . . . . . . . . . . . . . . . . . . 6
3.7. VMAF . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.7. VMAF . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4. Comparing and Interpreting Results . . . . . . . . . . . . . 7 4. Comparing and Interpreting Results . . . . . . . . . . . . . 7
4.1. Graphing . . . . . . . . . . . . . . . . . . . . . . . . 7 4.1. Graphing . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2. BD-Rate . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.2. BD-Rate . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.3. Ranges . . . . . . . . . . . . . . . . . . . . . . . . . 8 4.3. Ranges . . . . . . . . . . . . . . . . . . . . . . . . . 8
5. Test Sequences . . . . . . . . . . . . . . . . . . . . . . . 8 5. Test Sequences . . . . . . . . . . . . . . . . . . . . . . . 8
5.1. Sources . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.1. Sources . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.2. Test Sets . . . . . . . . . . . . . . . . . . . . . . . . 8 5.2. Test Sets . . . . . . . . . . . . . . . . . . . . . . . . 8
5.2.1. regression-1 . . . . . . . . . . . . . . . . . . . . 8 5.2.1. regression-1 . . . . . . . . . . . . . . . . . . . . 9
5.2.2. objective-2-slow . . . . . . . . . . . . . . . . . . 9 5.2.2. objective-2-slow . . . . . . . . . . . . . . . . . . 9
5.2.3. objective-2-fast . . . . . . . . . . . . . . . . . . 12 5.2.3. objective-2-fast . . . . . . . . . . . . . . . . . . 12
5.2.4. objective-1.1 . . . . . . . . . . . . . . . . . . . . 14 5.2.4. objective-1.1 . . . . . . . . . . . . . . . . . . . . 14
5.2.5. objective-1-fast . . . . . . . . . . . . . . . . . . 17 5.2.5. objective-1-fast . . . . . . . . . . . . . . . . . . 17
5.3. Operating Points . . . . . . . . . . . . . . . . . . . . 19 5.3. Operating Points . . . . . . . . . . . . . . . . . . . . 19
5.3.1. Common settings . . . . . . . . . . . . . . . . . . . 19 5.3.1. Common settings . . . . . . . . . . . . . . . . . . . 19
5.3.2. High Latency CQP . . . . . . . . . . . . . . . . . . 19 5.3.2. High Latency CQP . . . . . . . . . . . . . . . . . . 19
5.3.3. Low Latency CQP . . . . . . . . . . . . . . . . . . . 19 5.3.3. Low Latency CQP . . . . . . . . . . . . . . . . . . . 19
5.3.4. Unconstrained High Latency . . . . . . . . . . . . . 20 5.3.4. Unconstrained High Latency . . . . . . . . . . . . . 20
5.3.5. Unconstrained Low Latency . . . . . . . . . . . . . . 20 5.3.5. Unconstrained Low Latency . . . . . . . . . . . . . . 20
6. Automation . . . . . . . . . . . . . . . . . . . . . . . . . 20 6. Automation . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.1. Regression tests . . . . . . . . . . . . . . . . . . . . 21 6.1. Regression tests . . . . . . . . . . . . . . . . . . . . 21
6.2. Objective performance tests . . . . . . . . . . . . . . . 21 6.2. Objective performance tests . . . . . . . . . . . . . . . 21
6.3. Periodic tests . . . . . . . . . . . . . . . . . . . . . 22 6.3. Periodic tests . . . . . . . . . . . . . . . . . . . . . 22
7. Informative References . . . . . . . . . . . . . . . . . . . 22 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22
8. Security Considerations . . . . . . . . . . . . . . . . . . . 22
9. Informative References . . . . . . . . . . . . . . . . . . . 22
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23
1. Introduction 1. Introduction
When developing a video codec, changes and additions to the codec When developing a video codec, changes and additions to the codec
need to be decided based on their performance tradeoffs. In need to be decided based on their performance tradeoffs. In
addition, measurements are needed to determine when the codec has met addition, measurements are needed to determine when the codec has met
its performance goals. This document specifies how the tests are to its performance goals. This document specifies how the tests are to
be carried about to ensure valid comparisons when evaluating changes be carried about to ensure valid comparisons when evaluating changes
under consideration. Authors of features or changes should provide under consideration. Authors of features or changes should provide
the results of the appropriate test when proposing codec the results of the appropriate test when proposing codec
modifications. modifications.
2. Subjective quality tests 2. Subjective quality tests
Subjective testing is the preferable method of testing video codecs. Subjective testing uses human viewers to rate and compare the quality
of videos. It is the preferable method of testing video codecs.
Subjective testing results take priority over objective testing Subjective testing results take priority over objective testing
results, when available. Subjective testing is recommended results, when available. Subjective testing is recommended
especially when taking advantage of psychovisual effects that may not especially when taking advantage of psychovisual effects that may not
be well represented by objective metrics, or when different objective be well represented by objective metrics, or when different objective
metrics disagree. metrics disagree.
Selection of a testing methodology depends on the feature being Selection of a testing methodology depends on the feature being
tested and the resources available. Test methodologies are presented tested and the resources available. Test methodologies are presented
in order of increasing accuracy and cost. in order of increasing accuracy and cost.
Testing relies on the resources of participants. For this reason, Testing relies on the resources of participants. If a participant
even if the group agrees that a particular test is important, if no requires a subjective test for a particular feature or improvement,
one volunteers to do it, or if volunteers do not complete it in a they are responsible for ensuring that resources are available. This
timely fashion, then that test should be discarded. This ensures ensures that only important tests be done; in particular, the tests
that only important tests be done; in particular, the tests that are that are important to participants.
important to participants.
Subjective tests should use the same operating points as the Subjective tests should use the same operating points as the
objective tests. objective tests.
2.1. Still Image Pair Comparison 2.1. Still Image Pair Comparison
A simple way to determine superiority of one compressed image is to A simple way to determine superiority of one compressed image is to
visually compare two compressed images, and have the viewer judge visually compare two compressed images, and have the viewer judge
which one has a higher quality. For example, this test may be which one has a higher quality. For example, this test may be
suitable for an intra de-ringing filter, but not for a new inter suitable for an intra de-ringing filter, but not for a new inter
skipping to change at page 4, line 4 skipping to change at page 4, line 10
prediction mode. For this test, the two compressed images should prediction mode. For this test, the two compressed images should
have similar compressed file sizes, with one image being no more than have similar compressed file sizes, with one image being no more than
5% larger than the other. In addition, at least 5 different images 5% larger than the other. In addition, at least 5 different images
should be compared. should be compared.
Once testing is complete, a p-value can be computed using the Once testing is complete, a p-value can be computed using the
binomial test. A significant result should have a resulting p-value binomial test. A significant result should have a resulting p-value
less than or equal to 0.5. For example: less than or equal to 0.5. For example:
p_value = binom_test(a,a+b) p_value = binom_test(a,a+b)
where a is the number of votes for one video, b is the number of where a is the number of votes for one video, b is the number of
votes for the second video, and binom_test(x,y) returns the binomial votes for the second video, and binom_test(x,y) returns the binomial
PMF with x observed tests, y total tests, and expected probability PMF (probability mass function) with x observed tests, y total tests,
0.5. and expected probability 0.5.
If ties are allowed to be reported, then the equation is modified: If ties are allowed to be reported, then the equation is modified:
p_value = binom_test(a+floor(t/2),a+b+t) p_value = binom_test(a+floor(t/2),a+b+t)
where t is the number of tie votes. where t is the number of tie votes.
Still image pair comparison is used for rapid comparisons during Still image pair comparison is used for rapid comparisons during
development - the viewer may be either a developer or user, for development - the viewer may be either a developer or user, for
example. As the results are only relative, it is effective even with example. As the results are only relative, it is effective even with
skipping to change at page 5, line 50 skipping to change at page 6, line 7
This metric may be applied to both the luma and chroma planes, with This metric may be applied to both the luma and chroma planes, with
all planes reported separately. all planes reported separately.
3.2. Frame-averaged PSNR 3.2. Frame-averaged PSNR
PSNR can also be calculated per-frame, and then the values averaged PSNR can also be calculated per-frame, and then the values averaged
together. This is reported in the same way as overall PSNR. together. This is reported in the same way as overall PSNR.
3.3. PSNR-HVS-M 3.3. PSNR-HVS-M
The PSNR-HVS metric performs a DCT transform of 8x8 blocks of the The PSNR-HVS [PSNRHVS] metric performs a DCT transform of 8x8 blocks
image, weights the coefficients, and then calculates the PSNR of of the image, weights the coefficients, and then calculates the PSNR
those coefficients. Several different sets of weights have been of those coefficients. Several different sets of weights have been
considered [PSNRHVS]. The weights used by the dump_pnsrhvs.c tool in considered. The weights used by the dump_pnsrhvs.c tool in the Daala
the Daala repository have been found to be the best match to real MOS repository have been found to be the best match to real MOS scores.
scores.
3.4. SSIM 3.4. SSIM
SSIM (Structural Similarity Image Metric) is a still image quality SSIM (Structural Similarity Image Metric) is a still image quality
metric introduced in 2004 [SSIM]. It computes a score for each metric introduced in 2004 [SSIM]. It computes a score for each
individual pixel, using a window of neighboring pixels. These scores individual pixel, using a window of neighboring pixels. These scores
can then be averaged to produce a global score for the entire image. can then be averaged to produce a global score for the entire image.
The original paper produces scores ranging between 0 and 1. The original paper produces scores ranging between 0 and 1.
To linearize the metric for BD-Rate computation, the score is To linearize the metric for BD-Rate computation, the score is
skipping to change at page 6, line 38 skipping to change at page 6, line 43
CIEDE2000 is a metric based on CIEDE color distances [CIEDE2000]. It CIEDE2000 is a metric based on CIEDE color distances [CIEDE2000]. It
generates a single score taking into account all three chroma planes. generates a single score taking into account all three chroma planes.
It does not take into consideration any structural similarity or It does not take into consideration any structural similarity or
other psychovisual effects. other psychovisual effects.
3.7. VMAF 3.7. VMAF
Video Multi-method Assessment Fusion (VMAF) is a full-reference Video Multi-method Assessment Fusion (VMAF) is a full-reference
perceptual video quality metric that aims to approximate human perceptual video quality metric that aims to approximate human
perception of video quality [VMAF]. This metric is focused on perception of video quality [VMAF]. This metric is focused on
quality degradation due compression and rescaling. VMAF estimates quality degradation due to compression and rescaling. VMAF estimates
the perceived quality score by computing scores from multiple quality the perceived quality score by computing scores from multiple quality
assessment algorithms, and fusing them using a support vector machine assessment algorithms, and fusing them using a support vector machine
(SVM). Currently, three image fidelity metrics and one temporal (SVM). Currently, three image fidelity metrics and one temporal
signal have been chosen as features to the SVM, namely Anti-noise SNR signal have been chosen as features to the SVM, namely Anti-noise SNR
(ANSNR), Detail Loss Measure (DLM), Visual Information Fidelity (ANSNR), Detail Loss Measure (DLM), Visual Information Fidelity
(VIF), and the mean co-located pixel difference of a frame with (VIF), and the mean co-located pixel difference of a frame with
respect to the previous frame. respect to the previous frame.
The quality score from VMAF is used directly to calculate BD-Rate, The quality score from VMAF is used directly to calculate BD-Rate,
without any conversions. without any conversions.
skipping to change at page 8, line 48 skipping to change at page 8, line 48
scenarios the codec will be required to operate in. For easier scenarios the codec will be required to operate in. For easier
comparison, all videos in each set should have the same color comparison, all videos in each set should have the same color
subsampling, same resolution, and same number of frames. In subsampling, same resolution, and same number of frames. In
addition, all test videos must be publicly available for testing use, addition, all test videos must be publicly available for testing use,
to allow for reproducibility of results. All current test sets are to allow for reproducibility of results. All current test sets are
available for download [TESTSEQUENCES]. available for download [TESTSEQUENCES].
Test sequences should be downloaded in whole. They should not be Test sequences should be downloaded in whole. They should not be
recreated from the original sources. recreated from the original sources.
Each clip is labeled with its resolution, bit depth, color
subsampling, and length.
5.2.1. regression-1 5.2.1. regression-1
This test set is used for basic regression testing. It contains a This test set is used for basic regression testing. It contains a
very small number of clips. very small number of clips.
o kirlandvga (640x360, 8bit, 4:2:0, 300 frames) o kirlandvga (640x360, 8bit, 4:2:0, 300 frames)
o FourPeople (1280x720, 8bit, 4:2:0, 60 frames) o FourPeople (1280x720, 8bit, 4:2:0, 60 frames)
o Narrarator (4096x2160, 10bit, 4:2:0, 15 frames) o Narrarator (4096x2160, 10bit, 4:2:0, 15 frames)
o CSGO (1920x1080, 8bit, 4:4:4 60 frames) o CSGO (1920x1080, 8bit, 4:4:4 60 frames)
5.2.2. objective-2-slow 5.2.2. objective-2-slow
This test set is a comprehensive test set, grouped by resolution. This test set is a comprehensive test set, grouped by resolution.
These test clips were created from originals at [TESTSEQUENCES]. These test clips were created from originals at [TESTSEQUENCES].
They have been scaled and cropped to match the resolution of their They have been scaled and cropped to match the resolution of their
category. This test set requires compiling with high bit depth category. This test set requires a codec that supports both 8 and 10
support. bit video.
4096x2160, 4:2:0, 60 frames: 4096x2160, 4:2:0, 60 frames:
o Netflix_BarScene_4096x2160_60fps_10bit_420_60f o Netflix_BarScene_4096x2160_60fps_10bit_420_60f
o Netflix_BoxingPractice_4096x2160_60fps_10bit_420_60f o Netflix_BoxingPractice_4096x2160_60fps_10bit_420_60f
o Netflix_Dancers_4096x2160_60fps_10bit_420_60f o Netflix_Dancers_4096x2160_60fps_10bit_420_60f
o Netflix_Narrator_4096x2160_60fps_10bit_420_60f o Netflix_Narrator_4096x2160_60fps_10bit_420_60f
skipping to change at page 18, line 47 skipping to change at page 19, line 4
o speed_bag o speed_bag
o shields o shields
640x360, 8bit, 4:2:0, 60 frames: 640x360, 8bit, 4:2:0, 60 frames:
o red_kayak o red_kayak
o riverbed o riverbed
o kirlandvga o kirlandvga
o tacomascmvvga o tacomascmvvga
o mmmovingvga o mmmovingvga
o niklasvga o niklasvga
5.3. Operating Points 5.3. Operating Points
Four operating modes are defined. High latency is intended for on Four operating modes are defined. High latency is intended for on
demand streaming, one-to-many live streaming, and stored video. Low demand streaming, one-to-many live streaming, and stored video. Low
latency is intended for videoconferencing and remote access. Both of latency is intended for videoconferencing and remote access. Both of
these modes come in CQP and unconstrained variants. When testing these modes come in CQP (constant quantizer parameter) and
still image sets, such as subset1, high latency CQP mode should be unconstrained variants. When testing still image sets, such as
used. subset1, high latency CQP mode should be used.
5.3.1. Common settings 5.3.1. Common settings
Encoders should be configured to their best settings when being Encoders should be configured to their best settings when being
compared against each other: compared against each other:
o av1: -codec=av1 -ivf -frame-parallel=0 -tile-columns=0 -cpu-used=0 o av1: -codec=av1 -ivf -frame-parallel=0 -tile-columns=0 -cpu-used=0
-threads=1 -threads=1
5.3.2. High Latency CQP 5.3.2. High Latency CQP
skipping to change at page 22, line 11 skipping to change at page 22, line 11
disable-ext_partition and -disable-ext_partition_types can be passed disable-ext_partition and -disable-ext_partition_types can be passed
to the configure script to substantially speed up encoding, but the to the configure script to substantially speed up encoding, but the
usage of these options must be reported in the test results. usage of these options must be reported in the test results.
6.3. Periodic tests 6.3. Periodic tests
Periodic tests are run on a wide range of bitrates in order to gauge Periodic tests are run on a wide range of bitrates in order to gauge
progress over time, as well as detect potential regressions missed by progress over time, as well as detect potential regressions missed by
other tests. other tests.
7. Informative References 7. IANA Considerations
This document does not require any IANA actions.
8. Security Considerations
This document describes the methodologies an procedures for
qualitative testing, therefore does not iteself have implications for
network of decoder security.
9. Informative References
[AWCY] Xiph.Org, "Are We Compressed Yet?", 2016, [AWCY] Xiph.Org, "Are We Compressed Yet?", 2016,
<https://arewecompressedyet.com/>. <https://arewecompressedyet.com/>.
[BT500] ITU-R, "Recommendation ITU-R BT.500-13", 2012, [BT500] ITU-R, "Recommendation ITU-R BT.500-13", 2012,
<https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC- <https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-
BT.500-13-201201-I!!PDF-E.pdf>. BT.500-13-201201-I!!PDF-E.pdf>.
[CIEDE2000] [CIEDE2000]
Yang, Y., Ming, J., and N. Yu, "Color Image Quality Yang, Y., Ming, J., and N. Yu, "Color Image Quality
skipping to change at page 22, line 34 skipping to change at page 22, line 44
[COMPARECODECS] [COMPARECODECS]
Alvestrand, H., "Compare Codecs", 2015, Alvestrand, H., "Compare Codecs", 2015,
<http://compare-codecs.appspot.com/>. <http://compare-codecs.appspot.com/>.
[DAALA-GIT] [DAALA-GIT]
Xiph.Org, "Daala Git Repository", 2015, Xiph.Org, "Daala Git Repository", 2015,
<http://git.xiph.org/?p=daala.git;a=summary>. <http://git.xiph.org/?p=daala.git;a=summary>.
[I-D.ietf-netvc-requirements] [I-D.ietf-netvc-requirements]
Filippov, A. and A. Norkin, "<Video Codec Requirements and Filippov, A., Norkin, A., and j.
Evaluation Methodology>", draft-ietf-netvc-requirements-08 jose.roberto.alvarez@huawei.com, "Video Codec Requirements
(work in progress), May 2018. and Evaluation Methodology", draft-ietf-netvc-
requirements-10 (work in progress), November 2019.
[MSSSIM] Wang, Z., Simoncelli, E., and A. Bovik, "Multi-Scale [MSSSIM] Wang, Z., Simoncelli, E., and A. Bovik, "Multi-Scale
Structural Similarity for Image Quality Assessment", n.d., Structural Similarity for Image Quality Assessment", n.d.,
<http://www.cns.nyu.edu/~zwang/files/papers/msssim.pdf>. <http://www.cns.nyu.edu/~zwang/files/papers/msssim.pdf>.
[PSNRHVS] Egiazarian, K., Astola, J., Ponomarenko, N., Lukin, V., [PSNRHVS] Egiazarian, K., Astola, J., Ponomarenko, N., Lukin, V.,
Battisti, F., and M. Carli, "A New Full-Reference Quality Battisti, F., and M. Carli, "A New Full-Reference Quality
Metrics Based on HVS", 2002. Metrics Based on HVS", 2002.
[RD_TOOL] Xiph.Org, "rd_tool", 2016, <https://github.com/tdaede/ [RD_TOOL] Xiph.Org, "rd_tool", 2016,
rd_tool>. <https://github.com/tdaede/rd_tool>.
[SSIM] Wang, Z., Bovik, A., Sheikh, H., and E. Simoncelli, "Image [SSIM] Wang, Z., Bovik, A., Sheikh, H., and E. Simoncelli, "Image
Quality Assessment: From Error Visibility to Structural Quality Assessment: From Error Visibility to Structural
Similarity", 2004, Similarity", 2004,
<http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf>. <http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf>.
[TESTSEQUENCES] [TESTSEQUENCES]
Daede, T., "Test Sets", n.d., Daede, T., "Test Sets", n.d.,
<https://people.xiph.org/~tdaede/sets/>. <https://people.xiph.org/~tdaede/sets/>.
 End of changes. 25 change blocks. 
39 lines changed or deleted 55 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/