NETVC Working Group N. Egge Internet-Draft L. Trudeau Intended status: Informational Mozilla Expires: May 19, 2018 D. Barr Xiph.Org Foundation November 15, 2017 Chroma From Luma Intra Prediction for NETVC draft-egge-netvc-cfl-01 Abstract Chroma from luma (CfL) prediction is a new and promising chroma-only intra predictor that models chroma pixels as a linear function of the coincident reconstructed luma pixels. In this document, we propose the CfL predictor adopted in Alliance Video 1 (AV1) to the NETVC working group. The proposed CfL distinguishes itself from prior art not only by reducing decoder complexity, but also by producing more accurate predictions. On average, CfL reduces the BD-rate, when measured with CIEDE2000, by 5% for still images and 2% for video sequences. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on May 19, 2018. Copyright Notice Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of Egge, et al. Expires May 19, 2018 [Page 1] Internet-Draft cfl November 2017 publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. State of the Art in Chroma from Luma Prediction . . . . . . . 4 3. Model Fitting the "AC" Contribution . . . . . . . . . . . . . 5 4. Chroma "DC" Prediction for "DC" Contribution . . . . . . . . 6 5. Parameter Signaling . . . . . . . . . . . . . . . . . . . . . 7 6. Experimental Results . . . . . . . . . . . . . . . . . . . . 8 7. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 10 8. Informative References . . . . . . . . . . . . . . . . . . . 10 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12 1. Introduction Still image and video compression is typically not performed using red, green, and blue (RGB) color primaries, but rather with a color space that separates luma from chroma. There are many reasons for this, notably that luma and chroma are less correlated than RGB, which favors compression; and also that the human visual system is less sensitive to chroma allowing one to reduce the resolution in the chromatic planes, a technique know as chroma subsampling [Wang01]. Another way to improve compression in still images and videos is to subtract a predictor from the pixels. When this predictor is derived from previously reconstructed information inside the current frame, it is referred to as an intra prediction tool. In contrast, an inter prediction tool uses information from previously reconstructed frames. For example, "DC" prediction is an intra prediction tool that predicts the pixels values in a block by averaging the values of neighboring pixels adjacent to the above and left borders of the block [Li14]. Chroma from luma (CfL) prediction is a new and promising chroma-only intra predictor that models chroma pixels as a linear function of the coincident reconstructed luma pixels [Kim10]. It was proposed for the HEVC video coding standard [Chen11b], but was ultimately rejected, as the decoder model fitting caused a considerable complexity increase. More recently, CfL prediction was implemented in the Thor codec [Midtskogen16] as well as in the Daala codec [Egge15]. The Egge, et al. Expires May 19, 2018 [Page 2] Internet-Draft cfl November 2017 inherent conceptual differences in the Daala codec, when compared to HEVC, led to multiple innovative contributions by Egge and Valin [Egge15] to CfL prediction. Most notably a frequency domain implementation and the absence of decoder model fitting. As both Thor and Daala are part of NETVC working group, a research initiative was established regarding CfL, the results of which are presented in this draft. The proposed CfL implementation not only builds on the innovations of [Egge15], but does so in a way that is compatible with the more conventional compression tools found in Alliance Video 1 (AV1). The following table details the key differences between LM Mode [Chen11b], Thor CfL [Midtskogen16], and Daala CfL [Egge15] (the previous version of this draft): +-----------------------+---------+----------+-----------+---------+ | | LM Mode | Thor CfL | Daala CfL | AV1 CfL | +-----------------------+---------+----------+-----------+---------+ | Prediction Domain | Spatial | Spatial | Frequency | Spatial | | | | | | | | Bitsream Signaling | No | No | Sign bit | Signs | | | | | | | | | | | PVQ Gain | + Index | | | | | | | | Requires PVQ | No | No | Yes | No | | | | | | | | Encoder Model Fitting | Yes | Yes | Via PVQ | Search | | | | | | | | Decoder Model Fitting | Yes | Yes | No | No | +-----------------------+---------+----------+-----------+---------+ This new implementation is considerably different from its predecessors. Its key contributions are: o Parameter signaling, which avoids model fitting on the decoder and, as explained in Section 2, results in more precise predictions, as the chroma reference pixels are used for fitting (which is impossible when fitting on the decoder). The actual signaling is described in Section 5. o Model fitting the "AC" contribution of the reconstructed luma pixels, as shown in Section 3, which simplifies the model and allows for a more precise fit. o Chroma "DC" prediction for "DC" contribution, which requires no signaling and, as described in Section 4, is more precise. Finally, Section 6 presents detailed results of the compression gains of the proposed CfL prediction implementation in AV1. Egge, et al. Expires May 19, 2018 [Page 3] Internet-Draft cfl November 2017 2. State of the Art in Chroma from Luma Prediction As described in [Kim10], CfL prediction models chroma pixels as a linear function of the coincident reconstructed luma pixels. More precisely, Let L be an M X N matrix of pixels in the luma plane; we define C to be the chroma pixels spatially coincident to L. Since L is not available to the decoder, the reconstructed luma pixels, L^r, corresponding to L are used instead. The chroma pixel prediction, C^p, produced by CfL uses the following linear equation: C^p = alpha * L^r + beta Some implementations of CfL [Kim10], [Chen11b] and [Midtskogen16] determine the linear model parameters alpha and beta using linear least-squares regression ___ ___ ___ ___ ___ ___ \ \ \ \ \ \ alpha = (M*N) /__ /__ L^r(i,j)*C(i,j) - /__ /__ L^r(i,j) /__ /__ C(i,j) i=0 j=0 i=0 j=0 i=0 j=0 --------------------------------------------------------------- ___ ___ ___ ___ \ \ \ \ (M*N) /__ /__ (L^r(i,j))^2 - (/__ /__ L^r(i,j))^2 i=0 j=0 i=0 j=0 ___ ___ ___ ___ \ \ \ \ beta = /__ /__ C(i,j) - alpha /__ /__ L^r(i,j) i=0 j=0 i=0 j=0 --------------------------------------- (M*N) We classify [Kim10], [Chen11b], and [Midtskogen16] as implicit implementations of CfL, since alpha and beta are not signaled in the bitstream, but are implied from the bitstream. The main advantage of the implicit implementation is the absence of signaling. However, implicit implementations have numerous disadvantages. As mentioned before, computing least squares considerably increases decoder complexity. Another important disadvantage is that the chroma pixels, C, are not available when computing least squares on the decoder. As such, prediction error increases since neighboring reconstructed chroma pixels must be used instead. In [Egge15], the authors argue that the advantages of explicit signaling considerably outweigh the signaling cost. Based on these Egge, et al. Expires May 19, 2018 [Page 4] Internet-Draft cfl November 2017 findings, we propose a hybrid approach that signals alpha and implies beta. 3. Model Fitting the "AC" Contribution In [Egge15], Egge and Valin demonstrate the merits of separating the "DC" and "AC" contributions of the frequency domain CfL prediction. In the pixel domain, the "AC" contribution of a block can be obtained by subtracting it by its average. An important advantage of the "AC" contribution is that it is zero mean, which results in significant simplifications to the least squares model parameter equations. More precisely, let L^AC$ be the zero-meaned reconstructed luma pixels. Because ___ ___ \ \ /__ /__ L_AC(i,j) = 0 i=0 j=0 substituting L^r by L_AC yields the following simplified model parameters equations: ___ ___ \ \ alpha_AC = /__ /__ L_AC(i,j)*C(i,j) i=0 j=0 ------------------------ ___ ___ \ \ /__ /__ (L^r(i,j))^2 i=0 j=0 ___ ___ \ \ beta_AC = /__ /__ C(i,j) i=0 j=0 -------------- (M*N) We define the zero-mean chroma prediction, C_AC, like so C_AC = alpha_AC * L_AC + beta_AC When computing the zero-mean reconstructed pixels, the resulting values are stored using 1/8th precision fixed-point values. This ensures that even with 12-bit integer pixels, the average can be stored in a 16-bit signed integer. Egge, et al. Expires May 19, 2018 [Page 5] Internet-Draft cfl November 2017 By combining the luma subsampling step with the average subtraction step not only do the equations simplify, but the subsampling divisions and the corresponding rounding error are removed. The equation corresponding to the combination of both steps simplifies to: __sy-1__sx-1 \ \ L_AC(i,j) = 8 (/__ /__L^r(sy*i+y,sx*j+x)) ----- y=0 x=0 sy*sx ___ ___ __sy-1__sx-1 \ \ \ \ - /__ /__ 8 (/__ /__L^r(sy*i+y,sx*j+x)) i=0 j=0 ----- y=0 x=0 sy*sx ------------------------------------------- (M*N) Note that this equation uses an integer division. In the previous equation, sx and sy are the subsampling steps for the x and y axes, respectively. The proposed CfL only supports 4:2:0, 4:2:2, 4:4:0 and 4:4:4 chroma subsamplings [Wang01], for which: sy*sx in {1, 2, 4}. Also, because both M and N are powers of two, M * N is also a power of two. It follows that the previous integer divisions can be replaced by bit shift operations. 4. Chroma "DC" Prediction for "DC" Contribution Switching the linear model to use zero mean reconstructed luma pixels also changes beta_AC, to the extent that it now only depends on C. More precisely, beta_AC is the average of the chroma pixels. The chroma pixel average for a given block is not available in the decoder. However, there already exists an intra prediction tool that predicts this average. When applied to the chroma plane, the "DC" prediction predicts the pixel values in a block by averaging the values of neighboring pixels adjacent to the above and left borders of the block [Li14]. Concretely, the output of the chroma "DC" predictor can be injected inside the proposed CfL implementation as an approximation for beta_AC. Egge, et al. Expires May 19, 2018 [Page 6] Internet-Draft cfl November 2017 The proposed CfL prediction is expressed as follows: CfL(alpha) = alpha * L_AC + DC_PRED. 5. Parameter Signaling Signaling the scaling parameters allows encoder-only fitting of the linear model. This reduces decoder complexity and results in a more precise prediction, as the best scaling parameter can be determined based on the reference chroma pixels which are only available to the encoder. The scaling parameters for both chromatic planes are jointly coded using the following scheme. First, we signal the joint sign of both scaling parameters. A sign is either negative, zero, or positive. In the proposed scheme, signaling (zero, zero) is not permitted as it results in "DC" prediction. It follows that the joint sign requires an eight-value symbol. As for each scaling parameter, a 16-value symbol is used to represent values ranging from 0 to 2 with a step of 1/8th. The entropy coding details are beyond the scope of this document; however, it is important to note that a 16-value symbol fully utilizes the capabilities of the multi-symbol entropy encoder [Valin16]. Finally, scaling parameters are signaled only if they are non-zero. Signaling the scaling parameters fundamentally changes their selection. In this context, the least-squares regression used in [Kim10], [Chen11b], and [Midtskogen16] does not yield an RD-optimal solution as it ignores the trade-off between the rate and the distortion of the scaling parameters. For the proposed CfL prediction, the scaling parameter is determined using the same rate-distortion optimization mechanics as other coding tools and parameters of AV1. Concretely, given a set of scaling parameters A, the selected scaling parameter is the one that minimizes the trade-off between the rate and the distortion alpha = argmin ( D(CfL(a)) + lambda R(a) ). a in A In the previous equation, the distortion, D, is the sum of the squared error between the reconstructed chroma pixels and the reference chroma pixels. Whereas, the rate, R, is the number of bits required to encode the scaling parameter and the residual coefficients. Furthermore, lambda is the weighing coefficient between rate and distortion used by AV1. Egge, et al. Expires May 19, 2018 [Page 7] Internet-Draft cfl November 2017 6. Experimental Results To ensure a valid evaluation of coding efficiency gains, our testing methodology conforms to that of [Daede17]. All simulation parameters and a detailed sequence-by-sequence breakdown for all the results presented in this paper are available online at [AWCY]. Furthermore, the bitstreams generated in these simulations can be retrieved and analyzed online at [Analyzer]. The following tables show the average percent rate difference measured using the Bjontegaard rate difference, also known as BD- rate [Bjontegaard01]. The BD-rate is measured using the following objective metrics: PSNR, PSNR-HVS [Egiazarian2006], SSIM [Wang04], CIEDE2000 [Yang12] and MSSIM [Wang03]. Of all the previous metrics, only the CIEDE2000 considers both luma and chroma planes. It is also important to note that the distance measured by this metric is perceptually uniform [Yang12]. As required in [Daede17], for individual feature changes in libaom, we use quantizers: 20, 32, 43, and 55. We present results for three test sets: Objective-1-fast [Daede17], Subset1 [Testset] and Twitch [Testset]. In the following table, we present the results for the Subset1 test set[AWCYSubset1]. This test set contains still images, which are ideal to evaluate the chroma intra prediction gains of CfL when compared to other intra prediction tools in AV1. +---------+-------+--------+--------+-------+-------+-------+-------+ | | PSNR | PSNR | PSNR | PSNR | SSIM | MS | CIEDE | | | | Cb | Cr | HVS | | SSIM | 2000 | +---------+-------+--------+--------+-------+-------+-------+-------+ | Average | -0.53 | -12.87 | -10.75 | -0.31 | -0.34 | -0.34 | -4.87 | +---------+-------+--------+--------+-------+-------+-------+-------+ For still images, when compared to all of the other intra prediction tools of AV1 combined, CfL prediction reduces the rate by an average of 5% for the same level of visual quality measured by CIEDE2000. For video sequences, next table breaks down the results obtained over the objective-1-fast test set [AWCYObjective1]. Egge, et al. Expires May 19, 2018 [Page 8] Internet-Draft cfl November 2017 +---------+-------+--------+--------+-------+-------+-------+-------+ | | PSNR | PSNR | PSNR | PSNR | SSIM | MS | CIEDE | | | | Cb | Cr | HVS | | SSIM | 2000 | +---------+-------+--------+--------+-------+-------+-------+-------+ | Average | -0.43 | -5.85 | -5.51 | -0.42 | -0.38 | -0.40 | -2.41 | | | | | | | | | | | 1080p | -0.32 | -6.80 | -5.31 | -0.37 | -0.28 | -0.31 | -2.52 | | | | | | | | | | | 1080psc | -1.82 | -17.76 | -12.00 | -1.72 | -1.71 | -1.75 | -8.22 | | | | | | | | | | | 360p | -0.15 | -2.17 | -6.45 | -0.05 | -0.10 | -0.04 | -0.80 | | | | | | | | | | | 720p | -0.12 | -1.08 | -1.23 | -0.11 | -0.07 | -0.12 | -0.52 | +---------+-------+--------+--------+-------+-------+-------+-------+ Not only does CfL yield better intra frames, which produces a better reference for inter prediction tools, but it also improves chroma intra prediction in inter frames. We observed CfL predictions in inter frames when the predicted content was not available in the reference frames. As such, CfL prediction reduces the rate of video sequences by an average of 2% for the same level of visual quality when measured with CIEDE2000. The average rate reductions for 1080psc are considerably higher than those of other types of content. This indicates that CfL prediction considerably outperforms other AV1 predictors for screen content coding. As shown in the following table, the results on the Twitch test set [AWCYTwitch], which contains only gaming-based screen content, corroborates this finding. +---------+-------+--------+-------+-------+-------+-------+--------+ | | PSNR | PSNR | PSNR | PSNR | SSIM | MS | CIEDE | | | | Cb | Cr | HVS | | SSIM | 2000 | +---------+-------+--------+-------+-------+-------+-------+--------+ | Average | -1.01 | -15.58 | -9.96 | -0.93 | -0.90 | -0.81 | -5.74 | +---------+-------+--------+-------+-------+-------+-------+--------+ Furthermore, individual sequences in the Twitch test set show considerable gains. We present the results for Minecraft_10_120f (Mine), GTAV_0_120F (GTAV), and Starcraft_10_120f (Star) in the following table. It would appear that CfL prediction is particularly efficient for sequences of the game Minecraft both sequences reduces the average rate by 20% for the same level of visual quality measured by CIEDE2000. Egge, et al. Expires May 19, 2018 [Page 9] Internet-Draft cfl November 2017 +------+-------+--------+--------+-------+-------+--------+---------+ | | PSNR | PSNR | PSNR | PSNR | SSIM | MS | CIEDE | | | | Cb | Cr | HVS | | SSIM | 2000 | +------+-------+--------+--------+-------+-------+--------+---------+ | Mine | -3.76 | -31.44 | -25.54 | -3.13 | -3.68 | -3.28 | -20.69 | | | | | | | | | | | GTAV | -1.11 | -15.39 | -5.57 | -1.11 | -1.01 | -1.04 | -5.88 | | | | | | | | | | | Star | -1.41 | -6.18 | -6.21 | -1.43 | -1.38 | -1.43 | -4.15 | +------+-------+--------+--------+-------+-------+--------+---------+ 7. Conclusion In this document, we presented the chroma from luma prediction tool adopted in AV1 that we proposed for NETVC. This new implementation is considerably different from its predecessors. Its key contributions are: parameter signaling, model fitting the "AC" contribution of the reconstructed luma pixels, and chroma "DC" prediction for "DC" contribution. Not only do these contributions reduce decoder complexity, but they also reduce prediction error; resulting in a 5% average reduction in BD-rate, when measured with CIEDE2000, for still images, and 2% for video sequences. Possible improvements to CfL for AV2 include non-linear prediction models and motion-compensated CfL. 8. Informative References [Analyzer] Bebenita, M., "AV1 Bitstream Analyzer", Mozilla https://arewecompressedyet.com/analyzer/, n.d.. [AWCY] "Are We Compressed Yet?", Xiph.Org Foundation https://arewecompressedyet.com, n.d.. [AWCYObjective1] Trudeau, L., "Results of Chroma from Luma over the Objective-1-fast test set", Are We Compressed Yet? https://doi.org/10.6084/m9.figshare.5577778.v1, November 2017. [AWCYSubset1] Trudeau, L., "Results of Chroma from Luma over the Subset1 test set", Are We Compressed Yet? https://doi.org/10.6084/m9.figshare.5577661.v2, November 2017. Egge, et al. Expires May 19, 2018 [Page 10] Internet-Draft cfl November 2017 [AWCYTwitch] Trudeau, L., "Results of Chroma from Luma over the twitch test set", Are We Compressed Yet? https://doi.org/10.6084/m9.figshare.5577946.v1, November 2017. [Bjontegaard01] Bjontegaard, G., "Calculation of average PSNR differences between RD-curves", Video Coding Experts Group (VCEG) of ITU-T VCEG-M33, 2001. [Chen11b] Chen, J., Seregin, V., Han, W., Kim, J., and B. Jeon, "CE6.a.4: Chroma intra prediction by reconstructed luma samples", Joint Collaborative Team on Video Coding (JCT- VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JCTVC- E266, March 2011. [Daede17] Daede, T., Norkin, A., and I. Brailovsky, "Video Codec Testing and Quality Measurement", IETF NETVC Internet- Draft draft-ietf-netvc-testing-05, March 2017. [Egge15] Egge, N. and J. Valin, "Predicting chroma from luma with frequency domain intra prediction", Proceedings of SPIE 9410, Visual Information Processing and Communication VI, March 2015. [Egiazarian2006] Egiazarian, K., Astola, J., Ponomarenko, N., Lukin, V., Battisti, F., and M. Carli, "Two new full-reference quality metrics based on HVS", Proceedings of the Second International Workshop on Video Processing and Quality Metrics for Consumer Electronics VPQM, January 2006. [Kim10] Kim, J., Park, S., Choi, Y., Jeon, Y., and B. Jeon, "New intra chroma prediction using inter-channel correlation", Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, JCTVC-B021, January 2010. [Li14] Ze-Nian, L., Drew, M., and J. Liu, "Fundamentals of Multimedia", ISBN 3319052896, org Springer Publishing Company, Incorporated, edition 2nd, 2014. [Midtskogen16] Midtskogen, S., "Improved chroma prediction", draft- midtskogen-netvc-chromapred-02 IETF NETVC Internet-Draft, October 2016. Egge, et al. Expires May 19, 2018 [Page 11] Internet-Draft cfl November 2017 [Testset] Daede, T., "Test Sets", Hosted by the Xiph.org Foundation https://people.xiph.org/~tdaede/sets/, n.d.. [Valin16] Valin, J., Terriberry, T., Egge, N., Daede, T., Cho, Y., Montgomery, C., and M. Bebenita, "Daala: Building A Next- Generation Video Codec From Unconventional Technology", Multimedia signal processing (MMSP) workshop arXiv:1608.01947, September 2016. [Wang01] Wang, Y., Zhang, Y., and J. Ostermann, "Video Processing and Communications", ISBN 23132985, Prentice Hall PTR, Upper Saddle River, NJ, USA, edition 1st, 2001. [Wang03] Wang, Z., Simoncelli, E., and A. Bovik, "Multiscale structural similarity for image quality assessment", The 37th Asilomar Conference on Signals, Systems Computers Volume 2, November 2003. [Wang04] Wang, Z., Bovik, A., Sheikh, H., and E. Simoncelli, "Image Quality Assessment: From Error Visibility to Structural Similarity", issn 1057-7149, IEEE transactions on image processing Volume 13, number 4, April 2004. [Yang12] Yang, Y., Ming, J., and N. Yu, "Color Image Quality Assessment Based on CIEDE2000", Advances in multimedia Article ID 273723, 2012. Authors' Addresses Nathan E. Egge Mozilla 331 E Evelyn Ave Mountain View 94041 USA Email: negge@mozilla.com Luc N. Trudeau Mozilla 331 E Evelyn Ave Mountain View 94041 USA Email: luc@trud.ca Egge, et al. Expires May 19, 2018 [Page 12] Internet-Draft cfl November 2017 David M. Barr Xiph.Org Foundation 21 College Hill Road Somerville, MA 1124 USA Email: b@rr-dav.id.au Egge, et al. Expires May 19, 2018 [Page 13]