Skip to main content
SpringerLink
Log in

Efficient coding unit classifier for HEVC screen content coding based on machine learning

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

The Video Coding Joint Collaboration team (JCT-VC) has been working on an emerging standard for screen content coding (SCC) as an extension of high efficiency video coding (HEVC) standard known as HEVC-SCC. The two powerful coding mechanisms used in HEVC-SCC are intra block copy (IBC) and palette coding (PLT). These techniques achieve the best coding efficiency at the expense of extremely high computational complexity. Therefore, we propose a new technique to minimize computational complexity by skipping undesired modes and retaining coding efficiency. A fast intra mode decision approach is suggested based on efficient CU classification. Our proposed solution depends on categorizing a CU as a natural content block (NCB) or a screen content block (SCB). Two classifiers are used for the classification process. The first one is a neural network (NN) classifier, and the other is an AdaBoost classifier, which depends on a boosted decision stump algorithm. The two classifiers predict the CU type individually and the final decision for CU classification depends on both of them. The experimental results reveal that the suggested technique significantly decreases encoding time without sacrificing coding efficiency. The suggested framework can achieve a 26.13% encoding time reduction on average with just a 0.81% increase in Bjontegaard Delta bit-rate (BD-Rate). Furthermore, the suggested framework saves encoding time by 51.5% on average for a set of NC sequences recommended for standard HEVC tests with minimal performance degradation. The proposed strategy has been merged with an existing methodology to accelerate the process even further.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Lu, Y., Li, S., Shen, H.: Virtualized screen: A third element for cloud-mobile convergence. IEEE Multimed. 18, 4–11 (2011)

    Article  Google Scholar 

  2. Liu, S., Xu, X., Lei, S., Jou, K.: Overview of HEVC extensions on screen content coding. APSIPA Trans. Signal Inf. Process. (2015). https://doi.org/10.1017/ATSIP.2015.11

    Article  Google Scholar 

  3. Xu, J., Joshi, R., Cohen, R.A.: Overview of the emerging HEVC screen content coding extension. IEEE Trans. Circuits Syst. Video Technol. 26, 50–62 (2016). https://doi.org/10.1109/TCSVT.2015.2478706

    Article  Google Scholar 

  4. Peng, W.H., Walls, F.G., Cohen, R.A., Xu, J., Ostermann, J., MacInnis, A., Lin, T.: Overview of screen content video coding: technologies, standards, and beyond. IEEE. J. Emerg. Sel. Top. Circuits Syst. 6, 393–408 (2016). https://doi.org/10.1109/JETCAS.2016.2608971

    Article  Google Scholar 

  5. Sullivan, G.J., Ohm, J.R., Han, W.J., Wiegand, T.: Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol. 22, 1649–1668 (2012). https://doi.org/10.1109/TCSVT.2012.2221191

    Article  Google Scholar 

  6. Budagavi, M., Kwon, D.K.: Intra motion compensation and entropy coding improvements for HEVC screen content coding. Pict. Coding Symp. (2013). https://doi.org/10.1109/PCS.2013.6737759

    Article  Google Scholar 

  7. Zhu, W., Ding, W., Xu, J., Shi, Y., Yin, B.: 2-D dictionary based video coding for screen contents. Data Compress. Conf. Proc. (2014). https://doi.org/10.1109/DCC.2014.11

    Article  Google Scholar 

  8. Tsang, S.H., Chan, Y.L., Siu, W.C.: Hash based fast local search for Intra Block Copy (IntraBC) mode in HEVC screen content coding. Asia-Pac. Signal Inf. Process. Assoc. Annu. Summit Conf. (2016). https://doi.org/10.1109/APSIPA.2015.7415302

    Article  Google Scholar 

  9. Xu, X., Liu, S., Chuang, T.D., Huang, Y.W., Lei, S.M., Rapaka, K., Pang, C., Seregin, V., Wang, Y.K., Karczewicz, M.: Intra block copy in HEVC screen content coding extensions. IEEE J. Emerg. Sel. Top. Circuits Syst. 6, 409–419 (2016). https://doi.org/10.1109/JETCAS.2016.2597645

    Article  Google Scholar 

  10. Xu, X., Liu, S., Chuang, T.D., Lei, S.: Block vector prediction for intra block copying in HEVC screen content coding. Data Compress. Conf. Proc. (2015). https://doi.org/10.1109/DCC.2015.22

    Article  Google Scholar 

  11. Guo, L., Pu, W., Zou, F., Sole, J., Karczewicz, M., Joshi, R.: Color palette for screen content coding. IEEE Int. Conf. Image Process. (2014). https://doi.org/10.1109/ICIP.2014.7026124

    Article  Google Scholar 

  12. Xiu, X., He, Y., Joshi, R., Karczewicz, M., Onno, P., Gisquet, C., Laroche, G.: Palette-based coding in the screen content coding extension of the HEVC standard. Data Compress. Conf. Proc (2015). https://doi.org/10.1109/DCC.2015.79

    Article  Google Scholar 

  13. Pu, W., Karczewicz, M., Joshi, R., Seregin, V., Zou, F., Sole, J., Sun, Y.C., Chuang, T.D., Lai, P., Liu, S., Hsiang, S.T., Ye, J., Huang, Y.W.: Palette mode coding in HEVC screen content coding extension. IEEE J. Emerg. Sel. Top. Circuits Syst. 6, 420–432 (2016). https://doi.org/10.1109/JETCAS.2016.2605661

    Article  Google Scholar 

  14. Kuang, W., Chan, Y.L., Tsang, S.H., Siu, W.C.: Machine learning-based fast intra mode decision for HEVC screen content coding via decision trees. IEEE Trans. Circuits Syst. Video Technol. 30, 1481–1496 (2020). https://doi.org/10.1109/TCSVT.2019.2903547

    Article  Google Scholar 

  15. Duanmu, F., Ma, Z., Wang, Y.: Fast mode and partition decision using machine learning for intra-frame coding in HEVC screen content coding extension. IEEE J. Emerg. Sel. Top. Circuits Syst. 6, 517–531 (2016). https://doi.org/10.1109/JETCAS.2016.2597698

    Article  Google Scholar 

  16. Huang, C., Peng, Z., Chen, F., Jiang, Q., Jiang, G., Hu, Q.: Efficient CU and PU decision based on neural network and gray level co-occurrence matrix for intra prediction of screen content coding. IEEE Access. 6, 46643–46655 (2018)

    Article  Google Scholar 

  17. Kuang, W., Chan, Y.L., Tsang, S.H., Siu, W.C.: DeepSCC: Deep learning-based fast prediction network for screen content coding. IEEE Trans. Circuits Syst. Video Technol. 30, 1917–1932 (2020)

    Article  Google Scholar 

  18. Laude, T., Ostermann, J.: Deep learning-based intra prediction mode decision for HEVC. Pict. Coding Symp. (2017). https://doi.org/10.1109/PCS.2016.7906399

    Article  Google Scholar 

  19. Tsang, S.H., Chan, Y.L., Kuang, W.: Mode skipping for HEVC screen content coding via random forest. IEEE Trans. Multimed. 21, 2433–2446 (2019). https://doi.org/10.1109/TMM.2019.2907472

    Article  Google Scholar 

  20. Yang, H., Shen, L., An, P.: Efficient screen content intra coding based on statistical learning. Signal Process. Image Commun. 62, 74–81 (2018). https://doi.org/10.1016/j.image.2017.12.001

    Article  Google Scholar 

  21. Lu, Y., Liu, H., Lin, Y., Shen, L., Yin, H.: Efficient coding mode and partition decision for screen content intra coding. Signal Process. Image Commun. 68, 249–257 (2018). https://doi.org/10.1016/j.image.2018.08.008

    Article  Google Scholar 

  22. Badry, E., Shalaby, A., Sayed, M.S.: Fast algorithm with palette mode skipping and splitting early termination for HEVC screen content coding. Midwest Symp. Circuits Syst. (2019). https://doi.org/10.1109/MWSCAS.2019.8885351

    Article  Google Scholar 

  23. Badry, E., Shalaby, A., Sayed, M.S.: Intra mode decision acceleration for HEVC screen content coding. Proc. Int. Japan-Africa Conf. Electron. Commun. Comput. (2019). https://doi.org/10.1109/JAC-ECC48896.2019.9051167

    Article  Google Scholar 

  24. Tsang, S.H., Chan, Y.L., Kuang, W., Siu, W.C.: Reduced-complexity intra block copy (IntraBC) mode with early CU splitting and pruning for HEVC screen content coding. IEEE Trans. Multimed. 21, 269–283 (2019). https://doi.org/10.1109/TMM.2018.2856078

    Article  Google Scholar 

  25. Zhang, H., Zhou, Q., Shi, N., Yang, F., Feng, X., Ma, Z.: Fast intra mode decision and block matching for HEVC screen content compression. IEEE Int. Conf. Acoust. Speech Signal Process. (2016). https://doi.org/10.1109/ICASSP.2016.7471902

    Article  Google Scholar 

  26. Tsang, S.H., Kuang, W., Chan, Y.L., Siu, W.C.: Fast HEVC screen content coding by skipping unnecessary checking of intra block copy mode based on CU activity and gradient. Asia-Pacific Signal Inf. Process. Assoc. Annu. Summit Conf. (2017). https://doi.org/10.1109/APSIPA.2016.7820900

    Article  Google Scholar 

  27. Lei, J., Li, D., Pan, Z., Sun, Z., Kwong, S., Hou, C.: Fast intra prediction based on content property analysis for low complexity HEVC-based screen content coding. IEEE Trans. Broadcast. 63, 48–58 (2017). https://doi.org/10.1109/TBC.2016.2623241

    Article  Google Scholar 

  28. Elsawy, N., Sayed, M.S., Farag, F.: Mode skipping for screen content coding based on neural network classifier. J Real-Time Image Proc 18, 2453–2468 (2021). https://doi.org/10.1007/s11554-021-01137-4

    Article  Google Scholar 

  29. Sullivan, G., Ohm, J.-R.: Common test conditions for screen content coding. document, JCTVC-U1015, Oct. (2015)

  30. HEVC test model version 16.18 screen content model version 8.7. [Online]. (2021). https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.18+SCM-8.7/. Accessed 14 May 2021.

  31. Liu, X., Li, Y., Liu, D., Wang, P., Yang, L.T.: An adaptive CU size decision algorithm for HEVC intra prediction based on complexity classification using machine learning. IEEE Trans. Circuits Syst. Video Technol. 29, 144–155 (2019)

    Article  Google Scholar 

  32. Schapire, R.E.: The boosting approach to machine learning: an overview. In: Denison, D.D., Hansen, M.H., Holmes, C.C., Mallick, B., Yu, B. (eds.) Nonlinear estimation and classification, pp. 149–171. Springer, New York (2003)

    Chapter  Google Scholar 

  33. Bakkouri, S., Elyousfi, A.: Machine learning-based fast CU size decision algorithm for 3D-HEVC inter-coding. J. Real-Time Image Process. (2021). https://doi.org/10.1007/s11554-020-01059-7

    Article  Google Scholar 

  34. Bjontegaard, G.: Calculation of average PSNE differences between R-D Curves, document VCEG-M33, ITU-T VCEG (2001)

  35. Kuang, W., Chan, Y.L., Tsang, S.H., Siu, W.C.: Fast HEVC to SCC transcoder by early CU partitioning termination and decision tree-based flexible mode decision for intra-frame coding. IEEE Access. 7, 8773–8788 (2019). https://doi.org/10.1109/ACCESS.2018.2890720

    Article  Google Scholar 

  36. Bossen, F.: Common Test Conditions. Doc. JCTVC-H1100 (2012)

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communications Engineering, Zagazig University, Zagazig, Egypt

    Nabila Elsawy, Mohammed S. Sayed & Fathi Farag

  2. Department of Electronics and Communications Engineering, Egypt-Japan University of Science and Technology, Borg El Arab, Egypt

    Mohammed S. Sayed

Authors
  1. Nabila Elsawy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammed S. Sayed
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fathi Farag
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nabila Elsawy.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Elsawy, N., Sayed, M.S. & Farag, F. Efficient coding unit classifier for HEVC screen content coding based on machine learning. J Real-Time Image Proc 19, 375–390 (2022). https://doi.org/10.1007/s11554-021-01189-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01189-6

Keywords

Search

Navigation