Skip to main content
SpringerLink
Log in

Fast CU patition based on image similarity using neural network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

High Efficiency Video Coding (HEVC) has introduced a quad-tree (QT) based coding unit (CU) partition structure, which has significantly improved the compression performance compared with Advanced Video Coding (AVC). However, the use of rate-distortion optimization (RDO) techniques in the search for the optimal CU partition has increased the encoding complexity of the video. In this paper, we propose a fast CU partitioning algorithm based on image similarity, which makes decisions on the partitioning of the parent CU by comparing the similarity of the image content of four sub-CUs. We propose four different neural networks based on this algorithm, and experimental results demonstrate that our proposed network structure reduces encoding time by 59.8%, 58.7%, 58.5%, and 59.3% respectively, while increasing the Bjøntegaard delta bit-rate (BDBR) by 2.32%, 1.99%, 1.82%, and 1.91%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The datasets generated during or analyzed during the current study are available from the corresponding author upon reasonable request.

References

  1. Wiegand T, Sullivan GJ, Bjontegaard G, Luthra A (2003) Overview of the h.264/avc video coding standard. IEEE Trans Circ Syst Vid Technol 13(7):560–576

  2. Sullivan GJ, Ohm JR, Han WJ, Wiegand T (2013) Overview of the high efficiency video coding (hevc) standard. IEEE Trans Circ Syst Vid Technol 22(12):1649–1668

    Article  Google Scholar 

  3. Menon VV, Amirpour H, Timmerer C, Ghanbari M (2021) Incept: Intra cu depth prediction for hevc. In: 2021 IEEE 23rd International workshop on multimedia signal processing (MMSP), pp 1–6

  4. Hou J, Li D, Li Z, Jiang X (2013) Fast cu size decision based on texture complexity for hevc intra coding. In: Proceedings 2013 International conference on mechatronic sciences, electric engineering and computer (MEC), pp 1096–1099

  5. Shen Y, Zhang S, Yang C (2014) Image texture based fast cu size selection algorithm for hevc intra coding. In: 2014 IEEE International conference on signal processing, communications and computing (ICSPCC), pp 363–367

  6. Shen L, Zhang Z, An P (2013) Fast cu size decision and mode decision algorithm for hevc intra coding. IEEE Trans Consum Electron 59(1):207–213

    Article  Google Scholar 

  7. Cho S, Kim M (2013) Fast cu splitting and pruning for suboptimal cu partitioning in hevc intra coding. IEEE Trans Circ Syst Vid Technol 23(9):1555–1564

    Article  Google Scholar 

  8. Shi H, Fan L, Chen H (2015) A fast cu size decision algorithm based on adaptive depth selection for hevc encoder. In: 2014 International conference on audio, language and image processing

  9. Kim N, Jeon S, Shim HJ, Jeon B, Lim S-C, Ko H (2016) Adaptive keypointbased cu depth decision for hevc intra coding. In: 2016 IEEE International symposium on broadband multimedia systems and broadcasting (BMSB), pp 1–3

  10. Zao L, Fan X, Ma S, Zhao D (2014) Fast intra-encoding algorithm for high efficiency video coding. Signal Process Image Commun 29(9):935–944

    Article  Google Scholar 

  11. Lee D, Jeong J (2017) Fast intra coding unit decision for high efficiency video coding based on statistical information. Signal Process Image Commun 55:121–129

    Article  Google Scholar 

  12. Li Z, Zhao Y, Dai Z, Rogeany K, Cen Y, Xiao Z, Yang W (2019) A fast cu partition method based on cu depth spatial correlation and rd cost characteristics for hevc intra coding. Signal Process Image Commun 75:141–146

    Article  Google Scholar 

  13. Zhang M, Lai D, Liu Z, An C (2019) A novel adaptive fast partition algorithm based on cu complexity analysis in hevc. Multimed Tools Appl 78:1035–1051

    Article  Google Scholar 

  14. Zhang Y, Li N, Kwong S, Jiang G, Zeng H (2019) Statistical early termination and early skip models for fast mode decision in hevc intra coding. ACM Trans Multimed Comput, Commun, Appl (TOMM) 15(3):1–23

  15. Ruiz D, VeliborAdzic Fernández-Escribano G, Kalva H, Cuenca P (2014) Fast partitioning algorithm for hevc intra frame coding using machine learning. In: Image processing (ICIP):2014 IEEE international conference on

  16. Hu N, Yang EH (2015) Fast mode selection for hevc intra-frame coding with entropy coding refinement based on a transparent composite model. IEEE Trans Circ Syst Vid Technol 25(9):1521–1532

    Article  Google Scholar 

  17. Hu Q, Shi Z, Zhang X, Gao Z (2016) Fast hevc intra mode decision based on logistic regression classification. In: 2016 IEEE International symposium on broadband multimedia systems and broadcasting (BMSB), pp 1–4

  18. Duanmu F, Ma Z, Wang Y (2016) Fast mode and partition decision using machine learning for intra-frame coding in hevc screen content coding extension. IEEE J Emerg Sel Top Circ Syst 6(4):517–531

    Article  Google Scholar 

  19. Zhang T, Sun MT, Zhao D, Gao W (2017) Fast intra-mode and cu size decision for hevc. IEEE Trans Circ Syst Vid Technol 27(8):1714–1726

    Article  Google Scholar 

  20. Liu X, Li Y, Liu D, Wang P, Yang LT (2019) An adaptive cu size decision algorithm for hevc intra prediction based on complexity classification using machine learning. IEEE Trans Circ Syst Vid Technol 29(1):144–155

    Article  Google Scholar 

  21. Kuang W, Chan Y-L, Tsang S-H, Siu W-C (2020) Online-learning-based bayesian decision rule for fast intra mode and cu partitioning algorithm in hevc screen content coding. IEEE Trans Image Process 29:170–185. https://doi.org/10.1109/TIP.2019.2924810

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  22. Zhang Y, Pan Z, Li N, Wang X, Jiang G, Kwong S (2018) Effective data driven coding unit size decision approaches for hevc intra coding. IEEE Trans Circ Syst Vid Technol 28(11):3208–3222

    Article  Google Scholar 

  23. Erabadda B, Mallikarachchi T, Kulupana G, Fernando A (2019) Content adaptive fast cu size selection for hevc intra-prediction. In: 2019 IEEE International conference on consumer electronics (ICCE)

  24. Werda I, Maraoui A, Sayadi FE, Masmoudi N (2022) Fast cu partition and intra mode prediction method for hevc. In: 2022 IEEE 9th International conference on sciences of electronics, technologies of information and telecommunications (SETIT)

  25. Amna M, Imen W, Nacir O, Ezahra SF (2022) Svm-based method to reduce hevc cu partition complexity. In: 2022 19th International multi-conference on systems, signals and devices (SSD), pp 480–484

  26. Amna M, Imen W, Bouaafia S, Sayadi FE (2022) Machine learning-based approaches to reduce hevc intra coding unit partition decision complexity. Multimed Tools Appl 81(2):2777–2802

    Article  Google Scholar 

  27. Feng Z, Liu P, Jia K, Duan K (2018) Fast intra ctu depth decision for hevc. IEEE Access 6:45262–45269

    Article  Google Scholar 

  28. Kim K, Ro WW (2018) Fast cu depth decision for hevc using neural networks. IEEE Trans Circ Syst Vid Technol 29(5):1462–1473

    Article  Google Scholar 

  29. Xu M, Li T, Wang Z, Deng X, Yang R, Guan Z (2018) Reducing complexity of hevc: A deep learning approach. IEEE Trans Image Process 27(10):5044–5059

    Article  ADS  MathSciNet  Google Scholar 

  30. Shi J, Gao C, Chen Z (2019) Asymmetric-kernel cnn based fast ctu partition for hevc intra coding. In: 2019 IEEE International symposium on circuits and systems (ISCAS)

  31. Wang Z, Li F (2021) Convolutional neural network based low complexity HEVC intra encoder. Multimed Tools Appl 80(2):2441–2460

    Article  Google Scholar 

  32. Amna M, Imen W, Sayadi FE (2021) Lenet5-based approach for fast intra coding. In: 2020 10th International symposium on signal, image, video and communications (ISIVC)

  33. Chen Z, Shi J, Li W (2020) Learned fast hevc intra coding. IEEE Trans Image Process 29:5431–5446

    Article  ADS  Google Scholar 

  34. Zhao R, Huang H, Zhang R, Jing X (2023) Fast hevc intra ctu partition algorithm based on lightweight cnn. In: Signal and information processing, networking and computers, pp 996–1003

  35. Zaki F, Mohamed AE, Sayed SG (2021) Ctunet: A deep learning-based framework for fast ctu partitioning of h265/hevc intra- coding. Ain Shams Eng J 12(2):1859–1866

    Article  Google Scholar 

  36. Hari P, Jadhav V, Rao BKNS (2022) Ctu partition for intra-mode hevc using convolutional neural network. In: 2022 IEEE International symposium on smart electronic systems (iSES), pp 548–551

  37. Imen W, Amna M, Fatma B, Ezahra SF, Masmoudi N (2022) Fast hevc intra-cu decision partition algorithm with modified lenet-5 and alexnet 16(7):1811–1819

  38. Zagoruyko S, Komodakis N (2015) Learning to compare image patches via convolutional neural networks. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 4353–4361

Download references

Acknowledgements

The work is supported by the National Key Research and Development Program of China(2022YFF0607000), National Natural Science Foundation of China (61871188), Guangdong Basic and Applied Basic Research Foundation (2023A1515010993), Guangdong Provincial Key Laboratory of Human Digital Twin (2022B1212010004), Guangzhou City Science and Technology Research Projects (2023B01J0011).

Author information

Authors and Affiliations

  1. Department, Guangzhou City University of Technology, Guangzhou, 510030, China

    Yinglie Cao

  2. Department, South China University of Technology, Guangzhou, 510641, China

    Wenjin Wu, Zhiheng Zhou, Haoqi Xu, Wanlin Yue & Shang Zhuge

Authors
  1. Yinglie Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenjin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhiheng Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haoqi Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wanlin Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shang Zhuge
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiheng Zhou.

Ethics declarations

Conflict of Interests

We declared there is no conflict of interest in terms of this submission.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A

 

Table 4 Some symbols and their description
Full size table
Table 5 Information of test set
Full size table

Appendix B different architecture with different sizes of CUs

Table 6 FIWS network architecture with different sizes of CUs
Full size table
Table 7 FF-CNN network architecture with different sizes of CUs
Full size table
Table 8 TSSN network architecture with different sizes of CUs
Full size table
Table 9 FF-TSSN network architecture with different sizes of CUs
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cao, Y., Wu, W., Zhou, Z. et al. Fast CU patition based on image similarity using neural network. Multimed Tools Appl 83, 33185–33205 (2024). https://doi.org/10.1007/s11042-023-16962-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16962-x

Keywords

Search

Navigation