Skip to main content
SpringerLink
Log in

Fast multi-type tree partitioning for versatile video coding using machine learning

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The emerging versatile video coding (VVC) standard adopted an innovated multi-type tree (MTT) versatile block structure, comprising binary trees (BTs) and ternary trees (TTs) pruning. This new tree structures’ flexibility, induced by the MTT module, significantly improved the compression performance. However, it dramatically increased the coding complexity due to the brute force search for rate distortion optimization (RDO). To cope with this issue, we proposed a fast decision approach using a lightweight neural network (LNN) with an early direction determination scheme to avoid redundant MTT pruning and hence, reduced considerable computing complexity. The I-frame processing significantly affected the coding efficiency. Thus, the main goal of the suggested LNN-based approach is to substitute the brute force RDO search, used to check all block decision candidates, without affecting the compression efficiency performance. Based on the BT RD cost, the TT splitting direction was selected in a first step. Subsequently, an adequate LNN-based model was applied to predict the corresponding VVC TT partition, which deeply optimized the VVC coding unit partition module. Experiments over various test sequences showed that the proposed method substantially decreased the total encoding time by up to 46% with negligible compression efficiency loss under the all-intra configuration.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Gary, J.S., Jens-Rainer, O., Woo-Jin, H., Thomas, W.: Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol. 22, 1649–1668 (2012)

    Article  Google Scholar 

  2. Bossen, F., Li, X., Suehring, K.: AHG Report: Test Model Software Development (AHG3), JVET-J0003 (2018)

  3. Amna, M., Imen, W., Ezahra, S.F., et al.: Fast intra-coding unit partition decision in H. 266/FVC based on deep learning. J. Real Time Image Process. 17(6), 1971–1981 (2020)

    Article  Google Scholar 

  4. Jin, Z., An, P., Yang, C., et al.: Fast QTBT partition algorithm for intra frame coding through convolutional neural network. IEEE Access 6, 54660–54673 (2018)

    Article  Google Scholar 

  5. Yang, H., Shen, L., Dong, X., et al.: Low-complexity CTU partition structure decision and fast intra mode decision for versatile video coding. IEEE Trans. Circuits Syst. Video Technol. 30(6), 1668–1682 (2019)

    Article  Google Scholar 

  6. Cho, S., Kim, M.: Fast CU splitting and pruning for suboptimal CU partitioning in HEVC intra coding. IEEE Trans. Circuits Syst. Video Technol. 23(9), 1555–1564 (2013)

    Article  Google Scholar 

  7. Lee, D., Jeong, J.: Fast intra coding unit decision for high efficiency video coding based on statistical information. Signal Process. Image Commun. 55, 121–129 (2017)

    Article  Google Scholar 

  8. Sun, X., Chen, X., Xu, Y., et al.: Fast CU partition strategy for HEVC based on Haar wavelet. IET Image Process. 11(9), 717–723 (2017)

    Article  Google Scholar 

  9. Zhang, M., Lai, D., Liu, Z., et al.: A novel adaptive fast partition algorithm based on CU complexity analysis in HEVC. Multimed. Tools Appl. 78(1), 1035–1051 (2019)

    Article  Google Scholar 

  10. Huang, C., Peng, Z., Chen, F., et al.: Efficient CU and PU decision based on neural network and gray level co-occurrence matrix for intra prediction of screen content coding. IEEE Access 6, 46643–46655 (2018)

    Article  Google Scholar 

  11. Kim, K., Ro, W.W.: Fast CU depth decision for HEVC using neural networks. IEEE Trans. Circuits Syst. Video Technol. 29(5), 1462–1473 (2018)

    Article  Google Scholar 

  12. Amna, M., Imen, W., Ezahra, S.F.: Lenet5-based approach for fast intra coding. In: 2020 10th International Symposium on Signal, Image, Video and Communications (ISIVC), pp. 1–4. IEEE (2021)

  13. Xu, M., Li, T., Wang, Z., et al.: Reducing complexity of HEVC: a deep learning approach. IEEE Trans. Image Process. 27(10), 5044–5059 (2018)

    Article  Google Scholar 

  14. Amna, M., Imen, W., Soulef, B., Fatma, E.Z.: Machine Learning-Based approaches to reduce HEVC intra coding unit partition decision complexity. Multimed. Tools Appl. 81, 1–26 (2021)

    Google Scholar 

  15. Wang, Z., Wang, S., Zhang, J., et al.: Probabilistic decision based block partitioning for future video coding. IEEE Trans. Image Process. 27(3), 1475–1486 (2017)

    Article  MATH  Google Scholar 

  16. Ryu, S., Kang, J.: Machine learning-based fast angular prediction mode decision technique in video coding. IEEE Trans. Image Process. 27(11), 5525–5538 (2018)

    Article  Google Scholar 

  17. Lin, T.L., Jiang, H.Y., Huang, J.Y., et al.: Fast intra coding unit partition decision in H. 266/FVC based on spatial features. J. Real Time Image Process. 17(3), 493–510 (2020)

    Article  Google Scholar 

  18. Park, S.H., Kang, J.: Fast multi-type tree partitioning for versatile video coding using a lightweight neural network. IEEE Trans. Multimed. 23, 4388–4399 (2020)

    Article  Google Scholar 

  19. Park, S.H., Kang, J.W.: Context-based ternary tree decision method in versatile video coding for fast intra coding. IEEE Access 7, 172597–172605 (2019)

    Article  Google Scholar 

  20. Chen, F., Ren, Y., Peng, Z., et al.: A fast CU size decision algorithm for VVC intra prediction based on support vector machine. Multimed. Tools Appl. 79(37), 27923–27939 (2020)

    Article  Google Scholar 

  21. VVC Test Model (VTM) Software. Accessed: 24 July 2019. Available: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/

  22. Frank, B., Jill, B., Karsten, S., Xiang, L., Vadim, S.: JVET Common Test Conditions and Software Reference Configurations for SDR Video, Document JVET-M1010, Joint Video Experts Team (JVET) of ITU-T and ISO/IEC (2019)

  23. Fernández, D.G., Del Barrio, A.A., Botella, G., Meyer-Baese,U., Meyer-Baese, A., Grecos, C.: Information fusion based techniques for HEVC. In: Real-Time Image and Video Processing 2017, vol. 10223, p. 102230M (2017)

Download references

Author information

Authors and Affiliations

  1. Electronics and Microelectronics Laboratory, University of Monastir, Monastir, Tunisia

    Maraoui Amna

  2. Electronic and Information Technology Laboratory, University of Sfax, Sfax, Tunisia

    Werda Imen

  3. Networked Objects, Control and Communication Systems, University of Sousse, Sousse, Tunisia

    Sayadi Fatma Ezahra

Authors
  1. Maraoui Amna
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Werda Imen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sayadi Fatma Ezahra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maraoui Amna.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Amna, M., Imen, W. & Fatma Ezahra, S. Fast multi-type tree partitioning for versatile video coding using machine learning. SIViP 17, 67–74 (2023). https://doi.org/10.1007/s11760-022-02204-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02204-4

Keywords

Search

Navigation