Skip to main content
SpringerLink
Log in

A frame-level MLP-based bit-rate controller for real-time video transmission using VVC standard

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Real-time video transmission is one of the most popular applications that are included in the versatile video coding (VVC) standard. However, real-time applications are encountered with practical limitations, including the buffer size and available bandwidth. In these applications, the buffer overflow and underflow should be strictly prevented and also the bit-rate fluctuation should be suppressed. In this paper, a video bit-rate controller is proposed that completely conforms with the constraints of real-time applications. The proposed controller is based on a multi-layer perceptron (MLP) neural network which estimates the proper quantization parameter (QP) modification at the frame level. The buffer occupancy is directly included in the QP derivation process for robust buffer control. Experimental results show that the proposed bit-rate controller fulfils the buffering constraints and controls the bit-rate accurately. The average bit-rate error of the proposed method is 0.29% while providing a low initial buffering delay of about 0.21 s. Also, the rate-distortion analysis shows that the performance of the proposed method is close to those of the conventional algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Humberto Ochoa Domınguez, K.R.R.: Versatile video coding latest advances in video coding standards. In: Signal, Image And Speech Processing, River, Netherlands (2019)

  2. Sullivan, G.J.: Video coding standards progress report: joint video experts team launches the versatile video coding project. SMPTE Motion Imaging J. 127(8), 94–98 (2018)

    Article  Google Scholar 

  3. Chen, J., Karczewicz, M., Huang, Y.-W., Choi, K., Ohm, J.-R., Sullivan, G.J.: The joint exploration model (JEM) for video compression with capability beyond HEVC. IEEE Trans. Circuits Syst. Video Technol. 30, 1208–1225 (2019)

    Article  Google Scholar 

  4. Bross, B.: General video coding technology in responses to the joint call for proposals on video compression with capability beyond HEVC. IEEE Trans. Circuits Syst. Video Technol. 30, 1226–1240 (2019)

    Article  Google Scholar 

  5. Joginipelly, A., Varela, A., Charalampidis, D., Schott, R., Fitzsimmons, Z.: Efficient FPGA implementation of steerable Gaussian smoothers. In: Proceedings of the 2012 44th Southeastern Symposium on System Theory (SSST) 2012, pp. 78–82. IEEE.

  6. Joginipelly, A.K., Charalampidis, D.: Efficient separable convolution using field programmable gate arrays. Microprocess. Microsyst. 71, 102852 (2019)

    Article  Google Scholar 

  7. Choi, H., Yoo, J., Nam, J., Sim, D., Bajić, I.V.: Pixel-wise unified rate-quantization model for multi-level rate control. IEEE J. Sel. Top. Signal Process. 7(6), 1112–1123 (2013)

    Article  Google Scholar 

  8. Lee, B., Kim, M., Nguyen, T.Q.: A frame-level rate control scheme based on texture and nontexture rate models for high efficiency video coding. IEEE Trans. Circuits Syst. Video Technol. 24(3), 465–479 (2013)

    Article  Google Scholar 

  9. Seo, C.-W., Moon, J.-H., Han, J.-K.: Rate control for consistent objective quality in high efficiency video coding. IEEE Trans. Image Process. 22(6), 2442–2454 (2013)

    Article  MathSciNet  Google Scholar 

  10. Wang, S., Ma, S., Wang, S., Zhao, D., Gao, W.: Rate-GOP based rate control for high efficiency video coding. IEEE J. Sel. Top. Signal Process. 7(6), 1101–1111 (2013)

    Article  Google Scholar 

  11. Yan, T., Ra, I.-H., Wen, H., Weng, M.-H., Zhang, Q., Che, Y.: CTU layer rate control algorithm in scene change video for free-viewpoint video. IEEE Access 8, 24549–24560 (2020)

    Article  Google Scholar 

  12. Li, B., Li, H., Li, L., Zhang, J.: λ-Domain rate control algorithm for high efficiency video coding. IEEE Trans. Image Process. 23(9), 3841–3854 (2014)

    Article  MathSciNet  Google Scholar 

  13. Li, L., Li, B., Liu, D., Li, H.: λ-Domain rate control algorithm for HEVC scalable extension. IEEE Trans. Multimed. 18(10), 2023–2039 (2016)

    Article  Google Scholar 

  14. Li, L., Li, B., Li, H., Chen, C.W.: λ-Domain optimal bit allocation algorithm for high efficiency video coding. IEEE Trans. Circuits Syst. Video Technol. 28(1), 130–142 (2016)

    Article  Google Scholar 

  15. Wang, M., Ngan, K.N., Li, H.: Low-delay rate control for consistent quality using distortion-based Lagrange multiplier. IEEE Trans. Image Process. 25(7), 2943–2955 (2016)

    Article  MathSciNet  Google Scholar 

  16. Chen, Z., Pan, X.: An optimized rate control for low-delay H. 265/HEVC. IEEE Trans. Image Process. 28(9), 4541–4552 (2019)

    Article  MathSciNet  Google Scholar 

  17. Guo, H., Zhu, C., Xu, M., Li, S.: Inter-block dependency-based CTU level rate control for HEVC. IEEE Trans. Broadcast. 66, 113–126 (2020)

    Article  Google Scholar 

  18. He, J., Yang, F.: Efficient frame-level bit allocation algorithm for H. 265/HEVC. IET Image Process. 11(4), 245–257 (2017)

    Article  Google Scholar 

  19. Liu, M., Ren, P., Xiang, Z.: Frame-level bit allocation for hierarchical coding of H. 265/HEVC considering dependent rate-distortion characteristics. Signal Image Video Process. 10(8), 1457–1463 (2016)

    Article  Google Scholar 

  20. Li, S., Xu, M., Wang, Z., Sun, X.: Optimal bit allocation for CTU level rate control in HEVC. IEEE Trans. Circuits Syst. Video Technol. 27(11), 2409–2424 (2016)

    Article  Google Scholar 

  21. Wang, P., Ni, C., Li, Z., Zhang, G.: Optimal CTU-level bit allocation in HEVC for low bit-rate applications. Multimed. Tools Appl. 78, 23733–23747 (2019)

    Article  Google Scholar 

  22. Zeng, H., Yang, A., Ngan, K.N., Wang, M.: Perceptual sensitivity-based rate control method for high efficiency video coding. Multimed. Tools Appl. 75(17), 10383–10396 (2016)

    Article  Google Scholar 

  23. Zhou, M., Wei, X., Wang, S., Kwong, S., Fong, C.-K., Wong, P.H., Yuen, W.Y., Gao, W.: SSIM-based global optimization for CTU-level rate control in HEVC. IEEE Trans. Multimed. 21(8), 1921–1933 (2019)

    Article  Google Scholar 

  24. Lim, W., Sim, D.: A perceptual rate control algorithm based on luminance adaptation for HEVC encoders. Signal Image Video Process. 14, 887–895 (2020)

    Article  Google Scholar 

  25. Li, S., Xu, M., Deng, X., Wang, Z.: Weight-based R-λ rate control for perceptual HEVC coding on conversational videos. Signal Process. Image Commun. 38, 127–140 (2015)

    Article  Google Scholar 

  26. Zhu, L., Wang, G., Teng, G., Yang, Z., Zhang, L.: A deep learning based perceptual bit allocation scheme on conversational videos for HEVC λ-domain rate control. In: International Forum on Digital TV and Wireless Multimedia Communications 2017, pp. 515–524. Springer

  27. Gao, W., Kwong, S., Jia, Y.: Joint machine learning and game theory for rate control in high efficiency video coding. IEEE Trans. Image Process. 26(12), 6074–6089 (2017)

    Article  MathSciNet  Google Scholar 

  28. Gao, W., Kwong, S., Zhou, Y., Yuan, H.: SSIM-based game theory approach for rate-distortion optimized intra frame CTU-level bit allocation. IEEE Trans. Multimed. 18(6), 988–999 (2016)

    Article  Google Scholar 

  29. Gao, W., Kwong, S., Jiang, Q., Fong, C.-K., Wong, P.H., Yuen, W.Y.: Data-driven rate control for rate-distortion optimization in HEVC Based on simplified effective initial QP learning. IEEE Trans. Broadcast. 65(1), 94–108 (2018)

    Article  Google Scholar 

  30. Zupancic, I., Naccari, M., Mrak, M., Izquierdo, E.: Two-pass rate control for improved quality of experience in UHDTV delivery. IEEE J. Sel. Top. Signal Process. 11(1), 167–179 (2016)

    Article  Google Scholar 

  31. Wang, S., Rehman, A., Zeng, K., Wang, J., Wang, Z.: SSIM-motivated two-pass VBR coding for HEVC. IEEE Trans. Circuits Syst. Video Technol. 27(10), 2189–2203 (2016)

    Article  Google Scholar 

  32. Wien, M.: High efficiency video coding. Coding tools and specification. Berlin (2015)

  33. Fani, D., Rezaei, M.: Novel PID-fuzzy video rate controller for high-delay applications of the HEVC standard. IEEE Trans. Circuits Syst. Video Technol. 28(6), 1379–1389 (2017)

    Article  Google Scholar 

  34. Shojaei, M., Rezaei, M.: FJND-based fuzzy rate control of scalable video for streaming applications. Multimed. Tools Appl. 79, 13753–13773 (2020)

    Article  Google Scholar 

  35. Yiming Li, Z.C.: Rate control for VVC JVET-K0390(ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11) (J2018)

  36. Y.T. Peng, A.M.T.: Closed GOP support for the random access common conditions. JVET-D0135 (of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11) (2016)

  37. Demuth, H.B., Beale, M.H., De Jess, O., Hagan, M.T.: Neural network design. Martin Hagan, New York (2014)

    Google Scholar 

  38. Suehring, K.: JVET common test conditions and software reference configurations. JVET-B1010(ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11) (2016).

  39. VTM-1.0 reference software. https://jvet.hhi.fraunhofer.de/svn/svn_VVCSoftware_VTM/tags/VTM-1.0/. Accessed 15 Sept 2020

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Electronics Engineering, Shiraz University of Technology, Shiraz, Iran

    Farhad Raufmehr, Mohammad Reza Salehi & Ebrahim Abiri

  2. Regional Information Center for Science and Technology, Shiraz, Iran

    Mohammad Reza Salehi

Authors
  1. Farhad Raufmehr
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Reza Salehi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ebrahim Abiri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Farhad Raufmehr.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Raufmehr, F., Salehi, M.R. & Abiri, E. A frame-level MLP-based bit-rate controller for real-time video transmission using VVC standard. J Real-Time Image Proc 18, 751–763 (2021). https://doi.org/10.1007/s11554-020-01018-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-020-01018-2

Keywords

Search

Navigation