Skip to main content
SpringerLink
Log in

PVC-STIM: Perceptual video coding based on spatio-temporal influence map

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Since the ultimate consumers and judgers of most video applications are human subjects, there has been growing interest in incorporating characteristics of the human visual system (HVS) in video coding for the further development of coding technology, called perceptual video coding (PVC). Although there have been numerous PVC methods reported in the literature to date, exploring various factors affecting the performance of PVC still remains challenging due to the complexity of HVS and its perceptual mechanisms. In this paper, we propose a perceptual video coding scheme based on a novel spatio-temporal influence map model (PVC-STIM). In the first step, we develop a novel perceptual model by considering multiple perceptual characteristics of HVS, with a special focus on the fusion of several spatial and temporal features, i.e. spatial masking effect, spatial stimuli, visual saliency and temporal motion attention. In the second step, the proposed perceptual model is incorporated into the classic video coding framework to adjust the Lagrange multiplier in order to reasonably allocate visual quality, which improves the rate and perceived distortion performance. Experimental results show that the proposed PVC-STIM method can achieve on average 8.76% bitrate savings while retaining similar perceived quality, compared to HEVC, and can also outperform two PVC approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Wiegand, T., Sullivan, G.J., Bjontegaard, G., Luthra, A.: Overview of the H.264/AVC video coding standard. IEEE Trans. Circuits Syst. Video Technol. 13(7), 560–576 (2003)

    Article  Google Scholar 

  2. Sullivan, G.J., Ohm, J.R., Han, W.J., Wiegand, T.: Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol. 22(12), 1649–1668 (2012)

    Article  Google Scholar 

  3. Bross, B., Wang, Y.K., Ye, Y., Liu, S., Chen, J., Sullivan, G.J., Ohm, J.R.: Overview of the versatile video coding (VVC) standard and its applications. IEEE Trans. Circuits Syst. Video Technol. 31(10), 3736–3764 (2021)

    Article  Google Scholar 

  4. Chen, Z., Lin, W., Ngan, K.N.: Perceptual video coding: challenges and approaches. In: IEEE International Conference on Multimedia and Expo (ICME), pp. 784–789. IEEE, Singapore (2010)

  5. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  6. Ban, J., Lai, H., Lin, X.: A novel method rate distortion optimization for HEVC based on improved SSIM. In: 9th International Symposium on Computational Intelligence and Design (ISCID), vol. 2, pp. 260–263. IEEE, Hangzhou (2016)

  7. Rouis, K., Larabi, M.C., Tahar, J.B.: Perceptually adaptive Lagrangian multiplier for HEVC guided rate-distortion optimization. IEEE Access 6, 33589–33603 (2018)

    Article  Google Scholar 

  8. Harshalatha, Y., Biswas, P.K.: SSIM-based joint-bit allocation for 3D video coding. Multimedia Tools Appl. 77(15), 19051–19069 (2018)

    Article  Google Scholar 

  9. Zhou, M., Wei, X., Wang, S., Kwong, S., Fong, C.K., Wong, P.H., Yuen, W.Y., Gao, W.: SSIM-based global optimization for CTU-level rate control in HEVC. IEEE Trans. Multimedia 21(8), 1921–1933 (2019)

    Article  Google Scholar 

  10. Valizadeh, S., Nasiopoulos, P., Ward, R.: Perceptually-friendly rate distortion optimization in high efficiency video coding. In: 23rd European Signal Processing Conference (EUSIPCO), pp. 115–119. IEEE, Nice (2015)

  11. Xu, L., Lin, W., Ma, L., Zhang, Y., Fang, Y., Ngan, K.N., Li, S., Yan, Y.: Free-energy principle inspired video quality metric and its use in video coding. IEEE Trans. Multimedia 18(4), 590–602 (2016)

    Article  Google Scholar 

  12. Zhai, G., Min, X., Liu, N.: Free-energy principle inspired visual quality assessment: an overview. Digital Signal Process. 91, 11–20 (2019)

    Article  Google Scholar 

  13. Kim, J., Bae, S.H., Kim, M.: An HEVC-compliant perceptual video coding scheme based on JND models for variable block-sized transform kernels. IEEE Trans. Circuits Syst. Video Technol. 25(11), 1786–1800 (2015)

    Article  Google Scholar 

  14. Takeuchi, M., Saika, S., Sakamoto, Y., Nagashima, T., Cheng, Z., Kanai, K., Katto, J., Wei, K., Zengwei, J., Wei, X.: Perceptual quality driven adaptive video coding using JND estimation. In: Picture Coding Symposium (PCS), pp. 179–183. IEEE, San Francisco, CA (2018)

  15. Zhang, X., Wang, H., Tian, T.: Perceptual video coding with block-level staircase just noticeable distortion. In: IEEE International Conference on Image Processing (ICIP), pp. 4140–4144. IEEE, Taipei (2019)

  16. Zhou, M., Wei, X., Kwong, S., Jia, W., Fang, B.: Just noticeable distortion-based perceptual rate control in HEVC. IEEE Trans. Image Process. 29, 7603–7614 (2020)

    Article  Google Scholar 

  17. Xu, M., Deng, X., Li, S., Wang, Z.: Region-of-interest based conversational HEVC coding with hierarchical perception model of face. IEEE J. Sel. Top. Signal Process. 8(3), 475–489 (2014)

    Article  Google Scholar 

  18. Li, S., Xu, M., Deng, X., Wang, Z.: Weight-based R-\(\lambda \) rate control for perceptual HEVC coding on conversational videos. Signal Process.: Image Commun. 38, 127–140 (2015)

    Google Scholar 

  19. Liu, Y., Hu, H., Xu, M.: Subjective rate-distortion optimization in HEVC with perceptual model of multiple faces. In: 2015 Visual Communications and Image Processing (VCIP), pp. 1–4. IEEE (2015)

  20. Wang, B., Peng, Q., Wang, E., Han, K., Xiang, W.: Region-of-interest compression and view synthesis for light field video streaming. IEEE Access 7, 41183–41192 (2019)

    Article  Google Scholar 

  21. Wei, H., Zhou, X., Zhou, W., Yan, C., Duan, Z., Shan, N.: Visual saliency based perceptual video coding in HEVC. In: IEEE International Symposium on Circuits and Systems (ISCAS), pp. 2547–2550. IEEE, Montreal, QC (2016)

  22. Jiang, X., Song, T., Zhu, D., Katayama, T., Wang, L.: Quality-oriented perceptual HEVC based on the spatio-temporal saliency detection model. Entropy 21(2), 165 (2019)

    Article  Google Scholar 

  23. Lim, W., Sim, D.: A perceptual rate control algorithm based on luminance adaptation for HEVC encoders. SIViP 14(5), 887–895 (2020)

    Article  Google Scholar 

  24. Wu, Y., Xiang, G., Li, Y., Yan, W., Xie, X.: A novel RDO based on perceptual algorithm. In: IEEE International Conference on Consumer Electronics (ICCE), pp. 1–4. IEEE, Las Vegas, NV (2020)

  25. Enns, J.T., Di Lollo, V.: What’s new in visual masking? Trends Cogn. Sci. 4(9), 345–352 (2000)

    Article  Google Scholar 

  26. Li, S., Ma, L., Ngan, K.N.: Full-reference video quality assessment by decoupling detail losses and additive impairments. IEEE Trans. Circuits Syst. Video Technol. 22(7), 1100–1112 (2012)

    Article  Google Scholar 

  27. Hu, S., Jin, L., Wang, H., Zhang, Y., Kwong, S., Kuo, C.C.J.: Compressed image quality metric based on perceptually weighted distortion. IEEE Trans. Image Process. 24(12), 5594–5608 (2015)

    Article  MathSciNet  Google Scholar 

  28. Harel, J., Koch, C., Perona, P.: Graph-based visual saliency. Adv. Neural Inf. Process. Syst. 19, 545–552 (2007)

    Google Scholar 

  29. Rath, G.B., Makur, A.: Iterative least squares and compression based estimations for a four-parameter linear global motion model and global motion compensation. IEEE Trans. Circuits Syst. Video Technol. 9(7), 1075–1099 (1999)

    Article  Google Scholar 

  30. Purnachand, N., Alves, L.N., Navarro, A.: Improvements to TZ search motion estimation algorithm for multiview video coding. In: 19th International Conference on Systems. Signals and Image Processing (IWSSIP), pp. 388–391. IEEE, Vienna (2012)

  31. McLeod, P., Driver, J., Dienes, Z., Crisp, J.: Filtering by movement in visual search. J. Exp. Psychol. Hum. Percept. Perform. 17(1), 55 (1991)

    Article  Google Scholar 

  32. HM reference software: version 16.7. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.7/

  33. Pinson, M.H., Wolf, S.: A new standardized method for objectively measuring video quality. IEEE Trans. Broadcast. 50(3), 312–322 (2004)

    Article  Google Scholar 

  34. Seshadrinathan, K., Bovik, A.C.: Motion tuned spatio-temporal quality assessment of natural videos. IEEE Trans. Image Process. 19(2), 335–350 (2009)

    Article  MathSciNet  Google Scholar 

  35. Laboratory for image and video engineering (LIVE). https://live.ece.utexas.edu/index.php

  36. Akamine, W.Y., Farias, M.C.: Video quality assessment using visual attention computational models. J. Electron. Imaging 23(6), 061–107 (2014)

    Article  Google Scholar 

  37. Loh, W.T., Bong, D.B.L.: A just noticeable difference-based video quality assessment method with low computational complexity. Sens. Imaging 19(1), 1–20 (2018)

    Article  Google Scholar 

  38. Bjøntegaard, G.: Calculation of average PSNR differences between RD-curves. In: VCEG-M33 (2001)

Download references

Author information

Authors and Affiliations

  1. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, 611756, China

    Bing Wang, Qiang Peng, Qianwen Zhang & Xiao Wu

  2. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Pan Gao

  3. School of Engineering and Mathematical Science, La Trobe University, Melbourne, VIC, 3686, Australia

    Wei Xiang

Authors
  1. Bing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiang Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qianwen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wei Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, B., Gao, P., Peng, Q. et al. PVC-STIM: Perceptual video coding based on spatio-temporal influence map. SIViP 16, 1841–1849 (2022). https://doi.org/10.1007/s11760-022-02143-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02143-0

Keywords

Search

Navigation