Skip to main content
SpringerLink
Log in

Deep region segmentation-based intra prediction for depth video coding

  • 1190: Depth-Related Processing and Applications in Visual Systems
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Depth information plays a vital role in 3D video systems. Since the depth video has large smooth areas segmented by sharp edges, preserving the sharp edges becomes a crucial task for depth video coding. Thus, depth modelling modes (DMMs) are integrated as partition prediction tools in 3D-HEVC. However, both DMM1 and DMM4 have limitations in processing diverse depth regions. To improve the performance of intra prediction for depth video coding, a novel deep region segmentation-based intra prediction (DRSIP) mode is proposed in this paper. Compared with traditional hand-crafted partition prediction methods, the proposed DRSIP mode introduces a deep region segmentation network (DRS-Net) to directly predict the segmentation result from reference texture frame. Besides, a frame-level training strategy is developed to effectively learn both local and global information for informative edge representation. Finally, the frame-level partition results are divided into block partitions to guide the reconstruction of depth blocks. Experimental results demonstrate that the proposed method achieves significant coding gains compared with the 3D-HEVC.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Assunçao PA, Marcelino S, Soares S, de Faria SM (2017) Spatial error concealment for intra-coded depth maps in Multiview video-plus-depth. Multimed Tools Appl 76(12):13835–C13858

    Article  Google Scholar 

  2. Birman R, Segal Y, Hadar O (2020) Overview of research in the field of video compression using deep neural networks. Multimed Tools Appl 79 (17):11699–C11722

    Article  Google Scholar 

  3. Bjontegard G (2001) Calculation of average PSNR differences between RD curves. In: Proceedings of the ITU-T Video Coding Experts Group (VCEG) Thirteens Meeting

  4. Chen Y, Lin J, Huang Y, Lei S (2015) Single depth intra coding mode in 3d-HEVC. IEEE International Symposium on Circuits and Systems (ISCAS) pp 1130–1133

  5. Cong R, Lei J, Fu H, Hou J, Huang Q, Kwong S (2020) Going From RGB to RGBD Saliency: A Depth-Guided Transformation Model. IEEE Trans Cybern 50(8):3627–3639

    Article  Google Scholar 

  6. Cong R, Lei J, Fu H, Huang Q, Cao X, Hou C (2018) Co-saliency detection for RGBD images based on multi-constraint feature matching and cross label propagation. IEEE Trans Image Process 27(2):568–579

    Article  MathSciNet  Google Scholar 

  7. Cui W, Zhang T, Zhang S, Jiang F, Zuo W, Wan Z, Zhao D (2017) Convolutional neural networks based intra prediction for HEVC. Data Compression Conference (DCC) pp 436–436

  8. Duan C, Shen Y, Zhang Y, Wang S, Zhu C, Yang M (2017) Enhancing wedgelet-based depth modeling in 3d-HEVC . Asia-pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Kuala Lumpur 2017 pp 516–519

  9. Helle P, Pfaff J, Schäfer M, Rischke R, Schwarz H, Marpe D, Wiegand T (2019) Intra picture prediction for video coding with neural networks. Data Compression Conference (DCC) pp 448–457

  10. Hu J, Peng W, Chung C (2018) Reinforcement learning for HEVC/h.265 Intra-frame rate control. IEEE International Symposium on Circuits and Systems (ISCAS) pp 1–5

  11. Hu Y, Yang W, Li M, Liu J (2019) Progressive spatial recurrent neural network for intra prediction. Trans Multimed 21(12):3024–3037

    Article  Google Scholar 

  12. Huang H, Schiopu I, Munteanu A (2019) Deep learning based angular intra-prediction for lossless HEVC video coding. Data Compression Conference (DCC) pp 579–579

  13. Huang H, Schiopu I, Munteanu A (2020) Frame-wise CNN-based filtering for intra-frame quality enhancement of HEVC videos. IEEE Transactions on Circuits and Systems for Video Technology:pp 1–1

  14. Huo S, Liu D, Wu F, Li H (2018) Convolutional neural network-based motion compensation refinement for video coding. IEEE International Symposium on Circuits and Systems (ISCAS) pp 1–4

  15. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM international conference on Multimedia (ACM MM) pp 675–678.

  16. Jia C, Wang S, Zhang X, Wang S, Liu J, Pu S, Ma S (2019) Content-aware convolutional neural network for in-loop filtering in high efficiency video coding. IEEE Transactions on Circuits and Systems for Video Technology 28(7):3343–3356

    MathSciNet  MATH  Google Scholar 

  17. Jin D, Lei J, Peng B, Li W, Ling N, Huang Q (2021) Deep affine motion compensation network for inter prediction in VVC. IEEE Transactions on Circuits and Systems for Video Technology. https://doi.org/10.1109/TCSVT.2021.3107135

  18. Ju R, Ge L, Geng W, Ren T, Wu G (2014) Depth saliency based on anisotropic center-surround difference. IEEE International Conference on Image Processing (ICIP) pp 1115–1119

  19. Kauff P, Atzpadin N, Fehn C, Miller M, Schreer O, Smolic A, Tanger R (2007) Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability. Signal Process Image Commu 22(2):217–234

    Article  Google Scholar 

  20. Lainema J, Bossen F, Han W, Min J, Ugur K (2012) Intra coding of the HEVC standard. IEEE Trans Circuits Syst Video Technol 22(12):1792–1801

    Article  Google Scholar 

  21. Lan C, Xu J, Zeng W, Shi G, Wu F (2018) Variable block-sized signal-dependent transform for video coding. IEEE Trans Circuits Syst Video Technol 28(8):1920–1933

    Article  Google Scholar 

  22. Laude T, Haub F, Ostermann J (2019) HEVC Inter coding using deep recurrent neural networks and artificial reference pictures. Picture Coding Symposium (PCS) pp 1–5

  23. Lee J, Park M, Kim C (2015) 3D-CE1: depth intra skip (DIS) mode. ITUT SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, document JCT3v-k0033, Geneva, Switzerland

  24. Lei J, Li X, Peng B, Fang L, Ling N, Huang Q (2020) Deep spatial-spectral subspace clustering for hyperspectral image. IEEE Trans Circuits Syst Video Technol 30(7):2686–2697

    Article  Google Scholar 

  25. Lei J, Liu X, Zhang K, Li G, Ling N (2019) Convolutional neural network based up-sampling for depth video intra coding. IEEE Visual Communications and Image Processing (VCIP) pp 1–4

  26. Lei J, Luo X, Fang L, Wang M, Gu Y (2020) Region-enhanced convolutional neural network for object detection in remote sensing images. IEEE Trans Geosci Remote Sens 58(8):5693–5702

    Article  Google Scholar 

  27. Li Y, Li B, Liu D, Chen Z (2017) A convolutional neural network-based approach to rate control in HEVC intra coding. IEEE Visual Communications and Image Processing (VCIP) pp 1–4

  28. Li Y, Liu D, Li H, Li L, Wu F, Zhang H, Yang H (2018) Convolutional neural network-based block up-sampling for intra frame coding. IEEE Trans Circuits Syst Video Technol 28(9):2316–2330

    Article  Google Scholar 

  29. Lin J, Liu D, Yang H, Li H, Wu F (2019) Convolutional neural network-based block up-sampling for HEVC. IEEE Trans Circuits Syst Video Technol 29 (12):3701–3715

    Article  Google Scholar 

  30. Lu X, Zhou B, Jin X, Martin G (2020) A rate control scheme for HEVC intra coding using convolution neural network (CNN). Data Compression Conference (DCC) pp 382–382

  31. Lucas LFR, Wegner K, Rodrigues NMM, Pagliari CL, da Silva EAB, de Faria SMM (2015) Intra predictive depth map coding using flexible block partitioning. IEEE Trans Image Process 24(11):4055–4068

    Article  MathSciNet  Google Scholar 

  32. Ma C, Liu D, Peng X, Li L, Wu F (2020) Convolutional neural network-based arithmetic coding for HEVC intra-predicted residues. IEEE Trans Circuits Syst Video Technol 30(7):1901–1916

    Google Scholar 

  33. Merkle P, Bartnik C, Miller K, Marpe D, Wiegand T (2012) 3D video: depth coding based on inter-component prediction of block partitions. Picture Coding Symposium (PCS) pp 149–152

  34. Merkle P, Miller K, Marpe D, Wiegand T (2016) Depth intra coding for 3D video based on geometric primitives. IEEE Trans Circuits Syst Video Technol 26(3):570–582

    Article  Google Scholar 

  35. Meyer M, Wiesner J, Schneider J, Rohlfing C (2019) convolutional neural networks for video intra prediction using cross-component adaptation. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp 1607-1611

  36. Miller K, Merkle P, Tech G, Wiegand T (2012) 3D video coding with depth modeling modes and view synthesis optimization. In: Proceedings of the 2012 asia pacific signal and information processing association annual summit and conference, pp 1–4

  37. Pan Z, Yu W, Lei J, Ling N, Kwong S (2021) TSAN: Synthesized View quality enhancement via two-stream attention network for 3d-HEVC. IEEE Transactions on Circuits and Systems for Video Technology pp 1–14

  38. Peng B, Lei J, Fu H, Jia HY, Zhang Z, Li Y (2021) Deep video action clustering via spatio-temporal feature learning. Neurocomputing pp 1–9

  39. Schiopu I, Huang H, Munteanu A (2020) CNN-Based intra-prediction for lossless HEVC. IEEE Trans Circuits Syst Video Technol 30(7):1816–1828

    Google Scholar 

  40. Sullivan GJ, Boyce JM, Chen Y, Ohm J-R, Segall CA, Vetro A (2013) Standardized extensions of high efficiency video coding (HEVC). EEE J Sel Top Signal Process 7(6):1001–1016

    Article  Google Scholar 

  41. Yokoyama R, Tahara M, Takeuchi M, Sun H, Matsuo Y, Katto J (2020) CNN Based optimal intra prediction mode estimation in video coding. IEEE International Conference on Consumer Electronics (ICCE) pp 1-2

  42. Zhang K, An J, Huang H, Lin J, Huang Y, Lei S (2017) Segmental prediction for video coding. IEEE Trans Circuits Syst Video Technol 27 (11):2425–2436

    Article  Google Scholar 

  43. Zhang Y, Wang Y, Zhu C, Lin Y, Zheng J (2017) Optimization of depth modeling modes in 3d-HEVC depth intra coding. J Real-Time Image Proc 13(1):85–100

    Article  Google Scholar 

  44. Zhang Y, Zhu C, Lin Y, Zheng J, Wang X (2015) Simplified reference pixel selection for constant partition value coding in 3d-HEVC. IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP)

  45. Zhao Z, Wang S, Wang S, Zhang X, Ma S, Yang J (2018) CNN-Based bi-directional motion compensation for high efficiency video coding. IEEE International Symposium on Circuits and Systems (ISCAS)

  46. Zhu L, Kwong S, Zhang Y, Wang S, Wang X (2020) Generative adversarial network-based intra prediction for video coding. Trans Multimed 22 (1):45–58

    Article  Google Scholar 

  47. Zhu X, Li Y, Fu H, Fan X, Shi Y, Lei J (2020) RGB-D salient object detection via cross-modal joint feature extraction and low-bound fusion loss. Neurocomputing, pp 0925–2312

Download references

Acknowledgments

This work was supported in part by the National Key R&D Program of China (No.2018YFE0203900), National Natural Science Foundation of China (No. 61931014), and Natural Science Foundation of Tianjin (No.18JCJQJC45800).

Author information

Authors and Affiliations

  1. School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China

    Jing Zhang, Yonghong Hou, Zhe Zhang, Dengchao Jin, Peihan Zhang & Ge Li

Authors
  1. Jing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yonghong Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhe Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dengchao Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Peihan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ge Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhe Zhang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Hou, Y., Zhang, Z. et al. Deep region segmentation-based intra prediction for depth video coding. Multimed Tools Appl 81, 35953–35964 (2022). https://doi.org/10.1007/s11042-022-13344-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13344-7

Keywords

Search

Navigation