Skip to main content
SpringerLink
Log in

An enhanced depth map based rendering method with directional depth filter and image inpainting

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Depth image-based rendering (DIBR), which is used to render virtual views with a color image and the corresponding depth map, is one of the key techniques in the 2D to 3D video conversion process. In this paper, a novel method is proposed to partially solve two puzzles of DIBR, i.e. visual image generation and hole filling. The method combines two different approaches for synthesizing new views from an existing view and a corresponding depth map. Disoccluded parts of the synthesized image are first classified as either smooth or highly structured. At structured regions, inpainting is used to preserve the background structure. In other regions, an improved directional depth smoothing is used to avoid disocclusion. Thus, more details and straight line structures in the generated virtual image are preserved. The key contributions include an enhanced adaptive directional filter and a directional hole inpainting algorithm. Experiments show that the disocclusion is removed and the geometric distortion is reduced efficiently. The proposed method can generate more visually satisfactory results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Zhang, L., Vazquez, C., Knorr, S.: 3D-TV content creation: automatic 2D-to-3D video conversion. IEEE Trans. Broadcast. 57(2), 372–383 (2011)

    Article  Google Scholar 

  2. Liu, W., Wu, Y., Guo, F., et al.: An efficient method for 2D to 3D conversion based on structure from motion. Vis. Comput. (2013). doi:10.1007/s00371-013-0904-3

  3. Knorr, S., Smolic, A., Sikora, T.: From 2D-to stereo-to multi-view video. In: Proceedings of 3DTV, pp. 1–4 (2007)

  4. Rotem, E., Wolowelsky, K., Pelz, D.: Automatic video to stereoscopic video conversion. In: Proceedings of SPIE Conference on Stereoscopic Displays and Virtual Reality Systems, vol. 5664, pp. 198–206 (2005)

  5. Zhang, G., Hua, W., Qin, X., et al.: Stereoscopic video synthesis from a monocular video. IEEE Trans. Vis. Comput. Graph. 13(4), 686–696 (2007)

    Article  Google Scholar 

  6. Fehn, C.: Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV. In: Proceedings of SPIE Conference on Stereoscopic Displays and Virtual Reality Systems vol. 5291, pp. 93–104 (2004)

  7. Zhang, L., Tam, W.J.: Stereoscopic image generation based on depth images for 3D TV. IEEE Trans. Broadcast. 51(2), 191–199 (2005)

    Article  Google Scholar 

  8. Wang, L., Huang, X., Xi, M., et al.: An asymmetric edge adaptive filter for depth generation and hole filling in 3DTV. IEEE Trans. Broadcast. 56(3), 425–431 (2010)

    Article  Google Scholar 

  9. Lee, S., Ho, Y.: Discontinuity-adaptive depth map filtering for 3D view generation. In: Proceedings of the 2nd International Conference on Immersive Telecommunications, pp. 1–6 (2009)

  10. Fehn, C.: A 3D-TV approach using depth-image-based-rendering (DIBR). In: Proceedings of Visualization, Imaging, and Image Processing, pp. 482–487 (2003)

  11. Hartley, R., Zisserman, A.: Multiple View Geometry. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  12. Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  13. Gabriel, P.: Texture synthesis with grouplets. Pattern Anal. Mach. Intell. 32(4), 733–746 (2010)

    Article  Google Scholar 

  14. Choi, J., Choe, Y., Kim, Y.: Sparsity based depth estimation and hole-filling algorithm for 2D to 3D video conversion. In: Proceedings of International Conference on Signals and Electronic Systems, pp. 1–4 (2012)

  15. Shen, J., Jin, X., Zhou, C., et al.: Gradient based image completion by solving the Poisson equation. Comput. Graph. 31(1), 119–126 (2007)

    Article  Google Scholar 

  16. Hwang, J., Lee, K., Kim, J., et al.: A novel hole filling method using image segmentation-based image in-painting. In: Proceedings of IEEE International Conference on Consumer Electronics (ICCE), pp. 470–471 (2013)

  17. Reel, S., Cheung, G., Wong, P., et al.: Joint texture-depth pixel inpainting of disocclusion holes in virtual view synthesis. In: Proceedings of Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp. 1–7 (2013)

  18. Tam, W., Alain, G., Zhang, L., et al.: Smoothing depth maps for improved stereoscopic image quality. In: Proceedings of SPIE Conference on Three-Dimensional TV, Video, and Display III, vol. 5599, pp. 162–172 (2004)

  19. Daribo, I., Tillier, C., Pesquet-Popescu, B.: Distance dependent depth filtering in 3D warping for 3DTV. In: Proceedings of the IEEE Workshop on Multimedia Signal Processing (MMSP), pp. 312–315 (2007)

  20. Chen, W., Chang, Y., Lin, S., et al.: Efficient depth image based rendering with edge dependent depth filter and interpolation. In: Proceedings of IEEE International Conference on Multimedia and Expo, pp. 1314–1317 (2005)

Download references

Acknowledgments

This work was supported in part by Ministry of Science and Technology of China under National 973 Basic Research Program (Grants No. 2013CB228206 and No. 2011CB302505) and National Natural Science Foundation of China (Grant No. U1404614 ).

Author information

Authors and Affiliations

  1. Nanyang Normal University, Nanyang, 473061, Henan, China

    Wei Liu & Mingyue Cui

  2. Center for Internet of Things, Institute of Microelectronics of Chinese Academy of Sciences, Beijing, 100029, China

    Wei Liu

  3. Tsinghua University, Beijing, 100190, China

    Dehua Zhang

  4. People’s Public Security University of China, Beijing, 100190, China

    Jianwei Ding

Authors
  1. Wei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dehua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingyue Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianwei Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, W., Zhang, D., Cui, M. et al. An enhanced depth map based rendering method with directional depth filter and image inpainting. Vis Comput 32, 579–589 (2016). https://doi.org/10.1007/s00371-015-1074-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-015-1074-2

Keywords

Search

Navigation