Skip to main content
SpringerLink
Log in

2D to 3D conversion with motion-type adaptive depth estimation

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

2D to 3D conversion is an important task for 3DTV broadcasting services due to the lack of stereoscopic 3D contents. In this paper, we propose 2D to 3D conversion with motion-type adaptive depth estimation. Because the most important depth cue is motion parallax in our method, we first perform motion estimation between sequential video frames. Then, we adopt a motion-type adaptive approach to depth map estimation because videos have different depth structures according to the type of motion. To be specific, depth from motion is exploited to estimate depth maps in the case of global motion while the depth maps are generated based on the depth from template with the local motion-guided refinement in the case of local motion. Finally, we employ depth image-based rendering (DIBR) to generate stereoscopic virtual views from the depth maps. Experimental results demonstrate that the proposed 2D to 3D conversion is very effective in generating accurate depth maps and providing realistic 3D effects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Tam, W.J., Zhang, L.: “3D-TV content generation: 2d-to-3d conversion”. In: Proceedings of IEEE International Conference on Multimedia and Expo (ICME), pp. 1869–1872 (2006)

  2. Wiegand, T., Sullivan, G.J., Bjontegaard, G., Luthra, A.: Overview of the H. 264/AVC video coding standard. IEEE Trans. Circuits Syst. Video Technol. 13(7), 560–576 (2003)

    Article  Google Scholar 

  3. Varodayan, D., Chen, D., Flierl, M., Girod, B.: “Wyner–Ziv coding of video with unsupervised motion vector learning,” Signal Processing: Image Communication, pp. 369–378 (2008)

  4. Chen, D., Tsai, S., Chandrasekhar, V., Takacs, G., Vedantham, R., Grzeszczuk, R., Girod, B.: Residual enhanced visual vector as a compact signature for mobile visual search. Sig. Process. 93(8), 2316–2327 (2013)

    Article  Google Scholar 

  5. Wang, H., Schuster, G.M., Katsaggelos, A.K.: Rate-distortion optimal bit allocation for object-based video coding. IEEE Trans. Circuits Syst. Video Technol. 15(9), 1113–1123 (2005)

    Article  Google Scholar 

  6. Zhu, L., Fan, Z., Aggelos, K.K.: “Joint video summarization and transmission adaptation for energy-efficient wireless video streaming,” EURASIP J Adv Signal Process, vol. 2008, Article ID 657032 (2008)

  7. Smolic, A., Kauff, P., Knorr, S., Hornung, A., Kunter, M., Muller, M., Lang, M.: Three-dimensional video postproduction and processing. Proc. IEEE 99(4), 607–625 (2011)

    Article  Google Scholar 

  8. Pitas, I., Nikolaidis, N.:“Anthropocentric video analysis for film and games postproduction”. In: Proceedings of the 11th International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing on International Conference on Computer Systems and Technologies pp. 11–18 (2010)

  9. Daribo, I., Saito, H.: A novel inpainting-based layered depth video for 3DTV. IEEE Trans. Broadcast. 57(2), 533–541 (2011)

    Article  Google Scholar 

  10. Holte, M.B., Moeslund, T.B., Nikolaidis, N., Pitas, I.: “3D human action recognition for multi-view camera systems”. In: Proceedings of International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), pp. 342–349 (2011)

  11. Liao, M., Gao, J., Yang, R., Gong, M.: Video stereolization: combining motion analysis with user interaction. IEEE Trans. Visual Comput. Graphics 18(7), 1079–1088 (2012)

    Article  Google Scholar 

  12. Yang, N.E., Lee, J.W., Park, R-H.: “Depth map generation from a single image using local depth hypothesis”. In: Proceedings of IEEE International Consumer Electronics (ICCE), pp. 311–312 (2012)

  13. Zhang, L., Vazquez, C., Knorr, S.: 3D-TV content creation: automatic 2D-to-3D video conversion. IEEE Trans. Broadcast. 99, 1–12 (2011)

    Google Scholar 

  14. Kim, D., Min, D., Sohn, K.: A stereoscopic video generation method using stereoscopic display characterization and motion analysis. IEEE Trans. Broadcast. 54, 188–197 (2008)

    Article  Google Scholar 

  15. Pourazad, M.T., Nasiopoulos, P., Ward, R.K.: An H. 264-based scheme for 2D to 3D video conversion. IEEE Trans. Consum. Electron. 55, 742–748 (2008)

    Article  Google Scholar 

  16. Lai, Y.K., Lai, Y.F., Chen, Y.C.: “An effective hybrid depth-perception algorithm for 2D-to-3D conversion in 3D display systems”. In: Proceedings IEEE ICCE, pp. 612–613 (2012)

  17. Yu, F., Liu, J., Ren, Y., Sun, J., Gao, Y., Liu, W.: “Depth generation method for 2D to 3D conversion”. In: Proceedings 3DTV-Con (2011)

  18. Jung, Y.J., Baik, A., Kim, J., Park, D.: “A novel 2D-to-3D conversion technique based on relative height depth cue”. In: Proceedings SPIE Electronics Imaging, Stereoscopic Displays and Applications (2009)

  19. Cheng, C.C., Li, C.T., Chen, L.-G.: A novel 2d-to-3d conversion system using edge information. IEEE Trans. Consum. Electron. 56(3), 1739–1745 (2010)

    Article  Google Scholar 

  20. Zhang, Z., Wang, Y., Jiang, T., Gao, W.: “Visual pertinent 2D-to-3D video conversion by multi-cue fusion”. In: Proceedings of IEEE International Conference on Image Processing (ICIP), pp. 909–912 (2011)

  21. Kraemer, P., Benois-Pineau, J.: “Camera motion detection in the rough indexing paradigm”. In: TREC Video Retrieval Evaluation Online Proceedings, TRECVID05 (2005)

  22. Tao, M.W., Bai, J., Kohli, P., Paris, S.: SimpleFlow: a non-iterative, sublinear optical flow algorithm. Comput Graphics Forum 31(2), 345–353 (2012)

    Article  Google Scholar 

  23. Liu, C., Christopher, L.: “Depth map estimation from motion for 2D to 3D conversion”. In: Proceedings of IEEE International Conference on Electro/Information Technology (EIT), pp. 1–4 (2012)

  24. Han, K., Hong, K.: “Geometric and texture cue based depth-map estimation for 2D to 3D image conversion”. In: Proceedings of IEEE International Conference on Consumer Electronics (ICCE), (2011)

  25. Duda, R.O., Hart, P.E.: “Use of the hough transformation to detect lines and curves in pictures”, In: AI Center, SRI International (1971)

  26. Zhang, X., Yang, Y.: “Minimum spanning tree and color image segmentation”. In: Proceedings of IEEE International Conference on Networking, Sensing and Control, pp. 900–904 (2008)

  27. Tomasi, C., Manduchi, R.: “Bilateral filtering for gray and color images”. In: Proceedings of International Conference on Computer Vision (ICCV) (1998)

  28. Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: “Digital photography with flash and no-flash image pairs”. In: Proceedings of ACM SIGGRAPH, pp. 664–672 (2004)

  29. Fehn, C.: “A 3D-TV approach using depth-image-based rendering (DIBR)”. In: Proceedings Of Visualization, Imaging, and Image Processing, pp. 482–487 (2003)

  30. Fehn, C.: “Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV”. In: Proceedings SPIE 5291, Stereoscopic Displays and Virtual Reality Systems XI (2004)

  31. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  32. Wang, Z., Bovik, A.C.: A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2001)

    Article  Google Scholar 

  33. Sheikh, H.R., Bovik, A.C.: Image information and visual quality. IEEE Trans. Image Process. 15(2), 430–444 (2006)

    Article  Google Scholar 

  34. Moorthy, A.K., Bovik, A.C.: A two-step framework for constructing blind image quality indices. IEEE Signal Process. Lett. 17(5), 513–516 (2010)

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers for their valuable comments that have led to improvements in the quality and presentation of the paper. This work was supported by the National Natural Science Foundation of China (No. 61271298) and the International S&T Cooperation Program of China (No. 2014DFG12780).

Author information

Authors and Affiliations

  1. Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, International Research Center for Intelligent Perception and Computation, Xidian University, Xi’an, 710071, China

    Cheolkon Jung, Lei Wang, Xiaohua Zhu & Licheng Jiao

Authors
  1. Cheolkon Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaohua Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Licheng Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cheolkon Jung.

Additional information

Communicated by P. Pala.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jung, C., Wang, L., Zhu, X. et al. 2D to 3D conversion with motion-type adaptive depth estimation. Multimedia Systems 21, 451–464 (2015). https://doi.org/10.1007/s00530-014-0375-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-014-0375-z

Keywords

Search

Navigation