Skip to main content
Springer Nature Link
Log in

Real-time stereo to multi-view conversion system based on adaptive meshing

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

The stereo to multi-view conversion technology plays an important role in the development and promotion of three-dimensional television, which can provide adequate supply of high-quality 3D content for autostereoscopic displays. This paper focuses on a real-time implementation of the stereo to multi-view conversion system, the major parts of which are adaptive meshing, sparse stereo correspondence, energy equation construction and virtual-view rendering. To achieve the real-time performance, we make three main contributions. First, we introduce adaptive meshing to reduce the computational complexity at the expense of slight decrease in quality. Second, we use a simple and effective method based on block matching algorithm to generate the sparse disparity map. Third, for the module of block-saliency calculation, sparse stereo correspondence and view synthesis, novel parallelization strategies and fine-grained optimization techniques based on graphic processing units are used to accelerate the executing speed. Experimental results show that the system can achieve real-time and semi-real-time performance when rendering 8 views with the image resolution of 1280 × 720 and 1920 × 1080 on Tesla K20. The images and videos presented finally are both visually realistic and comfortable.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  1. Konrad, J., Halle, M.: 3-D displays and signal processing. Sig. Process. Mag. IEEE 24(6), 97–111 (2007)

    Article  Google Scholar 

  2. Smolic, A., Mueller, K., Stefanoski, N., et al.: Coding algorithms for 3DTV—a survey[J]. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1606–1621 (2007)

    Article  Google Scholar 

  3. Merkle, P., Smolic, A., Muller, K., et al.: Efficient prediction structures for multiview video coding[J]. IEEE Trans. Circuits Syst. Video Technol. 17(11), 1461–1473 (2007)

    Article  Google Scholar 

  4. Zitnick, C., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23(3), 600–608 (2004)

    Article  Google Scholar 

  5. Fehn, C.: A 3D-TV system based on video plus depth information, Thirty-Seventh Asilomar Conference on signals, systems and computers, pp. 1529–1533. (2003)

  6. Riechert, C., Zilly, F., Kauff, P., et al.: Fully automatic stereo-to-multiview conversion in autostereoscopic displays. Proc. IBC. pp. 8–14 (2012)

  7. Doyen, D., Thiebaud, S., Drazic, V., Thebault, C.: A real-time 3D multi-view rendering from a real-time 3D capture, SID 2013. Digest. pp. 578–581 (2013)

  8. Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., Gross, M.: Nonlinear disparity mapping for stereoscopic 3D. ACM Trans. Grap. 29(4), (2010)

  9. Farre M, Wang O, Lang M, et al.: Automatic content creation for multiview autostereoscopic displays using image domain warping[C] Multimedia and Expo (ICME), 2011 IEEE International Conference on. IEEE. 1–6 (2011)

  10. Stefanoski, N., Wang, O., Lang, M., et al.: Automatic view synthesis by image-domain-warping[J]. IEEE Trans. Image Process. 22(9), 3329–3341 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  11. Horowitz, S.L., Pavlidis, T.: Picture segmentation by a tree traversal algorithm[J]. J. ACM (JACM) 23(2), 368–388 (1976)

    Article  MATH  Google Scholar 

  12. Garland, M.: Parallel computing with CUDA. IEEE Int. Symp. Parallel Distrib. Process. pp. 1 (2010)

  13. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis[J]. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Article  Google Scholar 

  14. Woods, A.J, Docherty, T., Koch, R.: Image distortions in stereoscopic video systems[C]/IS&T/SPIE’s symposium on electronic imaging: science and technology. Int. Soc. Opt. Photonics. pp. 36–48 (1993)

  15. Fehn, C.A.: 3D-TV approach using depth-image-based rendering (DIBR)[C]. Proc. of VIIP.pp. 3 (2003)

  16. Shamir, A., Sorkine, O.: Visual media retargeting[C] ACM SIGGRAPH ASIA 2009 Courses. ACM. pp. 11 (2009)

  17. Wang, Z., Bovik, A.C., Sheikh, H.R., et al.: Image quality assessment: from error visibility to structural similarity[J]. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  18. Middlebury College. Middlebury stereo datasets [EB/OL]. [2011-02-11]. http://vision.middlebury.edu/stereo/

  19. Mobile 3DTV. Stereo video data-sets [EB/OL]. [2013-10-24]. http://sp.cs.tut.fi/mobile3dtv/stereo-video/

  20. Jin, P.F., Yao, S.J., Li, D.X., Wang, L.H., Zhang, M.: Real-time multi-view rendering based on FPGA. Int. Conf. Syst. Inform. (ICSAI). pp. 1981–1984 (2012)

  21. Fu, H., Yao, S.J., Li, D.X., Wang, L.H., Zhang, M.: A real-time multi-view interlacing architecture for auto-stereoscopic 3DTV display based on FPGA. Int. Conf. Consum. Electron., Commun. Netw. (CECNet). pp. 1569–1572 (2012)

  22. Yao, S.J., Jin, P.F., Fu, H., Wang, L.H., Li, D.X., Zhang, M.: Real-time 3DTV system for autostereoscopic displays. Int. Conf. Audio, Lang. Image Process. (ICALIP). pp. 621–626 (2012)

  23. Yao, S.J., Wang, L.H., Li, D.X., Zhang, M.: A real-time full HD 2D-to-3D video conversion system based on FPGA. Seventh Int. Conf. Image Gr. pp. 774–778 (2013)

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (Grant No. 61271338, 61401390), the National High Technology Research and Development Program (863) of China (Grant No. 2012AA011505), the Zhejiang Provincial Natural Science Foundation of China (Grant No. LQ14F010005), and the Open Projects Program of National Laboratory of Pattern Recognition of China (Grant No. 201306308).

Author information

Authors and Affiliations

  1. Department of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, 310027, China

    Shao-Jun Yao, Liang-Hao Wang, Cheng-Liang Lin & Ming Zhang

  2. Zhejiang Provincial Key Laboratory of Information Network Technology, Hangzhou, 310027, China

    Shao-Jun Yao, Liang-Hao Wang, Cheng-Liang Lin & Ming Zhang

Authors
  1. Shao-Jun Yao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Liang-Hao Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Cheng-Liang Lin
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Ming Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Liang-Hao Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yao, SJ., Wang, LH., Lin, CL. et al. Real-time stereo to multi-view conversion system based on adaptive meshing. J Real-Time Image Proc 14, 481–499 (2018). https://doi.org/10.1007/s11554-015-0490-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-015-0490-x

Keywords