Skip to main content
SpringerLink
Log in

Stereoscopic image completion and depth recovery

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, we have proposed a novel patch-based method for automatic completion of stereoscopic images and the corresponding depth/disparity maps simultaneously. The missing depths are estimated in local feature space and a patch distance metric is designed to take the appearance, depth gradients and depth inconsistency into account. To ensure the proper stereopsis, we first search for the proper stereoscopic patch in both left and right images according to the distance metric, and then iteratively refine the images. Our method is capable of dealing with general scenes including both frontal-parallel and non-frontal-parallel objects. Experimental results show that our method is superior to previous ones with better stereoscopically consistent content and more plausible completion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24:1–24:11 (2009)

    Article  Google Scholar 

  2. Basha, T., Moses, Y., Avidan, S.: Geometrically consistent stereo seam carving. In: ICCV, pp. 1816–1823 (2011).

  3. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: SIGGRAPH, pp. 417–424 (2000).

  4. Bleyer, M., Rhemann, C., Rother, C.: Patchmatch stereo - stereo matching with slanted support windows. In: BMVC, pp. 1–11 (2011).

  5. Celikcan, U., Cimen, G., Kevinc, E.B., Capin, T.: Attention-aware disparity control in interactive environments. Vis. Comput. 29(6–8), 685–694 (2013)

    Article  Google Scholar 

  6. Chang, C.H., Liang, C.K., Chuang, Y.Y.: Content-aware display adaptation and interactive editing for stereoscopic images. IEEE Trans. Multimedia 13(4), 589–601 (2011)

    Article  Google Scholar 

  7. Criminisi, A., Pérez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: IEEE CVPR, pp. 721–728 (2003).

  8. Dahan, M.J., Chen, N., Shamir, A., Cohen-Or, D.: Combining color and depth for enhanced image segmentation and retargeting. Vis. Comput. 28(12), 1181–1193 (2012)

    Article  Google Scholar 

  9. Didyk, P., Sitthi-Amorn, P., Freeman, W.T., Durand, F., Matusik, W.: Joint view expansion and filtering for automultiscopic 3d displays. ACM Trans. Graph. 32(6), 221:1–221:8 (2013)

    Article  Google Scholar 

  10. Du, S.P., Hu, S.M., Martin, R.R.: Changing perspective in stereoscopic images. IEEE Trans. Vis. Comput. Graph. 19(8), 1288–1297 (2013)

    Article  Google Scholar 

  11. Du, S.P., Masia, B., Hu, S.M., Gutierrez, D.: A metric of visual comfort for stereoscopic motion. ACM Trans. Graph. 32(6), 222:1–222:9 (2013)

    Article  Google Scholar 

  12. He, K., Sun, J.: Computing nearest-neighbor fields via propagation-assisted kd-trees. In: IEEE CVPR, pp. 111–118 (2012).

  13. He, K., Sun, J.: Statistics of patch offsets for image completion. In: ECCV, pp. 16–29 (2012).

  14. He, L., Bleyer, M., Gelautz, M.: Object removal by depth-guided inpainting. ÖAGM / AAPR Workshop 2011, 1–8 (2011)

  15. Hervieu, A., Papadakis, N., Bugeau, A., Gargallo, P., Caselles, V.: Stereoscopic image inpainting: distinct depth maps and images inpainting. In: ICPR, pp. 4101–4104 (2010).

  16. Hervieux, A., Papadakis, N., Bugeau, A., Gargallo, P., Caselles, V.: Stereoscopic image inpainting using scene geometry. In: ICME, pp. 1–6 (2011).

  17. Hoffman, D.M., Girshick, A.R., Akeley, K., Banks, M.S.: Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. J. Vis. 8(3), 33:1–33:30 (2008)

    Article  Google Scholar 

  18. Hu, S.M., Chen, T., Xu, K., Cheng, M.M., Martin, R.R.: Internet visual media processing: a survey with graphics and vision applications. Vis. Comput. 29(5), 393–405 (2013)

    Article  Google Scholar 

  19. Kellnhofer, P., Ritschel, T., Myszkowski, K., Seidel, H.P.: Optimizing disparity for motion in depth. Comput. Graph. Forum 32(4), 143–152 (2013)

    Article  Google Scholar 

  20. Kim, Y., Winnemoller, H., Lee, S.: Wysiwyg stereo painting with usability enhancements. IEEE Trans. Vis. Comput. Graph. PrePrint(99), 1–1 (2014).

  21. Komodakis, N., Tziritas, G.: Image completion using efficient belief propagation via priority scheduling and dynamic pruning. Trans. Image Proc. 16(11), 2649–2661 (2007)

    Article  MathSciNet  Google Scholar 

  22. Lambooij, M.T.M., IJsselsteijn, W.A., Heynderickx, I.: Visual discomfort and visual fatigue of stereoscopic displays: a review. J. Imaging Sci. Technol. 53(3), 030,201–030,201–14 (2009).

  23. Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., Gross, M.: Nonlinear disparity mapping for stereoscopic 3d. ACM Trans. Graph. 29(4), 75:1–75:10 (2010)

    Article  Google Scholar 

  24. Lee, K.Y., Chung, C.D., Chuang, Y.Y.: Scene warping: layer-based stereoscopic image resizing. In: IEEE CVPR, pp. 49–56 (2012).

  25. Lee, S., Kim, Y., Lee, J., Kim, K., Lee, K., Noh, J.: Depth manipulation using disparity histogram analysis for stereoscopic 3d. Vis. Comput. 30(4), 455–465 (2014)

    Article  Google Scholar 

  26. Liao, M., Gao, J., Yang, R., Gong, M.: Video stereolization: combining motion analysis with user interaction. IEEE Trans. Vis. Comput. Graph. 18(7), 1079–1088 (2012)

    Article  Google Scholar 

  27. Lo, W.Y., van Baar, J., Knaus, C., Zwicker, M., Gross, M.: Stereoscopic 3d copy and paste. ACM Trans. Graph. 29(6), 147:1–147:10 (2010)

    Article  Google Scholar 

  28. Luo, S.J., Shen, I.C., Chen, B.Y., Cheng, W.H., Chuang, Y.Y.: Perspective-aware warping for seamless stereoscopic image cloning. ACM Trans. Graph. 31(6), 182:1–182:8 (2012)

    Google Scholar 

  29. Morse, B., Howard, J., Cohen, S., Price, B.: Patchmatch-based content completion of stereo image pairs. In: 3DIMPVT, pp. 555–562 (2012).

  30. Niu, Y., Feng, W.C., Liu, F.: Enabling warping on stereoscopic images. ACM Trans. Graph. 31(6), 183:1–183:7 (2012)

    Article  Google Scholar 

  31. Pollock, B., Burton, M., Kelly, J., Gilbert, S., Winer, E.: The right view from the wrong location: depth perception in stereoscopic multi-user virtual environments. IEEE Trans. Vis. Comput. Graph. 18(4), 581–588 (2012)

  32. Raimbault, F., Kokaram, A.: Stereo-video inpainting. J. Electron. Imaging 21(1), 1–10 (2012)

    Article  Google Scholar 

  33. Shibata, T., Kim, J., Hoffman, D.M., Banks, M.S.: The zone of comfort: predicting visual discomfort with stereo displays. J. Vis. 11(8), 11:1–11:29 (2011)

  34. Simakov, D., Caspi, Y., Shechtman, E., Irani, M.: Summarizing visual data using bidirectional similarity. In: IEEE CVPR, pp. 1–8 (2008).

  35. Smith, B., Zhang, L., Jin, H.: Stereo matching with nonparametric smoothness priors in feature space. In: IEEE CVPR, pp. 485–492 (2009).

  36. Sun, J., Li, Y., Kang, S.B., Shum, H.Y.: Symmetric stereo matching for occlusion handling. In: IEEE CVPR, pp. 399–406 (2005).

  37. Sun, J., Yuan, L., Jia, J., Shum, H.Y.: Image completion with structure propagation. ACM Trans. Graph. 24(3), 861–868 (2005)

  38. Tong, R.F., Zhang, Y., Cheng, K.L.: Stereopasting: interactive composition in stereoscopic images. IEEE Trans. Vis. Comput. Graph. 19(8), 1375–1385 (2013)

  39. Wang, L., Jin, H., Yang, R., Gong, M.: Stereoscopic inpainting: joint color and depth completion from stereo images. In: IEEE CVPR, pp. 1–8 (2008).

  40. Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 463–476 (2007)

    Article  Google Scholar 

  41. Yan, T., Lau, R., Xu, Y., Huang, L.: Depth mapping for stereoscopic videos. Int. J. Comput. Vis. 102(1–3), 293–307 (2013)

    Article  MATH  Google Scholar 

  42. Zhang, G., Hua, W., Qin, X., Wong, T.T., Bao, H.: Stereoscopic video synthesis from a monocular video. IEEE Trans. Vis. Comput. Graph. 13(4), 686–696 (2007)

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank the anonymous reviewers for their helpful comments. This work was supported by the National Basic Research Project of China (Project Number 2011CB302205), the Natural Science Foundation of China (Project Number 61272226/ 61120106007), the National High Technology Research and Development Program of China (Project Number 2013AA013903) and Tsinghua University Initiative Scientific Research Program.

Author information

Authors and Affiliations

  1. TNList, Department of Computer Science and Technology, Tsinghua University, Beijing, 100084, China

    Tai-Jiang Mu, Song-Pei Du & Shi-Min Hu

  2. Tsinghua-Tencent Joint Laboratory for Internet Innovation Technology, Beijing, 100084, China

    Ju-Hong Wang

Authors
  1. Tai-Jiang Mu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ju-Hong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Song-Pei Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shi-Min Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tai-Jiang Mu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mu, TJ., Wang, JH., Du, SP. et al. Stereoscopic image completion and depth recovery. Vis Comput 30, 833–843 (2014). https://doi.org/10.1007/s00371-014-0961-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-014-0961-2

Keywords

Search

Navigation