Skip to main content
Log in

Robust spatial–temporal Bayesian view synthesis for video stitching with occlusion handling

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Occlusion is visible in only one frame and cannot be seen in the other frame which is a vital challenge in video stitching. Occlusion always brings ghost artifacts in the blended area. Meanwhile, the traditional image stitching approaches ignore temporal consistency and cannot avoid flicking problem. To solve these challenges, we propose a unified framework in which the stitching quality and stabilization both perform well. Specifically, we explicitly detect the potential occlusion regions to indicate blending information. Then, based on the occlusion maps, we choose a proper strip in the overlapped region as the blending area. With spatial–temporal Bayesian view synthesis, spatial ghost-like artifacts can be significantly eliminated and the output videos can be kept stable. The experimental results show the out performance of the proposed approach compared to state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)

    Article  Google Scholar 

  2. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vision 74(1), 59–73 (2007)

    Article  Google Scholar 

  3. El-Saban, M., Izz, M., Kaheel, A., Refaat, M.: Improved optimal seam selection blending for fast video stitching of videos captured from freely moving devices. In: ICIP, IEEE (2011)

  4. Fitzgibbon, A., Wexler, Y., Zisserman, A.: Image-based rendering using image-based priors. In: ICCV (2003)

  5. Fitzgibbon, A., Wexler, Y., Zisserman, A.: Image-based rendering using image-based priors. Int. J. Comput. Vision 63(2), 141–151 (2005)

    Article  Google Scholar 

  6. Gao, J.H., Li, Y., Chin, T.J., Brown, M.S.: Seam-driven image stitching. In: Eurographics Short Papers. IEEE (2013)

  7. Hsiao, E., Hebert, M.: Occlusion reasoning for object detectionunder arbitrary viewpoint. IEEE Trans. Pattern Anal. Mach. Intell. 36(9), 1803–1815 (2014)

    Article  Google Scholar 

  8. Hu, J., Zhang, D.Q., Yu, H., Chen, C.W.: High resolution free-view interpolation of planar structure. In: ICME (2014)

  9. Huang, K.C., Chien, P.Y., Chien, C.A., Chang, H.C., Guo, J.I.: A 360-degree panoramic video system design. In: International Symposium on VLSI Design. IEEE (2014)

  10. Jang, W.S., Ho, Y.S.: Discontinuity preserving disparity estimation with occlusion handling. J. Vis. Commun. Image Represent. 25(7), 1595–1603 (2014)

    Article  Google Scholar 

  11. Jiang, W., Gu, J.: Video stitching with spatial-temporal content-preserving warping. In: CVPR (2015)

  12. Kang, S.B., Szeliski, R., Uyttendaele, M.: Seamless stitching using multi-perspective plane sweep. Technical Report. Microsoft Research, Redmond (2004)

  13. Kauff, P., Atzpadin, N., Fehn, C., Müller, M., Schreer, O., Smolic, A., Tanger, R.: Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability. Sig. Process. Image Commun. 22(2), 217–234 (2007)

    Article  Google Scholar 

  14. Leordeanu, M., Zanfir, A., Sminchisescu, C.: Locally affine sparse-to-dense matching for motion and occlusion estimation. In: ICCV. IEEE (2013)

  15. Lin, C.C., Pankanti, S.U., Ramamurthy, K.N., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. In: CVPR. IEEE (2015)

  16. McCloskey, S.: Masking light fields to remove partial occlusion. In: ICPR. IEEE (2014)

  17. McGuire, M., Stone, H.S.: Techniques for multiresolution image registration in the presence of occlusions. IEEE Trans. Geosci. Remote Sens. 38(3), 1476–1479 (2000)

    Article  Google Scholar 

  18. Oh, J.D., Kuo, C.C.J.: Robust stereo matching with improved graph and surface models and occlusion handling. J. Vis. Commun. Image Represent. 21(5), 404–415 (2010)

    Article  Google Scholar 

  19. Okumura, K., Raut, S., Gu, Q., Aoyama, T., Takaki, T., Ishii, I.: Real-time feature-based video mosaicing at 500 fps. In: International Conference on Intelligent Robots and Systems. IEEE (2013)

  20. Owens, A., Barnes, C., Flint, A., Singh, H., Freeman, W.: Camouflaging an object from many viewpoints. In: CVPR. IEEE (2014)

  21. Pujades, S., Devernay, F., Goldluecke, B.: Bayesian view synthesis and image-based rendering principles. In: CVPR (2014)

  22. Srinivasan, A., Balamurugan, V.: Occlusion detection and image restoration in 3d face image. In: TENCON. IEEE (2014)

  23. Su, J., Luo, A., Yang, L., Cheng, H.: Video stitching with spatial-temporal content-preserving warping. In: ICME Workshop (2016)

  24. Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, Berlin (2010)

    MATH  Google Scholar 

  25. Thomas, B., Jitendra, M.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)

    Article  Google Scholar 

  26. Uyttendaele, M., Eden, A., Skeliski, R.: Eliminating ghosting and exposure artifacts in image mosaics. In: CVPR. IEEE (2001)

  27. Vladimir, K., Ramin, Z.: Computing visual correspondence with occlusions using graph cuts. In: ICCV. IEEE (2001)

  28. Xiao, X., Daneshpanah, M., Javidi, B.: Occlusion removal using depth mapping in three-dimensional integral imaging. J. Disp. Technol. 8(8), 483–490 (2012)

    Article  Google Scholar 

  29. Xiong, H., Wang, Z., He, R., Feng, D.D.: Video object segmentation with occlusion map. In: DICTA. IEEE (2012)

  30. Yang, L., Cheng, H., Su, J.A.: Pixel-to-model background modeling in crowded scenes. In: ICME (2014)

  31. Yang, L., Yendo, T., Tehrani, M.P.: Artifact reduction using reliability reasoning for image generation of FTV. J. Vis. Commun. Image Represent. 21, 542–560 (2010)

    Article  Google Scholar 

  32. Yang, L., Yendo, T., Tehranil, M.P., Fujii, T.: Probabilistic reliability based view synthesis for FTV. In: ICIP (2010)

  33. Yoo, J.C., Ahn, C.W.: Image matching using peak signal-to-noise ratio-based occlusion detection. Image Process. 6(5), 483–495 (2012)

    Article  MathSciNet  Google Scholar 

  34. Yuan, Z., Kebin, J., Peng-Yu, L.: Video stitch algorithm based on dynamic foreground extraction. In: CISP. IEEE (2009)

  35. Zagrouba, E., Barhoumi, W., Amri, S.: An efficient image-mosaicing method based on multifeature matching. Mach. Vis. Appl. 20(3), 139–162 (2009)

    Article  Google Scholar 

  36. Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: Asprojective-as-possible image stitching with moving DLT. In: CVPR. IEEE (2013)

  37. Zeng, L., Zhang, S., Zhang, J., Zhang, Y.: Dynamic image mosaic via sift and dynamic programming. Mach. Vis. Appl. 25(5), 1271–1282 (2014)

    Article  Google Scholar 

  38. Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: CVPR. IEEE (2014)

  39. Zhang, F., Liu, F.: Casual stereoscopic panorama stitching. In: CVPR. IEEE (2015)

  40. Zhang, H.M., Zeng, W., Chen, X.: Foreground based borderline adjusting for real time multi-camera video stitching. In: ICIG. IEEE (2009)

  41. Zhang, T., Jia, K., Xu, C., Ma, Y., Ahuja, N.: Partial occlusion handling for visual tracking via robust part matching. In: CVPR. IEEE (2014)

  42. Zhao, Y.J., Lu, Z.Q., Liu, Y.K.: Video image stitching based on moving object detection and motion prediction compensation. In: CISP. IEEE (2010)

Download references

Acknowledgements

The authors would like to thank all the reviewers for their insightful comments. This work was supported by the National Natural Science Foundation of China (Grant Nos. U1613223, 61673088 and 61603078), National Key Research and Development Plan: New Energy Vehicles focus on special projects (No. 2017YFB0102500).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hong Cheng.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Su, J., Cheng, H., Yang, L. et al. Robust spatial–temporal Bayesian view synthesis for video stitching with occlusion handling. Machine Vision and Applications 29, 219–232 (2018). https://doi.org/10.1007/s00138-017-0888-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-017-0888-5

Keywords

Navigation