Skip to main content

Advertisement

Springer Nature Link
Log in

Panoramic video stitching of dual cameras based on spatio-temporal seam optimization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a panoramic video stitching algorithm based on seam optimization, which aims at stitching two videos taken by two wide-angle cameras into a single 720-degree video. The use of only two cameras makes the parallax of the dual videos very large, while previous stitching methods based on deformation or seams incur problems like distortion, blur and ghost. To solve these problems, we improve the graph-cut algorithm to compute the optimal seams in the overlapped regions. For the spatial and temporal consistency of the panoramic video, foreground detection and Gaussian filter are employed to generate a sequence of smooth seams. Besides, a quantitative evaluation on the seam quality is proposed for the linear fusion of the stitched frames. Compared with previous methods, our work can effectively reduce the distortion, blur and ghost artifacts, as well as maintain good spatial and temporal consistency of the panoramic video as evidenced by the experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. https://github.com/facebook/Surround360.

References

  1. Boykov Y, Kolmogorov V (2004) An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis & Machine Intelligence 26(9):1124–1137

    Article  Google Scholar 

  2. Brown M, Lowe DG (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–73

    Article  Google Scholar 

  3. Chang CH, Sato Y, Chuang YY (2014) Shape-preserving half-projective warps for image stitching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3254–3261

  4. Chen T, Cheng MM, Tan P, Hu SM and et al. (2009) Sketch2photo: internet image montage. ACM Transactions on Graphics (TOG), pages 89–97

  5. F P, A S-H, H Z and et al. (2015) Panoramic video from unstructured camera arrays. Computer Graphics Forum, pages 57–68

  6. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  7. H B, T T, Surf GLV (2006) Speeded up robust features. Comput Vis Image Underst 110(3):404–417

    Google Scholar 

  8. H KC, C PY, C CA and et al. (2014) A 360-degree panoramic video system design. International Symposium on VLSI Design, Automation and Test, pages 1–4

  9. He B, Yu S (2015) Parallax-robust surveillance video stitching. Sensors 16(1):1–7

    Article  MathSciNet  Google Scholar 

  10. Jia Q, Fan X, Liu Y et al (2016) Hierarchical projective invariant contexts for shape recognition. Pattern Recogn 52:358–374

    Article  Google Scholar 

  11. K L, S L, L C et al (2016) Seamless video stitching from handheld camera inputs. Computer Graphics Forum 35(2):479–487

    Article  Google Scholar 

  12. Kwatra V, Schdl A, Essa I et al (2003) Graphcut textures: image and video synthesis using graph cuts. ACM Transactions on Graphics (ToG) 22(3):277–286

    Article  Google Scholar 

  13. Lee J, et al. (2016) Rich360: optimized spherical representation from structured panoramic camera arrays. ACM Transactions on Graphics (TOG) 35(4):63–73

    Google Scholar 

  14. Lee W-T, Chen H-I, Chen M-S et al (2017) High-resolution 360 video foveated stitching for realtime VR. Computer Graphics Forum 36(7):115–123

    Article  Google Scholar 

  15. Li H, Tang J, Wang Y et al (2012) Looking into the world on Google maps with view direction estimated photos. Neurocomputing 95(14):72–77

    Article  Google Scholar 

  16. Liu Y, Nie L, Han L, et al. (2015) Action2Activity: Recognizing Complex Activities from Sensor Data. International Joint Conference on Artificial Intelligence (IJCAI), pages 1617–1623

  17. Liu Y, Nie L, Liu L et al (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115

    Article  Google Scholar 

  18. Liu Y, Zhang L, Nie L, et al. (2016) Fortune Teller: Predicting Your Career Path. AAAI Conference on Artificial Intelligence, pages 201–207

  19. Liu Y, Zheng Y, Liang Y, et al. (2016) Urban water quality prediction based on multi-task multi-view learning. International Joint Conference on Artificial Intelligence (IJCAI)

  20. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  21. Lu J, Zhu Z (2017) Real-time 4k panoramic video stitching based on gpu acceleration. Computer Science 44(8):18–21

    Google Scholar 

  22. Sun D, Roth S, Black MJ (2010) Secrets of optical flow estimation and their principles. Computer Vision and Pattern Recognition (CVPR), pages 2432–2439

  23. T S, Y N, Z Z, et al. (2016) Video stitching for handheld inputs via combined video stabilization. SIGGRAPH ASIA

  24. W J, J G. (2015) Video stitching with spatial-temporal content preserving warping. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 42–48

  25. Wei X, Mulligan J (2013) Panoramic video stitching from commodity hdtv cameras. Multimedia Systems 19(5):407–426

    Article  Google Scholar 

  26. Y SJ, C LY, J LG et al (2014) Dynamic image stitching for panoramic video. International Journal of Engineering and Technology Innovation 4(4):260–268

    Google Scholar 

  27. Yong J, Wang Y, Lei X, Wang S (2017) Panoramic video stitching method based on improved orb feature detection. Computer Applications and Software 34(5):182–188

    Google Scholar 

  28. Zaragoza J, Chin TJ, Brown MS, et al. (2013) As projective-as-possible image stitching with moving dlt. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2339–2346

  29. Zhang F, Liu F (2014) Parallax-tolerant image stitching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3262–3269

  30. Zhu Z, Lu JM, Wang MX, et al. (2016) A comparative study of algorithms for realtime panoramic video blending. Computer Vision and Pattern Recognition (CVPR)

Download references

Funding

This work was supported by the National Natural Science Foundation of China under Grant 61472035 and Grant 61425013.

Author information

Authors and Affiliations

  1. School of Computer Science, Beijing Institute of Technology, Beijing, 100081, China

    Qiongxin Liu, Xiangyang Su, Lei Zhang & Hua Huang

Authors
  1. Qiongxin Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Xiangyang Su
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Lei Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Hua Huang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Lei Zhang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(MP4 44,000 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Q., Su, X., Zhang, L. et al. Panoramic video stitching of dual cameras based on spatio-temporal seam optimization. Multimed Tools Appl 79, 3107–3124 (2020). https://doi.org/10.1007/s11042-018-6337-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6337-2

Keywords