Skip to main content
SpringerLink
Log in

Model-guided 3D stitching for augmented virtual environment

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Image and video stitching have made tremendous progress in the construction of wide field-of-view (FOV). However, some long-term challenges still exist, including wide baselines between cameras, large parallaxes, and low texture in overlapping areas. The augmented virtual environment (AVE) captures videos as live textures of 3D models in a virtual environment, and provides another 3D solution to overcome the aforementioned challenges. Existing AVE methods primarily follow from video projection, and cannot produce satisfactory stitching results compared with image stitching. In this paper, we propose a novel model-guided 3D stitching algorithm for AVE. The algorithm recovers an approximate 3D model for each video streaming and optimizes the warping of the models to meet the requirements of feature point matching of the 3D models from adjacent videos. Compared with previous state-of-the-art methods, experiment results illustrate that our method significantly improves the stitching quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Anderson R, Gallup D, Barron J T, et al. Jump: virtual reality video. ACM Trans Graph, 2016, 35: 1–13

    Article  Google Scholar 

  2. Matzen K, Cohen M F, Evans B, et al. Low-cost 360 stereo photography and video capture. ACM Trans Graph, 2017, 36: 148

    Article  Google Scholar 

  3. Zhu Z, Lu J, Wang M, et al. A comparative study of algorithms for realtime panoramic video blending. IEEE Trans Image Process, 2018, 27: 2952–2965

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  4. Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible image stitching with moving DLT. In: Proceedings of 2013 IEEE Conference on Computer Vision and Pattern Recognition, 2013. 2339–2346

  5. Chang C H, Sato Y, Chuang Y Y. Shape-preserving half-projective warps for image stitching. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014. 3254–3261

  6. Lin C C, Pankanti S U, Ramamurthy K N, et al. Adaptive as-natural-as-possible image stitching. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, 2015. 1155–1163

  7. Zhang F, Liu F. Parallax-tolerant image stitching. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014. 3262–3269

  8. Chen Y S, Chuang Y Y. Natural image stitching with the global similarity prior. In: Proceedings of European Conference on Computer Vision, 2016. 186–201

  9. Li J, Wang Z, Lai S, et al. Parallax-tolerant image stitching based on robust elastic warping. IEEE Trans Multimedia, 2018, 20: 1672–1687

    Article  Google Scholar 

  10. Zhang G, He Y, Chen W, et al. Multi-viewpoint panorama construction with wide-baseline images. IEEE Trans Image Process, 2016, 25: 3099–3111

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  11. Lin K, Jiang N, Cheong L F, et al. SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Proceedings of European Conference on Computer Vision, 2016. 370–385

  12. Sawhney H, Arpa A, Kumar R, et al. Video flashlights: real time rendering of multiple videos for immersive model visualization. In: Proceedings of Eurographics Workshop on Rendering, 2002. 157–168

  13. Neumann U, You S, Hu J, et al. Augmented virtual environments (AVE): dynamic fusion of imagery and 3D models. In: Proceedings of IEEE Virtual Reality, 2003. 61–67

  14. Sebe I O, Hu J, You S, et al. 3D video surveillance with augmented virtual environments. In: Proceedings of ACM SIGMM Workshop on Video Surveillance, 2003. 107–112

  15. DeCamp P, Shaw G, Kubat R, et al. An immersive system for browsing and visualizing surveillance video. In: Proceedings of ACM International Conference on Multimedia, 2010. 371–380

  16. Zhong Z, Jingdi Y, Jin Y, et al. Method for 3D scene structure modeling and camera registration from single image. US 20160249041 A1, 2016

  17. Szeliski R. Image alignment and stitching: a tutorial. FNT Comput Graph Vision, 2007, 2: 1–104

    Article  Google Scholar 

  18. Lyu W, Zhou Z, Chen L, et al. A survey on image and video stitching. Virtual Reality Intell Hardware, 2019, 1: 55–83

    Article  Google Scholar 

  19. Zheng J, Wang Y, Wang H, et al. A novel projective-consistent plane based image stitching method. IEEE Trans Multimedia, 2019, 21: 2561–2575

    Article  Google Scholar 

  20. Zhang Y, Lai Y K, Zhang F L. Content-preserving image stitching with piecewise rectangular boundary constraints. IEEE Trans Visual Comput Graph, 2021, 27: 3198–3212

    Article  Google Scholar 

  21. Lee K-Y, Sim J-Y. Warping residual based image stitching for large parallax. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 8198–8206

  22. Li J, Deng B, Tang R, et al. Local-adaptive image alignment based on triangular facet approximation. IEEE Trans Image Process, 2020, 29: 2356–2369

    Article  ADS  Google Scholar 

  23. Li A, Guo J, Guo Y. Image stitching based on semantic planar region consensus. IEEE Trans Image Process, 2021, 30: 5545–5558

    Article  ADS  PubMed  Google Scholar 

  24. Jiang W, Gu J. Video stitching with spatial-temporal content-preserving warping. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015. 42–48

  25. Perazzi F, Sorkine-Hornung A, Zimmer H, et al. Panoramic video from unstructured camera arrays. Comput Graph Forum, 2015, 34: 57–68

    Article  Google Scholar 

  26. Huang H, Liu H, Zhang L. VideoWeb: space-time aware presentation of a videoclip collection. IEEE J Emerg Sel Top Circ Syst, 2014, 4: 142–152

    Article  Google Scholar 

  27. Tompkin J, Kim K I, Kautz J, et al. Videoscapes: exploring sparse, unstructured video collections. ACM Trans Graph, 2012, 31: 1–12

    Article  Google Scholar 

  28. Li C, Liu Z, Zhao Z, et al. A fast fusion method for multi-videos with three-dimensional GIS scenes. Multimed Tools Appl, 2021, 80: 1671–1686

    Article  Google Scholar 

  29. Meng M, Zhou Y, Tan C, et al. Viewpoint quality evaluation for augmented virtual environment. In: Pacific-Rim Conference on Multimedia. Cham: Springer, 2018. 223–234

    Google Scholar 

  30. Zhou Y, Cao M, You J, et al. MR video fusion: interactive 3D modeling and stitching on wide-baseline videos. In: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology, 2018. 17

  31. Barnich O, van Droogenbroeck M. ViBe: a universal background subtraction algorithm for video sequences. IEEE Trans Image Process, 2011, 20: 1709–1724

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  32. Hoiem D, Efros A A, Hebert M. Automatic photo pop-up. ACM Trans Graph, 2005, 24: 577–584

    Article  Google Scholar 

  33. Hoiem D, Efros A A, Hebert M. Recovering surface layout from an image. Int J Comput Vis, 2007, 75: 151–172

    Article  Google Scholar 

  34. Guillou E, Meneveaux D, Maisel E, et al. Using vanishing points for camera calibration and coarse 3D reconstruction from a single image. Visual Comput, 2000, 16: 396–410

    Article  Google Scholar 

  35. Lowe D G. Distinctive image features from scale-invariant keypoints. Int J Comput Vision, 2004, 60: 91–110

    Article  Google Scholar 

  36. Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM, 1981, 24: 381–395

    Article  MathSciNet  Google Scholar 

  37. von Gioi R G, Jakubowicz J, Morel J M, et al. LSD: a fast line segment detector with a false detection control. IEEE Trans Pattern Anal Mach Intell, 2010, 32: 722–732

    Article  Google Scholar 

  38. Zhu Z, Huang H Z, Tan Z P, et al. Faithful completion of images of scenic landmarks using internet images. IEEE Trans Visual Comput Graph, 2016, 22: 1945–1958

    Article  Google Scholar 

  39. Kwatra V, Schödl A, Essa I, et al. Graphcut textures: image and video synthesis using graph cuts. ACM Trans Graph, 2003, 22: 277–286

    Article  Google Scholar 

  40. Segal M, Korobkin C, Widenfelt R V. Fast shadows and lighting effects using texture mapping. In: Proceedings of ACM International Conference on Computer Graphics and Interactive Techniques, 1992. 249–252

  41. Debevec P, Yu Y, Borshukov G. Efficient view-dependent image-based rendering with projective texture-mapping. In: Proceedings of the 9th Eurographics Rendering Workshop, 1998. 105–116

Download references

Acknowledgements

This work was supported by National Key Research and Development Program of China (Grant Nos. 2018YFB2100601, 2018YFB2100602) and National Natural Science Foundation of China (Grant No. 61872023).

Author information

Authors and Affiliations

  1. State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, 100191, China

    Zhong Zhou, Ming Meng & Yi Zhou

  2. Beijing BigView Technology Co., Ltd., Beijing, 100088, China

    Jingdi You

  3. Department of Radiology, Duke University, Durham, NC, 27708-0187, USA

    Zhe Zhu

Authors
  1. Zhong Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhe Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jingdi You
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi Zhou.

Additional information

Supporting information

Appendixes A-E. The supporting information is available online at info.scichina.com and link.springer.com. The supporting materials are published as submitted, without typesetting or editing. The responsibility for scientific accuracy and content remains entirely with the authors.

Supplementary File

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Z., Meng, M., Zhou, Y. et al. Model-guided 3D stitching for augmented virtual environment. Sci. China Inf. Sci. 66, 112106 (2023). https://doi.org/10.1007/s11432-021-3323-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-021-3323-2

Keywords

Search

Navigation