Skip to main content
SpringerLink
Log in

Integrating TPS, cylindrical projection, and plumb-line constraint for natural stitching of multiple images

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

This paper presents a novel approach for natural stitching multiple images by integrating thin-plate spline (TPS), cylindrical projection, and plumb-line constraint. Firstly, the homography estimated under plumb-line constraint is used to transform each image to keep the scene in the image upward as much as possible, so as to suppress the accumulation of image projection deformation and make the transformed images approximately available for cylindrical projection. Then, by introducing cylindrical projection into TPS as a global transformation, a multiple image alignment framework called cylindrical projection thin-plate spline (CP-TPS) is established to accurately align the transformed images. In this step, the virtual control points (VCP) are set in the non-overlapping area of images so that the CP-TPS can produce desired deformation in the final stitched image. Finally, a seam-line intersecting the significant structure in the aligned image is automatically generated by combining TPS, dynamic programming matching, and control points triangulation. In this step, the seam-line itself is used to estimate CP-TPS parameter. Experiments were conducted on four public image sets. The results show that the proposed approach can realize the natural stitching of public multiple image sets and has the best performance, compared with Autostitch, APAP, NISwGSP, ELA, and GES-GSP.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data availability

Data not available due to commercial restrictions.

References

  1. Stavros, T., Aggelos, K.K.: A multi-camera setup for generating stereo panoramic video. IEEE Trans. Multimed. 7(5), 880–890 (2005)

    Article  Google Scholar 

  2. Lee, J., Kim, B., Kim, K., Kim, Y., Noh, J.: Rich360: optimized spherical representation from structured panoramic camera arrays. ACM Trans. Graph. 35(4), 1–11 (2016)

    Google Scholar 

  3. Tarak, G., Manubhai, T.M.: Vehicle surround capture: survey of techniques and a novel omni-video-based approach for dynamic panoramic surround maps. IEEE Trans. Intell. Transp. Syst. 7(3), 293–308 (2006)

    Article  Google Scholar 

  4. Tobias, E., Tomá, P., Dieter, A.: Eliminating blind spots for assisted driving. IEEE Trans. Intell. Transp. Syst. 9(4), 657–665 (2008)

    Article  Google Scholar 

  5. Shum, H.Y., Ng, K.T., Chan, S.C.: A virtual reality system using the concentric mosaic: construction, rendering, and data compression. IEEE Trans. Multimed. 7(1), 85–95 (2005)

    Article  Google Scholar 

  6. Zhao, Q., Wan, L., Feng, W., Zhang, J., Wong, T.T.: Cube2Video: navigate between cubic panoramas in real-time. IEEE Trans. Multimed. 15(8), 1745–1754 (2013)

    Article  Google Scholar 

  7. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)

    Article  Google Scholar 

  8. Li, N., Xu, Y., Wang, C.: Quasi-homography warps in image stitching. IEEE Trans. Multimed. 20(6), 1365–1375 (2018)

    Article  Google Scholar 

  9. Liu, Y., Yu, D., Chen, X., et al.: TOP-SIFT: the selected SIFT descriptor based on dictionary learning. Vis. Comput. 35, 667–677 (2019)

    Article  Google Scholar 

  10. Gao, J., Kim, S. J., Brown, M.S.: Constructing image panoramas using dual-homography warping. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 49–56 (2011)

  11. Zaragoza, J., Chin, T., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2339–2346 (2013)

  12. Zaragoza, J., Chin, T., Tran, Q.H., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1285–1298 (2014)

    Article  Google Scholar 

  13. Lin, W.Y., Liu, S., Matsushita, Y., Ng, T.T., Cheong, L.F.: Smoothly varying affine stitching. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 345–352 (2011)

  14. Chang, C.H., Sato, Y., Chuang, Y.Y.: Shape-preserving half-projective warps for image stitching. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3254–3261 (2015)

  15. Lin, C.C., Pankanti, S.U., Ramamurthy, K.N., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1155–1163 (2015)

  16. Chen, Y.S., Chuang, Y.Y.: Natural image stitching with the global similarity prior. In: Computer Vision—14th European Conference, pp. 186–201 (2016)

  17. Chen, S.E.: QuickTime VR—an image-based approach to virtual environment navigation. In: Proceedings of ACM SIGGRAPH Conference on Computer Graphics, pp. 29–38 (1995)

  18. Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3262–3269 (2014)

  19. Szeliski, R.: Image alignment and stitching: a tutorial. Found Trends Comput. Graph. Vis. 2(1), 1–104 (2007)

    Article  Google Scholar 

  20. Jia, J., Tang, C.K.: Image stitching using structure deformation. IEEE Trans. Pattern Anal. Mach. Intell. 30(4), 617–631 (2008)

    Article  Google Scholar 

  21. Bookstein, F.L.: Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans. Pattern Anal. Mach. Intell. 11(6), 567–585 (1992)

    Article  Google Scholar 

  22. Zheng, J., Wang, Y., Wang, H., Li, B., Hu, H.M.: A novel projective-consistent plane based image stitching method. IEEE Trans. Multimed. 21(10), 2561–2575 (2019)

    Article  Google Scholar 

  23. Nie, L., Lin, C., Liao, K., Liu, S., Zhao, Y.: Unsupervised deep image stitching: reconstructing stitched features to images. IEEE Trans. Image Process. 30, 6184–6197 (2021)

    Article  MathSciNet  Google Scholar 

  24. Nie, L., Lin, C., Liao, K., Zhao, Y.: Learning edge-preserved image stitching from multi-scale deep homography. Neurocomputing 491(28), 533–543 (2022)

    Article  Google Scholar 

  25. Lee, K.Y., Sim, J.Y.: Warping residual based image stitching for large parallax. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 8195–8203 (2020)

  26. Li, J., Wang, Z., Lai, S., Zhai, Y., Zhang, M.: Parallax-tolerant image stitching based on robust elastic warping. IEEE Trans. Multimed. 20(7), 1672–1687 (2018)

    Article  Google Scholar 

  27. Zhang, Z., Yang, X., Xu, C.: Natural image stitching with layered warping constraint. IEEE Trans. Multimed. 25, 329–338 (2023)

    Article  Google Scholar 

  28. Du, P., Ning, J., Cui, J., Huang, S., Wang, X., Wang, J.: Geometric structure preserving warp for natural image stitching. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3678–3686 (2022)

  29. Vivek, K., Arno, S., Irfan, E., Turk, G., Bobick, A.: Graphcut textures: image and video synthesis using graph cuts. Proc. ACM SIGGRAPH Conf. Comput. Graph. 22(3), 277–286 (2003)

    Google Scholar 

  30. Gu, H., Yu, Y., Sun, W.: A new optimal seam selection method for airborne image stitching. In: IEEE International Workshop on Imaging Systems and Techniques, pp. 159–163 (2009)

  31. Lin, K., Jiang, N., Cheong, L.F., Do, M., Lu, J.: SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Computer Vision—14th European Conference, ECCV 2016, Proceedings, pp. 370–385 (2016)

  32. Zhang, J., Gao, Y., Xu, Y. et al.: A simple yet effective image stitching with computational suture zone. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02637-5

    Article  Google Scholar 

  33. Zhang, J., Xiu, Y.: Image stitching based on human visual system and SIFT algorithm. Vis. Comput. (2023). https://doi.org/10.1007/s00371-023-02791-4.

    Article  Google Scholar 

  34. Levin, A., Zomet, A., Peleg, S., Weiss, Y.: Seamless image stitching in the gradient domain. In: 8th European Conference on Computer Vision, ECCV 2004, pp. 377–389 (2004)

  35. Brian, S., Julien, T., Valerio, P.: Panorama weaving: fast and flexible seam processing. ACM Trans. Graph. 31(4), 1–11 (2012)

    Google Scholar 

  36. Zhang, M., Zhang, R., Zhang, J., Guo, J., Li, Y., Gao, X.: Dim2Clear network for infrared small target detection. IEEE Trans. Geosci. Remote Sens. 61, 1–14 (2023)

    Google Scholar 

  37. Zhang, M., Xin, J., Zhang, J., Tao, D., Gao, X.: Curvature consistent network for microscope chip image super-resolution. In: IEEE Trans. Neural Netw. Learn. Syst. 1–14 (2022). https://doi.org/10.1109/TNNLS.2022.3168540

  38. Zhang, M., Wu, Q., Zhang, J., Gao, X., Guo, J., Tao, D.: Fluid micelle network for image super-resolution reconstruction. IEEE Trans. Cybern. 53(1), 578–591 (2023)

    Article  Google Scholar 

  39. Zhang, M., Wu, Q., Guo, J., Li, Y., Gao, X.: Heat transfer-inspired network for image super-resolution reconstruction. IEEE Trans. Neural Netw. Learn. Syst. 1–11 (2022). https://doi.org/10.1109/TNNLS.2022.3185529

  40. Li, J., Xu, W., Zhang, J., Wang, Z., Li, X.: Efficient video stitching based on fast structure deformation. IEEE Trans. Cybern. 45(12), 2707–2719 (2015)

    Article  Google Scholar 

  41. Rohr, K., Stiehl, H.S., Sprengel, R., Buzug, T.M., Weese, J., Kuhn, M.H.: Landmark-based elastic registration using approximating thin-plate splines. IEEE Trans. Med. Imaging 20(6), 526–534 (2001)

    Article  Google Scholar 

  42. Wang, Z.Z.: Principles of Photogrammetry. House of Surveying and Mapping, Beijing (1990)

    Google Scholar 

  43. Caprile, B., Torre, V.: Using vanishing points for camera calibration. Int. J. Comput. Vis. 4(2), 127–139 (1990)

    Article  Google Scholar 

  44. Wu, J., Xu, G., Dong, Z., Wang, J.: Calibration of digital camera IOP using radial alignment constrain and vanish point geometry. In: Proceedings of SPIE—The International Society for Optical Engineering, 80092F (2001)

  45. Lowe, D.G.: Distinctive image features from scale invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  46. Ma, Q., Zou, Q., Huang, Y., Wang, N.: Dynamic pedestrian trajectory forecasting with LSTM-based Delaunay triangulation. Appl. Intell. 52, 3018–3028 (2022)

    Article  Google Scholar 

  47. Aaron, F.B., Stephen, S.I.: Large occlusion stereo. Int. J. Comput. Vis. 33, 181–200 (1999)

    Article  Google Scholar 

  48. Fu, J., Zhao, J., Li, F.: Infrared sea-sky line detection utilizing self-adaptive Laplacian of Gaussian filter and visual-saliency-based probabilistic Hough transform. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2022)

    Google Scholar 

  49. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  50. Szeliski, R.: Video mosaics for virtual environments. IEEE Comput. Graph. Appl. 16(2), 22–30 (1996)

    Article  Google Scholar 

  51. Brown, M., Lowe, D.G.: Autostitch. http://matthewalunbrown.com/autostitch/autostitch.html. Accessed 06 Jul 2018

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 42361071, and the Ningbo Science and Technology Innovation Project under Grant 2020Z013.

Author information

Authors and Affiliations

  1. School of Electronic Engineering and Automation, Guilin University of Electronic Technology, Guilin, 541004, China

    Jiongli Gao, Jun Wu & Xuemei Zhao

  2. Ningbo Institute of Materials Technology and Engineering, CAS, Ningbo, 315201, Zhejiang, China

    Gang Xu

Authors
  1. Jiongli Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jun Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuemei Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Wu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, J., Wu, J., Zhao, X. et al. Integrating TPS, cylindrical projection, and plumb-line constraint for natural stitching of multiple images. Vis Comput 40, 3795–3824 (2024). https://doi.org/10.1007/s00371-023-03065-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03065-9

Keywords

Search

Navigation