Abstract
Generating naturally stitched images is a challenging task due to the presence of the parallax in images. In this paper, we propose a semantic-based method to handle the distortions and artifacts generated in the process of stitching. Firstly, a semantic-based random sample consensus algorithm is proposed to obtain the corresponding feature points, which can effectively eliminate mismatch points. The obtained corresponding features are used to align the image and establish the global structure information. Then, we propose semantics-preserving warps based on mesh optimization. The global structure information, semantic information and the corresponding features are combined to warp the images. Experimental results show that our algorithm can provide accurate alignment results while effectively reducing image distortions in both overlapping and non-overlapping regions.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Figa_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig8_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig10_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig11_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig12_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig13_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig14_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig15_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03585-4/MediaObjects/11760_2024_3585_Fig16_HTML.png)
Similar content being viewed by others
Data availibility statement
This article uses a public dataset: TuSimnpleLane Detection Challenge Dataset.
References
Wang, L., Yu, W., Li, B.: Multi-scenes image stitching based on autonomous driving. In: 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) 1, pp. 694–698 (2020)
Zhang, J., Liu, T., Yin, X., Wang, X., Zhang, K., Xu, J., Wang, D.: An improved parking space recognition algorithm based on panoramic vision. Multimedia Tools Appl. 80, 18181–18209 (2021)
Luo, X., Li, Y., Yan, J., Guan, X.: Image stitching with positional relationship constraints of feature points and lines. Pattern Recognit. Lett. 135, 431–440 (2020)
Zhang, J., Yin, X., Luan, J., Liu, T.: An improved vehicle panoramic image generation algorithm. Multimedia Tools Appl. 78, 27663–27682 (2019)
Zhang, J., Pan, D., Zhang, K., Jin, J.J., Ma, Y.-Y., Chen, M.: Underwater single-image restoration based on modified generative adversarial net. SIViP 17, 1153–1160 (2022)
Brown, M.A., Lowe, D.G.: Recognising panoramas. In: Proceedings Ninth IEEE International Conference on Computer Vision, pp. 1218–12252 (2003)
Brown, M.A., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74, 59–73 (2007)
Gao, J., Kim, S.J., Brown, M.S.: Constructing image panoramas using dual-homography warping. CVPR 2011, 49–56 (2011)
Zaragoza, J., Chin, T.-J., Tran, Q.-H., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving dlt. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2339–2346 (2013)
Jia, Q., Li, Z., Fan, X., Zhao, H., Teng, S., Ye, X., Latecki, L.J.: Leveraging line-point consistence to preserve structures for wide parallax image stitching. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12181–12190 (2021)
Li, N., Xu, Y., Wang, C.: Quasi-homography warps in image stitching. IEEE Trans. Multimedia 20, 1365–1375 (2017)
Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 381–395 (1981)
Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262–3269 (2014)
Lin, C.-C., Pankanti, S., Ramamurthy, K.N., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1155–1163 (2015)
Chang, C.-H., Sato, Y., Chuang, Y.-Y.: Shape-preserving half-projective warps for image stitching. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3254–3261 (2014)
Chen, Y.-S., Chuang, Y.-Y.: Natural image stitching with the global similarity prior. In: European Conference on Computer Vision (2016)
Liao, T., Li, N.: Single-perspective warps in natural image stitching. IEEE Trans. Image Process. 29, 724–735 (2018)
Du, P., Ning, J., Cui, J., Huang, S., Wang, X., Wang, J.: Geometric structure preserving warp for natural image stitching. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3678–3686 (2022)
Zhang, J., Xiu, Y.: Image stitching based on human visual system and sift algorithm. Visual Comput. 1–13 (2023)
Lin, K., Jiang, N., Cheong, L.F., Do, M.N., Lu, J.: Seagull: Seam-guided local alignment for parallax-tolerant image stitching. In: European Conference on Computer Vision (2016)
Li, A., Guo, J., Guo, Y.: Image stitching based on semantic planar region consensus. IEEE Trans. Image Process. 30, 5545–5558 (2020)
Zheng, J., Wang, Y., Wang, H., Li, B., Hu, H.-M.: A novel projective-consistent plane based image stitching method. IEEE Trans. Multimedia 21, 2561–2575 (2019)
Zhang, Y., Lai, Y.-K., Zhang, F.-L.: Content-preserving image stitching with piecewise rectangular boundary constraints. IEEE Trans. Visual Comput. Graph. 27(7), 3198–3212 (2021). https://doi.org/10.1109/TVCG.2020.2965097
Nie, L., Lin, C., Liao, K., Liu, S., Zhao, Y.: Deep rectangling for image stitching: a learning baseline. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5730–5738 (2022). https://doi.org/10.1109/CVPR52688.2022.00565
DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation. CoRR abs/1606.03798 (2016) arXiv:1606.03798
Nguyen, T., Chen, S.W., Shivakumar, S.S., Taylor, C.J., Kumar, V.: Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model
Nie, L., Lin, C., Liao, K., Liu, S., Zhao, Y.: Unsupervised deep image stitching: Reconstructing stitched features to images. IEEE Trans. Image Process. 30, 6184–6197 (2021)
Nie, L., Lin, C., Liao, K., Liu, S., Zhao, Y.: Parallax–Tolerant Unsupervised Deep Image Stitching (2023)
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through the ade20k dataset. Int. J. Comput. Vis. 127, 302–321 (2016)
Zhang, Z.: Parameter estimation techniques: a tutorial with application to conic fitting. Image Vis. Comput. 15, 59–76 (1997)
Liu, F., Gleicher, M., Jin, H., Agarwala, A.: Content-preserving warps for 3d video stabilization. ACM Trans. Graph. 28, 44 (2009)
Liu, S., Yuan, L., Tan, P., Sun, J.: Bundled camera paths for video stabilization. ACM Trans. Graph. (TOG) 32, 1–10 (2013)
Funding
(1) National Key Research and Development Program of China (2017YFB0102500), (2) National Natural Science Foundation of China (62172186).
Author information
Authors and Affiliations
Contributions
(1) Jindong Zhang is responsible for providing experimental platform and paper review. (2) Ninan Jiang is responsible for writing the paper and completing the experiment.
Corresponding author
Ethics declarations
Conflict of interest
I declare that the authors have no Conflict of interest as defined by Springer, or other interests that might be perceived to influence the results and/or discussion reported in this paper.
Ethics approval
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, J., Jiang, N. Image stitching algorithm based on semantics-preserving warps. SIViP 19, 40 (2025). https://doi.org/10.1007/s11760-024-03585-4
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11760-024-03585-4