Skip to main content
Springer Nature Link
Log in

Weakly-Supervised Stitching Network for Real-World Panoramic Image Generation

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13676))

Included in the following conference series:

Abstract

Recently, there has been growing attention on an end-to-end deep learning-based stitching model. However, the most challenging point in deep learning-based stitching is to obtain pairs of input images with a narrow field of view and ground truth images with a wide field of view captured from real-world scenes. To overcome this difficulty, we develop a weakly-supervised learning mechanism to train the stitching model without requiring genuine ground truth images. In addition, we propose a stitching model that takes multiple real-world fisheye images as inputs and creates a 360\(^{\circ }\) output image in an equirectangular projection format. In particular, our model consists of color consistency corrections, warping, and blending, and is trained by perceptual and SSIM losses. The effectiveness of the proposed algorithm is verified on two real-world stitching datasets.

Project page is at https://eadcat.github.io/WSSN.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Kandao. https://www.kandaovr.com/. Accessed 05 Mar 2022

  2. Dualfisheye (2016). https://github.com/ooterness/DualFisheye

  3. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Intl. J. Comput. Vis. (IJCV) 74(1), 59–73 (2007). https://doi.org/10.1007/s11263-006-0002-3

    Article  Google Scholar 

  4. Cai, D., He, X., Han, J.: Isometric projection. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 528–533. AAAI Press (2007)

    Google Scholar 

  5. Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (ELUS). In: International Conference on Learning Representation (ICLR) (2016)

    Google Scholar 

  6. Coates, A., et al.: The PanCam instrument for the ExoMars rover. Astrobiology 17(6–7), 511–541 (2017)

    Article  Google Scholar 

  7. Dai, Q., Fang, F., Li, J., Zhang, G., Zhou, A.: Edge-guided composition network for image stitching. Pattern Recogn. (PR) 118, 108019 (2021)

    Article  Google Scholar 

  8. DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation. CoRR abs/1606.03798 (2016)

    Google Scholar 

  9. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: Proceedings of Conference on Robot Learning (CoRL), pp. 1–16 (2017)

    Google Scholar 

  10. Doutre, C., Nasiopoulos, P.: Fast vignetting correction and color matching for panoramic image stitching. In: IEEE International Conference on Image Processing (ICIP), pp. 709–712 (2009)

    Google Scholar 

  11. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  12. Flores, A., Belongie, S.: Removing pedestrians from Google street view images. In: Proceedings of Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 53–58. IEEE (2010)

    Google Scholar 

  13. Gao, J., Kim, S.J., Brown, M.S.: Constructing image panoramas using dual-homography warping. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 49–56. IEEE (2011)

    Google Scholar 

  14. Gao, J., Li, Y., Chin, T.J., Brown, M.S.: Seam-driven image stitching. In: Eurographics, pp. 45–48 (2013)

    Google Scholar 

  15. Guo, C., et al.: Zero-reference deep curve estimation for low-light image enhancement. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 1780–1789 (2020)

    Google Scholar 

  16. He, B., Yu, S.: Parallax-robust surveillance video stitching. Sensors 16(1), 7 (2016)

    Article  Google Scholar 

  17. He, K., Chang, H., Sun, J.: Rectangling panoramic images via warping. ACM Trans. Graph. (ToG) 32(4), 1–10 (2013)

    MATH  Google Scholar 

  18. Herrmann, C., Wang, C., Bowen, R.S., Keyder, E., Zabih, R.: Object-centered image stitching. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 846–861. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_50

    Chapter  Google Scholar 

  19. Herrmann, C., et al.: Robust image stitching with multiple registrations. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 53–69. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_4

    Chapter  Google Scholar 

  20. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Proceedings of Neural Information Processing Systems (NeurIPS), pp. 6626–6637 (2017)

    Google Scholar 

  21. Jia, Q., et al.: Leveraging line-point consistence to preserve structures for wide parallax image stitching. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 12186–12195 (2021)

    Google Scholar 

  22. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representation (ICLR) (2014)

    Google Scholar 

  23. Lai, W.S., Gallo, O., Gu, J., Sun, D., Yang, M.H., Kautz, J.: Video stitching for linear camera arrays. In: British Machine Vision Conference (BMVC), pp. 1–12 (2019)

    Google Scholar 

  24. Le, H., Liu, F., Zhang, S., Agarwala, A.: Deep homography estimation for dynamic scenes. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 7652–7661 (2020)

    Google Scholar 

  25. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 4681–4690 (2017)

    Google Scholar 

  26. Lee, K.Y., Sim, J.Y.: Warping residual based image stitching for large parallax. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 8198–8206 (2020)

    Google Scholar 

  27. Li, J., Yu, K., Zhao, Y., Zhang, Y., Xu, L.: Cross-reference stitching quality assessment for 360 omnidirectional images. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 2360–2368 (2019)

    Google Scholar 

  28. Li, J., Zhao, Y., Ye, W., Yu, K., Ge, S.: Attentive deep stitching and quality assessment for 360\(^{\circ }\) omnidirectional images. IEEE J. Sel. Top. Sig. Process. 14(1), 209–221 (2019)

    Article  Google Scholar 

  29. Li, N., Liao, T., Wang, C.: Perception-based seam cutting for image stitching. Sig. Image Video Process. 12, 967–974 (2018). https://doi.org/10.1007/s11760-018-1241-9

    Article  Google Scholar 

  30. Liao, T., Li, N.: Single-perspective warps in natural image stitching. IEEE Trans. Image Process. (TIP) 29, 724–735 (2019)

    Article  MathSciNet  Google Scholar 

  31. Lin, K., Jiang, N., Cheong, L.-F., Do, M., Lu, J.: SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 370–385. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_23

    Chapter  Google Scholar 

  32. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  33. Lin, W.Y., Liu, S., Matsushita, Y., Ng, T.T., Cheong, L.F.: Smoothly varying affine stitching. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 345–352. IEEE (2011)

    Google Scholar 

  34. Ling, S., Cheung, G., Le Callet, P.: No-reference quality assessment for stitched panoramic images using convolutional sparse coding and compound feature selection. In: IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6 (2018). https://doi.org/10.1109/ICME.2018.8486545

  35. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Intl. J. Comput. Vis. (IJCV) 60(2), 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  36. Madhusudana, P.C., Soundararajan, R.: Subjective and objective quality assessment of stitched images for virtual reality. IEEE Trans. Image Process. (TIP) 28(11), 5620–5635 (2019)

    Article  MathSciNet  Google Scholar 

  37. Maneshgar, B., Sujir, L., Mudur, S., Poullis, C.: A long-range vision system for projection mapping of stereoscopic content in outdoor areas. In: VISIGRAPP (1: GRAPP), pp. 290–297, January 2017. https://doi.org/10.5220/0006258902900297

  38. Nguyen, T., Chen, S.W., Shivakumar, S.S., Taylor, C.J., Kumar, V.: Unsupervised deep homography: a fast and robust homography estimation model. IEEE Robot. Autom. Lett. (RAL) 3(3), 2346–2353 (2018)

    Article  Google Scholar 

  39. Nie, L., Lin, C., Liao, K., Liu, M., Zhao, Y.: A view-free image stitching network based on global homography. J. Vis. Commun. Image Represent. 73, 102950 (2020)

    Article  Google Scholar 

  40. Nie, L., Lin, C., Liao, K., Liu, S., Zhao, Y.: Unsupervised deep image stitching: reconstructing stitched features to images. IEEE Trans. Image Process. (TIP) 30, 6184–6197 (2021)

    Article  MathSciNet  Google Scholar 

  41. Ozawa, T., Kitani, K.M., Koike, H.: Human-centric panoramic imaging stitching. In: Proceedings of the 3rd Augmented Human International Conference, pp. 1–6 (2012)

    Google Scholar 

  42. Patil, T., Turkowski, K.: Calibrating stitched videos with VRWorks 360 video SDK (2018). https://developer.nvidia.com/blog/calibrating-videos-vrworks-360-video/

  43. Perazzi, F., et al.: Panoramic video from unstructured camera arrays. In: Computer Graphics Forum, vol. 34, pp. 57–68. Wiley Online Library (2015)

    Google Scholar 

  44. Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: SuperGlue: learning feature matching with graph neural networks. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 4938–4947 (2020)

    Google Scholar 

  45. Shen, C., Ji, X., Miao, C.: Real-time image stitching with convolutional neural networks. In: 2019 IEEE International Conference on Real-Time Computing and Robotics (RCAR), pp. 192–197. IEEE (2019)

    Google Scholar 

  46. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representation (ICLR) (2015)

    Google Scholar 

  47. Song, D.Y., Um, G.M., Lee, H.K., Cho, D.: End-to-end image stitching network via multi-homography estimation. IEEE Sig. Process. Lett. (SPL) 28, 763–767 (2021)

    Article  Google Scholar 

  48. Wang, C., Wang, X., Bai, X., Liu, Y., Zhou, J.: Self-supervised deep homography estimation with invertibility constraints. Pattern Recogn. Lett. (PRL) 128, 355–360 (2019)

    Article  Google Scholar 

  49. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. (TIP) 13(4), 600–612 (2004)

    Article  Google Scholar 

  50. Xia, M., Yao, J., Xie, R., Zhang, M., Xiao, J.: Color consistency correction based on remapping optimization for image stitching. In: Proceedings of the International Conference on Computer Vision Workshops (ICCVW), pp. 2977–2984 (2017)

    Google Scholar 

  51. Xiang, T.Z., Xia, G.S., Bai, X., Zhang, L.: Image stitching by line-guided local warping with global similarity constraint. Pattern Recogn. (PR) 83, 481–497 (2018)

    Article  Google Scholar 

  52. Xu, B., Jia, Y.: Wide-angle image stitching using multi-homography warping. In: IEEE International Conference on Image Processing (ICIP), pp. 1467–1471 (2017)

    Google Scholar 

  53. Ye, W., Yu, K., Yu, Y., Li, J.: Logical stitching: a panoramic image stitching method based on color calibration box. In: 2018 14th IEEE International Conference on Signal Processing (ICSP), pp. 1139–1143 (2018). https://doi.org/10.1109/ICSP.2018.8652363

  54. Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR), pp. 2339–2346 (2013)

    Google Scholar 

  55. Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 3262–3269 (2014)

    Google Scholar 

  56. Zhang, J., et al.: Content-aware unsupervised deep homography estimation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 653–669. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_38

    Chapter  Google Scholar 

  57. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 586–595 (2018)

    Google Scholar 

Download references

Acknowledgement

This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. 2018-0-00207, Immersive Media Research Laboratory) and the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No.2021R1A4A1032580, No.2022R1C1C1009334).

Author information

Authors and Affiliations

  1. Chungnam National University, Daejeon, South Korea

    Dae-Young Song, Geonsoo Lee & Donghyeon Cho

  2. Electronics and Telecommunication Research Institute, Daejeon, South Korea

    HeeKyung Lee & Gi-Mun Um

Authors
  1. Dae-Young Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Geonsoo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. HeeKyung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gi-Mun Um
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Donghyeon Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Donghyeon Cho .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 8286 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Song, DY., Lee, G., Lee, H., Um, GM., Cho, D. (2022). Weakly-Supervised Stitching Network for Real-World Panoramic Image Generation. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13676. Springer, Cham. https://doi.org/10.1007/978-3-031-19787-1_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19787-1_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19786-4

  • Online ISBN: 978-3-031-19787-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics