Skip to main content

Advertisement

Springer Nature Link
Log in

Stitching images from a conventional camera and a fisheye camera based on nonrigid warping

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Conventional cameras and fisheye cameras are often used together to capture clear target images and large scene background images in many applications, such as mobile robotic telepresence systems and large scene monitoring systems. In this paper, we propose to stitch images from these cameras for offering remote operators a large field of view to perceive a local environment. To provide a clear view of targets for face-to-face communication and a complete view of a robot’s surroundings for safe teleoperation of the robot, we stitch these images by keeping the original conventional image. The image stitching is formulated as a nonrigid motion estimation problem and images are stitched based on nonrigid warping, e.g., the thin-plate spline. To improve the algorithmic efficiency of image stitching, we exploit a region-based point correspondence selection method to reduce the number of point correspondences that are used for thin-plate spline interpolation. The experiments conducted on collected images and images captured from a telepresence system show the effectiveness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Notes

  1. http://www.vlfeat.org/

  2. Efficiency here means the total time that needs to perform TPS parameter calculation and image warping.

References

  1. Bhagavatula C, Zhu C, Luu K, Savvides M (2017) Faster than real-time facial alignment: A 3D spatial transformer network approach in unconstrained poses. In: Proceedings of international conference of computer vision, pp 3980–3989

  2. Bookstein F L (1989) Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Trans Pattern Anal Mach Intell 11(6):567–585

    Article  Google Scholar 

  3. Brown M, Lowe D G (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–73

    Article  Google Scholar 

  4. Byröd M, Brown M A, Åström K (2009) Minimal solutions for panoramic stitching with radial distortion. In: Proceedings of British machine vision conference, pp 1–11

  5. Chang C-H, Chou C-N, Chang E Y (2017) CLKN: Cascaded Lucas-Kanade networks for image alignment. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2213–2221

  6. Chang C-H, Sato Y, Chuang Y-Y (2014) Shape-preserving half-projective warps for image stitching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3254–3261

  7. Cheng H-T, Chao C-H, Dong J-D, Wen H-K, Liu T-L, Sun M (2018) Cube padding for weakly-supervised saliency prediction in 360? videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1420–1429

  8. Cuevas C, Quilón D, García N (2020) Automatic soccer field of play registration. Pattern Recogn 103:107278

    Article  Google Scholar 

  9. DeTone D, Malisiewicz T, Rabinovich A (2016) Deep image homography estimation. In: RSS workshop on limits and potentials of deep learning in robotics

  10. Dong Y, Jia Y, Shen W, Wu Y (2020) Can you easily perceive the local environment? a user interface with one stitched live video for mobile robotic telepresence systems. Int J Human–Comput Interact 36(8):736–747

    Article  Google Scholar 

  11. Dong Y, Pei M, Zhang L, Xu B, Wu Y, Jia Y (2019) Stitching videos from a fisheye lens camera and a wide-angle lens camera for telepresence robots. arXiv:1903.06319

  12. Erlik Nowruzi F, Laganiere R, Japkowicz N (2017) Homography estimation from image pairs with hierarchical convolutional networks. In: Proceedings of the IEEE international conference on computer vision workshops, pp 913–920

  13. Furnari A, Farinella G M, Bruna A R, Battiato S (2017) Affine covariant features for fisheye distortion local modeling. IEEE Trans Image Process 26(2):696–710

    Article  MathSciNet  Google Scholar 

  14. Gao J, Kim S J, Brown M S (2011) Constructing image panoramas using dual-homography warping. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 49–56

  15. Goshtasby A A (2012) Image registration: Principles, tools and methods. Springer Science & Business Media

  16. Harder R L, Desmarais R N (1972) Interpolation using surface splines. J Aircraft 9(2):189–191

    Article  Google Scholar 

  17. Ho T, Budagavi M (2017) Dual-fisheye lens stitching for 360-degree imaging. In: Proceedings of the IEEE International conference on acoustics, speech and signal processing, pp 2172–2176

  18. Ji S, Qin Z, Shan J, Lu M (2020) Panoramic SLAM from a multiple fisheye camera rig. ISPRS J Photogramm Remote Sens 159:169–183

    Article  Google Scholar 

  19. Jia Y, Xu B, Shen J, Pei M, Dong Z, Hou J, Yang M (2015) Telepresence interaction by touching live video images. arXiv:1512.04334

  20. Jin H (2008) A three-point minimal solution for panoramic stitching with lens distortion. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–8

  21. Ju M-H, Kang H-B (2013) Panoramic image generation with lens distortions. In: Proceedings of the 20th IEEE international conference on image processing, pp 1296–1300

  22. Ju M-H, Kang H-B (2014) Stitching images with arbitrary lens distortions. Int J Adv Robot Syst 11(1):2

    Article  Google Scholar 

  23. Kukelova Z, Heller J, Bujnak M, Pajdla T (2015) Radial distortion homography. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 639–647

  24. Li J, Wang Z, Lai S, Zhai Y, Zhang M (2018) Parallax-tolerant image stitching based on robust elastic warping. IEEE Trans Multimed 20 (7):1672–1687

    Article  Google Scholar 

  25. Li Y, Tofighi M, Monga V (2019) Robust alignment for panoramic stitching via an exact rank constraint. IEEE Trans Image Process 28(10):4730–4745

    Article  MathSciNet  Google Scholar 

  26. Liao K, Lin C, Zhao Y, Xu M (2020) Model-free distortion rectification framework bridged by distortion distribution map. IEEE Trans Image Process 29:3707–3718

    Article  Google Scholar 

  27. Lin W-Y, Liu S, Matsushita Y, Ng T-T, Cheong L-F (2011) Smoothly varying affine stitching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 345–352

  28. Liu S, Chai Q (2019) Shape-optimizing and illumination-smoothing image stitching. IEEE Trans Multimed 21(3):690–703

    Article  Google Scholar 

  29. Lo I-C, Shih K-T, Chen H H (2018) Image stitching for dual fisheye cameras. In: Proceedings of the IEEE international conference on image processing, pp 3164–3168

  30. Lou Z, Gevers T (2014) Image alignment by piecewise planar region matching. IEEE Trans Multimed 16(7):2052–2061

    Article  Google Scholar 

  31. Nguyen T, Chen S W, Skandan S, Taylor C J, Kumar V (2018) Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robot Autom Lett 3(3):2346–2353

    Article  Google Scholar 

  32. Research M (2016) Image composite editor. https://www.microsoft.com/en-us/research/product/computational-photography-applications/image-composite-editor/

  33. Robotics D (2018) Double2. Website. Retrieved July 31, 2018 from http://www.doublerobotics.com

  34. Si X, Feng J, Yuan B, Zhou J (2017) Dense registration of fingerprints. Pattern Recogn 63:87–101

    Article  Google Scholar 

  35. Sprengel R, Rohr K, Stiehl H S (1996) Thin-plate spline approximation for image registration. In: Proceedings of 18th annual international conference of the IEEE engineering in medicine and biology society, vol 3. IEEE, pp 1190–1191

  36. Szeliski R (2006) Image alignment and stitching: A tutorial. Found Trends®; Compu Graph Vision 2(1):1–104

    MathSciNet  MATH  Google Scholar 

  37. Szeliski R (2010) Computer vision: algorithms and applications. Springer Science & Business Media

  38. Szeliski R, Shum H-Y (1997) Creating full view panoramic image mosaics and environment maps. In: Proceedings of the 24th international conference on computer graphics and interactive techniques, pp 251–258

  39. Technologies S (2018) BeamPro. Website. Retrieved July 31, 2018 from https://suitabletech.com

  40. Wang G, Zhou Q, Chen Y (2017) Robust non-rigid point set registration using spatially constrained gaussian fields. IEEE Trans Image Process 26 (4):1759–1769

    Article  MathSciNet  Google Scholar 

  41. Wang Z, Bovik A C, Sheikh H R, Simoncelli E P (2004) Image quality assessment: From error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  42. Xiang T-Z, Xia G-S, Bai X, Zhang L (2018) Image stitching by line-guided local warping with global similarity constraint. Pattern Recogn 83:481–497

    Article  Google Scholar 

  43. Xu B, Jia Y (2017) Wide-angle image stitching using multi-homography warping. In: Proceedings of the IEEE international conference on image processing, pp 1467–1471

  44. Ye Y, Yang K, Xiang K, Wang J, Wang K (2020) Universal semantic segmentation for fisheye urban driving images. arXiv:2002.03736

  45. Zaragoza J, Chin T-J, Brown M S, Suter D (2013) As-projective-as-possible image stitching with moving DLT. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2339–2346

  46. Zheng J, Wang Y, Wang H, Li B, Hu H-M (2019) A novel projective-consistent plane based image stitching method. IEEE Transactions on Multimedia

Download references

Author information

Authors and Affiliations

  1. Tus College of Digit, Guangxi University of Science and Technology, Liuzhou, 545006, People’s Republic of China

    Yanmei Dong

  2. Beijing Laboratory of Intelligent Information Technology, School of Computer Science, Beijing Institute of Technology, Beijing, 100081, People’s Republic of China

    Yanmei Dong, Mingtao Pei, Yuwei Wu & Yunde Jia

Authors
  1. Yanmei Dong
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Mingtao Pei
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yuwei Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Yunde Jia
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Mingtao Pei.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dong, Y., Pei, M., Wu, Y. et al. Stitching images from a conventional camera and a fisheye camera based on nonrigid warping. Multimed Tools Appl 81, 18417–18435 (2022). https://doi.org/10.1007/s11042-022-12236-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12236-0

Keywords