Skip to main content
Springer Nature Link
Log in

A hybrid descriptor for low-textural image stitching in real-time surface inspection systems

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Surface inspection systems in the steel industry use multiple machine-vision (MV) cameras to inspect steel sheets for real-time quality control. Conventional approaches are classified into direct, deep-learning-based, and feature-based methodologies. Direct techniques perform poorly on parallax, while deep-learning-based algorithms require higher execution times and are ineffective for real-time applications. We propose a hybrid descriptor that uses defect detection to effectively stitch low-textural images captured by multiple cameras that are evaluated based on matching accuracy, execution time, and quality of stitched images and compared to popular feature-based image descriptor algorithms. Experimental results show that the proposed hybrid descriptor outperforms existing feature descriptors with 91% matching accuracy and an execution time of 49 milliseconds, producing a seamlessly stitched output.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availibility Statement

The datasets generated during and/or analysed during the current study are not publicly available since this data was generated from the mills of Tata Steel, Jamshedpur and are propreitary in nature and they can be made available after proper consent from the authorities at Tata Steel Limited, India.

References

  1. Agarwala A, Dontcheva M, Agrawala M, Drucker S, Colburn A, Curless B, Salesin D, Cohen M (2004) Interactive digital photomontage. In: ACM SIGGRAPH 2004 Papers, pp. 294–302

  2. Agarwala A, Dontcheva M, Agrawala M, Drucker S, Colburn A, Curless B, Salesin D, Cohen M (2004) Interactive digital photomontage. In: Proceedings of SIGGRAPH04

  3. Alcantarilla PF, Nuevo J, Bartoli A (2013) Fast explicit diffusion for accelerated features in nonlinear scale spaces. In: British Machine Vision Conf. (BMVC)

  4. Alzohairy TA, El-Dein E (2016) Image mosaicing based on neural networks. Int. J. Comput. Appl 975:8887

    Google Scholar 

  5. Awad AI, Hassaballah M (2016) Image Feature Detectors and Descriptors; Foundations and Applications vol. 630. https://doi.org/10.1007/978-3-319-28854-3

  6. Bay H, Tuytelaars T, Gool LV (2008) Speeded-up robust features (surf. Computer vision and image understanding 110(3):346–359

    Article  Google Scholar 

  7. Bonny MZ, Uddin MS (2016) Feature-based image stitching algorithms. IWCI (December) 198–203. https://doi.org/10.1109/IWCI.2016.7860365

  8. Bonny MZ, Uddin MS (2016) Feature-based image stitching algorithms. IWCI (December), 198–203. https://doi.org/10.1109/IWCI.2016.7860365

  9. Brown M, G LD (2003) Recognizing panoramas. In: Proceedings of the IEEE International Conference on Computer Vision, vol. 3. Nice, France, p. 1218

  10. Brown M, G LD (2007) Automatic panoramic image stitching using invariant features. International Journal of Computer Vision 74(1):59–73

  11. Brown M, Lowe DG (2003) Recognising panoramas. In: ICCV’03: Proceedings of the Ninth IEEE International Conference on Computer Vision, Washington, DC, USA, p. 1218

  12. Burt P, Adelson E (1983) A multiresolution spline with application to image mosaics. ACM Trans. Graph 2(4):217–236

    Article  Google Scholar 

  13. Burt P, Adelson E (1983) A multiresolution spline with application to image mosaics. ACM Trans. Graph 2(4):217–236

  14. Chang C-H, Chuang Y-Y (2012) A line-structure-preserving approach to image resizing. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1075–1082

  15. Chen Y-S, Chuang Y-Y (2016) Natural image stitching with the global similarity prior. In: European Conference on Computer Vision, pp. 186–201

  16. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  17. Gao J, Kim SJ, Brown MS (2011) Constructing image panoramas using dual-homography warping. CVPR 2011:49–56

    Google Scholar 

  18. Gao J, J KS, S BM (2011) Constructing image panoramas using dual-homography warping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 201. Colorado Springs, CO, USA, pp. 49–56

  19. Gao J, Li Y, Chin T-J, Brown MS (2013) Seam-driven image stitching. In: Eurographics (Short Papers, pp. 45–48

  20. Ghorai S, Mukherjee A, Gangadaran M, Dutta PK (2013) Automatic defect detection on hot-rolled flat steel products. IEEE Transactions on Instrumentation and Measurement 62:612–621. https://doi.org/10.1109/TIM.2012.2218677

    Article  ADS  Google Scholar 

  21. Ghorai S, Mukherjee A, Gangadaran M, Dutta PK (2013) Automatic defect detection on hot-rolled flat steel products. IEEE Transactions on Instrumentation and Measurement 62:612–621. https://doi.org/10.1109/TIM.2012.2218677

  22. Hoang V-D, Tran D-P, Nhu NG, Pham V-H (2020) Deep feature extraction for panoramic image stitching. In: Asian Conference on Intelligent Information and Database Systems, pp. 141–151

  23. Hoang V-D, Tran D-P, Nhu NG, Pham V-H (2020) Deep feature extraction for panoramic image stitching. In: Asian Conference on Intelligent Information and Database Systems, pp. 141–151

  24. Jain R, Kasturi R, Schunck BG (1995) MACHINE VISION, pp. 249–256

  25. Jia J, Tang KC (2005) Eliminating structure and intensity misalignment in image stitching. In: Tenth IEEE International Conference on Computer Vision (ICCV’05, Beijing, China, pp. 1651–1658

  26. Joshi K (2020) OPEN ACCESS A Survey on Real-Time Image Stitching. https://doi.org/10.9790/9622-1005011924

  27. Joshi K (2020). OPEN ACCESS A Survey on Real-Time Image Stitching. https://doi.org/10.9790/9622-1005011924

    Article  Google Scholar 

  28. Kaynig V, Fischer B, M BJ (2008) Probabilistic image registration and anomaly detection by nonlinear warping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska, USA, pp. 1–8

  29. Lamkin MA, Ringgenberg KM, Lamkin JD (2019) DISTRIBUTED MULTI - APERTURE CAMERA ARRAY

  30. Le Moigne J, Campbell WJ, Cromp RF () An automated parallel image registration technique based on the correlation of wavelet features. IEEE Transactions on Geoscience and Remote Sensing 40(8):1849–1864

  31. Lee K-Y, Sim J-Y (2020) Warping residual based image stitching for large parallax. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8198–8206

  32. Leutenegger S, Chli M, Siegwart RY (2011) Brisk: Binary robust invariant scalable keypoints. In: International Conference on Computer Vision, pp. 2548–2555. https://doi.org/10.1109/ICCV.2011.6126542

  33. Levin SPA, Zomet A, Weiss Y (2004) Seamless image stitching in the gradient domain. In: Proceedings of the European Conference on Computer Vision (ECCV04, Prague, Czech Republic

  34. Li J, Wang Z, Lai S, Zhai Y, Zhang M (2017) Parallax-tolerant image stitching based on robust elastic warping. IEEE Transactions on Multimedia 20(7):1672–1687

    Article  Google Scholar 

  35. Li J, Deng B, Tang R, Wang Z, Yan Y (2019) Local-adaptive image alignment based on triangular facet approximation. IEEE Transactions on Image Processing 29:2356–2369

    Article  ADS  Google Scholar 

  36. Li J, Zhao Y, Ye W, Yu K, Ge S (2019) Attentive deep stitching and quality assessment for 360 omnidirectional images. IEEE J. Select. Top. Signal Process 14:209–221

    Article  ADS  Google Scholar 

  37. Li J, Zhao Y, Ye W, Yu K, Ge S (2019) Attentive deep stitching and quality assessment for 360 omnidirectional images. IEEE Journal of Selected Topics in Signal Processing 14(1):209–221

    Article  ADS  Google Scholar 

  38. Liao K, Lin C, Zhao Y, Xu M (2020) Model-free distortion rectification framework bridged by distortion distribution map. IEEE Transactions on Image Processing 29:3707–3718

    Article  ADS  Google Scholar 

  39. Li J, Deng B, Tang R, Wang Z, Yan Y (2019) Local-adaptive image alignment based on triangular facet approximation. IEEE Transactions on Image Processing 29, 2356–2369

  40. Li Z, Mahapatra D, Tielbeek JA, Stoker J, Vliet LJ. Vos FM () Image registration based on autocorrelation of local structure. IEEE Transactions on Image Processing 35(1):63–75

  41. Lin W-Y, Liu S, Matsushita Y, Ng T-T, Cheong L-F (2011) Smoothly varying affine stitching. In: CVPR 2011:345–352

    Google Scholar 

  42. Lin K, Jiang N, Cheong L-F, Do M, Lu J (2016) Seagull: Seam-guided local alignment for parallax-tolerant image stitching. In: European Conference on Computer Vision, pp. 370–385

  43. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M () A survey on deep learning in medical image analysis. Medical Image Analysis 42, 60–88

  44. Li J, Wang Z, Lai S, Zhai Y, Zhang M (2017) Parallax-tolerant image stitching based on robust elastic warping. IEEE Transactions on Multimedia 20(7):1672–1687

  45. Li J, Zhao Y, Ye W, Yu K, Ge S (2019) Attentive deep stitching and quality assessment for 360 omnidirectional images. IEEE J. Select. Top. Signal Process 14, 209–221

  46. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2):91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94 Accessed 12-30-2021

  47. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2):91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94 Accessed 12-30-2021

  48. Luo Q, Fang X, Liu L, Yang C, Sun Y (2020) Automated visual defect detection for flat steel surface: A survey. IEEE Transactions on Instrumentation and Measurement 69(3):626–644

    Article  CAS  ADS  Google Scholar 

  49. Mistry S, Patel A (2016) Image stitching using harris feature detection. Int. Res. J. Eng. Technol 03(04):1363–1369. Available: www.irjet.net

  50. Nayar SK (1997) Catadioptric omnidirectional camera. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition 482–488. https://doi.org/10.1109/CVPR.1997.609369

  51. Nayar SK (1997) Catadioptric omnidirectional camera. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 482–488. https://doi.org/10.1109/CVPR.1997.609369

  52. Ono Y, Trulls E, Fua P, Yi KM () Lf-net: Learning local features from images. In: Advances in Neural Information Processing Systems, pp. 6234–6244

  53. Peleg S, Rousso B, Rav-Acha A, Zomet A (2000) Mosaicing on adaptive manifolds. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(10):1144–1154

    Article  Google Scholar 

  54. Peleg S, Rousso B, Rav-Acha A, Zomet A (2000) Mosaicing on adaptive manifolds. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(10):1144–1154

  55. Sanders-Reed JN, Koon PL (2010) Vision systems for manned and robotic ground vehicles. Proc. SPIE 7692:1–12

    Google Scholar 

  56. Sarnoff (2015) Distributed Aperture Systems. http://www.sarnoffimaging.com/research-and-develop-ment/vision-technologies/embeddedvision/distributed-aperture-systems Accessed date: 06 May 2020

  57. Shen C, Ji X, Miao C (2019) Real-time image stitching with convolutional neural networks, pp. 192–197

  58. Shi Z, Li H, Cao Q, Ren H, Fan B (2020) An image mosaic method based on convolutional neural network semantic features extraction. J. Signal Process. Syst 92:435–444

    Article  Google Scholar 

  59. Shi Z, Li H, Cao Q, Ren H, Fan B (2020) An image mosaic method based on convolutional neural network semantic features extraction. Journal of Signal Processing Systems 92(4):435–444

    Article  Google Scholar 

  60. Shi Z, Li H, Cao Q, Ren H, Fan B (2020) An image mosaic method based on convolutional neural network semantic features extraction. Journal of Signal Processing Systems 92(4):435–444

  61. Shi Z, Li H, Cao Q, Ren H, Fan B (2020) An image mosaic method based on convolutional neural network semantic features extraction. J. Signal Process. Syst 92, 435–444

  62. Silva R, Bruno F, Gomes P, Frensh T, Monteiro D (2016) Real time 360 video stitching and streaming. ACM SIGGRAPH 2016 Posters. Anaheim, California, pp 1–2

    Google Scholar 

  63. Silva R, Bruno F, Gomes P, Frensh T, Monteiro D (2016) Real time 360 video stitching and streaming. In: ACM SIGGRAPH 2016 Posters, Anaheim, California, pp. 1–2

  64. Stojanovic R, Mitropulos P, Koulamas C, Karayiannis Y, Koubias S, Papadopoulos G (2001) Real-time vision-based system for textile fabric inspection. Real-Time Imaging 7:507–518. https://doi.org/10.1006/rtim.2001.0231

    Article  Google Scholar 

  65. Stojanovic R, Mitropulos P, Koulamas C, Karayiannis Y, Koubias S, Papadopoulos G (2001) Real-time vision-based system for textile fabric inspection. Real-Time Imaging 7, 507–518. https://doi.org/10.1006/rtim.2001.0231

  66. Su M, Hwang W, Cheng K (2004) Analysis on multiresolution mosaic images. IEEE Trans. Image Process 13(7):952–959

    Article  PubMed  ADS  Google Scholar 

  67. Sugimoto T, Kawaguchi T (1998) Development of a surface defect inspection system using radiant light from steel products in a hot rolling line. IEEE Transactions on Instrumentation and Measurement 47:409–416

    Article  ADS  Google Scholar 

  68. Su M, Hwang W, Cheng K (2004) Analysis on multiresolution mosaic images. IEEE Trans. Image Process 13(7):952–959

  69. Szeliski R (2006) Image alignment and stitching: A tutorial. Found. Trends Comput. Graph. Vis 2(1):1–10. https://doi.org/10.1561/0600000009

  70. Szeliski R (2006) Image alignment and stitching: A tutorial. Found. Trends Comput. Graph. Vis 2(1):1–10. https://doi.org/10.1561/0600000009

    Article  MathSciNet  Google Scholar 

  71. Tian L, Tu Z, Zhang D, Liu J, Li B, Yuan J (2020) Unsupervised learning of optical flow with cnn-based non-local filtering. IEEE Transactions on Image Processing 29:8429–8442

    Article  ADS  Google Scholar 

  72. Tian L, Tu Z, Zhang D, Liu J, Li B, Yuan J (2020) Unsupervised learning of optical flow with cnn-based non-local filtering. IEEE Transactions on Image Processing 29, 8429–8442

  73. Wang L, Yu, W, Li B (2020) Multi-scenes image stitching based on autonomous driving, vol. 1, pp. 694–698

  74. Yan M, Yin Q, Guo P (2016) Image stitching with single-hidden layer feedforward neural networks, pp. 4162–4169

  75. Zaragoza J, Chin T-J, Brown MS, Suter D (2013) As-projective-as possible image stitching with moving dlt. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2339–2346

  76. Zhang J, Chen G, Jia Z (2017) An image stitching algorithm based on histogram matching and sift algorithm. Int. J. Pattern Recognit. Artif. Intell 31(4):1–14. https://doi.org/10.1142/S0218001417540064

    Article  ADS  Google Scholar 

  77. Zhang Z, Xu C, Yang J, Gao J, Cui Z (2018) Progressive hard-mining network for monocular depth estimation. IEEE Transactions on Image Processing 27(8):3691–3702

    Article  MathSciNet  PubMed  ADS  Google Scholar 

  78. Zhang F, Liu F (2015) Casual stereoscopic panorama stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2002–2010

  79. Zitova B. Flusser J () Image registration methods: A survey. Image and Vision Computing 21(11):977–1000

  80. Zomet A, Levin A, Peleg S, Weiss Y (2006) Seamless image stitching by minimizing false edges. IEEE Transactions on Image Processing 15(4):969–977

    Article  PubMed  ADS  Google Scholar 

Download references

Acknowledgements

The authors would like to express their gratitude and thank the Automation Division of Tata Steel, Jamshedpur, Jharkhand, India for giving us the opportunity and allowing us to use their state-of-the-art laboratory facilities to conduct this research.

Author information

Authors and Affiliations

  1. Automation, Tata Steel, Jamshedpur, 831005, Jharkhand, India

    Vasanth Subramanyam

  2. Electronics and Communication, National Institute of Technology, Jamshedpur, 831015, Jharkhand, India

    Vasanth Subramanyam, Jayendra Kumar & Shiva Nand Singh

Authors
  1. Vasanth Subramanyam
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jayendra Kumar
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Shiva Nand Singh
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Vasanth Subramanyam.

Ethics declarations

Conflict of Interests/Competing Interests

The first author, Mr. Vasanth Subramanyam is an employed as a Principal Technologist of Tata Steel, India and also pursuing his Phd from National Institute of Technology, Jamshedpur, India. The Authors have the necessary permission from Tata Steel, India for publishing this paper and there are no potential conflicts of interest.

Research involving Human Participants and/or Animals

This research does not involve any human participants or animals.

Informed consent

Informed consent has been obtained from all the authors and Tata Steel,India for publishing this research.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Jayendra Kumar and Shiva Nand Singh contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Subramanyam, V., Kumar, J. & Singh, S.N. A hybrid descriptor for low-textural image stitching in real-time surface inspection systems. Multimed Tools Appl 83, 20653–20675 (2024). https://doi.org/10.1007/s11042-023-16357-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16357-y

Keywords