Skip to main content

Advertisement

Springer Nature Link
Log in

Loop closure detection based on image semantic feature and bag-of-words

  • 1230: Sentient Multimedia Systems and Visual Intelligence
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Loop closure detection is a key component of visual SLAM(Simultaneous Localization and Mapping). However, the existing loop closure detection algorithms are easily affected by the illumination change and object change of the scene. Since semantic features of images can improve the accuracy of object location recognition, a loop closure detection algorithm based on image semantic features and bag-of-words model is proposed in this paper. Because of the evenly distributed image features can better reflect the content of the image. So firstly, the ORB feature extraction algorithm is improved to make the extracted feature points more evenly distributed in the image, and then the extracted feature points are used to build the bag-of-words model. Then the L2 norm is adopted to calculate the similarity between images, and according to which the loop closure candidate images are determined quickly. In order to reduce the adverse effects of illumination changes and object changes on loop closure detection, YOLOv4 is used to extract semantic features of images in this paper, and real loop closure will be screened from the candidate images according to cosine values of included angles between similar objects in different images, so as to complete the loop closure detection. Experiments on TUM dataset and actual images show that the proposed algorithm can effectively reduce the adverse effects of illumination changes and object changes on loop closure detection, and effectively improve the accuracy and adaptability of loop closure detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Data availability

The datasets used or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. An P, Wang GP, Yu JD et al (2021) An efficient and accurate visual SLAM loop closure detection algorithm[J]. J Beijing Univ Aeronaut Astronaut 1:24–30

    Google Scholar 

  2. Bochkovskiy A, Wang CY, Liao HYM (2020) YOLOv4: optimal speed and accuracy of object detection[C]. IEEE Conference on Computer Vision and Pattern Recognition (VPR)

  3. Cummins M, Newman P (2008) FAB-MAP: probabilistic localization and mapping in the space of appearance[J]. Int J Robot Res 27(6):647–665

    Article  Google Scholar 

  4. Ding WD, Xu D, Liu XL et al (2018) Review on visual odometry for mobile robots[J]. Acta Automat Sin 44(3):385

    Google Scholar 

  5. Fuentes-Pacheco J, Ruiz-Ascencio J, Rendón-Mancha JM (2015) Visual simultaneous localization and mapping: a survey[J]. Artif Intell Rev 43(1):55–81

    Article  Google Scholar 

  6. Galvez-López D, Tardos JD (2012) Bags of binary words for fast place recognition in image sequences[J]. IEEE Trans Robot 28(5):1188–1197

    Article  Google Scholar 

  7. Gao L, Yuan P Wang T et al (2017) New research on SLAM algorithm based on feature matching[C]. 2nd international conference on advanced robotics and mechatronics. Newyork IEEE Press 1–5

  8. Gao X, Zhang T (2017) Unsupervised learning to detect loops using deep neural networks for visual SLAM system[J]. Auton Robot 41(1):1–18

    Article  MathSciNet  Google Scholar 

  9. Ge Z, Xiao QY, Yang DY (2019) Loop closure detection via maximization of mutual information[J]. IEEE Access 7:124217–124232

    Article  Google Scholar 

  10. Glover A, Maddern W, Warren M et al Open FAB-MAP: An open source toolbox for appearancebased loop closure detection. In Proceedings of the IEEE International Conference on Robotics and Automation, Saint Paul, MN, USA, 14–18 May 2012; pp 4730–4735

  11. Guclu O, Can AB (2019) Fast and effective loop closure detection to improve SLAM performance[J]. J Intell Robot Syst 93(3–4):495–517

    Article  Google Scholar 

  12. Hudson MSB, Esther LC (2021) LIFT-SLAM: a deep-learning feature-based monocular visual SLAM method[J]. Neurocomputing 455:97–110

    Article  Google Scholar 

  13. Ke LH, Wang XH (2019) Review of closed loop detection based on mobile robot visual SLAM[J] 1:12–14

  14. Khodatars M, Shoeibi A, Sadeghi D et al Deep learning for neuroimaging-based diagnosis and rehabilitation of autism spectrum disorder: a review[J/OL]. https://arxiv.org/abs/2007.01285.

  15. Li XQ, He W, Zhu SQ et al (2021) Survey of simultaneous localization and mapping based on environmental semantic information[J]. Chin J Eng 43(6):754–767

    Google Scholar 

  16. Li T, Zhang ZQ (2017) Research of SLAM loop-closure based on ORB bag of words[J]. Inf Commun 10:20–25

    Google Scholar 

  17. Liang ZW, Chen YY, Zhu SH et al (2013) Loop closure detection algorithm based on monocular vision using visual dictionary[J]. Pattern Recogn Arificial Intell 26(6):561–570

    Google Scholar 

  18. Liu Q, Duan FH, Sang Y et al (2017) A survey of loop-closure detection method of visual SLAM in complex environments[J]. Robot 41(1):112–123

    Google Scholar 

  19. Liu G, Hu ZZ (2017) Fast loop closure detection based on holistic features from SURF and ORB[J]. Robot 39(1):36–45

    Google Scholar 

  20. Liu W, Qian L (2018) Comparative analysis of SIFT and SURF and ORB algorithms based on OpenCV environment[J]. Control Inst Chem Ind 45(9):714–716

    Google Scholar 

  21. McCormac J, Handa A, Davison A et al (2017) Semantic fusion: dense 3D semantic mapping with convolutional neural networks[C]//IEEE international conference on robotics and automation. Piscataway, USA: IEEE 4628–4635

  22. Mu B, Liu SY, Paull L et al (2016) SLAM with objects using a nonparametric pose graph[C]. IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, USA: IEEE, 4602–4609

  23. Mur-Artal R, Tardos JD (2017) ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras[J]. IEEE Trans Robot 33(5):1255–1262

    Article  Google Scholar 

  24. Naseer T, Oliveira GL, Brox T et al (2017) Semantics-aware visual localization under challenging perceptual conditions[C]. IEEE international conference on Robotics & Automation. IEEE

  25. Nasim K, Shervan FE (2021) Content based image retrieval based on weighted fusion of texture and color features derived from modified local binary patterns and local neighborhood difference patterns[J]. Multimed Tools Appl 80(21–23):32763–32790

    Google Scholar 

  26. Peng ZD, Xuan SB (2018) Image segmentation method based on graph cut combining minimum spanning tree and quadtree[J]. Comput Technol Dev 28(12):102–105

    Google Scholar 

  27. Qin C, Zhang YZ, Liu YD et al (2021) Semantic loop closure detection based on graph matching in multi-objects scenes[J]. J Vis Commun Image Represent 76:103072

    Article  Google Scholar 

  28. Redmon J, Farhadi A YOLOv3: An incremental improvement[EB/OL]. (2018-04-08) [2019-01-01]. https://arxiv.org/abs/1804.02767

  29. Rublee E, Rabaud V, Konolige K et al (2012) ORB: An efficient alternative to SIFT o SURF [C]. International conference on computer vision. IEEE 2563–2570

  30. Rui ME, Hacker T, Rong C (2014) Competitive K-means, a new accurate and distributed K-means algorithm for large datasets[C]// IEEE international conference on cloud computing technology and science. IEEE 17–24

  31. Rui FD, Zhan GW, Chang AL et al (2019) A novel loop closure detection method using line features[J]. IEEE Access 7:111245–111256

    Article  Google Scholar 

  32. Salas-Moreno RF, Newcombe RA, Strasdat H et al (2013) SLAM++: Simultaneous localisation and mapping at the level of objects[C]. 2013 IEEE conference on computer vision and pattern recognition. IEEE

  33. Shoeibi A, Khodatars M, Alizadehsani R et al Automated detection and forecasting of COVID-19 using deep learning techniques: a review [J/OL]. https://arxiv.org/abs/2007.10785.

  34. Shoeibi A, Khodatars M, Jafari M et al Applications of deep learning techniques for automated multiple sclerosis detection using magnetic resonance imaging: a review[J/OL]. https://arxiv.org/abs/2105.04881.

  35. Sivic J, Zisserman A (2003) Video google: A text retrieval approach to object matching in videos[C]. Proc IEEE Int Conf Comput Vis (2):1470–1477

  36. Tsintotas KA, Loukas B, Antonios G (2021) Modest-vocabulary loop-closure detection with incremental bag of tracked words[J]. Robot Auton Syst 141:103782

    Article  Google Scholar 

  37. Vineet V, Miksik O, Lidegaard M et al (2015) Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction[C]//IEEE international conference on robotics and automation. Piscataway, USA: IEEE 75–82

  38. Wang K, Lin YM, Wang LW et al (2019) A unified framework for mutual improvement of SLAM and semantic segmentation[C]. 2019 international conference on robotics and automation (ICRA). Montreal 5224

  39. Xia LL, Cui JS, Shen R et al (2020) A survey of image semantics-based visual simultaneous localization and mapping: application-oriented solutions to autonomous navigation of mobile robots[J]. Int J Adv Robot Syst 17(3):4158

    Article  Google Scholar 

  40. Yu JS, Wu H, Tian GH et al (2016) Semantic database design and semantic map construction of robots based on the cloud[J]. Robot 38(4):410–419

    Google Scholar 

  41. Zhang LQ, Chen GL, Xu XD et al (2019) An improved ORB-SLAM algorithm for feature extraction[J]. Bull Surveying Mapp 3:16–20

    Google Scholar 

  42. Zhang G, Yan X, Ye Y (2019) Loop closure detection via maximization of mutual information [J ]. IEEE Access 7:124217–124232

    Article  Google Scholar 

  43. Zhang KJ, Zhang YZ, Lv GH et al (2019) Loop closure detection based on local semantic topology for visual SLAM system[J]. Robot 41(5):649–659

    Google Scholar 

  44. Zhi SF, Bloesch M, Leutenegger S et al (2019) Scene Code: monocular dense semantic reconstruction using learned encoded scene representations[C]. Proceedings of the 2019 IEEE/CVF conference on computer vision and pattern recognition. Long Beach 11768

Download references

Funding

This work was partially supported by the National Natural Science Funds of China (Grant No. 61502277)and Shandong Provincial Transportation Science and Technology Project (Grant No. 2021B120).

Author information

Author notes

  1. Hao Sun, Cui Ni and Jin ming Li contributed equally to this work.

Authors and Affiliations

  1. School of Information Science and Electric Engineering, Shandong Jiao Tong University, Jinan, 250357, China

    Hao Sun, Peng Wang, Cui Ni & Jin ming Li

Authors
  1. Hao Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cui Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jin ming Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Hao Sun contributed significantly to analysis and wrote the manuscript, Peng Wang contributed to the conception of the study, Cui Ni contributed to performed the data analyses and manuscript preparation, Jinming Li performed the experiment.

Corresponding author

Correspondence to Peng Wang.

Ethics declarations

Ethics approval

Not applicable

Consent to participate

Not applicable

Consent to publish

The work described has not been published before, and its publication has been approved by the responsible authorities at the institution where the work is carried out.

Competing interests

The authors declare that there is no competing interests regarding the publication of this article.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, H., Wang, P., Ni, C. et al. Loop closure detection based on image semantic feature and bag-of-words. Multimed Tools Appl 83, 36377–36398 (2024). https://doi.org/10.1007/s11042-022-13353-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13353-6

Keywords