Skip to main content
SpringerLink
Log in

Data association and loop closure in semantic dynamic SLAM using the table retrieval method

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Simultaneous localization and mapping (SLAM) plays an important role in the area of robotics and augmented reality to simultaneously obtain its location and construct environment maps in real-time. There are many challenges in SLAM, such as data association, loop closure, and dynamic environments. In this paper, we propose a table retrieval method for SLAM data association and loop closure using semantic information in a dynamic environment. The detected landmarks are stored in a table for retrieval, and each landmark has its own semantic and location information for data association and loop closure. The proposed method only checks the corresponding items, so it is not necessary to traverse all the landmarks in the reference frames, which is beneficial to real-time performance and cost efficiency. Experiments are performed to verify the effectiveness of our method on the public TUM and KITTI dataset. The results show that our system achieves considerable performance improvement compared with state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://github.com/RainerKuemmerle/g2o/blob/master/g2o/types/slam3d/isometry3d_mappings.cpp

References

  1. Fang BF, Mei GF, Yuan XH et al (2021) Visual SLAM for Robot Navigation in Healthcare Facility. Pattern Recognit 113:12

    Article  Google Scholar 

  2. Wu YK, Luo L, Yin SJ et al (2021) An FPGA Based Energy Efficient DS-SLAM Accelerator for Mobile Robots in Dynamic Environment. Applied Sciences-Basel 11:15

    Google Scholar 

  3. Bonin-Font F, Burguera A (2020) Towards Multi-Robot Visual Graph-SLAM for Autonomous Marine Vehicles. J. Mar. Sci. Eng. 8:25

    Article  Google Scholar 

  4. Chen YB, Huang SD, Fitch R (2020) Active SLAM for Mobile Robots with Area Coverage and Obstacle Avoidance. IEEE-ASME Trans Mechatron 25:1182–1192

    Article  Google Scholar 

  5. Chen YB, Zhao L, Lee KMB et al (2020) Broadcast Your Weaknesses: Cooperative Active Pose-Graph SLAM for Multiple Robots. IEEE Robot Autom Lett 5:2200–2207

    Article  Google Scholar 

  6. Girerd C, Kudryavtsev AV, Rougeot P et al (2020) SLAM-Based Follow-the-Leader Deployment of Concentric Tube Robots. IEEE Robot Autom Lett 5:548–555

    Article  Google Scholar 

  7. Lee TJ, Kim CH, Cho DID (2019) A Monocular Vision Sensor-Based Efficient SLAM Method for Indoor Service Robots. IEEE Trans Ind Electron 66:318–328

    Article  Google Scholar 

  8. Tang M, Chen Z, Yin FL (2020) Robot Tracking in SLAM with Masreliez-Martin Unscented Kalman Filter. Int J Control Autom Syst 18:2315–2325

    Article  Google Scholar 

  9. Li JL, Li ZJ, Feng Y et al (2019) Development of a Human-Robot Hybrid Intelligent System Based on Brain Teleoperation and Deep Learning SLAM. IEEE Trans Autom Sci Eng 16:1664–1674

    Article  Google Scholar 

  10. Pozna C, Troester F, Precup R-E et al (2009) On the design of an obstacle avoiding trajectory: method and simulation. Math Comput Simul 79:2211–2226

    Article  MathSciNet  MATH  Google Scholar 

  11. Haidegger T, Kovács L, Precup R-E et al (2011) Cascade control for telerobotic systems serving space medicine. The world congress of the international federation of automatic control. 44:3759–3764

    Google Scholar 

  12. Fiorini L, Mancioppi G, Semeraro F et al (2020) Unsupervised emotional state classification through physiological parameters for social robotics applications. Knowl-Based Syst 190:105217

    Article  Google Scholar 

  13. Munoz-Montoya F, Juan MC, Mendez-Lopez M et al (2021) SLAM-Based Augmented Reality for the Assessment of Short-Term Spatial Memory. A Comparative Study of Visual Versus Tactile Stimuli 16:30

    Google Scholar 

  14. Piao JC, Kim SD (2019) Real-Time Visual-Inertial SLAM Based on Adaptive Keyframe Selection for Mobile AR Applications. IEEE Trans Multimedia 21:2827–2836

    Article  Google Scholar 

  15. Chen L, Tang W, John NW et al (2018) SLAM-Based Dense Surface Reconstruction in Monocular Minimally Invasive Surgery and Its Application to Augmented Reality. Comput Meth Programs Biomed 158:135–146

    Article  Google Scholar 

  16. Piao JC, Kim SD (2017) Adaptive Monocular Visual-Inertial SLAM for Real-Time Augmented Reality Applications in Mobile Devices. Sensors 17:25

    Article  Google Scholar 

  17. Gálvez-López D, Tardos JD (2012) Bags of binary words for fast place recognition in image sequences. IEEE Trans Robot 28:1188–1197

    Article  Google Scholar 

  18. Li A, Ruan X, Huang J, et al (2019) Review of vision-based Simultaneous Localization and Mapping. In: 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). IEEE, pp 117–123

  19. Sualeh M, Kim G-W (2019) Simultaneous localization and mapping in the epoch of semantics: a survey. Int J Control Autom Syst 17:729–742

    Article  Google Scholar 

  20. Quimbita S, Chuquitarco D, Hallo V, et al (2019) Systematic and comparative analysis of techniques for SLAM development in mobile robotics. In: Eleventh International Conference on Machine Vision (ICMV 2018). International Society for Optics and Photonics, pp 110412X

  21. Mur-Artal R, Tardós JD (2017) ORB-SLAM2: An open-source SLAM system for monocular, stereo, and rgb-d cameras. IEEE Trans Robot 33:1255–1262

    Article  Google Scholar 

  22. Tang J, Ericson L, Folkesson J, Jensfelt P (2019) GCNv2: Efficient correspondence prediction for real-time SLAM. IEEE Robot Autom Lett 4:3505–3512

    Google Scholar 

  23. Sumikura S, Shibuya M, Sakurada K (2019) OpenVSLAM: A versatile visual SLAM framework. In: Proceedings of the 27th ACM International Conference on Multimedia. pp 2292–2295

  24. Carlevaris-Bianco N, Kaess M, Eustice RM (2014) Generic node removal for factor-graph SLAM. IEEE Trans Robot 30:1371–1385

    Article  Google Scholar 

  25. Schenk F, Fraundorfer F (2019) RESLAM: A real-time robust edge-based SLAM system. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE, pp 154–160

  26. Memon AR, Wang H, Hussain A (2020) Loop closure detection using supervised and unsupervised deep neural networks for monocular SLAM systems. Rob Auton Syst 126:103470

    Article  Google Scholar 

  27. Castro G, Nitsche MA, Pire T et al (2019) Efficient on-board Stereo SLAM through constrained-covisibility strategies. Rob Auton Syst 116:192–205

    Article  Google Scholar 

  28. Salas-Moreno RF, Newcombe RA, Strasdat H, et al (2013) SLAM++: Simultaneous localisation and mapping at the level of objects. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 1352–1359

  29. Tateno K, Tombari F, Laina I, et al (2017) CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 6243–6252

  30. Tian G, Liu L, Ri J et al (2019) ObjectFusion: An object detection and segmentation framework with RGB-D SLAM and convolutional neural networks. Neurocomputing 345:3–14

    Article  Google Scholar 

  31. Wang P, Cheng J, Feng W (2018) An Approach for construct semantic map with scene classification and object semantic Ssegmentation. In: 2018 IEEE International Conference on Real-time Computing and Robotics (RCAR). IEEE, pp 270–275

  32. Wang P, Cheng J, Feng W (2018) Efficient construction of topological semantic map with 3D information. J Intell Fuzzy Syst 35:3011–3020

    Article  Google Scholar 

  33. Cui L, Ma C (2019) SOF-SLAM: A semantic visual SLAM for dynamic environments. IEEE Access 7:166528–166539

    Article  Google Scholar 

  34. Xiao L, Wang J, Qiu X et al (2019) Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment. Rob Auton Syst 117:1–16

    Article  Google Scholar 

  35. Bescos B, Facil JM, Civera J, Neira J (2018) DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes. IEEE Robot Autom Lett 3:4076–4083

    Article  Google Scholar 

  36. Guan P, Cao Z, Chen E et al (2020) A real-time semantic visual SLAM approach with points and objects. Int J Adv Robot Syst 17:1729881420905443

    Article  Google Scholar 

  37. Li S, Zhang T, Gao X et al (2019) Semi-direct monocular visual and visual-inertial SLAM with loop closure detection. Rob Auton Syst 112:201–210

    Article  Google Scholar 

  38. Lourenço P, Batista P, Oliveira P, Silvestre C (2019) Strategies for uncertainty optimization through motion planning in GES sensor-based SLAM. Rob Auton Syst 113:38–55

    Article  Google Scholar 

  39. Mu B, Liu S-Y, Paull L, et al (2017) SLAM with objects using a nonparametric pose graph. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp 4602–4609

  40. Zhang J, Gui M, Wang Q et al (2019) Hierarchical topic model based object association for semantic SLAM. IEEE Trans Vis Comput Graph 25:3052–3062

    Article  Google Scholar 

  41. Iqbal A, Gans NR (2020) Data association and localization of classified objects in visual SLAM. J Intell Robot Syst 100:113–130

    Article  Google Scholar 

  42. Bernreiter L, Gawel A, Sommer H et al (2019) Multiple hypothesis semantic mapping for robust data association. IEEE Robot Autom Lett 4:3255–3262

    Google Scholar 

  43. Zhang J, Yuan L, Ran T et al (2021) Bayesian nonparametric object association for semantic SLAM. IEEE Robot Autom Lett 6:5493–5500

    Article  Google Scholar 

  44. Cao FK, Zhuang Y, Zhang H et al (2018) Robust place recognition and loop closing in laser-based SLAM for UGVs in urban environments. IEEE Sens J 18:4242–4252

    Article  Google Scholar 

  45. Ebadi K, Palieri M, Wood S et al (2021) Dare-SLAM: degeneracy-aware and resilient loop closing in perceptually-degraded environments. J Intell Robot Syst 102:25

    Article  Google Scholar 

  46. Im G, Kim M, Park J (2019) Parking line based SLAM approach using AVM/LiDAR sensor fusion for rapid and accurate loop closing and parking space detection. Sensors 19:17

    Article  Google Scholar 

  47. Chen MY (2019) Bionic SLAM based on MEMS pose measurement module and RTAB-Map closed loop detection algorithm. Cluster Comput 22:S5367–S5378

    Article  Google Scholar 

  48. Labbe M, Michaud F (2013) Appearance-based loop closure detection for online large-scale and long-term operation. IEEE Trans Robot 29:734–745

    Article  Google Scholar 

  49. Zhang ZQ, Zhang JT, Tang QR (2019) Mask R-CNN based semantic RGB-D SLAM for dynamic scenes. In: IEEE ASME International Conference on Advanced Intelligent Mechatronics, pp 1151-1156

  50. Yu C, Liu Z, Liu XJ et al. (2018) DS-SLAM: A Semantic visual SLAM towards dynamic environments. In: 25th IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 1168-1174

  51. Redmon J, Farhadi A (2018) YOLOv3: An incremental improvement. arXiv:1804.02767

  52. Hertzberg C, Wagner R, Frese U, Schröder L (2013) Integrating generic sensor fusion algorithms with sound state representations through encapsulation of manifolds. Inf Fusion 14(1):57–77

    Article  Google Scholar 

  53. Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 39:2481–2495

    Article  Google Scholar 

  54. Sturm J, Burgard W, Cremers D (2012) Evaluating egomotion and structure-from-motion approaches using the TUM RGB-D benchmark. In: Proc. of the Workshop on Color-Depth Camera Fusion in Robotics at the IEEE/RJS International Conference on Intelligent Robot Systems (IROS)

  55. Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision Meets Robotics: The KITTI dataset. Int J Rob Res 32(11):1231–1237

    Article  Google Scholar 

Download references

Acknowledgements

This work is partially supported by National Natural Science Foundation of China (U21A20487), Shenzhen Technology Project (JCYJ20180302145648171, JCYJ20180507182610734, KCXFZ20201221173411032) and CAS Key Technology Talent Program.

Author information

Authors and Affiliations

  1. The CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, 518055, Shenzhen, China

    Chengqun Song, Bo Zeng, Tong Su, Ke Zhang & Jun Cheng

  2. The Chinese University of Hong Kong, Hong Kong, China

    Chengqun Song, Bo Zeng, Tong Su, Ke Zhang & Jun Cheng

Authors
  1. Chengqun Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tong Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ke Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jun Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Cheng.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, C., Zeng, B., Su, T. et al. Data association and loop closure in semantic dynamic SLAM using the table retrieval method. Appl Intell 52, 11472–11488 (2022). https://doi.org/10.1007/s10489-021-03091-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-03091-x

Keywords

Search

Navigation