Skip to main content
SpringerLink
Log in

A tightly coupled monocular visual lidar odometry with loop closure

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Simultaneous localization and mapping (SLAM) is a fundamental requirement for mobile robots like self-driving cars. Vision-based methods have advantages in sensor cost and loop closure detection, but are sensitive to illumination change and texture deficiency. Lidar-based SLAM systems perform better in accuracy, field-of-view and robustness to environmental changes, but may easily fail in structure-less scenarios. To compensate for the deficiencies of standalone sensors and provide more efficient SLAM functions, in this paper we propose a tightly coupled monocular visual lidar odometry, which tightly fuses the measurements of a monocular camera and a 3D lidar in a joint optimization. The system starts with a data preprocessing module, which outputs 3D visual and laser features through feature extraction and data association. Furthermore, the tightly coupled visual lidar odometry tightly fuses the visual and laser features in a unified optimization framework to estimate the transformation between consecutive scans. Finally, we combine visual and vicinity loop detection to construct loop constraints and optimize a 6-DOF global pose graph to achieve global-consistent pose estimation and environment mapping. The performance of our system is verified in the public KITTI dataset, and the experimental results demonstrate that the proposed method can run in real time with the 64-line lidar data and achieve better in accuracy, runtime and mapping against other state-of-the-art lidar-based and fusion-based methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Besl PJ, McKay ND (1992) Method for registration of 3-d shapes. In: Sensor fusion IV: control paradigms and data structures, vol 1611. International Society for Optics and Photonics, pp 586–606

  2. Bogoslavskyi I, Stachniss C (2016) Fast range image-based segmentation of sparse 3d laser scans for online operation. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 163–169

  3. Calonder M, Lepetit V, Strecha C, Fua P (2010) Brief: binary robust independent elementary features. In: European conference on computer vision. Springer, pp 778–792

  4. Cheng J, Wang C, Meng MQH (2019) Robust visual localization in dynamic environments based on sparse motion removal. IEEE Trans Autom Sci Eng 1–12

  5. Curless B, Levoy M (1996) A volumetric method for building complex models from range images. In: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp 303–312

  6. De Croce M, Pire T, Bergero F (2019) Ds-ptam: distributed stereo parallel tracking and mapping slam system. J Intell Robotic Syst 95(2):365–377

    Article  Google Scholar 

  7. Deschaud JE (2018) Imls-slam: scan-to-model matching based on 3d data. In: 2018 IEEE international conference on robotics and automation (ICRA). IEEE, pp 2480–2485

  8. Engel J, Stückler J, Cremers D (2015) Large-scale direct slam with stereo cameras. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 1935–1942

  9. Gálvez-López D, Tardos JD (2012) Bags of binary words for fast place recognition in image sequences. IEEE Trans Robotics 28(5):1188–1197

    Article  Google Scholar 

  10. Gao X, Wang R, Demmel N, Cremers D (2018) Ldso: direct sparse odometry with loop closure, pp 2198–2204

  11. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 3354–3361

  12. Himmelsbach M, Hundelshausen FV, Wuensche HJ (2010) Fast segmentation of 3d point clouds for ground vehicles. In: 2010 IEEE intelligent vehicles symposium. IEEE, pp 560–565

  13. Kaess M, Johannsson H, Roberts R, Ila V, Leonard J, Dellaert F (2011) isam2: incremental smoothing and mapping with fluid relinearization and incremental variable reordering. In: 2011 IEEE international conference on robotics and automation. IEEE, pp 3281–3288

  14. Kim D, Shin S, Kweon IS (2018) On-line initialization and extrinsic calibration of an inertial navigation system with a relative preintegration method on manifold. IEEE Trans Autom Sci Eng 15(3):1272–1285

    Article  Google Scholar 

  15. Li G, Yu L, Fei S (2020) A binocular msckf-based visual inertial odometry system using lk optical flow. J Intell Robotic Syst 100(3):1179–1194

    Article  Google Scholar 

  16. Lucas BD, Kanade T et al (1981) An iterative image registration technique with an application to stereo vision

  17. Lynen S, Achtelik MW, Weiss S, Chli M, Siegwart R (2013) A robust and modular multi-sensor fusion approach applied to mav navigation. In: 2013 IEEE/RSJ international conference on intelligent robots and systems. IEEE, pp 3923–3929

  18. Mccormac J, Handa A, Davison AJ, Leutenegger S (2017) Semanticfusion: dense 3d semantic mapping with convolutional neural networks, pp 4628–4635

  19. Mourikis AI, Roumeliotis SI (2007) A multi-state constraint kalman filter for vision-aided inertial navigation. In: Proceedings 2007 IEEE international conference on robotics and automation. IEEE, pp 3565–3572

  20. Mur-Artal R, Tardós JD (2017) Orb-slam2: an open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Trans Robotics 33(5):1255–1262

    Article  Google Scholar 

  21. Qin T, Li P, Shen S (2018) Vins-mono: a robust and versatile monocular visual-inertial state estimator. IEEE Trans Robotics 34(4):1004–1020

    Article  Google Scholar 

  22. Ramasubramanian V, Paliwal KK (1992) Fast k-dimensional tree algorithms for nearest neighbor search with application to vector quantization encoding. IEEE Trans Sig Process 40(3):518–531

    Article  Google Scholar 

  23. Shan T, Englot B (2018) Lego-loam: lightweight and ground-optimized lidar odometry and mapping on variable terrain. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 4758–4765

  24. Shao W, Vijayarangan S, Li C, Kantor G (2019) Stereo visual inertial lidar simultaneous localization and mapping. arXiv preprint arXiv:1902.10741

  25. Shi J et al (1994) Good features to track. In: 1994 Proceedings of IEEE conference on computer vision and pattern recognition. IEEE, pp 593–600

  26. Wang Z, Zhang J, Chen S, Yuan C, Zhang J, Zhang J (2019) Robust high accuracy visual-inertial-laser slam system. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 6636–6641

  27. Yang Z, Shen S (2017) Monocular visual-inertial state estimation with online initialization and camera-imu extrinsic calibration. IEEE Trans Autom Sci Eng 14(1):39–51

    Article  Google Scholar 

  28. Zhang J, Kaess M, Singh S (2016) On degeneracy of optimization-based state estimation problems. In: 2016 IEEE international conference on robotics and automation (ICRA). IEEE, pp 809–816

  29. Zhang J, Singh S (2015) Visual-lidar odometry and mapping: low-drift, robust, and fast. In: 2015 IEEE international conference on robotics and automation (ICRA). IEEE, pp 2174–2181

  30. Zhang J, Singh S (2017) Low-drift and real-time lidar odometry and mapping. Autonom Robots 41(2):401–416

    Article  Google Scholar 

  31. Zhang J, Singh S (2018) Laser-visual-inertial odometry and mapping with high robustness and low drift. J Field Robotics 35(8):1242–1264

    Article  Google Scholar 

  32. Zuiderveld KJ (1994) Contrast limited adaptive histogram equalization. Graphics gems, pp 474–485

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China (61973099).

Author information

Authors and Affiliations

  1. Harbin Institute of Technology, Harbin, China

    Lingbo Meng, Chao Ye & Weiyang Lin

Authors
  1. Lingbo Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chao Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weiyang Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. All authors commented on previous versions of the manuscript, and approved the final manuscript.

Corresponding author

Correspondence to Weiyang Lin.

Ethics declarations

Conflicts of interest

The authors declare that there is no conflict of interests regarding the publication of this paper.

Ethics approval

The authors declare that there are no ethics issues regarding the publication of this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Meng, L., Ye, C. & Lin, W. A tightly coupled monocular visual lidar odometry with loop closure. Intel Serv Robotics 15, 129–141 (2022). https://doi.org/10.1007/s11370-021-00406-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-021-00406-2

Keywords

Search

Navigation