Skip to main content
SpringerLink
Log in

DVL-SLAM: sparse depth enhanced direct visual-LiDAR SLAM

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This paper presents a framework for direct visual-LiDAR SLAM that combines the sparse depth measurement of light detection and ranging (LiDAR) with a monocular camera. The exploitation of the depth measurement between two sensor modalities has been reported in the literature but mostly by a keyframe-based approach or by using a dense depth map. When the sparsity becomes severe, the existing methods reveal limitation. The key finding of this paper is that the direct method is more robust under sparse depth with narrow field of view. The direct exploitation of sparse depth is achieved by implementing a joint optimization of each measurement under multiple keyframes. To ensure real-time performance, the keyframes of the sliding window are kept constant through rigorous marginalization. Through cross-validation, loop-closure achieves the robustness even in large-scale mapping. We intensively evaluated the proposed method using our own portable camera-LiDAR sensor system as well as the KITTI dataset. For the evaluation, the performance according to the LiDAR of sparsity was simulated by sampling the laser beam from 64 to 16 and 8. The experiment proves that the presented approach is significantly outperformed in terms of accuracy and robustness under sparse depth measurements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Code is available at http://github.com/irapkaist/dvl_slam.

References

  • Agarwal, P., Tipaldi, G. D., Spinello, L., Stachniss, C., & Burgard, W. (2013). Robust map optimization using dynamic covariance scaling. In IEEE international conference on robotics and automation (ICRA) (pp. 62–69).

  • Bosse, M., Zlot, R., & Flick, P. (2012). Zebedee: Design of a spring-mounted 3-d range sensor with application to mobile mapping. IEEE Transactions on Robotics, 28(5), 1104–1119.

    Article  Google Scholar 

  • Civera, J., Davison, A. J., & Montiel, J. M. M. (2008). Inverse depth parametrization for monocular slam. IEEE Transactions on Robotics, 24(5), 932–945.

    Article  Google Scholar 

  • Concha, A., & Civera, J. (2017). Rgbdtam: A cost-effective and accurate rgb-d tracking and mapping system. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 6756–6763).

  • Davison, A. J., Reid, I. D., Molton, N. D., & Stasse, O. (2007). Monoslam: Real-time single camera slam. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6), 1052–1067.

    Article  Google Scholar 

  • Endres, F., Hess, J., Sturm, J., Cremers, D., & Burgard, W. (2014). 3-d mapping with an RGB-D camera. IEEE Transactions on Robotics, 30(1), 177–187.

    Article  Google Scholar 

  • Engel, J., Koltun, V., & Cremers, D. (2017). Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(99), 611–625.

    Google Scholar 

  • Engel, J., Schöps, T., & Cremers, D. (2014). LSD-SLAM: Large-scale direct monocular SLAM. In European conference on computer vision (ECCV).

  • Engel, J., Stückler, J., & Cremers, D. (2015). Large-scale direct slam with stereo cameras. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 1935–1942).

  • Forster, C., Pizzoli, M., & Scaramuzza, D. (2014). Svo: Fast semi-direct monocular visual odometry. In 2014 IEEE international conference on robotics and automation (ICRA) (pp. 15–22).

  • Gálvez-López, D., & Tardós, J. D. (2012). Bags of binary words for fast place recognition in image sequences. IEEE Transactions on Robotics, 28(5), 1188–1197.

    Article  Google Scholar 

  • Garrido-Jurado, S., noz Salinas, R. M., Madrid-Cuevas, F., & Marín-Jiménez, M. (2014). Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition, 47(6), 2280–2292.

    Article  Google Scholar 

  • Gauglitz, S., Sweeney, C., Ventura, J., Turk, M., & Höllerer, T. (2012). Live tracking and mapping from both general and rotation-only camera motion. In Proceedings of 11th IEEE international symposium on mixed and augmented reality (ISMAR), Atlanta, Georgia, (pp. 13–22).

  • Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? the KITTI vision benchmark suite. In Conference on computer vision and pattern recognition (CVPR).

  • Gräter, J., Wilczynski, A., & Lauer, M. (2018). LIMO: Lidar-monocular visual odometry. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 7872–7879).

  • Henry, P., Krainin, M., Herbst, E., Ren, X., & Fox, D. (2012). RGB-D mapping: Using kinect-style depth cameras for dense 3D modeling of indoor environments. The International Journal of Robotics Research, 31(5), 647–663.

    Article  Google Scholar 

  • Huang, A. S., Bachrach, A., Henry, P., Krainin, M., Maturana, D., Fox, D., et al. (2011). Visual odometry and mapping for autonomous flight using an RGB-D camera. In International symposium on robotics research (ISRR) (pp. 1–16).

  • Huang, K., & Stachniss, C. (2019). Accurate direct visual-laser odometry with explicit occlusion handling and plane detection. In Proceedings of the IEEE international conference on robotics and automation (ICRA), accepted (To appear).

  • Jin, H., Favaro, P., & Soatto, S. (2001). Real-time feature tracking and outlier rejection with changes in illumination. In Proceedings 8th IEEE international conference on computer vision. ICCV 2001 (Vol. 1, pp. 684–689).

  • Kassir, A., & Peynot, T. (2010). Reliable automatic camera-laser calibration. In G. Wyeth & B. Upcroft (Eds.), Australasian conference on robotics and automation (ACRA), ARAA. Queensland: Brisbane.

  • Kerl, C., Sturm, J., & Cremers, D. (2013a). Dense visual slam for RGB-D cameras. In IEEE/RSJ international conference on intelligent robots and systems (pp. 2100–2106).

  • Kerl, C., Sturm, J., & Cremers, D. (2013b). Robust odometry estimation for RGB-D cameras. In IEEE international conference on robotics and automation (pp. 3748–3754).

  • Kitt, B., Geiger, A., & Lategahn, H. (2010). Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In IEEE intelligent vehicles symposium (IV) (pp. 486–492).

  • Klein, G., & Murray, D. (2007). Parallel tracking and mapping for small AR workspaces. In 2007 6th IEEE and ACM international symposium on mixed and augmented reality (pp. 225–234).

  • Kuemmerle, R., Grisetti, G., Strasdat, H., Konolige, K., & Burgard, W. (2011). G2o: A general framework for graph optimization. In Proceedings of the IEEE international conference on robotics and automation, Shanghai, China (pp. 3607–3613).

  • Kuznietsov, Y., Stuckler, J., & Leibe, B. (2017). Semi-supervised deep learning for monocular depth map prediction. In The IEEE conference on computer vision and pattern recognition (CVPR).

  • Lange, K. L., Little, R. J., & Taylor, J. M. (1989). Robust statistical modeling using the t distribution. Journal of the American Statistical Association, 84(408), 881–896.

    MathSciNet  Google Scholar 

  • Li, S., & Lee, D. (2016). Fast visual odometry using intensity-assisted iterative closest point. IEEE Robotics and Automation Letters, 1(2), 992–999.

    Article  Google Scholar 

  • Lu, D. (2016). Vision-enhanced lidar odometry and mapping. Master’s thesis, Carnegie Mellon University, Pittsburgh, PA.

  • Malbezin, P., Piekarski, W., & Thomas, B. H. (2002). Measuring ARTootKit accuracy in long distance tracking experiments. In The 1st IEEE international workshop agumented reality toolkit (p. 2).

  • Mur-Artal, R., Montiel, J. M. M., & Tardós, J. D. (2015). ORB-SLAM: A versatile and accurate monocular slam system. IEEE Transactions on Robotics, 31(5), 1147–1163.

    Article  Google Scholar 

  • Newcombe, R. A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., et al. (2011a). Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE international symposium on mixed and augmented reality (pp. 127–136).

  • Newcombe, R. A., Lovegrove, S. J., & Davison, A. J. (2011b). DTAM: Dense tracking and mapping in real-time. In 2011 international conference on computer vision (pp. 2320–2327).

  • Park, C., Moghadam, P., Kim, S., Elfes, A., Fookes, C., & Sridharan, S. (2018). Elastic lidar fusion: Dense map-centric continuous-time slam. In Proceedings of the IEEE international conference on robotics and automation, Brisbane (pp. 1206–1213).

  • Pinggera, P., Pfeiffer, D., Franke, U., & Mester, R. (2014). Know your limits: Accuracy of long range stereoscopic object measurements in practice. In European conference on computer vision (pp. 96–111).

  • Pizzoli, M, Forster, C., & Scaramuzza, D. (2014). Remode: Probabilistic, monocular dense reconstruction in real time. In 2014 IEEE international conference on robotics and automation (ICRA) (pp. 2609–2616).

  • Qin, T., Li, P., & Shen, S. (2018). Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 34(4), 1004–1020.

    Article  Google Scholar 

  • Segal, A., Haehnel, D., & Thrun, S. (2009). Generalized-ICP. In Proceedings of robotics: Science and systems, Seattle, USA.

  • Serafin, J., & Grisetti, G. (2015). NICP: Dense normal based point cloud registration. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 742–749).

  • Shin, Y. S., Park, Y. S., & Kim, A. (2018). Direct visual slam using sparse depth for camera-lidar system. In Proceedings of the IEEE international conference on robotics and automation, Brisbane (pp. 5144–5151).

  • Sibley, G., Matthies, L., & Sukhatme, G. (2007). Bias reduction and filter convergence for long range stereo. In S. Thrun, R. Brooks, H. Durrant-Whyte (Eds.), Robotics research (pp. 285–294). Berlin: Springer.

  • Steinbrücker, F., Sturm, J., & Cremers, D. (2011). Real-time visual odometry from dense RGB-D images. In 2011 IEEE international conference on computer vision workshops (ICCV Workshops) (pp. 719–722).

  • Strasdat, H., Davison, A. J., Montiel, J. M. M., & Konolige, K. (2011). Double window optimisation for constant time visual slam. In 2011 international conference on computer vision (pp. 2352–2359).

  • Tateno, K., Tombari, F., Laina, I., & Navab, N. (2017). Cnn-slam: Real-time dense monocular slam with learned depth prediction. In The IEEE conference on computer vision and pattern recognition (CVPR).

  • Whelan, T., Kaess, M., Johannsson, H., Fallon, M., Leonard, J. J., & McDonald, J. (2015). Real-time large-scale dense RGB-D slam with volumetric fusion. The International Journal of Robotics Research, 34(4–5), 598–626.

    Article  Google Scholar 

  • Yang, N., Wang, R., Stuckler, J., & Cremers, D. (2018). Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry. In The European conference on computer vision (ECCV).

  • Yang, Z., & Shen, S. (2017). Monocular visual-inertial state estimation with online initialization and camera-IMU extrinsic calibration. IEEE Transactions on Automation Science and Engineering, 14(1), 39–51.

    Article  Google Scholar 

  • Yin, X., Wang, X., Du, X., & Chen, Q. (2017). Scale recovery for monocular visual odometry using depth estimated with deep convolutional neural fields. In The IEEE international conference on computer vision (ICCV).

  • Zhang, J., Kaess, M., & Singh, S. (2017). A real-time method for depth enhanced visual odometry. Autonomous Robots, 41(1), 31–43.

    Article  Google Scholar 

  • Zhang, J., & Singh, S. (2015). Visual-lidar odometry and mapping: Low-drift, robust, and fast. In IEEE international conference on robotics and automation (ICRA) (pp. 2174–2181).

  • Zhang, J., & Singh, S. (2017). Low-drift and real-time lidar odometry and mapping. Autonomous Robots, 41(2), 401–416.

    Article  Google Scholar 

  • Zhang, Z., & Scaramuzza, D. (2018). A tutorial on quantitative trajectory evaluation for visual(-inertial) odometry. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 7244–7251).

Download references

Acknowledgements

This research was supported by MOLIT (19TSRD-B151228-01) and by MOTIE (No. 10067202). Y. Shin is financially supported via ‘Innovative Talent Education Program for Smart City’ by MOLIT.

Author information

Authors and Affiliations

  1. Korea Advanced Institute of Science and Technology, Daejeon, Korea

    Young-Sik Shin, Yeong Sang Park & Ayoung Kim

Authors
  1. Young-Sik Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yeong Sang Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ayoung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayoung Kim.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shin, YS., Park, Y.S. & Kim, A. DVL-SLAM: sparse depth enhanced direct visual-LiDAR SLAM. Auton Robot 44, 115–130 (2020). https://doi.org/10.1007/s10514-019-09881-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-019-09881-0

Keywords

Search

Navigation