Skip to main content
SpringerLink
Log in

Robust stereo inertial odometry based on self-supervised feature points

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In the application of intelligent mobile robots, the odometry is the key system for implementing positioning. Traditional feature extraction algorithms can not work stably in challenging environments such as low-textured areas, and when the camera moves rapidly, the visual odometry can not track features. To solve the above problems, the paper proposes a robust stereo inertial odometry based on self-supervised feature points. An improved multi-task CNN is designed to extract the feature points in the images acquired by the stereo camera. In addition, we add the Inertial Measurement Unit (IMU) to cope with the rapid motion of the camera. Finally, the fixed number of key frames and IMU errors are optimized in the sliding window by minimizing a combined error function. The experimental results show that the proposed system can run in challenging scenes and maintain real-time performance. The overall performance of the proposed system is better than that of the classical stereo inertial odometry systems, and it is still competitive with the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Balntas V, Lenc K, Vedaldi A, Mikolajczyk K (2017) HPatches: a benchmark and evaluation of handcrafted and learned local descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5173–5182

  2. Bay H, Ess A, Tuytelaars T, van Gool L (2008) Speeded-up robust features (SURF). Comput Vis Image Underst 110(3):346–359

    Article  Google Scholar 

  3. Bloesch M, Omari S, et al. (2015) Robust visual inertial odometry using a direct EKF-based approach. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 298-304

  4. Burri M, Nikolic J, Gohl P, Schneider T, Rehder J, Omari S, Achtelik MW, Siegwart R (2016) The EuRoC micro aerial vehicle datasets. Int J Robot Res 35(10):1157–1163

    Article  Google Scholar 

  5. Chen Y, Jiang H, Li C, Jia X, Ghamisi P (2016) Deep feature extraction and classification of hyperspectral images based on convolutional neural networks. IEEE Trans Geosci Remote Sens 54(10):6232–6251

    Article  Google Scholar 

  6. Christiansen PH, Kragh MF, Brodskiy Y, et al. (2019) Unsuperpoint: end-to-end unsupervised interest point detector and descriptor. arXiv:1907.04011

  7. DeTone D, Malisiewicz T, Rabinovich A (2018) Superpoint: self-supervised interest point detection and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 224–236

  8. Engel J, Schöps T, Cremers D (2014) LSD-SLAM: Large-scale direct monocular SLAM. In: European Conference on Computer Vision, pp. 834–849

  9. Engel J, Koltun V, Cremers D (2017) Direct sparse odometry. IEEE Trans Pattern Anal Mach Intell 40(3):611–625

    Article  Google Scholar 

  10. Forster C, Carlone L, Dellaert F, Scaramuzza D (2016) On-manifold Preintegration for real-time visual--inertial Odometry. IEEE Trans Robot 33(1):1–21

    Article  Google Scholar 

  11. Gomez-Ojeda R, Moreno FA, Zuniga-Noël D et al (2019) PL-SLAM: a stereo SLAM system through the combination of points and line segments. IEEE Trans Robot 35(3):734–746

    Article  Google Scholar 

  12. Huang H, Ye H, Sun Y, et al. (2020) Monocular visual odometry using learned repeatability and description. In: IEEE international conference on robotics and automation (ICRA), pp. 913-8919

  13. Jang H, Plis SM, Calhoun VD, Lee JH (2017) Task-specific feature extraction and classification of fMRI volumes using a deep neural network initialized with a deep belief network: evaluation using sensorimotor tasks. NeuroImage 145:314–328

    Article  Google Scholar 

  14. Lee S, Lee S (2013) Embedded visual SLAM: applications for low-cost consumer robots. IEEE Robot Autom Mag 20(4):83–95

    Article  Google Scholar 

  15. Leutenegger S, Lynen S, Bosse M et al (2014) Keyframe-based visual-inertial Odometry using nonlinear optimization. Int J Robot Res 34(3):314–334

    Article  Google Scholar 

  16. Li G, Yu L, Fei S (2020) A binocular MSCKF-based visual inertial Odometry system using LK optical flow. J Intell Robot Syst 100(3):1179–1194

    Article  Google Scholar 

  17. Li G, Yu L, Fei S (2021) A deep-learning real-time visual SLAM system based on multi-task feature extraction network and self-supervised feature points. Measurement 168:108403

    Article  Google Scholar 

  18. Lowe DG (2004) Distinctive image features from scale-invariant Keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  19. Ma T, Kuang P, Tian W (2020) An improved recurrent neural networks for 3d object reconstruction. Appl Intell 50(3):905–923

    Article  Google Scholar 

  20. Morris KJ, Samonin V, Baltes J, Anderson J, Lau MC (2019) A robust interactive entertainment robot for robot magic performances. Appl Intell 49(11):3834–3844

    Article  Google Scholar 

  21. Mur-Artal R, Tardós JD (2017) Visual-inertial monocular SLAM with map reuse. IEEE Robot Autom Lett 2(2):796–803

    Article  Google Scholar 

  22. Mur-Artal R, Tardós JD (2017) Orb-slam2: an open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Trans Robot 33(5):1255–1262

    Article  Google Scholar 

  23. Nistér D, Naroditsky O, Bergen JR (2010) Visual odometry for ground vehicle applications. J Field Robot 23(1):3–20

    Article  MATH  Google Scholar 

  24. Oskiper T, Samarasekera S, Kumar R (2011) Tightly-coupled robust vision aided inertial navigation algorithm for augmented reality using monocular camera and IMU. In: 2011 10th IEEE international symposium on mixed and augmented reality, pp. 255-256

  25. Rosten E, Porter R, Drummond T (2008) Faster and better: a machine learning approach to corner detection. IEEE Trans Pattern Anal Mach Intell 32(1):105–119

    Article  Google Scholar 

  26. Rublee E, Rabaud V, Konolige K, et al. (2011) ORB: an efficient alternative to SIFT or SURF. In: 2011 international conference on computer vision, pp. 2564-2571

  27. Sun K, Mohta K, Pfrommer B, Watterson M, Liu S, Mulgaonkar Y, Taylor CJ, Kumar V (2018) Robust stereo visual inertial odometry for fast autonomous flight. IEEE Robot Autom Lett 3(2):965–972

    Article  Google Scholar 

  28. Tang J, Folkesson J, Jensfelt P (2018) Geometric correspondence network for camera motion estimation. IEEE Robot Autom Lett 3(2):1010–1017

    Article  Google Scholar 

  29. Tong Q, Li P, Shen S (2018) VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator. IEEE Trans Robot 34(4):1–17

    Google Scholar 

  30. Usenko V, Demmel N, Schubert D et al (2019) Visual-inertial mapping with non-linear factor recovery. IEEE Robot Autom Lett 5(2):422–429

    Article  Google Scholar 

  31. Von Stumberg L, Usenko V, Cremers D (2018) Direct sparse visual-inertial odometry using dynamic marginalization. In: 2018 IEEE international conference on robotics and automation (ICRA), pp. 2510-2517

  32. Yang S, Deng B, Wang J et al (2019) Scalable digital neuromorphic architecture for large-scale biophysically meaningful neural network with multi-compartment neurons. IEEE Trans Neural Netw Learn Syst 31(1):148–162

    Article  Google Scholar 

  33. Yang S, Gao T, Wang J, Deng B, Lansdell B, Linares-Barranco B (2021) Efficient spike-driven learning with dendritic event-based processing. Front Neurosci 15:97

    Article  Google Scholar 

  34. Yang S, Wang J, Deng B, Azghadi MR, Linares-Barranco B (2021) Neuromorphic context-dependent learning framework with fault-tolerant spike routing. IEEE Trans Neural Netw Learn Syst:1–15

  35. Yang S, Wang J, Hao X, Li H, Wei X, Deng B, Loparo KA (2021) BiCoSS: toward large-scale cognition brain with multigranular neuromorphic architecture. IEEE Trans Neural Netw Learn Syst:1–15

  36. Yang S, Wang J, Zhang N, et al. (2021) CerebelluMorphic: large-scale neuromorphic model and architecture for supervised motor learning. IEEE Trans Neural Netw Learn Syst. 1-15

  37. Yi KM, Trulls E, Lepetit V, et al. (2016) Lift: learned invariant feature transform. In: European Conference on Computer Vision, pp. 467–483

  38. Zhou H, Ummenhofer B, Brox T (2018) Deeptam: deep tracking and mapping. In: proceedings of the European conference on computer vision (ECCV), pp. 822-838

  39. Zou D, Tan P (2012) Coslam: collaborative visual slam in dynamic environments. IEEE Trans Pattern Anal Mach Intell 35(2):354–366

    Article  Google Scholar 

Download references

Acknowledgments

The work is supported by the National Natural Science Foundation of China (No.61873176;51705184); Natural Science Foundation of Jiangsu Province, China (BK20181433);The fund of Jiangsu CHINA_ISRAEL Industrial Technology Research Institute (JSIITRI202107); Tang Scholar Project of Soochow University(2021); Postgraduate Research & Practice Innovation Program of Jiangsu Province, China (KYCX21_2954); Natrual Science Foundation of Shandong Province (Grant No:ZR2017LF010); the Qing Lan Project of the Higher Education Institutions of Jiangsu Province(〔2019〕3,〔2020〕10,〔2021〕11); Basic science research project of Nantong (JC2020154). The authors would like to thank the referees for their constructive comments.

Author information

Authors and Affiliations

  1. School of Mechanical and Electric Engineering, Soochow University, Suzhou, China

    Guangqiang Li, Junyi Hou & Lei Yu

  2. Jiangsu Key Laboratory of Advanced Manufacturing Technology, Huaiyin Institute Of Technology, Huai’an, China

    Guangqiang Li, Zhong Chen & Lei Yu

  3. School of Automation, Southeast University, Nanjing, China

    Shumin Fei

Authors
  1. Guangqiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junyi Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lei Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shumin Fei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Yu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, G., Hou, J., Chen, Z. et al. Robust stereo inertial odometry based on self-supervised feature points. Appl Intell 53, 7093–7107 (2023). https://doi.org/10.1007/s10489-022-03278-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03278-w

Keywords

Search

Navigation