Skip to main content
SpringerLink
Log in

Towards Dynamic Monocular Visual Odometry Based on an Event Camera and IMU Sensor

  • Conference paper
  • First Online:
Intelligent Transport Systems. From Research and Development to the Market Uptake (INTSYS 2019)

Included in the following conference series:

Abstract

Visual odometry (VO) and visual simultaneous localization and mapping (V-SLAM) have gained a lot of attention in the field of autonomous robots due to the high amount of information per unit cost vision sensors can provide. One main problem in VO techniques is the high amount of data that a pixelated image has, affecting negatively the overall performance of such techniques. An event-based camera, as an alternative to a normal frame-based camera, is a prominent candidate to solve this problem by considering only pixel changes in consecutive events that can be observed with high time resolution. However, processing the event data that is captured by event-based cameras requires specific algorithms to extract and track features applicable for odometry. We propose a novel approach to process the data of an event-based camera and use it for odometry. It is a hybrid method that combines the abilities of event-based and frame-based cameras to reach a near-optimal solution for VO. Our approach can be split into two main contributions that are (1) using information theory and non-euclidean geometry to estimate the number of events that should be processed for efficient odometry and (2) using a normal pixelated frame to determine the location of features in an event-based camera. The obtained experimental results show that our proposed technique can significantly increase performance while keeping the accuracy of pose estimation in an acceptable range.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    This technique is widely used in cinema to show the importance or inferiority of a scene by applying slow-motion, fast motion, and time-elapsed photography.

  2. 2.

    The points are represented by homogeneous vectors in projective geometry.

References

  1. Morevec, H.P.: Towards automatic visual obstacle avoidance. In: Proceedings of the 5th International Joint Conference on Artificial Intelligence, ser. IJCAI 1977, vol. 2, pp. 584–584 (1977)

    Google Scholar 

  2. Harris, C.G., Pike, J.M.: 3d positional integration from image sequences. In: Proceedings of Alvey Vision Conference, Cambridge, England (1987)

    Google Scholar 

  3. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment - a modern synthesis. In: Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, ser. ICCV 1999, pp. 298–372 (2000)

    Chapter  Google Scholar 

  4. Lichtsteiner, P., Posch, C., Delbruck, T.: A 128\(\times \)128 120 db 15\(\mu \)s latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circ. 43(2), 566–576 (2008)

    Article  Google Scholar 

  5. Rebecq, H., Horstschaefer, T., Gallego, G., Scaramuzza, D.: EVO: a geometric approach to event-based 6-dof parallel tracking and mapping in real time. IEEE Robot. Autom. Lett. 2(2), 593–600 (2017)

    Article  Google Scholar 

  6. Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423 (1948)

    Article  MathSciNet  Google Scholar 

  7. Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430–443. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_34

    Chapter  Google Scholar 

  8. Alzugaray, I., Chli, M.: Asynchronous corner detection and tracking for event cameras in real time. IEEE Robot. Autom. Lett. 3(4), 3177–3184 (2018)

    Article  Google Scholar 

  9. Mur-Artal, R., Montiel, J.M.M., Tardós, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. CoRR (2015)

    Google Scholar 

  10. Klein, G., Murray, D.: Parallel tracking and mapping on a camera phone. In: Proceedings of Eigth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2009), Orlando, October 2009

    Google Scholar 

  11. Ranftl, R., Vineet, V., Chen, Q., Koltun, V.: Dense monocular depth estimation in complex dynamic scenes. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4058–4066, June 2016

    Google Scholar 

  12. Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: Dtam: dense tracking and mapping in real-time. In: 2011 International Conference on Computer Vision, pp. 2320–2327, November 2011

    Google Scholar 

  13. Pizzoli, M., Forster, C., Scaramuzza, D.: REMODE: probabilistic, monocular dense reconstruction in real time. In: 2014 IEEE International Conference on Robotics and Automation, ICRA 2014, Hong Kong, China, 31 May - 7 June 2014, pp. 2609–2616 (2014)

    Google Scholar 

  14. Engel, J., Sturm, J., Cremers, D.: Semi-dense visual odometry for a monocular camera. In: 2013 IEEE International Conference on Computer Vision, pp. 1449–1456, December 2013

    Google Scholar 

  15. Forster, C., Pizzoli, M., Scaramuzza, D.: SVO: fast semi-direct monocular visual odometry. In: Proceedings - IEEE International Conference on Robotics and Automation, pp. 15–22 (2014)

    Google Scholar 

  16. Engel, J., Koltun, V., Cremers, D.: Direct sparse odometry. IEEE Trans. Pattern Anal. Mach. Intell. 40(3), 611–625 (2018)

    Article  Google Scholar 

  17. Cook, M., Gugelmann, L., Jug, F., Krautz, C., Steger, A.: Interacting maps for fast visual interpretation. In: The 2011 International Joint Conference on Neural Networks, pp. 770–776, July 2011

    Google Scholar 

  18. Weikersdorfer, D., Hoffmann, R., Conradt, J.: Simultaneous localization and mapping for event-based vision systems. In: Chen, M., Leibe, B., Neumann, B. (eds.) ICVS 2013. LNCS, vol. 7963, pp. 133–142. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39402-7_14

    Chapter  Google Scholar 

  19. Weikersdorfer, D., Adrian, D.B., Cremers, D., Conradt, J.: Event-based 3d SLAM with a depth-augmented dynamic vision sensor. In: 2014 IEEE International Conference on Robotics and Automation, ICRA 2014, Hong Kong, China, 31 May - 7 June 2014, pp. 359–364 (2014)

    Google Scholar 

  20. Kim, H., Handa, A., Benosman, R., Ieng, S.-H., Davison, A.: Simultaneous mosaicing and tracking with an event camera. In: Proceedings of the British Machine Vision Conference. BMVA Press (2014)

    Google Scholar 

  21. Kueng, B., Mueggler, E., Gallego, G., Scaramuzza, D.: Low-latency visual odometry using event-based feature tracks. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2016, Daejeon, South Korea, 9–14 October 2016, pp. 16–23 (2016)

    Google Scholar 

  22. Kim, H., Leutenegger, S., Davison, A.J.: Real-time 3d reconstruction and 6-dof tracking with an event camera. In: Computer Vision - ECCV 2016 - 14th European Conference, Proceedings, Part VI, Amsterdam, The Netherlands, 11–14 October 2016, pp. 349–364 (2016)

    Google Scholar 

  23. Thum, C.: Measurement of the entropy of an image with application to image focusing. Opt. Acta: Int. J. Opt. 31(2), 203–211 (1984)

    Article  MathSciNet  Google Scholar 

  24. Grinberg, M.: Feature-based probabilistic data association for video-based multi-object tracking," Ph.D. dissertation, Karlsruhe Institute of Technology, Germany (2018)

    Google Scholar 

  25. Brandli, C., Berner, R., Yang, M., Liu, S., Delbruck, T.: A 240\(\times \)180 130 db 3\(\mu \)s latency global shutter spatiotemporal vision sensor. IEEE J. Solid-State Circ. 49(10), 2333–2341 (2014)

    Article  Google Scholar 

  26. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. PAMI–8(6), 679–698 (1986)

    Article  Google Scholar 

  27. Besl, P.J., McKay, N.D.: A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)

    Article  Google Scholar 

  28. Mueggler, E., Rebecq, H., Gallego, G., Delbrück, T., Scaramuzza, D.: The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM. Int. J. Robotics Res. 36(2), 142–149 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Turku (UTU), 20500, Turku, Finland

    Sherif A. S. Mohamed, Mohammad-Hashem Haghbayan, Jukka Heikkonen, Hannu Tenhunen & Juha Plosila

  2. Royal Institute of Technology (KTH), 11419, Stockholm, Sweden

    Hannu Tenhunen

  3. Kunsan National University (KNU), Gunsan, 54150, South Korea

    Mohammed Rabah

Authors
  1. Sherif A. S. Mohamed
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad-Hashem Haghbayan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammed Rabah
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jukka Heikkonen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hannu Tenhunen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Juha Plosila
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sherif A. S. Mohamed .

Editor information

Editors and Affiliations

  1. ISCTE-IUL, Lisbon, Portugal

    Ana Lúcia Martins

  2. ISCTE-IUL, Lisbon, Portugal

    Joao Carlos Ferreira

  3. University of Pisa, Pisa, Italy

    Alexander Kocian

Rights and permissions

Reprints and permissions

Copyright information

© 2020 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mohamed, S.A.S., Haghbayan, MH., Rabah, M., Heikkonen, J., Tenhunen, H., Plosila, J. (2020). Towards Dynamic Monocular Visual Odometry Based on an Event Camera and IMU Sensor. In: Martins, A., Ferreira, J., Kocian, A. (eds) Intelligent Transport Systems. From Research and Development to the Market Uptake. INTSYS 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 310. Springer, Cham. https://doi.org/10.1007/978-3-030-38822-5_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-38822-5_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-38821-8

  • Online ISBN: 978-3-030-38822-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation