Skip to main content

Part of the book series: Studies in Computational Intelligence ((SCI,volume 1093))

  • 769 Accesses

Abstract

Robotic automation has always been employed to optimize tasks that are deemed repetitive or hazardous for humans. One instance of such an application is within transportation, be it in urban environments or other harsh applications. In said scenarios, it is required for the platform’s operator to be at a heightened level of awareness at all times to ensure the safety of on-board materials being transported. Additionally, during longer journeys it is often the case that the driver might also be required to traverse difficult terrain under extreme conditions. For instance, low light, fog, or haze-ridden paths. To counter this issue, recent studies have proven that the assistance of smart systems is necessary to minimize the risk involved. In order to develop said systems, this chapter discusses a concept of a Deep Learning (DL) based Vision Navigation (VN) approach capable of terrain analysis and determining the appropriate steering angle within a margin of safety. Within the framework of Neuromorphic Vision (NV) and Event Cameras (EC), the proposed concept is tackling several issues within the development of autonomous systems. In particular, the use of Transformer based backbone for off-road depth estimation using an event camera for better accuracy result and processing time. The implementation of the above mentioned deep learning system, using event camera is leveraged through the necessary data processing techniques of the events prior to the training phase. Besides, binary convolutions (BN) and alternately spiking convolution paradigms using the latest technology trend have been deployed as acceleration methods, with efficiency in terms of energy latency, and environmental robustness. Initial results hold promising potential for the future development of real-time projects with event cameras.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Badue, C., Guidolini, R., Carneiro, R. V., Azevedo, P., Cardoso, V. B., Forechi, A., Jesus, L., Berriel, R., Paixão, T. M., Mutz, F., de Paula Veronese, L., Oliveira-Santos, T., & De Souza, A. F. (2021). Self-driving cars: A survey. Expert Systems with Applications, 165.

    Google Scholar 

  2. Ni, J., Chen, Y., Chen, Y., Zhu, J., Ali, D., & Cao, W. (2020) A survey on theories and applications for self-driving cars based on deep learning methods. Applied Sciences (Switzerland), 10.

    Google Scholar 

  3. Chen, G., Cao, H., Conradt, J., Tang, H., Rohrbein, F., & Knoll, A. (2020). Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Processing Magazine, 37.

    Google Scholar 

  4. Lin, M., Yoon, J., & Kim, B. (2020) Self-driving car location estimation based on a particle-aided unscented kalman filter. Sensors (Switzerland), 20.

    Google Scholar 

  5. Mugunthan, N., Naresh, V. H., & Venkatesh, P. V. (2020). Comparison review on lidar vs camera in autonomous vehicle. In International Research Journal of Engineering and Technology.

    Google Scholar 

  6. Ming, Y., Meng, X., Fan, C., & Yu, H. (2021) Deep learning for monocular depth estimation: A review. Neurocomputing, 438.

    Google Scholar 

  7. Li, X., Tang, B., Ball, J., Doude, M., & Carruth, D. W. (2019). Rollover-free path planning for off-road autonomous driving. Electronics (Switzerland), 8.

    Google Scholar 

  8. Pan, Y., Cheng, C. A., Saigol, K., Lee, K., Yan, X., Theodorou, E. A., & Boots, B. (2020). Imitation learning for agile autonomous driving. International Journal of Robotics Research, 39.

    Google Scholar 

  9. Liu, O., Yuan, S., & Li, Z. (2020). A survey on sensor technologies for unmanned ground vehicles. In Proceedings of 2020 3rd International Conference on Unmanned Systems, ICUS 2020.

    Google Scholar 

  10. Shin, J., Kwak, D. J., & Kim, J. (2021). Autonomous platooning of multiple ground vehicles in rough terrain. Journal of Field Robotics, 38.

    Google Scholar 

  11. Naranjo, J. E., Jiménez, F., Anguita, M., & Rivera, J. L. (2020). Automation kit for dual-mode military unmanned ground vehicle for surveillance missions. IEEE Intelligent Transportation Systems Magazine, 12.

    Google Scholar 

  12. Browne, M., Macharis, C., Sanchez-diaz, I., Brolinson, M., & Illsjö, R. (2017). Urban traffic congestion and freight transport : A comparative assessment of three european cities. Interdisciplinary Conference on Production Logistics and Traffic.

    Google Scholar 

  13. Zhong, H., Zhou, J., Du, Z., & Xie, L. (2018). A laboratory experimental study on laser attenuations by dust/sand storms. Journal of Aerosol Science, 121.

    Google Scholar 

  14. Koepke, P., Gasteiger, J., & Hess, M. (2015). Technical note: Optical properties of desert aerosol with non-spherical mineral particles: Data incorporated to opac. Atmospheric Chemistry and Physics Discussions, 15, 3995–4023.

    Google Scholar 

  15. Raja, A. R., Kagalwala, Q. J., Landolsi, T., & El-Tarhuni, M. (2007). Free-space optics channel characterization under uae weather conditions. In ICSPC 2007 Proceedings - 2007 IEEE International Conference on Signal Processing and Communications.

    Google Scholar 

  16. Vargasrivero, J. R., Gerbich, T., Buschardt, B., & Chen, J. (2021). The effect of spray water on an automotive lidar sensor: A real-time simulation study. IEEE Transactions on Intelligent Vehicles.

    Google Scholar 

  17. Strawbridge, K. B., Travis, M. S., Firanski, B. J., Brook, J. R., Staebler, R., & Leblanc, T. (2018). A fully autonomous ozone, aerosol and nighttime water vapor lidar: A synergistic approach to profiling the atmosphere in the canadian oil sands region. Atmospheric Measurement Techniques, 11.

    Google Scholar 

  18. Hummel, B., Kammel, S., Dang, T., Duchow, C., & Stiller, C. (2006). Vision-based path-planning in unstructured environments. In IEEE Intelligent Vehicles Symposium, Proceedings.

    Google Scholar 

  19. Mueller, G. R., & Wuensche, H. J. (2018). Continuous stereo camera calibration in urban scenarios. In IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, 2018-March.

    Google Scholar 

  20. Rankin, A. L., Huertas, A., & Matthies, L. H. (2009). Stereo-vision-based terrain mapping for off-road autonomous navigation. Unmanned Systems Technology X, I, 7332.

    Google Scholar 

  21. Litzenberger, M., Belbachir, A. N., Donath, N., Gritsch, G., Garn, H., Kohn, B., Posch, C., & Schraml, S. (2006). Estimation of vehicle speed based on asynchronous data from a silicon retina optical sensor. In IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC.

    Google Scholar 

  22. Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., & Scaramuzza, D. (2020). Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44.

    Google Scholar 

  23. DelbrĂĽck, T., Linares-Barranco, B., Culurciello, E., & Posch, C. (2010). Activity-driven, event-based vision sensors. In ISCAS 2010 - 2010 IEEE International Symposium on Circuits and Systems: Nano-Bio Circuit Fabrics and Systems.

    Google Scholar 

  24. Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D. (2021). High speed and high dynamic range video with an event camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43.

    Google Scholar 

  25. Lichtsteiner, P., Posch, C., & Delbruck, T. (2008). A 128\(\times \) 128 120 db 15 \(\upmu \)s latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, 43, 566–576.

    Article  Google Scholar 

  26. Brändli, C., Berner, R., Yang, M., Liu, S.-C., & Delbruck, T. (2014). A 240 \(\times \) 180 130 db 3 \(\upmu \)s latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49, 2333–2341.

    Article  Google Scholar 

  27. Scheerlinck, C., Barnes, N., & Mahony, R. (2019). Continuous-time intensity estimation using event cameras. Lecture notes in computer science (including subseries Lecture notes in artificial intelligence and lecture notes in bioinformatics), 11365 LNCS.

    Google Scholar 

  28. Gallego, G., Lund, J. E. A., Mueggler, E., Rebecq, H., Delbruck, T., & Scaramuzza, D. (2018). Event-based, 6-dof camera tracking from photometric depth maps. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40.

    Google Scholar 

  29. Mostafavi, M., Wang, L., & Yoon, K. J. (2021). Learning to reconstruct hdr images from events, with applications to depth and flow prediction. International Journal of Computer Vision, 129.

    Google Scholar 

  30. Mueggler, E., Huber, B., & Scaramuzza, D. (2014). Event-based, 6-dof pose tracking for high-speed maneuvers.

    Google Scholar 

  31. Posch, C., Matolin, D., & Wohlgenannt, R. (2011). A qvga 143 db dynamic range frame-free pwm image sensor with lossless pixel-level video compression and time-domain cds. IEEE Journal of Solid-State Circuits, 46.

    Google Scholar 

  32. Lee, S., Kim, H., & Kim, H. J. (2020). Edge detection for event cameras using intra-pixel-area events. In 30th British Machine Vision Conference 2019, BMVC 2019.

    Google Scholar 

  33. Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D. (2019). Events-to-video: Bringing modern computer vision to event cameras. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June.

    Google Scholar 

  34. Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017). End-to-end learning of driving models from large-scale video datasets. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January.

    Google Scholar 

  35. Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017). End-to-end learning of driving models from large-scale video datasets. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January.

    Google Scholar 

  36. Boahen, K. A. (2004). A burst-mode word-serial address-event link - i: Transmitter design (p. 51). IEEE Transactions on Circuits and Systems I: Regular Papers.

    Google Scholar 

  37. Wang, C., Buenaposada, J. M., Zhu, R., & Lucey, S. (2018). Learning depth from monocular videos using direct methods. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

    Google Scholar 

  38. Guo, S., Kang, Z., Wang, L., Zhang, L., Chen, X., Li, S., & Xu, W. (2020). A noise filter for dynamic vision sensors using self-adjusting threshold.

    Google Scholar 

  39. Gehrig, D., Ruegg, M., Gehrig, M., Hidalgo-Carrio, J., & Scaramuzza, D. (2021). Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction. IEEE Robotics and Automation Letters, 6.

    Google Scholar 

  40. Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., & Dai, Y. (2019). Bringing a blurry frame alive at high frame-rate with an event camera. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June.

    Google Scholar 

  41. Pan, L., Hartley, R., Scheerlinck, C., Liu, M., Yu, X., & Dai, Y. (2022). High frame rate video reconstruction based on an event camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44.

    Google Scholar 

  42. Gehrig, D., Rebecq, H., Gallego, G., & Scaramuzza, D. (2020). Eklt: Asynchronous photometric feature tracking using events and frames. International Journal of Computer Vision, 128.

    Google Scholar 

  43. Saner, D., Wang, O., Heinzle, S., Pritch, Y., Smolic, A., Sorkine-Hornung, A., & Gross, M. (2014). High-speed object tracking using an asynchronous temporal contrast sensor. In 19th International Workshop on Vision, Modeling and Visualization, VMV 2014.

    Google Scholar 

  44. Muglikar, M., Gehrig, M., Gehrig, D., & Scaramuzza, D. (2021). How to calibrate your event camera. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

    Google Scholar 

  45. Maqueda, A. I., Loquercio, A., Gallego, G., Garcia, N., & Scaramuzza, D. (2018). Event-based vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

    Google Scholar 

  46. Galluppi, F., Denk, C., Meiner, M. C., Stewart, T. C., Plana, L. A., Eliasmith, C., Furber, S., & Conradt, J. (2014). Event-based neural computing on an autonomous mobile platform. In Proceedings - IEEE International Conference on Robotics and Automation.

    Google Scholar 

  47. Hu, Y., Binas, J., Neil, D., Liu, S. C., & Delbruck, T. (2020). Ddd20 end-to-end event camera driving dataset: Fusing frames and events with deep learning for improved steering prediction. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems, ITSC 2020.

    Google Scholar 

  48. Zhong, H., Wang, H., Wu, Z., Zhang, C., Zheng, Y., & Tang, T. (2021). A survey of lidar and camera fusion enhancement. Procedia Computer Science, 183.

    Google Scholar 

  49. Song, R., Jiang, Z., Li, Y., Shan, Y., & Huang, K. (2018). Calibration of event-based camera and 3d lidar. In 2018 WRC Symposium on Advanced Robotics and Automation, WRC SARA 2018 - Proceeding.

    Google Scholar 

  50. Zhou, Y., Gallego, G., & Shen, S. (2021). Event-based stereo visual odometry. IEEE Transactions on Robotics, 37.

    Google Scholar 

  51. Dahlkamp, H., Kaehler, A., Stavens, D., Thrun, S., & Bradski, G. (2007). Self-supervised monocular road detection in desert terrain. Robotics: Science and Systems, 2.

    Google Scholar 

  52. Bayard, D. S., Conway, D. T., Brockers, R., Delaune, J., Matthies, L., Grip, H. F., Merewether, G., Brown, T., & Martin, A. M. S. (2019). Vision-based navigation for the nasa mars helicopter. AIAA Scitech 2019 Forum.

    Google Scholar 

  53. Hidalgo-Carrio, J., Gehrig, D., & Scaramuzza, D. (2020). Learning monocular dense depth from events. In Proceedings - 2020 International Conference on 3D Vision, 3DV 2020.

    Google Scholar 

  54. Li, Z., Asif, M. S., & Ma, Z. (2022). Event transformer.

    Google Scholar 

  55. Juefei-Xu, F., Boddeti, V. N., & Savvides, M. (2017). Local binary convolutional neural networks. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January.

    Google Scholar 

  56. Khodamoradi, A., & Kastner, R. (2021). O(n)-space spatiotemporal filter for reducing noise in neuromorphic vision sensors. IEEE Transactions on Emerging Topics in Computing, 9.

    Google Scholar 

  57. Feng, Y., Lv, H., Liu, H., Zhang, Y., Xiao, Y., & Han, C. (2020). Event density based denoising method for dynamic vision sensor. Applied Sciences (Switzerland), 10.

    Google Scholar 

  58. Meyer, L., Smíšek, M., Villacampa, A. F., Maza, L. O., Medina, D., Schuster, M. J., Steidle, F., Vayugundla, M., Müller, M. G., Rebele, B., Wedler, A., & Triebel, R. (2021). The madmax data set for visual-inertial rover navigation on mars. Journal of Field Robotics, 38.

    Google Scholar 

  59. Figurnov, M., Ibraimova, A., Vetrov, D., & Kohli, P. (2016). Perforatedcnns: Acceleration through elimination of redundant convolutions. Advances in Neural Information Processing Systems, 29.

    Google Scholar 

  60. Salman, A. M., Tulan, A. S., Mohamed, R. Y., Zakhari, M. H., & Mostafa, H. (2020). Comparative study of hardware accelerated convolution neural network on pynq board. In 2nd Novel Intelligent and Leading Emerging Sciences Conference, NILES 2020.

    Google Scholar 

  61. Yoshida, Y., Oiwa, R., & Kawahara, T. (2018). Ternary sparse xnor-net for fpga implementation. In Proceedings - 7th International Symposium on Next-Generation Electronics. ISNE, 2018.

    Google Scholar 

  62. Ding, C., Wang, S., Liu, N., Xu, K., Wang, Y., & Liang, Y. (2019). Req-yolo: A resource-aware, efficient quantization framework for object detection on fpgas. In FPGA 2019 - Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays.

    Google Scholar 

  63. Li, J. N., & Tian, Y. H. (2021). Recent advances in neuromorphic vision sensors: A survey. Jisuanji Xuebao/Chinese Journal of Computers, 44.

    Google Scholar 

  64. Chen, G., Cao, H., Aafaque, M., Chen, J., Ye, C., Röhrbein, F., Conradt, J., Chen, K., Bing, Z., Liu, X., Hinz, G., Stechele, W., & Knoll, A. (2018) Neuromorphic vision based multivehicle detection and tracking for intelligent transportation system. Journal of Advanced Transportation, 2018.

    Google Scholar 

  65. Gutierrez-Galan, D., Schoepe, T., Dominguez-Morales, J. P., Jiménez-Fernandez, A., Chicca, E., & Linares-Barranco, A. (2020). An event-based digital time difference encoder model implementation for neuromorphic systems.

    Google Scholar 

  66. Schuman, C. D., Kulkarni, S. R., Parsa, M., Mitchell, J. P., Date, P., & Kay, B. (2022). Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science, 2.

    Google Scholar 

  67. Richter, C., Jentzsch, S., Hostettler, R., Garrido, J. A., Ros, E., Knoll, A., et al. (2016). Musculoskeletal robots: Scalability in neural control. IEEE Robotics & Automation Magazine, 23(4), 128–137.

    Article  Google Scholar 

  68. Zenke, F., & Gerstner, W. (2014). Limits to high-speed simulations of spiking neural networks using general-purpose computers. Frontiers in Neuroinformatics, 8.

    Google Scholar 

  69. Dupeyroux, J., Hagenaars, J. J., Paredes-Vallés, F., & de Croon, G. C. H. E. (2021). Neuromorphic control for optic-flow-based landing of mavs using the loihi processor. In Proceedings - IEEE International Conference on Robotics and Automation, 2021-May.

    Google Scholar 

  70. Mitchell, J. P., Bruer, G., Dean, M. E., Plank, J. S. Rose, G. S., & Schuman, C. D. (2018). Neon: Neuromorphic control for autonomous robotic navigation. In Proceedings - 2017 IEEE 5th International Symposium on Robotics and Intelligent Sensors, IRIS 2017, 2018-January.

    Google Scholar 

  71. Tang, G., Kumar, N., & Michmizos, K. P. (2020). Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. In IEEE International Conference on Intelligent Robots and Systems.

    Google Scholar 

  72. Rajendran, B., Sebastian, A., Schmuker, M., Srinivasa, N., & Eleftheriou, E. (2019). Low-power neuromorphic hardware for signal processing applications: A review of architectural and system-level design approaches. IEEE Signal Processing Magazine, 36.

    Google Scholar 

  73. Lahbacha, K., Belgacem, H., Dghais, W., Zayer, F., & Maffucci, A. (2021) High density rram arrays with improved thermal and signal integrity. In 2021 IEEE 25th Workshop on Signal and Power Integrity (SPI) (pp. 1–4).

    Google Scholar 

  74. Fakhreddine, Z., Lahbacha, K., Melnikov, A., Belgacem, H., de Magistris, M., Dghais, W., & Maffucci, A. (2021). Signal and thermal integrity analysis of 3-d stacked resistive random access memories. IEEE Transactions on Electron Devices, 68(1), 88–94.

    Article  Google Scholar 

  75. Zayer, F., Mohammad, B., Saleh, H., & Gianini, G. (2020). Rram crossbar-based in-memory computation of anisotropic filters for image preprocessingloa. IEEE Access, 8, 127569–127580.

    Article  Google Scholar 

  76. Bettayeb, M., Zayer, F., Abunahla, H., Gianini, G., & Mohammad, B. (2022). An efficient in-memory computing architecture for image enhancement in ai applications. IEEE Access, 10, 48229–48241.

    Article  Google Scholar 

  77. Ajmi, H., Zayer, F., Fredj, A. H., Hamdi, B., Mohammad, B., Werghi, N., & Dias, J. (2022). Efficient and lightweight in-memory computing architecture for hardware security. arXiv:2205.11895.

  78. Zayer, F., Dghais, W., Benabdeladhim, M., & Hamdi, B. (2019). Low power, ultrafast synaptic plasticity in 1r-ferroelectric tunnel memristive structure for spiking neural networks. AEU-International Journal of Electronics and Communications, 100, 56–65.

    Google Scholar 

  79. Zayer, F., Dghais, W., & Belgacem, H. (2019). Modeling framework and comparison of memristive devices and associated stdp learning windows for neuromorphic applications. Journal of Physics D: Applied Physics, 52(39), 393002.

    Article  Google Scholar 

  80. Li, Z., Asif, M., & Ma, Z. (2022). Event transformerh.

    Google Scholar 

  81. Varma, A., Chawla, H., Zonooz, B., & Arani, E. (2022). Transformers in self-supervised monocular depth estimation with unknown camera intrinsics.

    Google Scholar 

  82. Hu, T., Wang, L., Xu, X., Liu, S., & Jia, J. (2021). Self-supervised 3d mesh reconstruction from single images. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

    Google Scholar 

  83. Li, S., Yan, Z., Li, H., & Cheng, K. T. (2021). Exploring intermediate representation for monocular vehicle pose estimation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

    Google Scholar 

Download references

Acknowledgements

This project is funded by Tawazun Technology & Innovation (TTI), under Tawazun Economic Council, through the collaboration with Khalifa University. The work shared is part of a MSc Thesis project by Hamad AlRemeithi, and all equipment is provided by TTI. Professional expertise is also a shared responsibility between both entities, and the authors extend their deepest gratitude for the opportunity to encourage research in this field.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Fakhreddine Zayer .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

AlRemeithi, H., Zayer, F., Dias, J., Khonji, M. (2023). Event Vision for Autonomous Off-Road Navigation. In: Azar, A.T., Koubaa, A. (eds) Artificial Intelligence for Robotics and Autonomous Systems Applications. Studies in Computational Intelligence, vol 1093. Springer, Cham. https://doi.org/10.1007/978-3-031-28715-2_8

Download citation

Publish with us

Policies and ethics