Abstract
Robotic automation has always been employed to optimize tasks that are deemed repetitive or hazardous for humans. One instance of such an application is within transportation, be it in urban environments or other harsh applications. In said scenarios, it is required for the platform’s operator to be at a heightened level of awareness at all times to ensure the safety of on-board materials being transported. Additionally, during longer journeys it is often the case that the driver might also be required to traverse difficult terrain under extreme conditions. For instance, low light, fog, or haze-ridden paths. To counter this issue, recent studies have proven that the assistance of smart systems is necessary to minimize the risk involved. In order to develop said systems, this chapter discusses a concept of a Deep Learning (DL) based Vision Navigation (VN) approach capable of terrain analysis and determining the appropriate steering angle within a margin of safety. Within the framework of Neuromorphic Vision (NV) and Event Cameras (EC), the proposed concept is tackling several issues within the development of autonomous systems. In particular, the use of Transformer based backbone for off-road depth estimation using an event camera for better accuracy result and processing time. The implementation of the above mentioned deep learning system, using event camera is leveraged through the necessary data processing techniques of the events prior to the training phase. Besides, binary convolutions (BN) and alternately spiking convolution paradigms using the latest technology trend have been deployed as acceleration methods, with efficiency in terms of energy latency, and environmental robustness. Initial results hold promising potential for the future development of real-time projects with event cameras.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Badue, C., Guidolini, R., Carneiro, R. V., Azevedo, P., Cardoso, V. B., Forechi, A., Jesus, L., Berriel, R., Paixão, T. M., Mutz, F., de Paula Veronese, L., Oliveira-Santos, T., & De Souza, A. F. (2021). Self-driving cars: A survey. Expert Systems with Applications, 165.
Ni, J., Chen, Y., Chen, Y., Zhu, J., Ali, D., & Cao, W. (2020) A survey on theories and applications for self-driving cars based on deep learning methods. Applied Sciences (Switzerland), 10.
Chen, G., Cao, H., Conradt, J., Tang, H., Rohrbein, F., & Knoll, A. (2020). Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Processing Magazine, 37.
Lin, M., Yoon, J., & Kim, B. (2020) Self-driving car location estimation based on a particle-aided unscented kalman filter. Sensors (Switzerland), 20.
Mugunthan, N., Naresh, V. H., & Venkatesh, P. V. (2020). Comparison review on lidar vs camera in autonomous vehicle. In International Research Journal of Engineering and Technology.
Ming, Y., Meng, X., Fan, C., & Yu, H. (2021) Deep learning for monocular depth estimation: A review. Neurocomputing, 438.
Li, X., Tang, B., Ball, J., Doude, M., & Carruth, D. W. (2019). Rollover-free path planning for off-road autonomous driving. Electronics (Switzerland), 8.
Pan, Y., Cheng, C. A., Saigol, K., Lee, K., Yan, X., Theodorou, E. A., & Boots, B. (2020). Imitation learning for agile autonomous driving. International Journal of Robotics Research, 39.
Liu, O., Yuan, S., & Li, Z. (2020). A survey on sensor technologies for unmanned ground vehicles. In Proceedings of 2020 3rd International Conference on Unmanned Systems, ICUS 2020.
Shin, J., Kwak, D. J., & Kim, J. (2021). Autonomous platooning of multiple ground vehicles in rough terrain. Journal of Field Robotics, 38.
Naranjo, J. E., Jiménez, F., Anguita, M., & Rivera, J. L. (2020). Automation kit for dual-mode military unmanned ground vehicle for surveillance missions. IEEE Intelligent Transportation Systems Magazine, 12.
Browne, M., Macharis, C., Sanchez-diaz, I., Brolinson, M., & Illsjö, R. (2017). Urban traffic congestion and freight transport : A comparative assessment of three european cities. Interdisciplinary Conference on Production Logistics and Traffic.
Zhong, H., Zhou, J., Du, Z., & Xie, L. (2018). A laboratory experimental study on laser attenuations by dust/sand storms. Journal of Aerosol Science, 121.
Koepke, P., Gasteiger, J., & Hess, M. (2015). Technical note: Optical properties of desert aerosol with non-spherical mineral particles: Data incorporated to opac. Atmospheric Chemistry and Physics Discussions, 15, 3995–4023.
Raja, A. R., Kagalwala, Q. J., Landolsi, T., & El-Tarhuni, M. (2007). Free-space optics channel characterization under uae weather conditions. In ICSPC 2007 Proceedings - 2007 IEEE International Conference on Signal Processing and Communications.
Vargasrivero, J. R., Gerbich, T., Buschardt, B., & Chen, J. (2021). The effect of spray water on an automotive lidar sensor: A real-time simulation study. IEEE Transactions on Intelligent Vehicles.
Strawbridge, K. B., Travis, M. S., Firanski, B. J., Brook, J. R., Staebler, R., & Leblanc, T. (2018). A fully autonomous ozone, aerosol and nighttime water vapor lidar: A synergistic approach to profiling the atmosphere in the canadian oil sands region. Atmospheric Measurement Techniques, 11.
Hummel, B., Kammel, S., Dang, T., Duchow, C., & Stiller, C. (2006). Vision-based path-planning in unstructured environments. In IEEE Intelligent Vehicles Symposium, Proceedings.
Mueller, G. R., & Wuensche, H. J. (2018). Continuous stereo camera calibration in urban scenarios. In IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, 2018-March.
Rankin, A. L., Huertas, A., & Matthies, L. H. (2009). Stereo-vision-based terrain mapping for off-road autonomous navigation. Unmanned Systems Technology X, I, 7332.
Litzenberger, M., Belbachir, A. N., Donath, N., Gritsch, G., Garn, H., Kohn, B., Posch, C., & Schraml, S. (2006). Estimation of vehicle speed based on asynchronous data from a silicon retina optical sensor. In IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC.
Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., & Scaramuzza, D. (2020). Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44.
DelbrĂĽck, T., Linares-Barranco, B., Culurciello, E., & Posch, C. (2010). Activity-driven, event-based vision sensors. In ISCAS 2010 - 2010 IEEE International Symposium on Circuits and Systems: Nano-Bio Circuit Fabrics and Systems.
Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D. (2021). High speed and high dynamic range video with an event camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43.
Lichtsteiner, P., Posch, C., & Delbruck, T. (2008). A 128\(\times \) 128 120 db 15 \(\upmu \)s latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, 43, 566–576.
Brändli, C., Berner, R., Yang, M., Liu, S.-C., & Delbruck, T. (2014). A 240 \(\times \) 180 130 db 3 \(\upmu \)s latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49, 2333–2341.
Scheerlinck, C., Barnes, N., & Mahony, R. (2019). Continuous-time intensity estimation using event cameras. Lecture notes in computer science (including subseries Lecture notes in artificial intelligence and lecture notes in bioinformatics), 11365 LNCS.
Gallego, G., Lund, J. E. A., Mueggler, E., Rebecq, H., Delbruck, T., & Scaramuzza, D. (2018). Event-based, 6-dof camera tracking from photometric depth maps. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40.
Mostafavi, M., Wang, L., & Yoon, K. J. (2021). Learning to reconstruct hdr images from events, with applications to depth and flow prediction. International Journal of Computer Vision, 129.
Mueggler, E., Huber, B., & Scaramuzza, D. (2014). Event-based, 6-dof pose tracking for high-speed maneuvers.
Posch, C., Matolin, D., & Wohlgenannt, R. (2011). A qvga 143 db dynamic range frame-free pwm image sensor with lossless pixel-level video compression and time-domain cds. IEEE Journal of Solid-State Circuits, 46.
Lee, S., Kim, H., & Kim, H. J. (2020). Edge detection for event cameras using intra-pixel-area events. In 30th British Machine Vision Conference 2019, BMVC 2019.
Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D. (2019). Events-to-video: Bringing modern computer vision to event cameras. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June.
Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017). End-to-end learning of driving models from large-scale video datasets. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January.
Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017). End-to-end learning of driving models from large-scale video datasets. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January.
Boahen, K. A. (2004). A burst-mode word-serial address-event link - i: Transmitter design (p. 51). IEEE Transactions on Circuits and Systems I: Regular Papers.
Wang, C., Buenaposada, J. M., Zhu, R., & Lucey, S. (2018). Learning depth from monocular videos using direct methods. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
Guo, S., Kang, Z., Wang, L., Zhang, L., Chen, X., Li, S., & Xu, W. (2020). A noise filter for dynamic vision sensors using self-adjusting threshold.
Gehrig, D., Ruegg, M., Gehrig, M., Hidalgo-Carrio, J., & Scaramuzza, D. (2021). Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction. IEEE Robotics and Automation Letters, 6.
Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., & Dai, Y. (2019). Bringing a blurry frame alive at high frame-rate with an event camera. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June.
Pan, L., Hartley, R., Scheerlinck, C., Liu, M., Yu, X., & Dai, Y. (2022). High frame rate video reconstruction based on an event camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44.
Gehrig, D., Rebecq, H., Gallego, G., & Scaramuzza, D. (2020). Eklt: Asynchronous photometric feature tracking using events and frames. International Journal of Computer Vision, 128.
Saner, D., Wang, O., Heinzle, S., Pritch, Y., Smolic, A., Sorkine-Hornung, A., & Gross, M. (2014). High-speed object tracking using an asynchronous temporal contrast sensor. In 19th International Workshop on Vision, Modeling and Visualization, VMV 2014.
Muglikar, M., Gehrig, M., Gehrig, D., & Scaramuzza, D. (2021). How to calibrate your event camera. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.
Maqueda, A. I., Loquercio, A., Gallego, G., Garcia, N., & Scaramuzza, D. (2018). Event-based vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
Galluppi, F., Denk, C., Meiner, M. C., Stewart, T. C., Plana, L. A., Eliasmith, C., Furber, S., & Conradt, J. (2014). Event-based neural computing on an autonomous mobile platform. In Proceedings - IEEE International Conference on Robotics and Automation.
Hu, Y., Binas, J., Neil, D., Liu, S. C., & Delbruck, T. (2020). Ddd20 end-to-end event camera driving dataset: Fusing frames and events with deep learning for improved steering prediction. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems, ITSC 2020.
Zhong, H., Wang, H., Wu, Z., Zhang, C., Zheng, Y., & Tang, T. (2021). A survey of lidar and camera fusion enhancement. Procedia Computer Science, 183.
Song, R., Jiang, Z., Li, Y., Shan, Y., & Huang, K. (2018). Calibration of event-based camera and 3d lidar. In 2018 WRC Symposium on Advanced Robotics and Automation, WRC SARA 2018 - Proceeding.
Zhou, Y., Gallego, G., & Shen, S. (2021). Event-based stereo visual odometry. IEEE Transactions on Robotics, 37.
Dahlkamp, H., Kaehler, A., Stavens, D., Thrun, S., & Bradski, G. (2007). Self-supervised monocular road detection in desert terrain. Robotics: Science and Systems, 2.
Bayard, D. S., Conway, D. T., Brockers, R., Delaune, J., Matthies, L., Grip, H. F., Merewether, G., Brown, T., & Martin, A. M. S. (2019). Vision-based navigation for the nasa mars helicopter. AIAA Scitech 2019 Forum.
Hidalgo-Carrio, J., Gehrig, D., & Scaramuzza, D. (2020). Learning monocular dense depth from events. In Proceedings - 2020 International Conference on 3D Vision, 3DV 2020.
Li, Z., Asif, M. S., & Ma, Z. (2022). Event transformer.
Juefei-Xu, F., Boddeti, V. N., & Savvides, M. (2017). Local binary convolutional neural networks. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January.
Khodamoradi, A., & Kastner, R. (2021). O(n)-space spatiotemporal filter for reducing noise in neuromorphic vision sensors. IEEE Transactions on Emerging Topics in Computing, 9.
Feng, Y., Lv, H., Liu, H., Zhang, Y., Xiao, Y., & Han, C. (2020). Event density based denoising method for dynamic vision sensor. Applied Sciences (Switzerland), 10.
Meyer, L., SmĂšek, M., Villacampa, A. F., Maza, L. O., Medina, D., Schuster, M. J., Steidle, F., Vayugundla, M., MĂĽller, M. G., Rebele, B., Wedler, A., & Triebel, R. (2021). The madmax data set for visual-inertial rover navigation on mars. Journal of Field Robotics, 38.
Figurnov, M., Ibraimova, A., Vetrov, D., & Kohli, P. (2016). Perforatedcnns: Acceleration through elimination of redundant convolutions. Advances in Neural Information Processing Systems, 29.
Salman, A. M., Tulan, A. S., Mohamed, R. Y., Zakhari, M. H., & Mostafa, H. (2020). Comparative study of hardware accelerated convolution neural network on pynq board. In 2nd Novel Intelligent and Leading Emerging Sciences Conference, NILES 2020.
Yoshida, Y., Oiwa, R., & Kawahara, T. (2018). Ternary sparse xnor-net for fpga implementation. In Proceedings - 7th International Symposium on Next-Generation Electronics. ISNE, 2018.
Ding, C., Wang, S., Liu, N., Xu, K., Wang, Y., & Liang, Y. (2019). Req-yolo: A resource-aware, efficient quantization framework for object detection on fpgas. In FPGA 2019 - Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays.
Li, J. N., & Tian, Y. H. (2021). Recent advances in neuromorphic vision sensors: A survey. Jisuanji Xuebao/Chinese Journal of Computers, 44.
Chen, G., Cao, H., Aafaque, M., Chen, J., Ye, C., Röhrbein, F., Conradt, J., Chen, K., Bing, Z., Liu, X., Hinz, G., Stechele, W., & Knoll, A. (2018) Neuromorphic vision based multivehicle detection and tracking for intelligent transportation system. Journal of Advanced Transportation, 2018.
Gutierrez-Galan, D., Schoepe, T., Dominguez-Morales, J. P., Jiménez-Fernandez, A., Chicca, E., & Linares-Barranco, A. (2020). An event-based digital time difference encoder model implementation for neuromorphic systems.
Schuman, C. D., Kulkarni, S. R., Parsa, M., Mitchell, J. P., Date, P., & Kay, B. (2022). Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science, 2.
Richter, C., Jentzsch, S., Hostettler, R., Garrido, J. A., Ros, E., Knoll, A., et al. (2016). Musculoskeletal robots: Scalability in neural control. IEEE Robotics & Automation Magazine, 23(4), 128–137.
Zenke, F., & Gerstner, W. (2014). Limits to high-speed simulations of spiking neural networks using general-purpose computers. Frontiers in Neuroinformatics, 8.
Dupeyroux, J., Hagenaars, J. J., Paredes-Vallés, F., & de Croon, G. C. H. E. (2021). Neuromorphic control for optic-flow-based landing of mavs using the loihi processor. In Proceedings - IEEE International Conference on Robotics and Automation, 2021-May.
Mitchell, J. P., Bruer, G., Dean, M. E., Plank, J. S. Rose, G. S., & Schuman, C. D. (2018). Neon: Neuromorphic control for autonomous robotic navigation. In Proceedings - 2017 IEEE 5th International Symposium on Robotics and Intelligent Sensors, IRIS 2017, 2018-January.
Tang, G., Kumar, N., & Michmizos, K. P. (2020). Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. In IEEE International Conference on Intelligent Robots and Systems.
Rajendran, B., Sebastian, A., Schmuker, M., Srinivasa, N., & Eleftheriou, E. (2019). Low-power neuromorphic hardware for signal processing applications: A review of architectural and system-level design approaches. IEEE Signal Processing Magazine, 36.
Lahbacha, K., Belgacem, H., Dghais, W., Zayer, F., & Maffucci, A. (2021) High density rram arrays with improved thermal and signal integrity. In 2021 IEEE 25th Workshop on Signal and Power Integrity (SPI) (pp. 1–4).
Fakhreddine, Z., Lahbacha, K., Melnikov, A., Belgacem, H., de Magistris, M., Dghais, W., & Maffucci, A. (2021). Signal and thermal integrity analysis of 3-d stacked resistive random access memories. IEEE Transactions on Electron Devices, 68(1), 88–94.
Zayer, F., Mohammad, B., Saleh, H., & Gianini, G. (2020). Rram crossbar-based in-memory computation of anisotropic filters for image preprocessingloa. IEEE Access, 8, 127569–127580.
Bettayeb, M., Zayer, F., Abunahla, H., Gianini, G., & Mohammad, B. (2022). An efficient in-memory computing architecture for image enhancement in ai applications. IEEE Access, 10, 48229–48241.
Ajmi, H., Zayer, F., Fredj, A. H., Hamdi, B., Mohammad, B., Werghi, N., & Dias, J. (2022). Efficient and lightweight in-memory computing architecture for hardware security. arXiv:2205.11895.
Zayer, F., Dghais, W., Benabdeladhim, M., & Hamdi, B. (2019). Low power, ultrafast synaptic plasticity in 1r-ferroelectric tunnel memristive structure for spiking neural networks. AEU-International Journal of Electronics and Communications, 100, 56–65.
Zayer, F., Dghais, W., & Belgacem, H. (2019). Modeling framework and comparison of memristive devices and associated stdp learning windows for neuromorphic applications. Journal of Physics D: Applied Physics, 52(39), 393002.
Li, Z., Asif, M., & Ma, Z. (2022). Event transformerh.
Varma, A., Chawla, H., Zonooz, B., & Arani, E. (2022). Transformers in self-supervised monocular depth estimation with unknown camera intrinsics.
Hu, T., Wang, L., Xu, X., Liu, S., & Jia, J. (2021). Self-supervised 3d mesh reconstruction from single images. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
Li, S., Yan, Z., Li, H., & Cheng, K. T. (2021). Exploring intermediate representation for monocular vehicle pose estimation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
Acknowledgements
This project is funded by Tawazun Technology & Innovation (TTI), under Tawazun Economic Council, through the collaboration with Khalifa University. The work shared is part of a MSc Thesis project by Hamad AlRemeithi, and all equipment is provided by TTI. Professional expertise is also a shared responsibility between both entities, and the authors extend their deepest gratitude for the opportunity to encourage research in this field.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
AlRemeithi, H., Zayer, F., Dias, J., Khonji, M. (2023). Event Vision for Autonomous Off-Road Navigation. In: Azar, A.T., Koubaa, A. (eds) Artificial Intelligence for Robotics and Autonomous Systems Applications. Studies in Computational Intelligence, vol 1093. Springer, Cham. https://doi.org/10.1007/978-3-031-28715-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-28715-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-28714-5
Online ISBN: 978-3-031-28715-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)