Skip to main content

Advertisement

SpringerLink
Log in

Driver information system: a combination of augmented reality, deep learning and vehicular Ad-hoc networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Improving traffic safety is one of the important goals of Intelligent Transportation Systems (ITS). In vehicle-based safety systems, it is more desirable to prevent an accident than to reduce severity of injuries. Critical traffic problems such as accidents and traffic congestion require the development of new transportation systems. Research in perceptual and human factors assessment is needed for relevant and correct display of this information for maximal road traffic safety as well as optimal driver comfort. One of the solutions to prevent accidents is to provide information on the surrounding environment of the driver. Augmented Reality Head-Up Display (AR-HUD) can facilitate a new form of dialogue between the vehicle and the driver; and enhance ITS by superimposing surrounding traffic information on the users view and keep drivers view on roads. In this paper, we propose a fast deep-learning-based object detection approaches for identifying and recognizing road obstacles types, as well as interpreting and predicting complex traffic situations. A single convolutional neural network predicts region of interest and class probabilities directly from full images in one evaluation. We also investigated potential costs and benefits of using dynamic conformal AR cues in improving driving safety. A new AR-HUD approach to create real-time interactive traffic animations was introduced in terms of types of obstacle, rules for placement and visibility, and projection of these on an in-vehicle display. The novelty of our approach is that both global and local context information are integrated into a unified framework to distinguish the ambiguous detection outcomes, enhance ITS by superimposing surrounding traffic information on the users view and keep drivers view on roads.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Accesed 22 may 2014. https://www.nsnam.org/

  2. Akhlaq M (2010) A smart-dashboard: Augmenting safe & smooth driving

  3. Batavia PH, Pomerleau D, Thorpe CE et al (1997) Overtaking vehicle detection using implicit optical flow. In: IEEE conference on intelligent transportation system, 1997. ITSC’97. IEEE, pp 729–734

    Google Scholar 

  4. Behrisch M, Bieker L, Erdmann J, Krajzewicz D (2011) Sumo–simulation of urban mobility. In: The 3rd international conference on advances in system simulation (SIMUL 2011). Barcelona, Spain

    Google Scholar 

  5. Bergenhem C, Coelingh E, Johansson R, Tehrani A (2014) V2v communication quality: measurements in a cooperative automotive platooning application. SAE Int J Passenger Cars Electron Electr Syst 7(2014-01-0302):462–470

    Article  Google Scholar 

  6. Bradai A, Ahmed T, Benslimane A (2014) Vicov: efficient video streaming for cognitive radio vanet. Veh Commun 1(3):105–122

    Article  Google Scholar 

  7. Caird J (2004) In-vehicle intelligent transportation systems. Transportation in an Aging Society, p 236

  8. Cheng HT, Shan H, Zhuang W (2011) Infotainment and road safety service support in vehicular networking: from a communication perspective. Mech Syst Signal Process 25(6):2020–2038

    Article  Google Scholar 

  9. Choi RA (2015) Automatic speed controllable vehicle and method for controlling speed thereof. US Patent 20,150,149,059

  10. Cireṡan D, Meier U, Masci J, Schmidhuber J (2012) Multi-column deep neural network for traffic sign classification. Neural Netw 32:333–338

    Article  Google Scholar 

  11. De Felice M, Cerqueira E, Melo A, Gerla M, Cuomo F, Baiocchi A (2015) A distributed beaconless routing protocol for real-time video dissemination in multimedia vanets. Comput Commun 58:40–52

    Article  Google Scholar 

  12. Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2013) A deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv:1310.1531

  13. Fang Z, Fei F, Fang Y, Lee C, Xiong N, Shu L, Chen S (2016) Abnormal event detection in crowded scenes based on deep learning. Multi Tools Appl 75(22):14,617–14,639

    Article  Google Scholar 

  14. Fu WT, Gasper J, Kim S (2013) Effects of an in-car augmented reality system on improving safety of younger and older drivers. In: 2013 IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 59–66

    Google Scholar 

  15. Ghoreyshi SM, Shahrabi A, Boutaleb T (2016) A novel cooperative opportunistic routing scheme for underwater sensor networks. Sensors 16(3):297

    Article  Google Scholar 

  16. Girshick R (2015) Fast r-cnn. arXiv preprint arXiv:1504.08083

  17. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 580–587

    Google Scholar 

  18. Haklay M, Weber P (2008) Openstreetmap: User-generated street maps. IEEE Pers Commun 7(4):12–18

    Google Scholar 

  19. Halit L, Kemeny A, Mohellebi H, Garbaya S, Mérienne F, Michelin S, Albaut V Augmented reality head-up-display for advanced driver assistance system: A driving simulation study

  20. Haloi M (2015) A novel plsa based traffic signs classification system. arXiv preprint arXiv:1503.06643

  21. Han P, Zhao G (2016) Line-based initialization method for mobile augmented reality in aircraft assembly. The Visual Computer, pp 1–12

  22. He K, Zhang X, Ren S, Sun J (2014) Spatial pyramid pooling in deep convolutional networks for visual recognition. In: Computer Vision–ECCV 2014. Springer, pp 346–361

    Google Scholar 

  23. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

    Google Scholar 

  24. Huang CM, Lin SY (2014) Cooperative vehicle collision warning system using the vector-based approach with dedicated short range communication data transmission. IET Intell Transp Syst 8(2):124–134

    Article  Google Scholar 

  25. Huang J, Li B, Zhu J, Chen J (2017) Age classification with deep learning face representation. Multimedia Tools and Applications, pp 1–17

  26. Jama M, Risaq A, Issa S, Khalifa OO (2010) Performance evaluation of mpeg-4 video transmission over ieee802. 11e. Int J Comput Netw Secur 2(5):11–15

    Google Scholar 

  27. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia, pp 675–678. ACM

    Google Scholar 

  28. Khaleda Y, Tsukadaa M, Santab J, Choia J, Ernsta T (2009) A usage oriented analysis of vehicular networks: from technologies to applications. J Commun 4(5):357–368

    Article  Google Scholar 

  29. Khandelwal SA, Abhale AB, Nagaraj U (2014) Accident prevention and air pollution control using vanet under cloud environment

  30. Kim H (2010) An efficient alert broadcasting scheme considering various densities in vanet. In: Networked digital technologies

    Google Scholar 

  31. Kim S, Dey AK (2009) Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 133–142

    Google Scholar 

  32. Kim S, Dey AK (2016) Augmenting human senses to improve the user experience in cars: applying augmented reality and haptics approaches to reduce cognitive distances. Multi Tools Appl 75(16):9587–9607

    Article  Google Scholar 

  33. Lee JW, Yoon CR, Kang J, Park BJ, Kim KH (2015) Development of lane-level guidance service in vehicle augmented reality system. In: 2015 17th international conference on advanced communication technology (ICACT). IEEE, pp 263–266

    Google Scholar 

  34. Lenc K, Vedaldi A (2015) R-cnn minus r. arXiv preprint arXiv:1506.06981

  35. Li H, Nashashibi F (2011) Multi-vehicle cooperative perception and augmented reality for driver assistance: a possibility to ’see’through front vehicle. In: 2011 14th international IEEE conference on intelligent transportation systems (ITSC). IEEE, pp 242–247

    Google Scholar 

  36. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S (2015) Ssd: single shot multibox detector. arXiv preprint arXiv:1512.02325

  37. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) Ssd: single shot multibox detector. In: European conference on computer vision. Springer, pp 21–37

    Google Scholar 

  38. Meng F, Spence C (2015) Tactile warning signals for in-vehicle systems. Accid Anal Prev 75:333–346

    Article  Google Scholar 

  39. Milanés V, Shladover SE, Spring J, Nowakowski C, Kawazoe H, Nakamura M (2014) Cooperative adaptive cruise control in real traffic situations. IEEE Trans Intell Transp Syst 15(1):296–305

    Article  Google Scholar 

  40. Mota S, Ros E, Ortigosa EM, Pelayo FJ (2004) Bio-inspired motion detection for a blind spot overtaking monitor. Int J Robot Autom 19(4):190–196

    Google Scholar 

  41. Moysset B, Adam P, Wolf C, Louradour J (2015) Space displacement localization neural networks to locate origin points of handwritten text lines in historical documents. In: ICDAR 2015 workshop on historical document imaging and processing

    Google Scholar 

  42. Naranjo JE, Gonzalez C, Garcia R, De Pedro T (2008) Lane-change fuzzy control in autonomous vehicles for the overtaking maneuver. IEEE Trans Intell Transp Syst 9(3):438–450

    Article  Google Scholar 

  43. Ng-Thow-Hing V, Bark K, Beckwith L, Tran C, Bhandari R, Sridhar S (2013) User-centered perspectives for automotive augmented reality. In: 2013 IEEE international symposium on mixed and augmented reality-arts, media, and humanities (ISMAR-AMH). IEEE, pp 13–22

    Google Scholar 

  44. Ngai DC, Yung NH (2007) Automated vehicle overtaking based on a multiple-goal reinforcement learning framework. In: Intelligent transportation systems conference, 2007. ITSC 2007. IEEE. IEEE, pp 818–823

    Google Scholar 

  45. Olaverri-Monreal C, Gomes P, Fernandes R, Vieira F, Ferreira M (2010) The see-through system: A vanet-enabled assistant for overtaking maneuvers. In: Intelligent vehicles symposium (IV), 2010 IEEE. IEEE, pp 123–128

    Google Scholar 

  46. Paul B, Ibrahim M, Bikas M, Naser A (2012) Experimental analysis of aodv & dsr over tcp & cbr connections with varying speed and node density in vanet. arXiv preprint arXiv:1204.1206

  47. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

    Google Scholar 

  48. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497

  49. Ren S, He K, Girshick R, Zhang X, Sun J (2015) Object detection networks on convolutional feature maps. arXiv preprint arXiv:1504.06066

  50. Rezende C (2014) Video streaming in vehicular ad hoc networks: challenges, protocols and the use of redundancy. Ph.D. thesis, University of Ottawa

  51. Richter P Ar-hud – technology for a dialog without words. http://www.continental-corporation.com/www/pressportal.comen/themepressreleases/3automotivegroup/interior/pressreleasespr20140721arhudtechniken.html

  52. Schall MC, Rusch ML, Lee JD, Dawson JD, Thomas G, Aksan N, Rizzo M (2012) Augmented reality cues and elderly driver hazard perception. Human Factors: The Journal of the Human Factors and Ergonomics Society p pp. 0018720812462029

  53. Schofield K (2013) Driver assistance system for a vehicle. US Patent App. 13/919,483

  54. Sermanet P, LeCun Y (2011) Traffic sign recognition with multi-scale convolutional networks. In: The 2011 international joint conference on neural networks (IJCNN). IEEE, pp 2809–2813

    Google Scholar 

  55. Silvéria M.K. (2014) Virtual windshields: merging reality and digital content to improve the driving experience. arXiv preprint arXiv:1405.0910

  56. Sivaraman S, Trivedi MM (2014) Dynamic probabilistic drivability maps for lane change and merge driver assistance. IEEE Trans Intell Transp Syst 15(5):2063–2073

    Article  Google Scholar 

  57. Stallkamp J, Schlipsing M, Salmen J, Igel C (2011) The german traffic sign recognition benchmark: a multi-class classification competition. In: The 2011 international joint conference on neural networks (IJCNN). IEEE, pp 1453–1460

    Google Scholar 

  58. Stallkamp J, Schlipsing M, Salmen J, Igel C (2012) Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Netw 32:323–332

    Article  Google Scholar 

  59. Stebbins MA, Kiefer RJ (2014) Vehicle seat back haptic alert systems and methods. US Patent App. 14/453,391

  60. Tian D, Zhou J, Wang Y, Zhang G, Xia H (2016) An adaptive vehicular epidemic routing method based on attractor selection model. Ad Hoc Netw 36:465–481

    Article  Google Scholar 

  61. Timofte R Kul belgium traffic signs and classification benchmark datasets. http://btsd.ethz.ch/shareddata

  62. Tonnis M, Sandor C, Lange C, Bubb H (2005) Experimental evaluation of an augmented reality visualization for directing a car driver’s attention. In: Proceedings of the 4th IEEE/ACM international symposium on mixed and augmented reality. IEEE Computer Society, pp 56–59

    Google Scholar 

  63. Wang F, Yang M, Yang R (2009) Conflict-probability-estimation-based overtaking for intelligent vehicles. IEEE Trans Intell Transp Syst 10(2):366–370

    Article  Google Scholar 

  64. Werneke J, Vollrath M (2012) What does the driver look at? The influence of intersection characteristics on attention allocation and driving behavior. Accid Anal Prev 45:610–619

    Article  Google Scholar 

  65. Xie H, Boukerche A, Loureiro A Mervs: A novel multi-channel error recovery video streaming scheme for vehicle ad-hoc networks

  66. Yang J, Xu R, Cui J, Ding Z (2016) Robust visual tracking using adaptive local appearance model for smart transportation. Mult Tools Appl 75(24):17,487–17,500

    Article  Google Scholar 

  67. Zaklouta F, Stanciulescu B, Hamdoun O (2011) Traffic sign classification using kd trees and random forests. In: The 2011 international joint conference on neural networks (IJCNN). IEEE, pp 2151–2155

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Networked Objects Control and Communication Systems Laboratory, National Engineering School of Tunis, University of Tunis El Manar, Tunis, Tunisia

    Lotfi Abdi

  2. Networked Objects Control and Communication Systems Laboratory, National Engineering School of Sousse, University of Sousse, Sousse, Tunisia

    Aref Meddeb

Authors
  1. Lotfi Abdi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aref Meddeb
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lotfi Abdi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abdi, L., Meddeb, A. Driver information system: a combination of augmented reality, deep learning and vehicular Ad-hoc networks. Multimed Tools Appl 77, 14673–14703 (2018). https://doi.org/10.1007/s11042-017-5054-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5054-6

Keywords

Search

Navigation