Skip to main content

A Multimodal Approach to Understand Driver’s Distraction for DMS

  • Conference paper
  • First Online:
Universal Access in Human-Computer Interaction (HCII 2024)

Abstract

This study introduces a multimodal approach for enhancing the accuracy of Driver Monitoring Systems (DMS) in detecting driver distraction. By integrating data from vehicle control units with vision-based information, the research aims to address the limitations of current DMS. The experimental setup involves a driving simulator and advanced computer vision, deep learning technologies for facial expression recognition, and head rotation analysis. The findings suggest that combining various data types—behavioral, physiological, and emotional—can significantly improve DMS’s predictive capability. This research contributes to the development of more sophisticated, adaptive, and real-time systems for improving driver safety and advancing autonomous driving technologies.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Antony, M.M., Whenish, R.: Advanced Driver Assistance Systems (ADAS) in AKathiresh, M., Neelaveni, R. (eds) Automotive Embedded Systems. EAI/Springer Innovations in Communication and Computing. Springer, Cham, pp. 165–181 (2021). https://doi.org/10.1007/978-3-030-59897-6_9

  2. Owens, J., Dingus, T., Guo, F., et al.: Prevalence of Drowsy-Driving Crashes: Estimates from a Large-Scale Naturalistic Driving Study. AAA Foundation for Traffic Safety, Washington, DC (2018)

    Google Scholar 

  3. Fitzharrris, M., Liu, S., Stephens, A.N., Lenné, M.G.: The relative importance of real-time in-cab and external feedback in managing fatigue in real-world commercial transport operations. Traffic Inj. Prev. 18(1), 71–78 (2017)

    Article  Google Scholar 

  4. Ortega, J.D., Canas, P.N., Nieto, M., Otaegui, O., Salgado, L.: Challenges of large-scale multi-camera datasets for driver monitoring systems. Sensors 22(7), 2554 (2022)

    Article  Google Scholar 

  5. Hasenjäger, M., Heckmann, M., Wersing, H.: A survey of personalization for advanced driver assistance systems. IEEE Trans. Intell. Veh. 5(2), 335–344 (2019)

    Article  Google Scholar 

  6. Ceccacci, S., Mengoni, M., Andrea, G., Giraldi, L., Carbonara, G., Castellano, A., Montanari, R.: A preliminary investigation towards the application of facial expression analysis to enable an emotion-aware car interface. In: Universal Access in Human-Computer Interaction. Applications and Practice: 14th International Conference, UAHCI 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings, Part II 22 (pp. 504–517). Springer International Publishing (2020). https://doi.org/10.1007/978-3-030-49108-6_36

  7. European Commission: Road safety thematic report – Serious injuries. European Road Safety Observatory. Brussels, European Commission, Directorate General for Transport (2021)

    Google Scholar 

  8. Gardony, A.L., Lindeman, R.W., Brunyé, T.T.: Eye-tracking for human-centered mixed reality: promises and challenges. In: Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR), (Vol. 11310, pp. 230–247). SPIE (Feb 2020)

    Google Scholar 

  9. Dehzangi, O., Sahu, V., Taherisadr, M., Galster, S.: Multi-modal system to detect on-the-road driver distraction. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), (pp. 2191–2196). IEEE (Nov 2018)

    Google Scholar 

  10. Muhammad, K., Ullah, A., Lloret, J., Del Ser, J., de Albuquerque, V.H.C.: Deep learning for safe autonomous driving: Current challenges and future directions. IEEE Trans. Intell. Transp. Syst. 22(7), 4316–4336 (2020)

    Article  Google Scholar 

  11. Daza, I.G., Bergasa, L.M., Bronte, S., Yebes, J.J., Almazán, J., Arroyo, R.: Fusion of optimized indicators from advanced driver assistance systems (adas) for driver drowsiness detection. Sensors 14, 1106–1131 (2014)

    Article  Google Scholar 

  12. Sandberg, D., Akerstedt, T., Anund, A., Kecklund, G., Wahde, M.: Detecting driver sleepiness using optimized nonlinear combinations of sleepiness indicators. IEEE Trans. Intell. Transp. Syst. 12(1), 97–108 (2010)

    Article  Google Scholar 

  13. Zhang, H., Wu, C., Huang, Z., Yan, X., Qiu, T.Z.: Sensitivity of lane position and steering angle measurements to driver fatigue. Transp. Res. Rec. 2585(1), 67–76 (2016)

    Article  Google Scholar 

  14. Khandakar, A., et al.: Portable system for monitoring and controlling driver behavior and the use of a mobile phone while driving. Sensors 19(7), 1563 (2019)

    Article  Google Scholar 

  15. Nemcova, A., et al.: Multimodal features for detection of driver stress and fatigue. IEEE Trans. Intell. Transp. Syst. 22(6) (2021)

    Google Scholar 

  16. Dinges, D.F.: An overview of sleepiness and accidents. J. Sleep Res. 4(2), 4–14 (1995)

    Article  Google Scholar 

  17. Kircher, A., Uddman, M., Sandin, J.: Vehicle control and drowsiness. Statens väg-och transportforskningsinstitut (2002)

    Google Scholar 

  18. Bunji, A: Evaluation of Mental Condition on Drivers by Analysis of Heart Rate Variability. Journal of the Society of Automotive Engineers of Japan No. 9437601, 1994

    Google Scholar 

  19. Milardo, S., Rathore, P., Amorim, M., Fugiglando, U., Santi, P., Ratti, C.: Understanding drivers’ stress and interactions with vehicle systems through naturalistic data analysis. IEEE Trans. Intell. Transp. Syst. 23(9), 14570–14581 (2021)

    Article  Google Scholar 

  20. Bergasa, L.M., Nuevo, J., Sotelo, M.A., Barea, R., Lopez, M.E.: Real-time system for monitoring driver vigilance. IEEE Trans. Intell. Transport. Syst. 7(1), 63–77 (2006)

    Article  Google Scholar 

  21. Çetinkaya, M., Acarman, T.: Driver impairment detection using decision tree based feature selection and classification. Results Eng. 18, 101025 (2023)

    Article  Google Scholar 

  22. Wylie, C.D., Shultz, T., Miller, J.C., Mitler, M.M., Mackie, R.R.: Commercial motor vehicle driver fatigue and alertness study: Technical summary (1996)

    Google Scholar 

  23. Doudou, M., Bouabdallah, A., Berge-Cherfaoui, V.: Driver drowsiness measurement technologies: current research, market solutions, and challenges. Int. J. ITS Res. 18, 297–319 (2020)

    Article  Google Scholar 

  24. Sagberg, F.: Road accidents caused by drivers falling asleep. Accid. Anal. Prev. 31, 639–649 (1999)

    Article  Google Scholar 

  25. Verwey, W.B., Zaidel, D.: Predicting Drowsiness Accidents from Personal Attributes, Eye Blinks, and Ongoing Driving Behaviour. Report TM-97-B009. Soesterberg. TNO Human Factors Research Institute, The Netherlands (1997)

    Google Scholar 

  26. Aksjonov, A., Nedoma, P., Vodovozov, V., Petlenkov, E., Herrmann, M.: Detection and evaluation of driver distraction using machine learning and fuzzy logic. IEEE Trans. Intell. Transp. Syst. 20(6), 2048–2059 (2018)

    Article  Google Scholar 

  27. Bittner, R., Hána, K., Poušek, L., Smrka, P., Schreib, P., Vysoký, P.: Detecting of Fatigue States of a Car Driver. In: Brause, R.W., Hanisch, E. (eds.) ISMDA 2000. LNCS, vol. 1933, pp. 260–273. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-39949-6_32

    Chapter  Google Scholar 

  28. Sârbescu, P.: Aggressive driving in Romania: psychometric properties of the driving anger expression inventory. Transport. Res. F: Traffic Psychol. Behav. 15(5), 556–564 (2012)

    Article  Google Scholar 

  29. Wu, Y.L., Tsai, H.Y., Huang, Y.C., Chen, B.H.: Accurate emotion recognition for driving risk prevention in driver monitoring system. In: 2018 IEEE 7TH Global Conference on Consumer Electronics (GCCE) (pp. 796–797). IEEE (Oct 2018)

    Google Scholar 

  30. Talipu, A., Generosi, A., Mengoni, M., Giraldi, L.: Evaluation of deep convolutional neural network architectures for emotion recognition in the wild. In: 2019 IEEE 23rd International Symposium on Consumer Technologies, pp. 25–27, 2019. IEEEhttps://doi.org/10.1109/ISCE.2019.8900994

  31. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops (2010). https://doi.org/10.1109/CVPRW.2010.5543262

  32. Barsoum, E., Zhang, C., Ferrer, C.-C., Zhang, Z.:“Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution” (2016). https://doi.org/10.1145/2993148.2993165

  33. Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)

    Article  Google Scholar 

  34. Feldman Barrett, L., Russell, J.A.: Independence and bipolarity in the structure of current affect. J. Pers. Soc. Psychol. 74(4), 967 (1998)

    Article  Google Scholar 

  35. Ceccacci, S., Generosi, A., Cimini, G., Faggiano, S., Giraldi, L., Mengoni, M.: Facial coding as a mean to enable continuous monitoring of student’s behavior in e-Learning. In: teleXbe (Jan 2021)

    Google Scholar 

  36. Mallick, S.: Head Pose Estimation using OpenCV and Dlib | LearnOpenCV #. LearnOpenCV – Learn OpenCV, PyTorch, Keras, Tensorflow With Examples and Tutorials (May 2021). https://learnopencv.com/head-pose-estimation-using-opencv-and-dlib/

  37. Soukupova, T., Cech, J.: “Eye Blink Detection Using Facial Landmarks.“ 21st Computer Vision Winter Workshop, Rimske Toplice, Slovenia (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Andrea Generosi .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Generosi, A., Villafan, J.Y., Montanari, R., Mengoni, M. (2024). A Multimodal Approach to Understand Driver’s Distraction for DMS. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. HCII 2024. Lecture Notes in Computer Science, vol 14696. Springer, Cham. https://doi.org/10.1007/978-3-031-60875-9_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-60875-9_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-60874-2

  • Online ISBN: 978-3-031-60875-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics