Skip to main content
SpringerLink
Log in

A Multimodal Approach to Psycho-Emotional State Detection of a Vehicle Driver

  • Conference paper
  • First Online:
Intelligent Systems and Applications (IntelliSys 2021)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 295))

Included in the following conference series:

Abstract

This paper aims to investigate the physio-emotional state of the driver in the vehicle cabin using a multimodal approach, comprising context, motion, visual, and audio data, collected beforehand. Driver behavior monitoring is implemented upon the data gained from different types of sensors, including accelerometer, magnetometer, gyroscope, GPS, front-facing camera, microphone, and information retrieved from third-party services. This data is intended to fully describe driver behavior and aid advanced driver assistant systems to fully classify and recognize dangerous driving behavior, and generate alerts on how to eliminate emergency situations. The emotional state of the driver is determined as six basic emotions, including sadness, fear, disgust, anger, surprise, and happiness. This work eventually presents driving style classification, dividing drivers into three groups: normal, ecological, urban, risky, and aggressive driving. This classification may potentially recognize aggressive vehicle drivers on public roads, and, therefore, undertake measures to reduce the risk of traffic accident occurrence.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Examination of the Traffic Safety Environment During the Second Quarter Of 2020 October 2020, DOT HS 813 011. https://rosap.ntl.bts.gov/view/dot/50940/dot_50940_DS1.pdf. Accessed 28 Jan 2021

  2. Haas, R.E., Bhattacharjee, S., Möller, D.P.F.: Advanced driver assistance systems. In: Akhilesh, K.B., Möller, D.P.F. (eds.) Smart Technologies, pp. 345–371. Springer, Singapore (2020). https://doi.org/10.1007/978-981-13-7139-4_27

    Chapter  Google Scholar 

  3. Rebecca, S., Vahabaghaie, A., Murakhovsky, D., Bahouth, G., Drayer, B., St Lawrence, S.: Effectiveness of Advanced Driver Assistance Systems in Preventing System-Relevant Crashes. No. 2021–01–0869. SAE Technical Paper (2021)

    Google Scholar 

  4. Morando, A., Gershon, P., Mehler, B., Reimer, B.: Driver-initiated tesla autopilot disengagements in naturalistic driving. In: 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '20). Association for Computing Machinery, New York, NY, USA, pp. 57–65 (2020)

    Google Scholar 

  5. Webb, N., et al.: Waymo's safety methodologies and safety readiness determinations. arXiv preprint arXiv:2011.00054 (2020)

  6. Islam, M., Mannering, F.: A temporal analysis of driver-injury severities in crashes involving aggressive and non-aggressive driving. Anal. Meth. Accid.t Res. 27, 100128 (2020)

    Google Scholar 

  7. Kashevnik, A., Lashkov, I., Gurtov, A.: Methodology and mobile application for driver behavior analysis and accident prevention. IEEE Trans. Intell. Transp. Syst. IEEE 21(6), 2427–2436 (2019)

    Article  Google Scholar 

  8. Kashevnik, A., Lashkov, I., Ponomarev, A., Teslya, N., Gurtov, A.: Cloud-based driver monitoring system using a smartphone. IEEE Sens. IEEE. 20(12), 6701–6715 (2020)

    Article  Google Scholar 

  9. Kashevnik, A., et al.: Multimodal Corpus Design for Audio-Visual Speech Recognition in Vehicle Cabin. IEEE Access (2021)

    Google Scholar 

  10. Jardin, P., Moisidis, I., Zetina, S.S., Rinderknecht, S.: Rule-based driving style classification using acceleration data profiles. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece, 2020, pp. 1–6 (2020). https://doi.org/10.1109/ITSC45102.2020.9294611

  11. Wang, Q., Zhang, R., Wang, Y., Lv, S.: Machine Learning-Based Driving Style Identification of Truck Drivers in Open-Pit Mines. Electronics 9(1), 1–23 (2020). https://doi.org/10.3390/electronics9010019

    Article  Google Scholar 

  12. Mohammadnazar, A., Arvin, R., Khattak, A.J.: Classifying travelers’ driving style using basic safety messages generated by connected vehicles: Application of unsupervised machine learning. Transp. Res. Part C: Emer. Technol. 122, 102917 (2021). https://doi.org/10.1016/j.trc.2020.102917

    Article  Google Scholar 

  13. Zhang, Y., Xu, Q., Wang, J., Wu, K., Zheng, Z., Lu, K.: A learning-based discretionary lane-change decision-making model with driving style awareness (2020)

    Google Scholar 

  14. Huang, C., Wang, X., Cao, J., Wang, S., Zhang, Y.: HCF: a hybrid CNN framework for behavior detection of distracted drivers. IEEE Access 8, 109335–109349 (2020). https://doi.org/10.1109/ACCESS.2020.3001159

    Article  Google Scholar 

  15. Mafeni Mase, J., Chapman, P., Figueredo, G.P., Torres Torres, M.: A hybrid deep learning approach for driver distraction detection. In: 2020 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Korea (South), pp. 1–6 (2020). https://doi.org/10.1109/ICTC49870.2020.9289588

  16. Lee, H., Lee, J., Shin, M.: Using wearable ECG/PPG sensors for driver drowsiness detection based on distinguishable pattern of recurrence plots. Electronics 8(2), 192 (2019). https://doi.org/10.3390/electronics8020192

    Article  Google Scholar 

  17. Naqvi, R.A., Arsalan, M., Rehman, A., Rehman, A.U., Loh, W.-K., Paul, A.: Deep learning-based drivers emotion classification system in time series data for remote applications. Remote Sens. 12(3), 587 (2020). https://doi.org/10.3390/rs12030587

    Article  Google Scholar 

  18. Choi, D.Y., Kim, D.-H., Song, B.C.: Multimodal attention network for continuous-time emotion recognition using video and EEG signals. IEEE Access 8, 203814–203826 (2020). https://doi.org/10.1109/ACCESS.2020.3036877

    Article  Google Scholar 

  19. Cordero, J., Aguilar, J., Aguilar, K., Chávez, D., Puerto, E.: Recognition of the driving style in vehicle drivers. Sensors (Basel) 20(9), 2597 (2020). https://doi.org/10.3390/s20092597.PMID:32370223;PMCID:PMC7249129

    Article  Google Scholar 

  20. Puerto, E., Aguilar, J., Vargas, R., Reyes, J.: An Ar2p deep learning architecture for the discovery and the selection of features. Neural Process. Lett. 50(1), 623–643 (2019). https://doi.org/10.1007/s11063-019-10062-4

    Article  Google Scholar 

  21. Huang, Y., Yang, J., Liu, S., Pan, J.: Combining facial expressions and electroencephalography to enhance emotion recognition. Future Internet 11(5), 105 (2019). https://doi.org/10.3390/fi11050105

    Article  Google Scholar 

  22. An, S., Ji, L.J., Marks, M., Zhang, Z.: Two sides of emotion: exploring positivity and negativity in six basic emotions across cultures. Front. Psychol. 8, 610 (2017)

    Article  Google Scholar 

  23. Niu, S.F., Liu, Y.J., Wang, L., Li, H.Q.: Effects of different intervention methods on novice drivers’ speeding. Sustainability 11(4), 1168 (2019)

    Article  Google Scholar 

  24. Mumcuoglu, M.E., et al.: Driver evaluation in heavy duty vehicles based on acceleration and braking behaviors. In: IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society, pp. 447–452. IEEE, October 2020

    Google Scholar 

  25. Minhad, K.N., Ali, S.H.M., Reaz, M.B.I.: Happy-anger emotions classifications from electrocardiogram signal for automobile driving safety and awareness. J. Transp. Health 7, 75–89 (2017)

    Article  Google Scholar 

  26. Google Play – Drive Safely. https://play.google.com/store/apps/details?id=ru.igla.drivesafely. Accessed 01 Feb 2021

  27. af Wåhlberg, A., Dorn, L., Kline, T.: The Manchester Driver Behaviour Questionnaire as a predictor of road traffic accidents. Theoret. Issues Ergon. Sci. 12(1), 66–86 (2011)

    Google Scholar 

  28. Spielberger, C.D.: State‐Trait anxiety inventory. Corsini Encyclopedia Psychol. 1 (2010)

    Google Scholar 

Download references

Acknowledgments

The research has been supported by the Russian Foundation for Basic Research project # 19–29-09081. The prototype (discussed in Sect. 6) has been supported by the Russian State Research # 0073–2019-0005.

Author information

Authors and Affiliations

  1. St. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS), 39, 14 Line, St. Petersburg, Russia

    Igor Lashkov & Alexey Kashevnik

Authors
  1. Igor Lashkov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexey Kashevnik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Igor Lashkov .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lashkov, I., Kashevnik, A. (2022). A Multimodal Approach to Psycho-Emotional State Detection of a Vehicle Driver. In: Arai, K. (eds) Intelligent Systems and Applications. IntelliSys 2021. Lecture Notes in Networks and Systems, vol 295. Springer, Cham. https://doi.org/10.1007/978-3-030-82196-8_42

Download citation

Publish with us

Policies and ethics

Search

Navigation