Skip to main content
SpringerLink
Log in

Multimodal Fatigue Detection in Drivers via Physiological and Visual Signals

  • Conference paper
  • First Online:
Artificial Intelligence Security and Privacy (AIS&P 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14509))

Abstract

Driving conditions such as stress, sleepiness and fatigue can easily lead to traffic accidents, and the detection of these unsafe conditions is an important means of ensuring driving safety. The fatigue detection system currently on the market suffers from a single detection feature and low detection accuracy. In view of the powerful feature extraction and fusion capabilities of neural networks, a multimodal driver fatigue detection method based on physiological and visual signals is proposed for processing physiological signals from wearable devices and visual signals from cameras during the driving process. Firstly, the model processes the physiological signal sequences and locates significant changes in physiological state, and extracts visual features based on visual signals, mainly eye features and complementary mouth and head features. Secondly, the fatigue features are obtained by fusing the physiological and visual features in the temporal dimension, based on which an effective fatigue detection model can be trained. We use publicly available datasets to segment physiological signal sequences such as heart rate and quantitatively capture change points. After testing in different visual environments such as day and night and with or without face occlusion, this model can meet the requirements of basic real-time fatigue detection with a high detection accuracy.

Supported by Jiangxi Province 03 Special Project (No. 20203ABC03W07).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Sarker, H., et al.: Finding significant stress episodes in a discontinuous time series of rapidly varying mobile sensor data. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 4489–4501 (2016)

    Google Scholar 

  2. Fogarty, J., Hudson, S.E., Lai, J.: Examining the robustness of sensor-based statistical models of human interruptibility. In: Proceedings of SIGCHI, pp. 207–214 (2004)

    Google Scholar 

  3. Ghoddoosian, R., Galib, M., Athitsos, V.: A realistic dataset and baseline temporal model for early drowsiness detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 178–187 (2019). https://doi.org/10.1109/CVPRW.2019.00027’

  4. Zhang, W., Murphey, Y.L., Wang, T., Xu, Q.: Driver yawning detection based on deep convolutional neural learning and robust nose tracking. In: 2015 International Joint Conference on Neural Networks (IJCNN), IEEE, pp. 1–8 (2015)

    Google Scholar 

  5. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2010)

    Article  Google Scholar 

  6. Schneegass, S., Pfleging, B., Broy, N., et al.: A data set of real world driving to assess driver workload. In: Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 150–157 (2013)

    Google Scholar 

  7. Saeed, A., Trajanovski, S., Van Keulen, M., et al.: Deep physiological arousal detection in a driving simulator using wearable sensors. In: 2017 IEEE International Conference on Data Mining Workshops (ICDMW). IEEE, pp. 486–493 (2017)

    Google Scholar 

  8. Sahayadhas, A., Sundaraj, K., Murugappan, M.: Detecting driver drowsiness based on sensors: a review. Sensors 12(12), 16937–16953 (2012)

    Article  Google Scholar 

  9. Kim, H.G., Cheon, E.J., Bai, D.S., et al.: Stress and heart rate variability: a meta-analysis and review of the literature. Psychiatry Investig. 15(3), 235 (2018)

    Article  Google Scholar 

  10. Munla, N., Khalil, M., Shahin, A., et al.: Driver stress level detection using HRV analysis. In: 2015 International Conference on Advances in Biomedical Engineering (ICABME). IEEE, pp. 61–64 (2015)

    Google Scholar 

  11. Malik, M.: Heart rate variability: standards of measurement, physiological interpretation, and clinical use: task force of the European society of cardiology and the north American society for pacing and electrophysiology. Ann. Noninvasive Electrocardiol. 1(2), 151–181 (1996)

    Article  Google Scholar 

  12. Widjaja, D., Orini, M., Vlemincx, E., et al.: Cardiorespiratory dynamic response to mental stress: a multivariate time-frequency analysis. Comput. Math. Methods Med. 2013, 451857 (2013)

    Article  Google Scholar 

  13. Chowdhury, A., Shankaran, R., Kavakli, M., et al.: Sensor applications and physiological features in drivers’ drowsiness detection: a review. IEEE Sens. J. 18(8), 3055–3067 (2018)

    Article  Google Scholar 

  14. Affanni, A., Bernardini, R., Piras, A., et al.: Driver’s stress detection using skin potential response signals. Measurement 122, 264–274 (2018)

    Article  Google Scholar 

  15. Stappen, L., Schumann, L., Sertolli, B., et al.: MuSe-Toolbox: the multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox. In: Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge, pp. 75–82 (2021)

    Google Scholar 

  16. Lapuschkin, S., Binder, A., Muller, K.R., et al.: Understanding and comparing deep neural networks for age and gender classification. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1629–1638 (2017)

    Google Scholar 

  17. Howard, A.G., Zhu, M., Chen, B., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  18. Greff, K., Srivastava, R.K., Koutnik, J., et al.: LSTM: a search space odyssey (2015). arXiv preprint arXiv:1503.04069 (2016)

  19. Hallac, D., Nystrup, P., Boyd, S.: Greedy Gaussian segmentation of multivariate time series. Adv. Data Anal. Classif. 13(3), 727–751 (2019)

    Article  MathSciNet  Google Scholar 

  20. Van Gent, P., Farah, H., Van Nes, N., et al.: HeartPy: a novel heart rate algorithm for the analysis of noisy signals. Transport. Res. F: Traffic Psychol. Behav. 66, 368–378 (2019)

    Article  Google Scholar 

  21. Makowski, D., et al.: NeuroKit2: a Python toolbox for neurophysiological signal processing. Behav. Res. Methods 53(4), 1689–1696 (2021). https://doi.org/10.3758/s13428-020-01516-y

    Article  Google Scholar 

  22. Feng, T., Booth, B.M., Narayanan, S.S.: Modeling behavior as mutual dependency between physiological signals and indoor location in large-scale wearable sensor study. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp. 1016–1020 (2020)

    Google Scholar 

  23. Chen, X., Gu, Q., Liu, W., Liu, S., Ni, C.: Research on static software defect prediction methods. J. Softw. 27(1), 1–25 (2016). https://www.jos.org.cn/1000-9825/4923.htm, https://doi.org/10.13328/j.cnki.jos.004923

  24. Hongjun, W., Hao, B., Hui, Z., et al.: Driver fatigue state detection and warning technology based on computer vision. Sci. Technol. Eng. 22(12), 4887–4894 (2022). https://doi.org/10.3969/j.issn.1671-1815.2022.12.027

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information Engineering, East China Jiao Tong University, Nanchang, 33000, China

    Weijia Li, Hong Tang & Junhang Qiu

  2. Jiangxi Artificial Intelligence Production-Education Integration Innovation Center, Nanchang, China

    Weijia Li, Hong Tang & Junhang Qiu

  3. School of Information Engineering, Jiangxi University of Science and Technology, No. 115, Yaohu University Park, Nanchang, Jiangxi, China

    Xunxun Pi

Authors
  1. Weijia Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xunxun Pi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hong Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junhang Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junhang Qiu .

Editor information

Editors and Affiliations

  1. Rutgers University, Newark, NJ, USA

    Jaideep Vaidya

  2. Tampere University, Tampere, Finland

    Moncef Gabbouj

  3. Guangzhou University, Guangzhou, China

    Jin Li

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, W., Pi, X., Tang, H., Qiu, J. (2024). Multimodal Fatigue Detection in Drivers via Physiological and Visual Signals. In: Vaidya, J., Gabbouj, M., Li, J. (eds) Artificial Intelligence Security and Privacy. AIS&P 2023. Lecture Notes in Computer Science, vol 14509. Springer, Singapore. https://doi.org/10.1007/978-981-99-9785-5_16

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-9785-5_16

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-9784-8

  • Online ISBN: 978-981-99-9785-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation