Skip to main content
SpringerLink
Log in

An fNIRS labeling image feature-based customized driving fatigue detection method

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

The practicality of driving fatigue detection approaches largely depends on social acceptance. Physiological-based methods perform well, but they are rarely accepted by drivers, while subjective evaluations cannot support real-time detection. However, image-based approaches with relatively high take-up rates, ace detection accuracy problems, thus causing driver dissatisfaction. To satisfy the demand of logistic companies, in this paper, a customized driving fatigue detection method that integrates a subjective evaluation, physiological features and image features is proposed. As a result, a real-time driving fatigue detection system, RefreshingDrive, is designed. First, the relationships between questionnaire results, near-infrared spectroscopy indices, and oxyhemoglobin (HBO) and deoxyhemoglobin (HHB) values are investigated to express the individual differences among fatigue thresholds. Then, a long short-term memory network is constructed to find an individual-oriented HBO+HHB threshold, which is used to divide an image dataset into two subdatasets, namely, a fatigue dataset and a nonfatigue dataset. After that, a facial feature-based fatigue detection method, in which a multitasking cascaded eye convolution network is proposed to extract eye features while a 3D ConvNet is employed to recognize fatigue actions, is presented. Moreover, a fatigue evaluation rule, the Action-Blink Fusion Rule (A-B fusion), is proposed to recognize driving fatigue. Experimental results show that the proposed method achieves a fatigue recognition accuracy of 95.3%. Finally, an on-road test is performed to verify the effectiveness of the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  • Ahmadi A, Bazregarzadeh H, Kazemi K (2021) Automated detection of driver fatigue from electroencephalography through wavelet-based connectivity. Biocybern Biomed Eng 41(1):316–332

    Google Scholar 

  • Ahn S, Nguyen T, Jang H, Kim JG, Jun SC (2016) Exploring neuro-physiological correlates of drivers’ mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and FNIRS data. Front Hum Neurosci 10:219

    Google Scholar 

  • Akerstedt T, Gillberg M (1990) Subjective and objective sleepiness in the active individual. Int J Neurosci 52(1–2):29–37

    Google Scholar 

  • Akrout B, Mahdi W (2021) A novel approach for driver fatigue detection based on visual characteristics analysis. J Ambient Intell Human Comput 20:25

    Google Scholar 

  • Alioua N, Amine A, Rziza M (2014) Driver’s fatigue detection based on yawning extraction. Int J Veh Technol 20:14

    Google Scholar 

  • Ayachi R, Afif M, Said Y, Abdelali AB (2021) Drivers fatigue detection using efficientdet in advanced driver assistance systems. In: 2021 18th international multi-conference on systems, signals and devices (SSD), pp 738–742. IEEE

  • Azarnoosh M, Mohammadi MR, Nasrabadi AM, Firoozabadi SMP (2010) Evaluating variability of frequency features of EEG signals during mental fatigue. In: 2010 17th Iranian conference of biomedical engineering (ICBME), pp 1–4. IEEE

  • Bakker B, Zabłocki B, Baker A, Riethmeister V, Marx B, Iyer G, Anund A, Ahlström C (2022) A multi-stage, multi-feature machine learning approach to detect driver sleepiness in naturalistic road driving conditions. IEEE Trans Intell Transp Syst 23(5):4791–4800

    Google Scholar 

  • Bhatt R, Naik N, Subramanian VK (2021) Ssim compliant modeling framework with denoising and deblurring applications. IEEE Trans Image Process 30:2611–2626

    MathSciNet  Google Scholar 

  • Chen M, Li F, Lei J, Zeng Z, Han Q, Chen Q (2017) Driving fatigue detecting method based on temperature insensitive ECG parameters. In: International conference on internet of vehicles. Springer, pp 105–118

  • Dang W, Gao Z, Lv D, Sun X, Cheng C (2020) Rhythm-dependent multilayer brain network for the detection of driving fatigue. IEEE J Biomed Health Inform 25(3):693–700

    Google Scholar 

  • Daza IG, Bergasa LM, Bronte S, Yebes JJ, Almazán J, Arroyo R (2014) Fusion of optimized indicators from advanced driver assistance systems (ADAS) for driver drowsiness detection. Sensors (Basel, Switzerland) 14(1):1106–1131

    Google Scholar 

  • Hara K, Kataoka H, Satoh Y (2018) Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6546–6555

  • Herscovitch J, Broughton R (1981) Sensitivity of the Stanford sleepiness scale to the effects of cumulative partial sleep deprivation and recovery oversleeping. Sleep 4(1):83–92

    Google Scholar 

  • Hoddes E, Dement W, Zarcone V (1972) The development and use of the Stanford sleepiness scale (SSS). Psychophysiology 10:431–436

    Google Scholar 

  • Huynh X-P, Park S-M, Kim Y-G (2016) Detection of driver drowsiness using 3d deep neural network and semi-supervised gradient boosting machine. In: Asian conference on computer vision. Springer, pp 134–145

  • Jia H, Xiao Z, Ji P (2021) Fatigue driving detection based on deep learning and multi-index fusion. IEEE Access 9:147054–147062

    Google Scholar 

  • Kaida K, Takahashi M, Åkerstedt T, Nakata A, Otsuka Y, Haratani T, Fukasawa K (2006) Validation of the karolinska sleepiness scale against performance and EEG variables. Clin Neurophysiol 117(7):1574–1581

    Google Scholar 

  • Klaiber M, Sauter D, Baumgartl H, Buettner R (2021) A systematic literature review on transfer learning for 3d-cnns. In: 2021 international joint conference on neural networks (IJCNN), pp 1–10. IEEE

  • Ku H, Dong W (2020) Face recognition based on mtcnn and convolutional neural network. Front Signal Process 4(1):37–42

    Google Scholar 

  • Lam C, Epps J, Chen S (2021) Wearable fatigue detection based on blink-saccade synchronisation. In: 2021 IEEE international conference on systems, man, and cybernetics (SMC), pp 1186–1191. IEEE

  • Lei J, Han Q, Chen L, Lai Z, Zeng L, Liu X (2017) A novel side face contour extraction algorithm for driving fatigue statue recognition. IEEE Access 5:5723–5730

    Google Scholar 

  • Lei J, Liu F, Han Q, Tang Y, Zeng L, Chen M, Ye L, Jin L (2018) Study on driving fatigue evaluation system based on short time period ECG signal. In: 2018 21st international conference on intelligent transportation systems (ITSC), pp 2466–2470. IEEE

  • Li Z, Zhang M, Zhang X, Dai S, Yu X, Wang Y (2009) Assessment of cerebral oxygenation during prolonged simulated driving using near infrared spectroscopy: its implications for fatigue development. Eur J Appl Physiol 107(3):281–287

    Google Scholar 

  • Li R, Chen YV, Zhang L (2021) A method for fatigue detection based on driver’s steering wheel grip. Int J Ind Ergon 82:103083

    Google Scholar 

  • Li X, Xia J, Cao L, Zhang G, Feng X (2021) Driver fatigue detection based on convolutional neural network and face alignment for edge computing device. Proc Inst Mech Eng Part D J Autom Eng 235(10–11):2699–2711

    Google Scholar 

  • Lin CT, King JT, Chuang CH, Ding W, Wang YK (2019) Exploring the brain responses to driving fatigue through simultaneous EEG and FNIRS measurements. Int J Neural Syst 30:9

    Google Scholar 

  • Liu Y, Zhang T, Li Z (2019) Dcnn-based real-time driver fatigue behavior detection in urban rail transit. IEEE Access 99:1

    Google Scholar 

  • Liu Z, Peng Y, Hu W (2019) Driver fatigue detection based on deeply-learned facial expression representation. J Vis Commun Image Represent 71:102723

    Google Scholar 

  • Lu Y, Wang Z (2007) Detecting driver yawning in successive images. In: 2007 1st international conference on bioinformatics and biomedical engineering, pp 581–583. IEEE

  • Mao H, Tang J, Zhao X, Tang M, Jiang Z (2022) A driver drowsiness detection scheme based on 3d convolutional neural networks. Int J Pattern Recognit Artif Intell 36(02):2252007

    Google Scholar 

  • Min J, Xiong C, Zhang Y, Cai M (2021) Driver fatigue detection based on prefrontal EEG using multi-entropy measures and hybrid model. Biomed Signal Process Control 69:102857

    Google Scholar 

  • NHTSA (2020) Preview of motor vehicle crashes in 2019. NHTSA’s National Center for Statistics and Analysis, 1

  • Nihashi T, Ishigaki T, Satake H, Ito S, Kaii O, Mori Y, Shimamoto K, Fukushima H, Suzuki K, Umakoshi H et al (2019) Monitoring of fatigue in radiologists during prolonged image interpretation using fnirs. Jpn J Radiol 37(6):437–448

    Google Scholar 

  • Pan T, Wang H, Si H, Liu H, Xu M (2022) Research on the identification of pilots’ fatigue status based on functional near-infrared spectroscopy. Aerospace 9(3):173

    Google Scholar 

  • Pei Z, Zhenghe S, Yiming Z (2002) Perclos-based recognition algorithms of motor driver fatigue. J China Agric Univ 7(2):104–109

    Google Scholar 

  • Penson A, van Deuren S, Worm-Smeitink M, Bronkhorst E, van den Hoogen FHJ, van Engelen BGM, Peters M, Bleijenberg G, Vercoulen JH, Blijlevens N, van Dulmen-den Broeder E, Loonen J, Knoop H (2020) Short fatigue questionnaire: screening for severe fatigue. J Psychosom Res 137:110229

    Google Scholar 

  • Połap D, Srivastava G (2021) Neural image reconstruction using a heuristic validation mechanism. Neural Comput Appl 33(17):10787–10797

    Google Scholar 

  • Połap D, Woźniak M (2021) Meta-heuristic as manager in federated learning approaches for image processing purposes. Appl Soft Comput 113:107872

    Google Scholar 

  • Połap D, Wawrzyniak N, Włodarczyk-Sielicka M (2022) Side-scan sonar analysis using roi analysis and deep neural networks. IEEE Trans Geosci Remote Sens 60:1–8

    Google Scholar 

  • Putilov AA, Donskaya OG (2013) Construction and validation of the EEG analogues of the karolinska sleepiness scale based on the karolinska drowsiness test. Clin Neurophysiol 124(7):1346–1352

    Google Scholar 

  • Tak S, Ye JC (2014) Statistical analysis of fnirs data: a comprehensive review. Neuroimage 85:72–91

    Google Scholar 

  • Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 4489–4497

  • Tuncer T, Dogan S, Ertam F, Subasi A (2021) A dynamic center and multi threshold point based stable feature extraction network for driver fatigue detection utilizing EEG signals. Cogn Neurodyn 15:7

    Google Scholar 

  • Wang Z, Bovik A, Sheikh H, Simoncelli E (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Google Scholar 

  • Wang F, Hong W, Fu R (2018) Real-time ECG-based detection of fatigue driving using sample entropy. Entropy 20(3):196

    MathSciNet  Google Scholar 

  • Wang H, Dragomir A, Abbasi NI, Li J, Thakor NV, Bezerianos A (2018) A novel real-time driving fatigue detection system based on wireless dry EEG. Cogn Neurodyn 12(4):365–376

    Google Scholar 

  • Wu EQ, Xiong P, Tang ZR, Li GJ, Song A, Zhu LM (2022) Detecting dynamic behavior of brain fatigue through 3-d-cnn-lstm. IEEE Trans Syst Man Cybern Syst 52(1):90–100

    Google Scholar 

  • Xu S, Zhao X-h, Zhang X-J, Rong J (2011) A study of the identification method of driving fatigue based on physiological signals. In: ICCTP 2011: towards sustainable transportation systems, pp 2296–2307

  • Yan P, Sun Y, Li Z, Zou J, Hong D (2020) Driver fatigue detection system based on colored and infrared eye features fusion. Comput Mater Contin 63(3):1563–1574

    Google Scholar 

  • Yang JH, Mao Z-H, Tijerina L, Pilutti T, Coughlin J, Feron E (2009) Detection of driver fatigue caused by sleep deprivation. IEEE Trans Syst Man Cybern Part A Syst Humans 39(4):694–705

    Google Scholar 

  • Ye M, Zhang W, Cao P, Liu K (2021) Driver fatigue detection based on residual channel attention network and head pose estimation. Appl Sci 11(19):9195

    Google Scholar 

  • Yin Z, Liu B, Hao D, Yang L, Feng Y (2022) Evaluation of vdt-induced visual fatigue by automatic detection of blink features. Sensors 22(3):916

    Google Scholar 

  • Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Google Scholar 

  • Zhang F, Su J, Geng L, Xiao Z (2017) Driver fatigue detection based on eye state recognition. In: 2017 international conference on machine vision and information technology (CMVIT), pp 105–110. IEEE

  • Zhao Z, Zhou N, Zhang L, Yan H, Xu Y, Zhang Z (2020) Driver fatigue detection based on convolutional neural networks using EM-CNN. Comput Intell Neurosci 2020:7251280

    Google Scholar 

  • Zhao G, He Y, Yang H, Tao Y (2022) Research on fatigue detection based on visual features. IET Image Proc 16(4):1044–1053

    Google Scholar 

  • Zhu T, Zhang C, Wu T, Ouyang Z, Li H, Na X, Liang J, Li W (2022) Research on a real-time driver fatigue detection algorithm based on facial video sequences. Appl Sci 12(4):2224

    Google Scholar 

Download references

Acknowledgements

Thank the anonymous reviewers for their insightful comments that resulted in significantly improved paper. This work was supported by the special key project of Chongqing Technology Innovation and Application Development under Grant No.csct 2021jscx-gksbX0057, in part by the National Nature Science Foundation of China, under Project 62172066 and U21A20448, in part by Central University Foundation of China, under project 2022CDJJJ-003.

Author information

Authors and Affiliations

  1. College of Computer Science, Chongqing University, Chongqing, 400044, China

    Lingqiu Zeng & Kun Zhou

  2. College of Communication Engineering, Chongqing University, Chongqing, 400044, China

    Qingwen Han, Yang Wang & Lei Ye

  3. College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing, 400044, China

    Gang Guo

Authors
  1. Lingqiu Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kun Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qingwen Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gang Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lei Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qingwen Han.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, L., Zhou, K., Han, Q. et al. An fNIRS labeling image feature-based customized driving fatigue detection method. J Ambient Intell Human Comput 14, 12493–12509 (2023). https://doi.org/10.1007/s12652-022-04325-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-022-04325-7

Keywords

Search

Navigation