Abstract
In recent years, fatigue driving has become the main cause of traffic accidents, leading to increased attention towards fatigue detection systems. However, the pooling and strided convolutional operations in fatigue detection algorithm based on traditional deep learning methods may led to the loss of some useful information. This paper proposed a novel \(\uptau\)-shaped convolutional network (\(\mathrm{\tau Net}\)) aiming to address this issue. Unlike traditional network structures, \(\mathrm{\tau Net}\) incorporates the operations of upsampling features and concatenating high- and low-level features, enabling full utilization of useful information. Moreover, considering that the fatigue state is a mental state involving temporal evolution, we proposed the novel long short-term memory (LSTM)–\(\uptau\)-shaped convolutional network (LSTM-\(\mathrm{\tau Net}\)), a parallel structure composed of LSTM and \(\mathrm{\tau Net}\) for fatigue detection, where \(\mathrm{\tau Net}\) extracts time-invariant features with location information, and LSTM extracts long temporal dependencies. We compared LSTM-\(\mathrm{\tau Net}\) with six competing methods based on two datasets. Results showed that the proposed algorithm achieved higher classification accuracy than the other methods, with 94.25% on EEG data (binary classification) and 82.19% on EOG data (triple classification). Additionally, the proposed algorithm exhibits low computational cost, good training stability, and robustness against insufficient training. Therefore, it is promising for further implementation of fatigue online detection systems.
Graphical abstract
Access this article
We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
Similar content being viewed by others
References
Vanlaar W, Simpson H, Mayhew D, Robertson R (2008) Fatigued and drowsy driving: a survey of attitudes, opinions and behaviors. J Safety Res 39:303–309. https://doi.org/10.1016/j.jsr.2007.12.007
Gevins A, Leong H, Du R et al (1995) Towards measurement of brain function in operational environments. Biol Psychol 40:169–186. https://doi.org/10.1016/0301-0511(95)05105-8
Khunpisuth O, Chotchinasri T, Koschakosai V, Hnoohom N (2017) Driver drowsiness detection using eye-closeness detection. 12th Int Conf Signal Image Technol Internet-Based Syst SITIS 2016, Naples, ITALY, Nov 28-Dec 01,2016, Proceedings. pp 661–668. https://doi.org/10.1109/SITIS.2016.110
Hashemi Nazari SS, Moradi A, Rahmani K (2017) A systematic review of the effect of various interventions on reducing fatigue and sleepiness while driving. Chin J Traumatol - English Ed 20:249–258. https://doi.org/10.1016/j.cjtee.2017.03.005
Du G, Zhang L, Su K et al (2022) A multimodal fusion fatigue driving detection method based on heart rate and PERCLOS. IEEE Trans Intell Transp Syst 23:21810–21820. https://doi.org/10.1109/TITS.2022.3176973
Ansari S, Naghdy F, Du H, Pahnwar YN (2022) Driver mental fatigue detection based on head posture using new modified reLU-BiLSTM deep neural network. IEEE Trans Intell Transp Syst 23:10957–10969. https://doi.org/10.1109/TITS.2021.3098309
Li Z, Chen L, Nie L, Yang SX (2022) A novel learning model of driver fatigue features representation for steering wheel angle. IEEE Trans Veh Technol 71:269–281. https://doi.org/10.1109/TVT.2021.3130152
Li R, Chen YV, Zhang L (2021) A method for fatigue detection based on driver’s steering wheel grip. Int J Ind Ergon 82:103083. https://doi.org/10.1016/j.ergon.2021.103083
Karchani M, Mazloumi A, Saraji GN et al (2015) Presenting a model for dynamic facial expression changes in detecting drivers’ drowsiness. Electron Physician 7:1073–1077. https://doi.org/10.14661/2015.1073-1077
Liu Q, Liu Y, Chen K et al (2021) Research on channel selection and multi-feature fusion of EEG signals for mental fatigue detection. Entropy 23:457. https://doi.org/10.3390/e23040457
Zheng WL, Lu BL (2017) A multimodal approach to estimating vigilance using EEG and forehead EOG. J Neural Eng 14:026017. https://doi.org/10.1088/1741-2552/aa5a98
Cao Z, Chuang CH, King JK, Lin CT (2019) Multi-channel EEG recordings during a sustained-attention driving task. Sci Data 6:19. https://doi.org/10.1038/s41597-019-0027-4
Liu CC, Hosking SG, Lenné MG (2009) Predicting driver drowsiness using vehicle measures: recent insights and future challenges. J Safety Res 40:239–245. https://doi.org/10.1016/j.jsr.2009.04.005
Horne JA, Baulk SD (2004) Awareness of sleepiness when driving. Psychophysiology 41:161–165. https://doi.org/10.1046/j.1469-8986.2003.00130.x
Tong W, Chen W, Han W et al (2020) Channel-attention-based densenet network for remote sensing image scene classification. IEEE J Sel Top Appl Earth Obs Remote Sens 13:4121–4132. https://doi.org/10.1109/JSTARS.2020.3009352
Gao Z, Wang X, Yang Y et al (2019) EEG-based spatio-temporal convolutional neural network for driver fatigue evaluation. IEEE Trans Neural Netw Learn Syst 30:2755–2763. https://doi.org/10.1109/TNNLS.2018.2886414
Yeo MVM, Li X, Shen K, Wilder-Smith EPV (2009) Can SVM be used for automatic EEG detection of drowsiness during car driving? Saf Sci 47:115–124. https://doi.org/10.1016/j.ssci.2008.01.007
Dong N, Li Y, Gao Z et al (2019) A WPCA-based method for detecting fatigue driving from EEG-based internet of vehicles system. IEEE Access 7:124702–124711. https://doi.org/10.1109/ACCESS.2019.2937914
Peng Y, Wong CM, Wang Z et al (2019) Fatigue evaluation using multi-scale entropy of EEG in SSVEP-based BCI. IEEE Access 7:108200–108210. https://doi.org/10.1109/ACCESS.2019.2932503
Papadelis C, Chen Z, Kourtidou-Papadeli C et al (2007) Monitoring sleepiness with on-board electrophysiological recordings for preventing sleep-deprived traffic accidents. Clin Neurophysiol 118:1906–1922. https://doi.org/10.1016/j.clinph.2007.04.031
Zheng WL, Gao K, Li G et al (2020) Vigilance estimation using a wearable EOG device in real driving environment. IEEE Trans Intell Transp Syst 21:170–184. https://doi.org/10.1109/TITS.2018.2889962
Jammes B, Sharabty H, Esteve D (2008) Automatic EOG analysis: a first step toward automatic drowsiness scoring during wake-sleep transitions. Somnologie 12:227–232. https://doi.org/10.1007/s11818-008-0351-y
Das AK, Kumar P, Halder S (2022) Experimentation on detection and analysis of drowsiness and fatigue based on permutation entropy and hurst exponent. Condition Assessment Techniques in Electrical Systems: IEEE 6th International Conference, IEEE CATCON 2022, Natl Inst Technol, Dept Elect Engn, Durgapur, INDIA, DEC 17-19, 2022, Proceedings. IEEE, pp 239–243. https://doi.org/10.1109/CATCON56237.2022.10077702
Lee DH, Jeong JH, Kim K et al (2020) Continuous EEG decoding of pilots’ mental states using multiple feature block-based convolutional neural network. IEEE Access 8:121929–121941. https://doi.org/10.1109/ACCESS.2020.3006907
Wang H, Xu L, Bezerianos A et al (2021) Linking attention-based multiscale CNN with dynamical GCN for driving fatigue detection. IEEE Trans Instrum Meas 70:2504811. https://doi.org/10.1109/TIM.2020.3047502
Falk T, Mai D, Bensch R et al (2019) U-Net: deep learning for cell counting, detection, and morphometry. Nat Methods 16:67–70. https://doi.org/10.1038/s41592-018-0261-2
Mai Y, Chen Z, Yu B et al (2022) Non-contact heartbeat detection based on ballistocardiogram using UNet and bidirectional long short-term memory. IEEE J Biomed Heal Informatics 26:3720–3730. https://doi.org/10.1109/JBHI.2022.3162396
Hersek S, Semiz B, Shandhi MMH et al (2020) A globalized model for mapping wearable seismocardiogram signals to whole-body ballistocardiogram signals based on deep learning. IEEE J Biomed Heal Informatics 24:1296–1309. https://doi.org/10.1109/JBHI.2019.2931872
Perslev M, Darkner S, Kempfner L et al (2021) U-sleep: resilient high-frequency sleep staging. npj Digit Med 4:1–12. https://doi.org/10.1038/s41746-021-00440-5
Khessiba S, Blaiech AG, Ben Khalifa K et al (2021) Innovative deep learning models for EEG-based vigilance detection. Neural Comput Appl 33:6921–6937. https://doi.org/10.1007/s00521-020-05467-5
Jiao Y, Deng Y, Luo Y, Lu BL (2020) Driver sleepiness detection from EEG and EOG signals using GAN and LSTM networks. Neurocomputing 408:100–111. https://doi.org/10.1016/j.neucom.2019.05.108
Karim F, Majumdar S, Darabi H, Chen S (2017) LSTM fully convolutional networks for time series classification. IEEE Access 6:1662–1669. https://doi.org/10.1109/ACCESS.2017.2779939
Teimouri N, Dyrmann M, Jørgensen RN (2019) A novel spatio-temporal FCN-LSTM network for recognizing various crop types using multi-temporal radar images. Remote Sens 11:1–18. https://doi.org/10.3390/rs11080893
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Shahid A, Wilkinson K, Marcu S, Shapiro CM (2012) STOP, THAT and one hundred other sleep scales. STOP, THAT One Hundred Other Sleep Scales 1–406. https://doi.org/10.1007/978-1-4419-9893-4
Navab N, Hornegger J, Wells WM, Frangi AF (2015) Medical image computing and computer-assisted intervention - MICCAI 2015: 18th International Conference Munich, Germany, October 5–9, 2015 proceedings, part III. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 9351:12–20. https://doi.org/10.1007/978-3-319-24574-4
Wang Z, Yan W, Oates T (2017) Time series classification from scratch with deep neural networks: a strong baseline. Proceedings of the IEEE International Joint Conference on Neural Networks. pp 1578–1585. https://doi.org/10.1109/IJCNN.2017.7966039
Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. Proceedings of the 27th International Conference on International Conference on Machine Learning. pp 807–814
Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. 32nd Int Conf Mach Learn ICML 1:448–456
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7132–7141
Lin M, Chen Q, Yan S (2014) Network in network. arXiv:1312.4400
Srivastava N, Hinton G, Krizhevsky A et al (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958
He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision. pp 1026–1034. https://doi.org/10.1109/ICCV.2015.123
Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. arXiv:1412.6980
Chollet F et al. Keras [Online]. Available: https://github.com/fchollet/keras
Cui J, Lan Z, Sourina O, Muller-Wittig W (2022) EEG-based cross-subject driver drowsiness recognition with an interpretable convolutional neural network. IEEE Trans Neural Networks Learn Syst 34:7921–7933. https://doi.org/10.1109/TNNLS.2022.3147208
Karim F, Majumdar S, Darabi H, Harford S (2019) Multivariate LSTM-FCNs for time series classification. Neural Netw 116:237–245. https://doi.org/10.1016/j.neunet.2019.04.014
Shi S (2021) Visualizing data using GTSNE. arXiv:2108.01301
Acknowledgements
The authors would like to thank the Prof. Bao-Liang Lu and his research team for providing the public SEED-VIG dataset. The authors also thank Dr. Fazle Karim for his demo code.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
He, L., Zhang, L., Lin, X. et al. A novel deep-learning model based on τ-shaped convolutional network (τNet) with long short-term memory (LSTM) for physiological fatigue detection from EEG and EOG signals. Med Biol Eng Comput 62, 1781–1793 (2024). https://doi.org/10.1007/s11517-024-03033-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11517-024-03033-y