Abstract
Driver fatigue is a major contributing factor to traffic accidents, and accurately detecting driver fatigue is crucial. Existing methods for driver fatigue detection face limitations in handling interference from pose changes in real-world scenarios and distinguishing between similar movements. This paper proposes a driver fatigue detection method based on facial multi-feature information fusion, which includes facial multi-granularity feature extraction, facial correlation feature fusion network, and temporal feature extraction network. Specifically, the method first extracts accurate facial fatigue features using a facial multi-granularity feature extraction method, followed by extracting spatial feature information of driver facial fatigue features using a compact facial correlation feature fusion extraction network. Temporal feature information is then extracted using a long short-term memory recurrent neural network. Finally, the spatiotemporal feature information is combined to identify the driver’s status. The method was evaluated on the National Tsing Hua University Driver Drowsiness Detection dataset and demonstrated outstanding detection performance.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03573-8/MediaObjects/11760_2024_3573_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03573-8/MediaObjects/11760_2024_3573_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03573-8/MediaObjects/11760_2024_3573_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03573-8/MediaObjects/11760_2024_3573_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03573-8/MediaObjects/11760_2024_3573_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03573-8/MediaObjects/11760_2024_3573_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03573-8/MediaObjects/11760_2024_3573_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03573-8/MediaObjects/11760_2024_3573_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03573-8/MediaObjects/11760_2024_3573_Fig9_HTML.png)
Similar content being viewed by others
Availability of data and material
The datasets used during the cur - rent study are available from the corresponding author on reasonable request.
References
Owens, J., Dingus, T., Guo, F., Fang, Y., Perez, M., McClafferty, J., Tefft, B.: Prevalence of drowsy-driving crashes: Estimates from a large-scale naturalistic driving study (2018)
Sikander, G., Anwar, S.: Driver fatigue detection systems: a review. IEEE Trans. Intell. Transp. Syst. 20, 2339–2352 (2018). https://doi.org/10.1109/TITS.2018.2868499
Maior, C.B.S., das Chagas Moura, M.J., Santana, J.M.M., Lins, I.D.: Real-time classification for autonomous drowsiness detection using eye aspect ratio. Exp. Syst. Appl. 158, 113505 (2020). https://doi.org/10.1016/j.eswa.2020.113505
Ahmed, M., Masood, S., Ahmad, M., Abd El-Latif, A.A.: Intelligent driver drowsiness detection for traffic safety based on multi cnn deep model and facial subsampling. IEEE Trans. Intell. Transp. Syst. 23, 19743–19752 (2021). https://doi.org/10.1109/TITS.2021.3134222
Bergasa, L.M., Nuevo, J., Sotelo, M.A., Barea, R., Lopez, M.E.: Real-time system for monitoring driver vigilance. IEEE Trans. Intell. Transp. Syst. 7, 63–77 (2006). https://doi.org/10.1109/TITS.2006.869598
Liu, M.-Z., Xu, X., Hu, J., Jiang, Q.-N.: Real time detection of driver fatigue based on cnn-lstm. IET Image Proc. 16, 576–595 (2022). https://doi.org/10.1049/ipr2.12373
Huang, R., Wang, Y., Li, Z., Lei, Z., Xu, Y.: Rf-dcm: multi-granularity deep convolutional model based on feature recalibration and fusion for driver fatigue detection. IEEE Trans. Intell. Transp. Syst. 23, 630–640 (2020). https://doi.org/10.1109/TITS.2020.3017513
Wang, Y., Huang, R., Guo, L.: Eye gaze pattern analysis for fatigue detection based on gp-bcnn with esm. Pattern Recogn. Lett. 123, 61–74 (2019). https://doi.org/10.1016/j.patrec.2019.03.013
Zhao, Z., Zhou, N., Zhang, L., Yan, H., Xu, Y., Zhang, Z., others.: Driver fatigue detection based on convolutional neural networks using em-cnn. Comput. Intell. Neurosci. (2020). https://doi.org/10.1155/2020/7251280
Li, X., Xia, J., Cao, L., Zhang, G., Feng, X.: Driver fatigue detection based on convolutional neural network and face alignment for edge computing device. Proc. Inst. Mech. Eng. D J. Automobile Eng. 235, 2699–2711 (2021). https://doi.org/10.1177/0954407021999485
Zhao, G., He, Y., Yang, H., Tao, Y.: Research on fatigue detection based on visual features. IET Image Proc. 16, 1044–1053 (2022). https://doi.org/10.1049/ipr2.12207
Yi, Y., Zhou, Z., Zhang, W., Zhou, M., Yuan, Y., Li, C.: Fatigue detection algorithm based on eye multifeature fusion. IEEE Sens. J. 23, 7949–7955 (2023). https://doi.org/10.1109/JSEN.2023.3247582
Ghoddoosian, R., Galib, M., Athitsos, V.: A realistic dataset and baseline temporal model for early drowsiness detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)
Alameen, S.A., Alhothali, A.M.: A lightweight driver drowsiness detection system using 3dcnn with lstm. Comput. Syst. Sci. Eng. 4, 1 (2023). https://doi.org/10.1177/0954407021999485
Bai, J., Yu, W., Xiao, Z., Havyarimana, V., Regan, A.C., Jiang, H., Jiao, L.: Two-stream spatial-temporal graph convolutional networks for driver drowsiness detection. IEEE Trans. Cybern. 52, 13821–13833 (2021). https://doi.org/10.1109/TCYB.2021.3110813
Zhang, L., Zheng, X., Chen, X., Ren, X., Ji, C.: Facial expression recognition based on spatial-temporal fusion with attention mechanism. Neural Process. Lett. pp. 1–16 (2022)
Vu, T.H., Dang, A., Wang, J.-C.: A deep neural network for real-time driver drowsiness detection. IEICE Trans. Inf. Syst. 102, 2637–2641 (2019). https://doi.org/10.1587/transinf.2019EDL8079
Jamshidi, S., Azmi, R., Sharghi, M., Soryani, M.: Hierarchical deep neural networks to detect driver drowsiness. Multimed. Tools. Appl. 80, 16045–16058 (2021)
Bajaj, J.S., Kumar, N., Kaushal, R.K., Gururaj, H., Flammini, F., Natarajan, R.: System and method for driver drowsiness detection using behavioral and sensor-based physiological measures. Sensors. 23, 1292 (2023). https://doi.org/10.3390/s23031292
Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23, 1499–1503 (2016). https://doi.org/10.1109/LSP.2016.2603342
Bakheet, S., Al-Hamadi, A.: A framework for instantaneous driver drowsiness detection based on improved hog features and naïve bayesian classification. Brain Sci. 11, 240 (2021). https://doi.org/10.3390/brainsci11020240
Maheswari, V.U., Aluvalu, R., Kantipudi, M.P., Chennam, K.K., Kotecha, K., Saini, J.R.: Driver drowsiness prediction based on multiple aspects using image processing techniques. IEEE Access. 10, 54980–54990 (2022). https://doi.org/10.1109/ACCESS.2022.3176451
He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37, 1904–1916 (2015). https://doi.org/10.1109/TPAMI.2015.2389824
Dua, M., Singla, R., Raj, S., Jangra, A.: Deep cnn models-based ensemble approach to driver drowsiness detection. Neural Comput. Appl. 33, 3155–3168 (2021)
Zhang, L., Saito, H., Yang, L., Wu, J.: Privacy-preserving federated transfer learning for driver drowsiness detection. IEEE Access. 10, 80565–80574 (2022)
Husain, S.S., Mir, J., Anwar, S.M., Rafique, W., Ullah, M.O.: Development and validation of a deep learning-based algorithm for drowsiness detection in facial photographs. Multimed. Tools Appl. 81, 20425–20441 (2022)
Weng, C.-H., Lai, Y.-H., Lai, S.-H.: Driver drowsiness detection via a hierarchical temporal deep belief network. In: Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part III 13, pp. 117–133 (2017)
Aydemir, G., Kurnaz, O., Bekiryazıcı, T., Avcı, A., Kocakulak, M.: Driver drowsiness detection using mobilenets and long short-term memory. In: 2021 13th International Conference on Electrical and Electronics Engineering (ELECO), pp. 220–223 (2021)
Mao, H., Tang, J., Zhao, X., Tang, M., Jiang, Z.: A driver drowsiness detection scheme based on 3d convolutional neural networks. Int. J. Pattern Recogn. Artific. Intell. 36, 2252007 (2022)
Kumar, V., Sharma, S.: Driver drowsiness detection using modified deep learning architecture. Evol. Intel. 158, 1–10 (2022)
Funding
This research was supported by the Research Foundation of the Institute of Environment-friendly Materials and Occupational Health (Wuhu), Anhui University of Science and Technology (No. ALW2021YF04), University Synergy Innovation Program of Anhui Province under Grant GXXT-2021-006.
Author information
Authors and Affiliations
Contributions
XL: conceived the idea for the study and provided guidance throughout the research process. They also contributed to the methodology development, experimental design, and data analysis. Additionally, they played a significant role in writing the manuscript. WY: conducted the majority of the experiments and data collection. They implemented the proposed method, performed the necessary computations, and analyzed the results. They also contributed to the writing and revision of the manuscript. XF: contributed to the literature review, gathering relevant research materials, and providing critical insights into the topic. They actively participated in discussions and provided valuable suggestions for improving the methodology. They also contributed to the manuscript writing and revision. CZ: assisted in data preprocessing, conducted additional experiments to validate the proposed method, and contributed to the analysis and interpretation of the results. They actively participated in discussions and provided critical feedback during the research process. They also contributed to the manuscript writing and revision.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no Conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Liang, X., Yao, W., Fang, X. et al. FMIF: facial multi-feature information fusion for driver fatigue detection. SIViP 19, 121 (2025). https://doi.org/10.1007/s11760-024-03573-8
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11760-024-03573-8