Skip to main content

Advertisement

Log in

FMIF: facial multi-feature information fusion for driver fatigue detection

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Driver fatigue is a major contributing factor to traffic accidents, and accurately detecting driver fatigue is crucial. Existing methods for driver fatigue detection face limitations in handling interference from pose changes in real-world scenarios and distinguishing between similar movements. This paper proposes a driver fatigue detection method based on facial multi-feature information fusion, which includes facial multi-granularity feature extraction, facial correlation feature fusion network, and temporal feature extraction network. Specifically, the method first extracts accurate facial fatigue features using a facial multi-granularity feature extraction method, followed by extracting spatial feature information of driver facial fatigue features using a compact facial correlation feature fusion extraction network. Temporal feature information is then extracted using a long short-term memory recurrent neural network. Finally, the spatiotemporal feature information is combined to identify the driver’s status. The method was evaluated on the National Tsing Hua University Driver Drowsiness Detection dataset and demonstrated outstanding detection performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and material

The datasets used during the cur - rent study are available from the corresponding author on reasonable request.

References

  1. Owens, J., Dingus, T., Guo, F., Fang, Y., Perez, M., McClafferty, J., Tefft, B.: Prevalence of drowsy-driving crashes: Estimates from a large-scale naturalistic driving study (2018)

  2. Sikander, G., Anwar, S.: Driver fatigue detection systems: a review. IEEE Trans. Intell. Transp. Syst. 20, 2339–2352 (2018). https://doi.org/10.1109/TITS.2018.2868499

    Article  MATH  Google Scholar 

  3. Maior, C.B.S., das Chagas Moura, M.J., Santana, J.M.M., Lins, I.D.: Real-time classification for autonomous drowsiness detection using eye aspect ratio. Exp. Syst. Appl. 158, 113505 (2020). https://doi.org/10.1016/j.eswa.2020.113505

  4. Ahmed, M., Masood, S., Ahmad, M., Abd El-Latif, A.A.: Intelligent driver drowsiness detection for traffic safety based on multi cnn deep model and facial subsampling. IEEE Trans. Intell. Transp. Syst. 23, 19743–19752 (2021). https://doi.org/10.1109/TITS.2021.3134222

    Article  MATH  Google Scholar 

  5. Bergasa, L.M., Nuevo, J., Sotelo, M.A., Barea, R., Lopez, M.E.: Real-time system for monitoring driver vigilance. IEEE Trans. Intell. Transp. Syst. 7, 63–77 (2006). https://doi.org/10.1109/TITS.2006.869598

    Article  MATH  Google Scholar 

  6. Liu, M.-Z., Xu, X., Hu, J., Jiang, Q.-N.: Real time detection of driver fatigue based on cnn-lstm. IET Image Proc. 16, 576–595 (2022). https://doi.org/10.1049/ipr2.12373

    Article  MATH  Google Scholar 

  7. Huang, R., Wang, Y., Li, Z., Lei, Z., Xu, Y.: Rf-dcm: multi-granularity deep convolutional model based on feature recalibration and fusion for driver fatigue detection. IEEE Trans. Intell. Transp. Syst. 23, 630–640 (2020). https://doi.org/10.1109/TITS.2020.3017513

    Article  MATH  Google Scholar 

  8. Wang, Y., Huang, R., Guo, L.: Eye gaze pattern analysis for fatigue detection based on gp-bcnn with esm. Pattern Recogn. Lett. 123, 61–74 (2019). https://doi.org/10.1016/j.patrec.2019.03.013

    Article  MATH  Google Scholar 

  9. Zhao, Z., Zhou, N., Zhang, L., Yan, H., Xu, Y., Zhang, Z., others.: Driver fatigue detection based on convolutional neural networks using em-cnn. Comput. Intell. Neurosci. (2020). https://doi.org/10.1155/2020/7251280

  10. Li, X., Xia, J., Cao, L., Zhang, G., Feng, X.: Driver fatigue detection based on convolutional neural network and face alignment for edge computing device. Proc. Inst. Mech. Eng. D J. Automobile Eng. 235, 2699–2711 (2021). https://doi.org/10.1177/0954407021999485

  11. Zhao, G., He, Y., Yang, H., Tao, Y.: Research on fatigue detection based on visual features. IET Image Proc. 16, 1044–1053 (2022). https://doi.org/10.1049/ipr2.12207

    Article  MATH  Google Scholar 

  12. Yi, Y., Zhou, Z., Zhang, W., Zhou, M., Yuan, Y., Li, C.: Fatigue detection algorithm based on eye multifeature fusion. IEEE Sens. J. 23, 7949–7955 (2023). https://doi.org/10.1109/JSEN.2023.3247582

    Article  MATH  Google Scholar 

  13. Ghoddoosian, R., Galib, M., Athitsos, V.: A realistic dataset and baseline temporal model for early drowsiness detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)

  14. Alameen, S.A., Alhothali, A.M.: A lightweight driver drowsiness detection system using 3dcnn with lstm. Comput. Syst. Sci. Eng. 4, 1 (2023). https://doi.org/10.1177/0954407021999485

    Article  Google Scholar 

  15. Bai, J., Yu, W., Xiao, Z., Havyarimana, V., Regan, A.C., Jiang, H., Jiao, L.: Two-stream spatial-temporal graph convolutional networks for driver drowsiness detection. IEEE Trans. Cybern. 52, 13821–13833 (2021). https://doi.org/10.1109/TCYB.2021.3110813

    Article  MATH  Google Scholar 

  16. Zhang, L., Zheng, X., Chen, X., Ren, X., Ji, C.: Facial expression recognition based on spatial-temporal fusion with attention mechanism. Neural Process. Lett. pp. 1–16 (2022)

  17. Vu, T.H., Dang, A., Wang, J.-C.: A deep neural network for real-time driver drowsiness detection. IEICE Trans. Inf. Syst. 102, 2637–2641 (2019). https://doi.org/10.1587/transinf.2019EDL8079

    Article  MATH  Google Scholar 

  18. Jamshidi, S., Azmi, R., Sharghi, M., Soryani, M.: Hierarchical deep neural networks to detect driver drowsiness. Multimed. Tools. Appl. 80, 16045–16058 (2021)

    Article  Google Scholar 

  19. Bajaj, J.S., Kumar, N., Kaushal, R.K., Gururaj, H., Flammini, F., Natarajan, R.: System and method for driver drowsiness detection using behavioral and sensor-based physiological measures. Sensors. 23, 1292 (2023). https://doi.org/10.3390/s23031292

    Article  Google Scholar 

  20. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23, 1499–1503 (2016). https://doi.org/10.1109/LSP.2016.2603342

    Article  MATH  Google Scholar 

  21. Bakheet, S., Al-Hamadi, A.: A framework for instantaneous driver drowsiness detection based on improved hog features and naïve bayesian classification. Brain Sci. 11, 240 (2021). https://doi.org/10.3390/brainsci11020240

    Article  MATH  Google Scholar 

  22. Maheswari, V.U., Aluvalu, R., Kantipudi, M.P., Chennam, K.K., Kotecha, K., Saini, J.R.: Driver drowsiness prediction based on multiple aspects using image processing techniques. IEEE Access. 10, 54980–54990 (2022). https://doi.org/10.1109/ACCESS.2022.3176451

    Article  Google Scholar 

  23. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37, 1904–1916 (2015). https://doi.org/10.1109/TPAMI.2015.2389824

    Article  MATH  Google Scholar 

  24. Dua, M., Singla, R., Raj, S., Jangra, A.: Deep cnn models-based ensemble approach to driver drowsiness detection. Neural Comput. Appl. 33, 3155–3168 (2021)

    Article  Google Scholar 

  25. Zhang, L., Saito, H., Yang, L., Wu, J.: Privacy-preserving federated transfer learning for driver drowsiness detection. IEEE Access. 10, 80565–80574 (2022)

    Article  Google Scholar 

  26. Husain, S.S., Mir, J., Anwar, S.M., Rafique, W., Ullah, M.O.: Development and validation of a deep learning-based algorithm for drowsiness detection in facial photographs. Multimed. Tools Appl. 81, 20425–20441 (2022)

    Article  Google Scholar 

  27. Weng, C.-H., Lai, Y.-H., Lai, S.-H.: Driver drowsiness detection via a hierarchical temporal deep belief network. In: Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part III 13, pp. 117–133 (2017)

  28. Aydemir, G., Kurnaz, O., Bekiryazıcı, T., Avcı, A., Kocakulak, M.: Driver drowsiness detection using mobilenets and long short-term memory. In: 2021 13th International Conference on Electrical and Electronics Engineering (ELECO), pp. 220–223 (2021)

  29. Mao, H., Tang, J., Zhao, X., Tang, M., Jiang, Z.: A driver drowsiness detection scheme based on 3d convolutional neural networks. Int. J. Pattern Recogn. Artific. Intell. 36, 2252007 (2022)

  30. Kumar, V., Sharma, S.: Driver drowsiness detection using modified deep learning architecture. Evol. Intel. 158, 1–10 (2022)

    MATH  Google Scholar 

Download references

Funding

This research was supported by the Research Foundation of the Institute of Environment-friendly Materials and Occupational Health (Wuhu), Anhui University of Science and Technology (No. ALW2021YF04), University Synergy Innovation Program of Anhui Province under Grant GXXT-2021-006.

Author information

Authors and Affiliations

Authors

Contributions

XL: conceived the idea for the study and provided guidance throughout the research process. They also contributed to the methodology development, experimental design, and data analysis. Additionally, they played a significant role in writing the manuscript. WY: conducted the majority of the experiments and data collection. They implemented the proposed method, performed the necessary computations, and analyzed the results. They also contributed to the writing and revision of the manuscript. XF: contributed to the literature review, gathering relevant research materials, and providing critical insights into the topic. They actively participated in discussions and provided valuable suggestions for improving the methodology. They also contributed to the manuscript writing and revision. CZ: assisted in data preprocessing, conducted additional experiments to validate the proposed method, and contributed to the analysis and interpretation of the results. They actively participated in discussions and provided critical feedback during the research process. They also contributed to the manuscript writing and revision.

Corresponding author

Correspondence to Wei Yao.

Ethics declarations

Conflict of interest

The authors declare no Conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liang, X., Yao, W., Fang, X. et al. FMIF: facial multi-feature information fusion for driver fatigue detection. SIViP 19, 121 (2025). https://doi.org/10.1007/s11760-024-03573-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11760-024-03573-8

Keywords

Navigation