Skip to main content

Advertisement

A real-time and lightweight driver fatigue detection model using anchor-free and visual-attention mechanisms

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Fatigue driving poses a serious threat to road safety, and research on how to effectively perceive driver fatigue and provide friendly reminders under the limited computing resources of vehicle-mounted platforms has attracted much attention. This study aims to address this limitation by developing edge computing-friendly operators and lightweight network structures. The proposed model, EMFastDet, enhances the efficiency and accuracy of driver fatigue detection. It integrates an attention module within edge computing-friendly operation blocks to capture features of the mouth and eyes. Anchor-free methods and a single detection head layer are employed for position and category predictions. The eye and mouth states in video streams are evaluated based on the metrics of Percentage of Eye Closure (PERCLOS) and Percentage of Yawning (POY). Extensive experiments were conducted using the YFDMS dataset collected in a real driving cabin environment with an infrared camera. Testing on the Qualcomm Snapdragon SA8155P chip, the DSP-accelerated EMFastDet 0.5\(\times \) version achieved an inference time of 3.02 ms and a quantized model size of 0.91 MB. The model achieved an mAP0.5 accuracy of 66.1\(\%\), meeting the deployment requirements of in-vehicle platforms.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Czeisler CA, Wickwire EM, Barger LK et al (2016) Sleep-deprived motor vehicle operators are unfit to drive: a multidisciplinary expert consensus statement on drowsy driving. Sleep Health 2(2):94–99

    Article  Google Scholar 

  2. Yang S, Wang H, Chen B (2023) Sibols: robust and energy-efficient learning for spike-based machine intelligence in information bottleneck framework. IEEE T Cogn Dev Syst 1–13

  3. Yang S, Pang Y, Wang H et al (2023) Spike-driven multi-scale learning with hybrid mechanisms of spiking dendrites. Neurocomputing 542:126–240

    Article  Google Scholar 

  4. Yang S, Chen B (2023) Effective surrogate gradient learning with high-order information bottleneck for spike-based machine intelligence. IEEE T Neur Net Lear 1–15

  5. Xu T, Wang H, Lu G et al (2021) E-key: an eeg-based biometric authentication and driving fatigue detection system. IEEE T Affect Comput 14(2):864–877

  6. Zhao L, Li M, He Z et al (2022) Data-driven learning fatigue detection system: a multimodal fusion approach of ecg (electrocardiogram) and video signals. Measurement 201:111648

  7. Zheng WL, Gao K, Li G et al (2019) Vigilance estimation using a wearable eog device in real driving environment. IEEE T Intell Transp 21(1):170–184

    Article  Google Scholar 

  8. Fan Y, Gu F, Wang J et al (2021) Safedriving: an effective abnormal driving behavior detection system based on emg signals. IEEE Internet Things 9(14):12338–12350

  9. Yang S, Chen B (2023) Snib: improving spike-based machine learning using nonlinear information bottleneck. IEEE Trans Syst Man Cybern Syst 53(12):7852–7863

  10. Chai M et al (2019) Drowsiness monitoring based on steering wheel status. Transp Res D-TR E 66:95–103

    Article  Google Scholar 

  11. Wang X, Xu C (2016) Driver drowsiness detection based on non-intrusive metrics considering individual specifics. Accident Anal Prev 95:350–357

    Article  Google Scholar 

  12. Xi J, Wang S, Ding T et al (2021) Detection model on fatigue driving behaviors based on the operating parameters of freight vehicles. Appl Sci 11(15):7132

    Article  Google Scholar 

  13. Ansari S, Naghdy F, Du H et al (2021) Driver mental fatigue detection based on head posture using new modified relu-bilstm deep neural network. IEEE T Intell Transp 23(8):10957–10969

    Article  Google Scholar 

  14. Yuan G, Wang Y, Yan H et al (2022) Self-calibrated driver gaze estimation via gaze pattern learning. Knowl-Based Syst 235:107630

    Article  Google Scholar 

  15. Yang H, Liu L, Min W et al (2020) Driver yawning detection based on subtle facial action recognition. IEEE T Multimed 23:572–583

    Article  Google Scholar 

  16. Akrout B, Mahdi W (2023) A novel approach for driver fatigue detection based on visual characteristics analysis. J Amb Intel Hum Comp 14(1):527–552

    Article  Google Scholar 

  17. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1867–1874

  18. Jin H, Liao S, Shao L (2021) Pixel-in-pixel net: towards efficient facial landmark detection in the wild. Int J Comput Vision 129(12):3174–3194

  19. Miyam M (2023) Real-time embedded system of multi-task cnn for advanced driving assistance. Int J Adv Sig Img Sci 9(2):129–138

    Google Scholar 

  20. Chen Z, Chen J, Ding G et al (2023) A lightweight cnn-based algorithm and implementation on embedded system for real-time face recognition. Multimedia Syst 29:129–138

    Article  Google Scholar 

  21. Knapik M, Cyganek B (2019) Driver’s fatigue recognition based on yawn detection in thermal images. Neurocomputing 338:274–292

    Article  Google Scholar 

  22. Purnamasari PD, Kriswoyo A, Ratna AAP et al (2022) Eye based drowsiness detection system for driver. J Electr Eng Technol 17:697–705

    Article  Google Scholar 

  23. Moujahid A, Dornaika F, Arganda-Carreras I et al (2021) Efficient and compact face descriptor for driver drowsiness detection. Expert Syst Appl 168:114334

    Article  Google Scholar 

  24. Huang B, Chen R, Xu W et al (2021) Improved fatigue detection using eye state recognition with hog-lbp. In: Proceedings of the 9th international conference on computer engineering and networks, Springer, pp 365–374

  25. Bakheet S, Al-Hamadi A (2021) A framework for instantaneous driver drowsiness detection based on improved hog features and naïve bayesian classification. Brain Sci 11(2):240

    Article  Google Scholar 

  26. Kır Savaşş B, Becerikli Y (2022) Behavior-based driver fatigue detection system with deep belief network. Neural Comput Appl 34(16):14053–14065

    Article  Google Scholar 

  27. Chen S, Wang Z, Chen W (2020) Driver drowsiness estimation based on factorized bilinear feature fusion and a long-short-term recurrent convolutional network. Information 12(1):3

    Article  Google Scholar 

  28. Ye M, Zhang W, Cao P et al (2021) Driver fatigue detection based on residual channel attention network and head pose estimation. Appl Sci 11(19):9195

    Article  Google Scholar 

  29. Yu J, Park S, Lee S et al (2018) Driver drowsiness detection using condition-adaptive representation learning framework. IEEE T Intell Transp 20(11):4206–4218

    Article  Google Scholar 

  30. Du G, Li T, Li C et al (2020) Vision-based fatigue driving recognition method integrating heart rate and facial features. IEEE T Intell Transp 22(5):3089–3100

    Article  Google Scholar 

  31. Civik E, Yuzgec U (2023) Real-time driver fatigue detection system with deep learning on a low-cost embedded system. Microprocess Microsy 99:104851

    Article  Google Scholar 

  32. Cui Z, Sun HM, Yin RN et al (2021) Real-time detection method of driver fatigue state based on deep learning of face video. Multimed Tools Appl 80:25495–25515

    Article  Google Scholar 

  33. Li X, Lin H, Du J et al (2024) Computer vision-based driver fatigue detection framework with personalization threshold and multi-feature fusion. Signal Image Video Process 18(1):505–514

    Article  Google Scholar 

  34. Qu J, Wei Z, Han Y (2024) An embedded device-oriented fatigue driving detection method based on a yolov5s. Neural Comput Appl 36(7):3711–3723

    Article  Google Scholar 

  35. Sandler M, Howard A, Zhu M et al (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  36. Howard A, Sandler M, Chu G et al (2019) Searching for mobilenetv3. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1314–1324

  37. Tan M, Le Q (2021) Efficientnetv2: smaller models and faster training. In: International conference on machine learning, PMLR, pp 10096–10106

  38. Zhang X, Zhou X, Lin M et al (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6848–6856

  39. Ma N, Zhang X, Zheng HT et al (2018) Shufflenet v2: practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV), pp 116–131

  40. xuehao.ma (2022) dog-qiuqiu/fastestdet: first commit. https://zenodo.org/records/6789729

  41. Law H, Deng J (2018) Cornernet: detecting objects as paired keypoints. In: Proceedings of the European conference on computer vision (ECCV), pp 734–750

  42. Tian Z, Shen C, Chen H et al (2019) Fcos: fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9627–9636

  43. Jocher G, Chaurasia A, Stoken A et al (2022) ultralytics/yolov5: v6.2 - YOLOv5 Classification Models, Apple M1, Reproducibility, ClearML and Deci.ai integrations. https://doi.org/10.5281/zenodo.7002879

  44. Zhang S, Chi C, Yao Y et al (2020) Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9759–9768

  45. He K, Zhang X, Ren S et al (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE T Pattern Anal 37(9):1904–1916

    Article  Google Scholar 

  46. Gevorgyan Z (2022) Siou loss: more powerful learning for bounding box regression. arXiv:2205.12740

  47. Rezatofighi H, Tsoi N, Gwak J et al (2019) Generalized intersection over union: a metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 658–666

  48. Zheng Z, Wang P, Liu W et al (2020) Distance-iou loss: faster and better learning for bounding box regression. In: Proceedings of the AAAI conference on artificial intelligence, pp 12993–13000

  49. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  50. Wang Q, Wu B, Zhu P et al (2020) Eca-net: efficient channel attention for deep convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11534–11542

  51. Woo S, Park J, Lee JY et al (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  52. Hou Q, Zhou D, Feng J (2021) Coordinate attention for efficient mobile network design. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13713–13722

Download references

Funding

This work was financially supported by the Natural Science Foundation of Shanghai [Grant number: 20ZR1422700].

Author information

Authors and Affiliations

Authors

Contributions

Methodology, J.W.; investigation, J.W.; B.L.; software, J.W.; B.L.; formal analysis, J.W.; B.L.; conceptualization, J.W.; P.X.; resources, B.L.; J.W.; P.X.; project administration, Z.L.; J.W.; visualization, J.W.; validation, J.W.; B.L.; writing-original draft preparation, J.W.; L.L.; writing-review and editing, L.L.; P.X.; J.W.; B.L.; Z.L.; supervision, P.X.; All authors reviewed the manuscript.

Corresponding authors

Correspondence to Peiquan Xu or Leijun Li.

Ethics declarations

Competing Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Consent for Publication

All authors have read and agreed to the published version of the manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 56658 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Li, B., Li, Z. et al. A real-time and lightweight driver fatigue detection model using anchor-free and visual-attention mechanisms. Appl Intell 54, 9811–9829 (2024). https://doi.org/10.1007/s10489-024-05696-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05696-4

Keywords