Abstract
Fatigue driving poses a serious threat to road safety, and research on how to effectively perceive driver fatigue and provide friendly reminders under the limited computing resources of vehicle-mounted platforms has attracted much attention. This study aims to address this limitation by developing edge computing-friendly operators and lightweight network structures. The proposed model, EMFastDet, enhances the efficiency and accuracy of driver fatigue detection. It integrates an attention module within edge computing-friendly operation blocks to capture features of the mouth and eyes. Anchor-free methods and a single detection head layer are employed for position and category predictions. The eye and mouth states in video streams are evaluated based on the metrics of Percentage of Eye Closure (PERCLOS) and Percentage of Yawning (POY). Extensive experiments were conducted using the YFDMS dataset collected in a real driving cabin environment with an infrared camera. Testing on the Qualcomm Snapdragon SA8155P chip, the DSP-accelerated EMFastDet 0.5\(\times \) version achieved an inference time of 3.02 ms and a quantized model size of 0.91 MB. The model achieved an mAP0.5 accuracy of 66.1\(\%\), meeting the deployment requirements of in-vehicle platforms.
Graphical abstract
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Czeisler CA, Wickwire EM, Barger LK et al (2016) Sleep-deprived motor vehicle operators are unfit to drive: a multidisciplinary expert consensus statement on drowsy driving. Sleep Health 2(2):94–99
Yang S, Wang H, Chen B (2023) Sibols: robust and energy-efficient learning for spike-based machine intelligence in information bottleneck framework. IEEE T Cogn Dev Syst 1–13
Yang S, Pang Y, Wang H et al (2023) Spike-driven multi-scale learning with hybrid mechanisms of spiking dendrites. Neurocomputing 542:126–240
Yang S, Chen B (2023) Effective surrogate gradient learning with high-order information bottleneck for spike-based machine intelligence. IEEE T Neur Net Lear 1–15
Xu T, Wang H, Lu G et al (2021) E-key: an eeg-based biometric authentication and driving fatigue detection system. IEEE T Affect Comput 14(2):864–877
Zhao L, Li M, He Z et al (2022) Data-driven learning fatigue detection system: a multimodal fusion approach of ecg (electrocardiogram) and video signals. Measurement 201:111648
Zheng WL, Gao K, Li G et al (2019) Vigilance estimation using a wearable eog device in real driving environment. IEEE T Intell Transp 21(1):170–184
Fan Y, Gu F, Wang J et al (2021) Safedriving: an effective abnormal driving behavior detection system based on emg signals. IEEE Internet Things 9(14):12338–12350
Yang S, Chen B (2023) Snib: improving spike-based machine learning using nonlinear information bottleneck. IEEE Trans Syst Man Cybern Syst 53(12):7852–7863
Chai M et al (2019) Drowsiness monitoring based on steering wheel status. Transp Res D-TR E 66:95–103
Wang X, Xu C (2016) Driver drowsiness detection based on non-intrusive metrics considering individual specifics. Accident Anal Prev 95:350–357
Xi J, Wang S, Ding T et al (2021) Detection model on fatigue driving behaviors based on the operating parameters of freight vehicles. Appl Sci 11(15):7132
Ansari S, Naghdy F, Du H et al (2021) Driver mental fatigue detection based on head posture using new modified relu-bilstm deep neural network. IEEE T Intell Transp 23(8):10957–10969
Yuan G, Wang Y, Yan H et al (2022) Self-calibrated driver gaze estimation via gaze pattern learning. Knowl-Based Syst 235:107630
Yang H, Liu L, Min W et al (2020) Driver yawning detection based on subtle facial action recognition. IEEE T Multimed 23:572–583
Akrout B, Mahdi W (2023) A novel approach for driver fatigue detection based on visual characteristics analysis. J Amb Intel Hum Comp 14(1):527–552
Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1867–1874
Jin H, Liao S, Shao L (2021) Pixel-in-pixel net: towards efficient facial landmark detection in the wild. Int J Comput Vision 129(12):3174–3194
Miyam M (2023) Real-time embedded system of multi-task cnn for advanced driving assistance. Int J Adv Sig Img Sci 9(2):129–138
Chen Z, Chen J, Ding G et al (2023) A lightweight cnn-based algorithm and implementation on embedded system for real-time face recognition. Multimedia Syst 29:129–138
Knapik M, Cyganek B (2019) Driver’s fatigue recognition based on yawn detection in thermal images. Neurocomputing 338:274–292
Purnamasari PD, Kriswoyo A, Ratna AAP et al (2022) Eye based drowsiness detection system for driver. J Electr Eng Technol 17:697–705
Moujahid A, Dornaika F, Arganda-Carreras I et al (2021) Efficient and compact face descriptor for driver drowsiness detection. Expert Syst Appl 168:114334
Huang B, Chen R, Xu W et al (2021) Improved fatigue detection using eye state recognition with hog-lbp. In: Proceedings of the 9th international conference on computer engineering and networks, Springer, pp 365–374
Bakheet S, Al-Hamadi A (2021) A framework for instantaneous driver drowsiness detection based on improved hog features and naïve bayesian classification. Brain Sci 11(2):240
Kır Savaşş B, Becerikli Y (2022) Behavior-based driver fatigue detection system with deep belief network. Neural Comput Appl 34(16):14053–14065
Chen S, Wang Z, Chen W (2020) Driver drowsiness estimation based on factorized bilinear feature fusion and a long-short-term recurrent convolutional network. Information 12(1):3
Ye M, Zhang W, Cao P et al (2021) Driver fatigue detection based on residual channel attention network and head pose estimation. Appl Sci 11(19):9195
Yu J, Park S, Lee S et al (2018) Driver drowsiness detection using condition-adaptive representation learning framework. IEEE T Intell Transp 20(11):4206–4218
Du G, Li T, Li C et al (2020) Vision-based fatigue driving recognition method integrating heart rate and facial features. IEEE T Intell Transp 22(5):3089–3100
Civik E, Yuzgec U (2023) Real-time driver fatigue detection system with deep learning on a low-cost embedded system. Microprocess Microsy 99:104851
Cui Z, Sun HM, Yin RN et al (2021) Real-time detection method of driver fatigue state based on deep learning of face video. Multimed Tools Appl 80:25495–25515
Li X, Lin H, Du J et al (2024) Computer vision-based driver fatigue detection framework with personalization threshold and multi-feature fusion. Signal Image Video Process 18(1):505–514
Qu J, Wei Z, Han Y (2024) An embedded device-oriented fatigue driving detection method based on a yolov5s. Neural Comput Appl 36(7):3711–3723
Sandler M, Howard A, Zhu M et al (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520
Howard A, Sandler M, Chu G et al (2019) Searching for mobilenetv3. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1314–1324
Tan M, Le Q (2021) Efficientnetv2: smaller models and faster training. In: International conference on machine learning, PMLR, pp 10096–10106
Zhang X, Zhou X, Lin M et al (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6848–6856
Ma N, Zhang X, Zheng HT et al (2018) Shufflenet v2: practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV), pp 116–131
xuehao.ma (2022) dog-qiuqiu/fastestdet: first commit. https://zenodo.org/records/6789729
Law H, Deng J (2018) Cornernet: detecting objects as paired keypoints. In: Proceedings of the European conference on computer vision (ECCV), pp 734–750
Tian Z, Shen C, Chen H et al (2019) Fcos: fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9627–9636
Jocher G, Chaurasia A, Stoken A et al (2022) ultralytics/yolov5: v6.2 - YOLOv5 Classification Models, Apple M1, Reproducibility, ClearML and Deci.ai integrations. https://doi.org/10.5281/zenodo.7002879
Zhang S, Chi C, Yao Y et al (2020) Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9759–9768
He K, Zhang X, Ren S et al (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE T Pattern Anal 37(9):1904–1916
Gevorgyan Z (2022) Siou loss: more powerful learning for bounding box regression. arXiv:2205.12740
Rezatofighi H, Tsoi N, Gwak J et al (2019) Generalized intersection over union: a metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 658–666
Zheng Z, Wang P, Liu W et al (2020) Distance-iou loss: faster and better learning for bounding box regression. In: Proceedings of the AAAI conference on artificial intelligence, pp 12993–13000
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141
Wang Q, Wu B, Zhu P et al (2020) Eca-net: efficient channel attention for deep convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11534–11542
Woo S, Park J, Lee JY et al (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19
Hou Q, Zhou D, Feng J (2021) Coordinate attention for efficient mobile network design. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13713–13722
Funding
This work was financially supported by the Natural Science Foundation of Shanghai [Grant number: 20ZR1422700].
Author information
Authors and Affiliations
Contributions
Methodology, J.W.; investigation, J.W.; B.L.; software, J.W.; B.L.; formal analysis, J.W.; B.L.; conceptualization, J.W.; P.X.; resources, B.L.; J.W.; P.X.; project administration, Z.L.; J.W.; visualization, J.W.; validation, J.W.; B.L.; writing-original draft preparation, J.W.; L.L.; writing-review and editing, L.L.; P.X.; J.W.; B.L.; Z.L.; supervision, P.X.; All authors reviewed the manuscript.
Corresponding authors
Ethics declarations
Competing Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Consent for Publication
All authors have read and agreed to the published version of the manuscript.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Supplementary file 1 (mp4 56658 KB)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, J., Li, B., Li, Z. et al. A real-time and lightweight driver fatigue detection model using anchor-free and visual-attention mechanisms. Appl Intell 54, 9811–9829 (2024). https://doi.org/10.1007/s10489-024-05696-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-024-05696-4