Skip to main content
SpringerLink
Log in

Real-time detection method of driver fatigue state based on deep learning of face video

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The use of face video information for driver fatigue detection has received extensive attention because of its low cost and non-invasiveness. However, the current vehicle-mounted embedded device has insufficient memory and limited computing power, which cannot complete the real-time detection of driver fatigue based on deep learning. Therefore, this paper designs a lightweight neural network model to solve this problem. The model includes object detection and fatigue detection. First, a lightweight object detection network is designed, which can quickly identify the opening and closing states of the driver’s eyes and mouth in the time series video. Secondly, the EYE-MOUTH (EM) driver fatigue detection model is designed, which encodes the driver’s eye and mouth opening and closing state, and calculates the driver’s PERCLOS (Percentage of Eyelid Closure over the Pupil) and FOM (Frequency of Open Mouth) according to the coding sequence. Finally, the multi-feature fusion judgment algorithm is used to realize the judgment of the driver’s fatigue state. The experimental results show that our method has an accuracy rate of 98.30% for drowsiness and yawning behaviors in a real vehicle environment, and a detection speed of 27FPS, which is better than other advanced methods and meets the requirements of real-time detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Anund A, Fors C, Ahlstrom C (2017) The severity of driver fatigue in terms of line crossing: a pilot study comparing day-and night time driving in simulator. Eur Transp Res Rev 9(31). https://doi.org/10.1007/s12544-017-0248-6

  2. Azim T, Jaffar M A, Mirza A M (2009) Automatic fatigue detection of drivers through pupil detection and yawning analysis, Int Conf Innov Comput, Inf Control (ICICIC), 441-445

  3. Bochkovskiy A, Wang C Y, Liao H Y M (2020) YOLOv4: optimal speed and accuracy of object detection, arXiv preprint arXiv:2004.10934

  4. Bosso A, Conficoni C, Raggini D (2020) A computational-effective field-oriented control strategy for accurate and efficient electric propulsion of unmanned aerial vehicles. IEEE/ASME Trans Mechatron 99:1–1. https://doi.org/10.1109/TMECH.2020.3022379

    Article  Google Scholar 

  5. Cashman D, Patterson G, Mosca A (2018) Rnnbow: visualizing learning via backpropagation gradients in rnns. IEEE Comput Graph Appl 38(6):39–50

    Article  Google Scholar 

  6. Dwivedi K, Biswaranjan K, Sethi A (2014) Drowsy driver detection using representation learning, IEEE Int Adv Comput Conf (IACC), 995-999

  7. Feldman D, Schmidt M, Sohler C (2020) Turning big data into tiny data: constant-size coresets for k-means, PCA, and projective clustering. SIAM J Comput 49(3):601–657. https://doi.org/10.1137/18M1209854

    Article  MathSciNet  MATH  Google Scholar 

  8. Flores MJ, Armingol JM, de la Escalera A (2010) Real-time warning system for driver drowsiness detection using visual information. J Intell Robot Syst 59(2):103–125

    Article  Google Scholar 

  9. Geng LF, Yuan ZT, Xiao ZT (2018) Driver fatigue detection method based on facial behavior analysis. Comput Eng 44(1):274–279

    Google Scholar 

  10. Girshick R (2015) Fast r-cnn. Proceedings of the IEEE international conference on computer vision (ICCV), pp 1440–1448. https://doi.org/10.1109/ICCV.2015.169

  11. Girshick R, Donahue J, Darrell T (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. IEEE Conf Comput Vis Pattern Recognition:580–587

  12. Gu WH, Zhu Y, Chen XD (2018) Hierarchical CNN-based real-time fatigue detection system by visual-based technologies using MSP model. IET Image Process 12(12):2319–2329

    Article  Google Scholar 

  13. He K (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37(9):1904–1916

    Article  Google Scholar 

  14. Huang J, Lin Z (1651) Multi-feature fatigue driving detection based on computer vision. J Phys Conf Ser 2020(1):012188

  15. Huang R, Pedoeem J, Chen C (2018) YOLO-LITE: a real-time object detection algorithm optimized for non-GPU computers, IEEE international conference on big data. pp 2503-2510. https://doi.org/10.1109/BigData.2018.8621865

  16. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning 37:338–356

  17. Ivanov Y (2015) Adaptive moving object segmentation algorithms in cluttered environments, The Exp Design Appl CAD Syst Microelectron, 97–99

  18. Khunpisuth O, Chotchinasri T, Koschakosai V (2016) Driver drowsiness detection using eye-closeness detection, international conference on signal-image Technology & Internet-Based Systems, 661-668

  19. Kim S, Wisanggeni I, Ros R (2020) Detecting fatigue driving through PERCLOS: a review. Int J Image Process (IJIP) 14(1):1

    Google Scholar 

  20. Koh S, Cho B R, Lee J (2017) Driver drowsiness detection via PPG biosignals by using multimodal head support, 2017 4th international conference on control, decision and information technologies (CoDIT), pp 0383–0388. https://doi.org/10.1109/CoDIT.2017.8102622

  21. Li K, Gong Y, Ren Z (2020) A fatigue driving detection algorithm based on facial multi-feature fusion. IEEE Access 8:101244–101259. https://doi.org/10.1109/ACCESS.2020.2998363

    Article  Google Scholar 

  22. Liu W, Anguelov D, Erhan D (2016) Ssd: single shot multibox detector. Eur Conf Comput Vis Springer:21–37

  23. LIU Z, LUO P, WANG X (2015) Deep learning face attributes in the wild. IEEE Comput Soc Conf Comput Vis Pattern Recog:3730–3738

  24. Lu X, Ma C, Ni B (2018) Deep regression tracking with shrinkage loss. Proceedings of the European Conference on Computer Vision (ECCV), pp 353–369). https://doi.org/10.1007/978-3-030-01264-9_22

  25. Lv Z, Qiao L (2020) Deep belief network and linear perceptron based cognitive computing for collaborative robots Appl Soft Comput, 92

  26. Lv Z, Qiao L, Li J (2020) Deep learning enabled security issues in the internet of things. IEEE Internet Things J 99:1–1

    Google Scholar 

  27. Lv Z, Zhang S, Xiu W (2020) Solving the security problem of intelligent transportation system with deep learning. IEEE Trans Intell Transp Syst 99:1–10. https://doi.org/10.1109/TITS.2020.2980864

    Article  Google Scholar 

  28. Ma N, Zhang X, Zheng H T (2018) Shufflenet v2: practical guidelines for efficient cnn architecture design, proceedings of the European conference on computer vision (ECCV). https://doi.org/10.1007/978-3-030-01264-9_8

  29. Mao Q, Sun H, Liu Y, Jia R (2019) Mini-YOLOv3: real-time object detector for embedded applications. IEEE Access 7:133529–133538

    Article  Google Scholar 

  30. Navastara DA, Putra WYM, Fatichah C (2020) Drowsiness detection based on facial landmark and uniform local binary pattern. J Phys Conf Ser 1529(5):052015. https://doi.org/10.1088/1742-6596/1529/5/052015

    Article  Google Scholar 

  31. Parekh V, Shah D, Shah M (2020) Fatigue detection using artificial intelligence framework. Augmented Hum Res 5(1):1–17. https://doi.org/10.1007/s41133-019-0023-4

    Article  Google Scholar 

  32. Peleshko D, Ivanov Y, Sharov B, Izonin I, Borzov Y (2016) Design and implementation of visitors queue density analysis and registration method for retail video surveillance purposes. IEEE First Int Conf Data Stream Min Process (DSMP) 2016:159–162

  33. Ravi A, Phanigna T R, Lenina Y (2020) Real time driver fatigue detection and smart rescue system, international conference on electronics and sustainable communication systems (ICESC), pp 434-439. https://doi.org/10.1109/ICESC48915.2020.9156021

  34. Redmon J, Divvala S, Girshick R (2016) You only look once: unified, real-time object detection. IEEE Conf Comput Vis Pattern Recognition (CVPR), pp 779–788. https://doi.org/10.1109/CVPR.2016.91

  35. Redmon J, Farhadi A (2017) YOLO9000: better, faster, stronger, IEEE Conf Comput Vis Pattern Recog, 7263–7271

  36. Redmon J, Farhadi A (2018) Yolov3: An incremental improvement, 1-6. [online]. Available: https://pjreddie.com/media/files/papers/YOLOv3.pdf

  37. Ren S, He K, Girshick R (2016) Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  38. Sandler M, Howard A, Zhu M (2018) Mobilenetv2: inverted residuals and linear bottlenecks, IEEE Conf Comput Vis Pattern Recog, 4510-4520

  39. Taigman Y, Yang M, Ranzato M A (2014) Deepface: closing the gap to human-level performance in face verification, IEEE Conf Comput Vis Pattern Recog, 1701-1708

  40. Tkachenko R, Tkachenko P, Izonin I (2018) Learning-based image scaling using neural-like structure of geometric transformation paradigm. Adv Soft Comput Mach Learn Image Process, 537–565. https://doi.org/10.1007/978-3-319-63754-9_25

  41. Wong A, Famuori M, Shafiee M J (2019) YOLO Nano: a Highly Compact You Only Look Once Convolutional Neural Network for Object Detection, arXiv preprint arXiv:1910.01271, 2019

  42. Xu H, Zhou X, Xue C (2020) Fatigue measurement of task: based on multiple eye-tracking parameters and task performance, international conference on intelligent human systems integration, pp 1263-1269. https://doi.org/10.1007/978-3-030-39512-4_193

  43. Yadav N, Banerjee K, Bali V (2020) A survey on fatigue detection of workers using machine learning. Int J E-Health Med Commun (IJEHMC) 11(3):1–8

    Article  Google Scholar 

  44. Zhang H, Li Y, Lv Z, Sangaiah AK, Huang T (2020) A real-time and ubiquitous network attack detection based on deep belief network and support vector machine. IEEE/CAA J Autom Sin 7(3):790–799. https://doi.org/10.1109/JAS.2020.1003099

    Article  Google Scholar 

  45. Zhang W, Murphey Y L, Wang T (2015) Driver yawning detection based on deep convolutional neural learning and robust nose tracking, Int Joint Conf Neural Netw (IJCNN), 1-8

  46. Zhang P, Zhong Y, Li X (2019) SlimYOLOv3: narrower, faster and better for real-time UAV applications. IEEE international conference on computer vision (ICCV). https://doi.org/10.1109/ICCVW.2019.00011

  47. Zhou Z, Cai Y, Ke R, Yang J (2017) A collision avoidance model for twopedestrian groups: considering random avoidance patterns. Physica A: Stat Mech Appl 475:142–154. https://doi.org/10.1016/j.physa.2016.12.041

  48. Zhou Z, Zhou Y, Pu Z (2019) Simulation of pedestrian behavior during the flashing green signal using a modified social force model. Transportmetrica A: Transport Sci 15(2):1019–1040. https://doi.org/10.1080/23249935.2018.1559895

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful for collaborative funding support from the Natural Science Foundation of Shandong Province, China (ZR2018MEE008), the Key Research and Development Project of Shandong Province, China (2019JZZY020326).

Author information

Authors and Affiliations

  1. College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, China

    Zhe Cui, Hong-Mei Sun, Ruo-Nan Yin, Li Gao, Hai-Bin Sun & Rui-Sheng Jia

  2. Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590, China

    Hong-Mei Sun, Hai-Bin Sun & Rui-Sheng Jia

Authors
  1. Zhe Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong-Mei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruo-Nan Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Li Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hai-Bin Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rui-Sheng Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hong-Mei Sun or Rui-Sheng Jia.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cui, Z., Sun, HM., Yin, RN. et al. Real-time detection method of driver fatigue state based on deep learning of face video. Multimed Tools Appl 80, 25495–25515 (2021). https://doi.org/10.1007/s11042-021-10930-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10930-z

Keywords

Search

Navigation