Skip to main content
SpringerLink
Log in

Driver drowsiness recognition via transferred deep 3D convolutional network and state probability vector

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Driver drowsiness is a major cause of road accidents. In this study, a novel approach that detects human drowsiness is proposed and investigated. First, driver face and facial landmarks are detected to extract facial region from each frame in a video. Then, a residual-based deep 3D convolution neural network (CNN) that learned from an irrelevant dataset is constructed to classify driver facial image sequences with a certain number of frames for obtaining its drowsiness output probability value. After that, a certain number of output probability values is concatenated to obtain the state probability vector of a video. Finally, a recurrent neural network is adopted to classify constructed probability vector and obtain the recognition result of driver drowsiness. The proposed method is tested and investigated using a public drowsy driver dataset. Experimental results demonstrate that similar to 2D CNN, 3D CNN can learn spatiotemporal features from irrelevant dataset to improve its performance obviously in driver drowsiness classification. Furthermore, the proposed method performs stably and robustly, and it can achieve an average accuracy of 88.6%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Deo N, Trivedi MM (2018) Looking at the driver/rider in autonomous vehicles to predict take-over readiness. arXiv preprint arXiv:181106047

  2. Zhao L, Wang Z, Wang X, Liu Q (2018) Driver drowsiness detection using facial dynamic fusion information and a DBN. IET Intell Transp Syst 12(2):127–133

    Article  Google Scholar 

  3. Sikander G, Anwar S (2018) Driver fatigue detection systems: a review. IEEE Trans Intell Transp Syst 20(6):2339–2352

    Article  Google Scholar 

  4. Mårtensson H, Keelan O, Ahlström C (2018) Driver sleepiness classification based on physiological data and driving performance from real road driving. IEEE Trans Intell Transp Syst 20(2):421–430

    Article  Google Scholar 

  5. Mcdonald AD, Lee JD, Schwarz C, Brown TL (2018) A contextual and temporal algorithm for driver drowsiness detection. Accid Anal Prev 113:25–37

    Article  Google Scholar 

  6. Ou C, Ouali C, Bedawi SM, Karray F Driver Behavior Monitoring Using Tools of Deep Learning and Fuzzy Inferencing. In: 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), 2018. IEEE, pp 1–7

  7. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  8. Tong M, Chen Y, Zhao M, Bu H, Xi S (2019) A deep discriminative and robust nonnegative matrix factorization network method with soft label constraint. Neural Comput & Applic 31(11):7447–7475

    Article  Google Scholar 

  9. Weng C-H, Lai Y-H, Lai S-H Driver drowsiness detection via a hierarchical temporal deep belief network. In: Asian Conference on Computer Vision, 2016. Springer, pp 117–133

  10. Guo J-M, Markoni H (2018) Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimed Tools Appl:1–29

  11. Park S, Pan F, Kang S, Yoo CD Driver drowsiness detection system based on feature representation learning using various deep networks. In: Asian Conference on Computer Vision, 2016. Springer, pp 154–164

  12. Shih T-H, Hsu C-T MSTN: Multistage spatial-temporal network for driver drowsiness detection. In: Asian Conference on Computer Vision, 2016. Springer, pp 146–153

  13. Mandal B, Li L, Wang GS, Lin J (2017) Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Trans Intell Transp Syst 18(3):545–557

    Article  Google Scholar 

  14. Cyganek B, Gruszczyński S (2014) Hybrid computer vision system for drivers’ eye recognition and fatigue monitoring. Neurocomputing 126:78–94

    Article  Google Scholar 

  15. You F, Y-h L, Huang L, Chen K, R-h Z, Xu J-m (2017) Monitoring drivers’ sleepy status at night based on machine vision. Multimed Tools Appl 76(13):14869–14886

    Article  Google Scholar 

  16. Ibrahim LF, Abulkhair M (2014) Using Haar classifiers to detect driver fatigue and provide alerts. Multimed Tools Appl 71(3):1857–1877

    Article  Google Scholar 

  17. Song F, Tan X, Liu X, Chen S (2014) Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients. Pattern Recogn 47(9):2825–2838. https://doi.org/10.1016/j.patcog.2014.03.024

    Article  Google Scholar 

  18. Gou C, Wu Y, Wang K, Wang K, Wang F-Y, Ji Q (2017) A joint cascaded framework for simultaneous eye detection and eye state estimation. Pattern Recogn 67(1):23–31

    Article  Google Scholar 

  19. Zhao L, Wang Z, Zhang G, Qi Y, Wang X (2018) Eye state recognition based on deep integrated neural network and transfer learning. Multimed Tools Appl 77(15):19415–19438

    Article  Google Scholar 

  20. Omidyeganeh M, Shirmohammadi S, Abtahi S, Khurshid A, Farhan M, Scharcanski J, Hariri B, Laroche D, Martel L (2016) Yawning detection using embedded smart cameras. IEEE Trans Instrum Meas 65(3):570–582

    Article  Google Scholar 

  21. Zhang W, Murphey YL, Wang T, Xu Q Driver yawning detection based on deep convolutional neural learning and robust nose tracking. In: 2015 International Joint Conference on Neural Networks (IJCNN), 2015. IEEE, pp 1–8

  22. Zhang W, Su J Driver yawning detection based on long short term memory networks. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI), 2017. IEEE, pp 1–5

  23. Akrout B, Mahdi W Yawning detection by the analysis of variational descriptor for monitoring driver drowsiness. In: 2016 International Image Processing, Applications and Systems (IPAS), 2016. IEEE, pp 1–5

  24. Zhao L, Wang Z, Wang X, Qi Y, Liu Q, Zhang G (2016) Human fatigue expression recognition through image-based dynamic multi-information and bimodal deep learning. J Electronic Imaging 25(5):053024

    Article  Google Scholar 

  25. Yu J, Park S, Lee S, Jeon M Representation learning, scene understanding, and feature fusion for drowsiness detection. In: Asian Conference on Computer Vision, 2016. Springer, pp 165–177

  26. Ji S, Xu W, Yang M, Yu K (2013) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Analysis Machine Intell 35(1):221–231

    Article  Google Scholar 

  27. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  28. He K, Zhang X, Ren S, Sun J Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. pp 770–778

  29. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017. pp 4700–4708

  30. Xie S, Girshick R, Dollár P, Tu Z, He K aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017. pp 1492–1500

  31. Hara K, Kataoka H, Satoh Y Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018. pp 6546–6555

  32. Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T decaf: a deep convolutional activation feature for generic visual recognition. In: International conference on machine learning, 2014. pp 647–655

  33. Yosinski J, Clune J, Bengio Y, Lipson H How transferable are features in deep neural networks? In: Advances in neural information processing systems, 2014. pp 3320–3328

  34. Kay W, Carreira J, Simonyan K, Zhang B, Hillier C, Vijayanarasimhan S, Viola F, Green T, Back T, Natsev P (2017) The kinetics human action video dataset. arXiv preprint arXiv:170506950

Download references

Acknowledgments

This work was supported by the Doctoral Foundation of Shandong Jianzhu University (China, Grant no. X18039Z), the Natural Science Foundation of Shandong Province (China, Grant no. ZR2018MEE015) and the Open Foundation of State Key Laboratory of Automotive Simulation and Control (China, Grant no. 20161105).

Author information

Authors and Affiliations

  1. School of Mechanical and Electronic Engineering, Shandong Jianzhu University, No. 1000 of Fengming Road, Jinan, China

    Lei Zhao

  2. Vehicle Engineering Research Institute, School of Mechanical Engineering, Shandong University, Jinan, China

    Zengcai Wang

  3. Department of Mechanical Engineering, Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong

    Guoxin Zhang

  4. School of Information and Electrical Engineering, Shandong Jianzhu University, No. 1000 of Fengming Road, Jinan, China

    Huanbing Gao

Authors
  1. Lei Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zengcai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guoxin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huanbing Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Zhao.

Ethics declarations

Conflict of interest

None.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, L., Wang, Z., Zhang, G. et al. Driver drowsiness recognition via transferred deep 3D convolutional network and state probability vector. Multimed Tools Appl 79, 26683–26701 (2020). https://doi.org/10.1007/s11042-020-09259-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09259-w

Keywords

Search

Navigation