Skip to main content
SpringerLink
Log in

A novel drowsiness detection model using composite features of head, eye, and facial expression

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Drowsiness is the principal cause of road crashes nowadays, as per the existing data. Drowsiness may put many precious lives in jeopardy. Drowsiness may be detected early and accurately, which can save lives. Using computer vision and deep learning techniques, this research proposes a new approach to detect driver drowsiness at an early stage with improved accuracy. In our developed model, we have considered the most significant temporal features such as head pose angles (Yaw, Pitch, and Roll), centers of pupil movement, and distance for the emotional feature that help in the detection of drowsiness state more accurately. Our method solves the possibility of occluded frames at initial stage via imposing the occlusion criteria depending on the relationship of distance between pupil centers and the horizontal length of the eye. As a result, it outperformed existing approaches in terms of overall system accuracy and consistency. Furthermore, retrieved features from correct frames are used as training and test data by the long short-term memory network to classify the driver's state. Here, results are elaborated in terms of area under the curve-receiver operating characteristic curve scores.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Sadeghniiat-Haghighi K, Yazdi Z (2015) Fatigue management in the workplace. Ind Psychiat J 24(1):12

    Article  Google Scholar 

  2. Dong Y, Hu Z, Uchimura K, Murayama N (2010) Driver inattention monitoring system for intelligent vehicles: a review. IEEE Trans Intell Transp Syst 12(2):596–614. https://doi.org/10.1109/TITS.2010.2092770

    Article  Google Scholar 

  3. Road Accidents in India (2018) https://morth.nic.in/sites/default/filesAccidednt.pdf, pp 1–125, Accessed [2 March 2021].

  4. Wheaton AG, Shults RA, Chapman DP, Ford ES, Croft JB (2014) Drowsy driving and risk behaviors—10 states and Puerto Rico, 2011–2012. MMWR. Morbidity and mortality weekly report, 63(26), 557. https://www.ncbi.nlm.nih.gov/pubmed/24990488

  5. CDC (2013) Drowsy driving 19 states and the district of Columbia, 2009–2010. MMWR Morb Mortal Wkly Rep., 63:1033. https://www.cdc.gov/mmwr/preview/mmwrhtml/mm6151a1.htm

  6. Wei CS, Wang YT, Lin CT, Jung TP (2018) Toward drowsiness detection using non-hair-bearing EEG-based braincomputer interfaces. IEEE Trans Neural Syst Rehabil Eng 26(2):400–406. https://doi.org/10.1109/TNSRE.2018.2790359

    Article  Google Scholar 

  7. Cui Y, Xu Y, Wu D (2019) EEG-based driver drowsiness estimation using feature weighted episodic training. IEEE Trans Neural Syst Rehabil Eng 27(11):2263–2273. https://doi.org/10.1109/TNSRE.2019.2945794

    Article  Google Scholar 

  8. Garg H (2020) Drowsiness detection of a driver using conventional computer vision application. In: 2020 international conference on power electronics & IoT applications in renewable energy and its control (PARC), pp 50–53. IEEE. https://doi.org/10.1109/PARC49193.2020.236556

  9. Pandey NN, Muppalaneni NB (2021) Temporal and spatial feature based approaches in drowsiness detection using deep learning technique. J Real-Time Image Proc. https://doi.org/10.1007/s11554-021-01114-x

    Article  Google Scholar 

  10. Ghoddoosian R, Galib M, Athitsos V (2019) A realistic dataset and baseline temporal model for early drowsiness detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, http://www.cv-foundation.org/

  11. Tsuzuki Y, Mizusako M, Yasushi M, Hashimoto H (2019) Sleepiness detection system based on facial expressions. In: IECON 2019–45th annual conference of the IEEE Industrial electronics society, vol 1, pp 6934–6939. IEEE. https://doi.org/10.1109/IECON.2019.8927215

  12. Dasgupta A, Rahman D, Routray A (2018) A smart phone based drowsiness detection and warning system for automotive drivers. IEEE Trans Intell Transp Syst 20(11):4045–4054. https://doi.org/10.1109/TITS.2018.2879609

    Article  Google Scholar 

  13. Wang Y, Huang R, Guo L (2019) Eye gaze pattern analysis for fatigue detection based on GP-BCNN with ESM. Pattern Recogn Lett 123:61–74. https://doi.org/10.1016/j.patrec.2019.03.013

    Article  Google Scholar 

  14. Joshi A, Kyal S, Banerjee S, Mishra T (2020) In-the-wild drowsiness detection from facial expressions. In: 2020 IEEE intelligent vehicles symposium (IV), pp 207–212. IEEE. https://doi.org/10.1109/IV47402.2020.9304579

  15. Johns MW (2003) The amplitude-velocity ratio of blinks: a new method for monitoring drowsiness. Sleep, 26(SUPPL.)

  16. McIntire LK, McKinley RA, Goodyear C, McIntire JP (2014) Detection of vigilance performance using eye blinks. Appl Ergon 45(2):354–362. https://doi.org/10.1016/j.apergo.2013.04.020

    Article  Google Scholar 

  17. Yan WJ, Wu Q, Liu YJ, Wang SJ, Fu X (2013) CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–7. https://doi.org/10.1109/FG.2013.6553799

  18. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) CASME II: An improved spontaneous micro expression database and the baseline evaluation. PLoS ONE 9(1):e86041. https://doi.org/10.1371/journal.pone.0086041

    Article  Google Scholar 

  19. Yin H, Su Y, Liu Y, Zhao D (2016) A driver fatigue detection method based on multi-sensor signals. In: 2016 IEEE winter conference on applications of computer vision (WACV), pp 1–7. https://doi.org/10.1109/WACV.2016.7477672

  20. RLDD: dataset created by The University of Texas at Arlington in 2019. https://sites.google.com/view/utarldd/home. Accessed 10 Jan 2020.

  21. Åkerstedt T, Gillberg M (1990) Subjective and objective sleepiness in the active individual. Int J Neurosci 52(1–2):29–37. https://doi.org/10.3109/00207459008994241

    Article  Google Scholar 

  22. Choi IH, Kim YG (2014) Head pose and gaze direction tracking for detecting a drowsy driver. In: 2014 international conference on big data and smart computing (BIGCOMP), pp 241–244. IEEE. https://doi.org/10.1109/BIGCOMP.2014.6741444

  23. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1867–1874. http://www.cv-foundation.org/

  24. skvar: Installation and Usage- Jan 2, 2020. https://pypi.org/project/opencv-python/ (2020). Accessed 15 Jan 2020

  25. Zhihong W, Xiaohong X (2011) Study on histogram equalization. In: International symposium on intelligence information processing and trusted computing, pp 177–179. IEEE Computer Society. https://doi.org/10.1109/IPTC.2011.52

  26. Kubinger W, Vincze M, Ayromlou M (1998) The role of gamma correction in colour image processing. In: 9th European signal processing conference (EUSIPCO 1998), pp 1–4. IEEE.

  27. Adrian Rosebrock (2019). Eye motion tracking, https://www.youtube.com/watch?v= kbdbZFT9NQI. Accessed 20 Jan 2020.

  28. Ji Y, Wang S, Lu Y, Wei J, Zhao Y (2018) Eye and mouth state detection algorithm based on contour feature extraction. J Electron Imaging 27(5):051205. https://doi.org/10.1117/1.JEI.27.5.051205

    Article  Google Scholar 

  29. Madarkar J, Sharma P (2020) Head pose estimation of face: angle of roll, yaw, and pitch of the face image. In: International conference on machine learning, image processing, network security and data sciences, pp 228–242. https://link.springer.com/book/https://doi.org/10.1007/978-981-15-6315-7

  30. Aditi Mittal (2019) Understanding RNN and LSTM,. https://towardsdatascience.com/understanding-rnn-andlstm-f7cdf6dfc 14e. Accessed 20 Jan 2020.

  31. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  32. Greff K, Srivastava RK, Koutník J, Steunebrink BR, Schmidhuber J (2016) LSTM: A search space odyssey. IEEE Trans Neural Netw Learn Syst 28(10):2222–2232. https://doi.org/10.1109/TNNLS.2016.2582924

    Article  MathSciNet  Google Scholar 

  33. Lei G, Xiaoyu L, Zhitao X, Yuelong L (2018) Real-time driver fatigue detection based on morphology infrared features and deep learning. Infrared Laser Eng 47(2): 203009–0203009. https://doi.org/10.3788/IRLA201847.0203009

  34. Guo JM, Markoni H (2019) Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimedia Tools Appl 78(20):29059–29087. https://doi.org/10.1007/s11042-018-6378-6

    Article  Google Scholar 

  35. Picot A, Charbonnier S, Caplier A (2010) Drowsiness detection based on visual signs: blinking analysis based on high frame rate video. In: IEEE instrumentation & measurement technology conference proceedings, pp 801–804. https://doi.org/10.1109/IMTC.2010.5488257

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, National Institute of Technology Silchar, Silchar, Assam, India

    Nageshwar Nath Pandey & Naresh Babu Muppalaneni

Authors
  1. Nageshwar Nath Pandey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Naresh Babu Muppalaneni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nageshwar Nath Pandey.

Ethics declarations

Conflict of interest

The authors state that they have no known competing financial interests or personal connections that might have influenced the research presented in this publication.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pandey, N.N., Muppalaneni, N.B. A novel drowsiness detection model using composite features of head, eye, and facial expression. Neural Comput & Applic 34, 13883–13893 (2022). https://doi.org/10.1007/s00521-022-07209-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07209-1

Keywords

Search

Navigation