Abstract
Driver drowsiness is one of the main reasons for deadly accidents, especially on suburban roads. Researchers have used many methods for analyzing videos and detecting drowsiness, and the most up-to-date methods among them are using deep learning. This paper proposes a hierarchical framework comprising deep networks with split spatial and temporal phases referred to as hierarchical deep drowsiness detection (HDDD) network. The proposed method uses ResNet to detect the driver’s face, lighting condition, and whether the driver is wearing glasses or not. This phase also causes a significant increase in eyes and mouth detection percentage in the next stage. Afterward, the LSTM network is used to take advantage of temporal information between the frames. The average accuracy of the drowsiness detection system is reached 87.19 percent.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-021-10542-7/MediaObjects/11042_2021_10542_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-021-10542-7/MediaObjects/11042_2021_10542_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-021-10542-7/MediaObjects/11042_2021_10542_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-021-10542-7/MediaObjects/11042_2021_10542_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-021-10542-7/MediaObjects/11042_2021_10542_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-021-10542-7/MediaObjects/11042_2021_10542_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-021-10542-7/MediaObjects/11042_2021_10542_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-021-10542-7/MediaObjects/11042_2021_10542_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-021-10542-7/MediaObjects/11042_2021_10542_Fig9_HTML.png)
Similar content being viewed by others
References
Alioua N, Amine A, Rziza M (2014) Driver’s fatigue detection based on yawning extraction. Int J Veh Technol, vol 2014
Bergasa LM, Nuevo J, Sotelo MA, Barea R, Lopez ME (2006) Real-time system for monitoring driver vigilance. IEEE Trans Intell Transp Syst 7(1):63–77
Bhandari G, Durge A, Bidwai A, Aware U (2014) Driver drowsiness monitoring. Int J Sci Eng Technol 3(2):203–206
Breuer R, Kimmel R (2017) A deep learning perspective on the origin of facial expressions. arXiv:1705.01842
Choi IH, Kim YG (2014) Head pose and gaze direction tracking for detecting a drowsy driver. In: 2014 International conference on big data and smart computing (BIGCOMP). IEEE, pp 241–244
Craye C, Karray F (2015) Driver distraction detection and recognition using rgb-d sensor. arXiv:1502.00250
Diaz-Chito K, Hernández-Sabaté A., López A. M. (2016) A reduced feature set for driver head pose estimation. Appl Soft Comput 45:98–107
Feng R, Zhang G, Cheng B (2009) An on-board system for detecting driver drowsiness based on multi-sensor data fusion using dempster-shafer theory. In: 2009 International conference on networking, sensing and control. IEEE, pp 897–902
González-Ortega D, Díaz-Pernas F, Antón-Rodríguez M, Martínez-Zarzuela M, Díez-Higuera J (2013) Real-time vision-based eye state detection for driver alertness monitoring. Pattern Anal Appl 16(3):285–306
Gritzman AD, Rubin DM, Pantanowitz A (2015) Comparison of colour transforms used in lip segmentation algorithms. SIViP 9(4):947–957
Guo JM, Markoni H (2019) Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimed Tools Appl 78(20):29,059–29,087
Hachisuka S (2013) Human and vehicle-driver drowsiness detection by facial expression. In: 2013 International conference on biometrics and Kansei engineering. IEEE, pp 320–326
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Ibrahim MM, Soraghan JJ, Petropoulakis L, Di Caterina G (2015) Yawn analysis with mouth occlusion detection. Biomed Sig Process Control 18:360–369
Ingre M, Åkerstedt T, Peters B, Anund A, Kecklund G (2006) Subjective sleepiness, simulated driving performance and blink duration: examining individual differences. J Sleep Res 15(1):47–53
Ishii Y, Ogitsu T, Takemura H, Mizoguchi H (2014) Real-time eyelid open/closed state recognition based on hlac towards driver drowsiness detection. In: 2014 IEEE international conference on robotics and biomimetics (ROBIO 2014). IEEE, pp 2449–2454
Jones M, Viola P (2003) Fast multi-view face detection. Mitsubishi Electr Res Lab TR-20003-96 3(14):2
Kalbkhani H, Amirani MC (2012) An efficient algorithm for lip segmentation in color face images based on local information. J World’s Electr Eng Technol 1(1):12–16
Kaplan S, Guvensan MA, Yavuz AG, Karalurt Y (2015) Driver behavior analysis for safe driving: A survey. IEEE Trans Intell Transp Syst 16 (6):3017–3032
Li G, Lee BL, Chung WY (2015) Smartwatch-based wearable eeg system for driver drowsiness detection. IEEE Sensors J 15(12):7169–7180
Li Z, Li SE, Li R, Cheng B, Shi J (2017) Online detection of driver fatigue using steering wheel angles for real driving conditions. Sensors 17(3):495
Li H, Lin Z, Shen X, Brandt J, Hua G (2015) A convolutional neural network cascade for face detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 5325–5334
Lin CT, Chang CJ, Lin BS, Hung SH, Chao CF, Wang IJ (2010) A real-time wireless brain–computer interface system for drowsiness detection. IEEE Trans Biomed Circ Syst 4(4):214–222
Lu X, Wang W, Shen J, Tai YW, Crandall DJ, Hoi SC (2020) Learning video object segmentation from unlabeled videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8960–8970
Mandal B, Li L, Wang GS, Lin J (2016) Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Trans Intell Transp Syst 18(3):545–557
Mbouna RO, Kong SG, Chun MG (2013) Visual analysis of eye state and head pose for driver alertness monitoring. IEEE Trans Intell Transp Syst 14(3):1462–1469
Mehta S, Dadhich S, Gumber S, Jadhav Bhatt A (2019) Real-time driver drowsiness detection system using eye aspect ratio and eye closure ratio. In: Proceedings of international conference on sustainable computing in science, technology and Management (SUSCOM), Amity University Rajasthan, Jaipur-India
Omidyeganeh M, Shirmohammadi S, Abtahi S, Khurshid A, Farhan M, Scharcanski J, Hariri B, Laroche D, Martel L (2016) Yawning detection using embedded smart cameras. IEEE Trans Instrum Meas 65(3):570–582
Park S, Pan F, Kang S, Yoo CD (2016) Driver drowsiness detection system based on feature representation learning using various deep networks. In: Asian conference on computer vision. Springer, pp. 154–164
Pratama BG, Ardiyanto I, Adji TB (2017) A review on driver drowsiness based on image, bio-signal, and driver behavior. In: 2017 3rd international conference on science and technology-computer (ICST). IEEE, pp 70–75
Saini V, Saini R (2014) Driver drowsiness detection system and techniques: a review. Int J Comput Sci Inf Technol 5(3):4245–4249
Shih TH, Hsu CT (2016) Mstn: Multistage spatial-temporal network for driver drowsiness detection. In: Asian conference on computer vision. Springer, pp 146–153
Tansakul W, Tangamchit P (2016) Fatigue driver detection system using a combination of blinking rate and driving inactivity. J Autom Control Eng, vol 4(1)
Tu Y, Zeng C, Yeh C, Huang S, Cheng T, Ouhyoung M (2011) Real-time head pose estimation using depth map for avatar control. In: Proceedings of IPPR conference on computer vision, graphics, and image processing
Weng CH, Lai YH, Lai SH (2016) Driver drowsiness detection via a hierarchical temporal deep belief network. In: Asian conference on computer vision. Springer, pp 117–133
Wood R, Olszewska JI (2012) Lighting-variable adaboost based-on system for robust face detection. In: Proceedings of the 5th international conference on bio-inspired systems and signal processing. SciTePress digital library, pp. 494–497
Yu J, Park S, Lee S, Jeon M (2016) Representation learning, scene understanding, and feature fusion for drowsiness detection. In: Asian conference on computer vision. Springer, pp 165–177
Zhang L, Liu F, Tang J (2015) Real-time system for driver fatigue detection by rgb-d camera. ACM Trans Intell Syst Technol (TIST) 6(2):1–17
Zhang Z, Luo P, Loy CC, Tang X (2014) Facial landmark detection by deep multi-task learning. In: European conference on computer vision. Springer, pp 94–108
Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig Process Lett 23(10):1499–1503
Zhao L, Wang Z, Wang X, Qi Y, Liu Q, Zhang G (2016) Human fatigue expression recognition through image-based dynamic multi-information and bimodal deep learning. J Electron Imaging 25(5):053,024
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Jamshidi, S., Azmi, R., Sharghi, M. et al. Hierarchical deep neural networks to detect driver drowsiness. Multimed Tools Appl 80, 16045–16058 (2021). https://doi.org/10.1007/s11042-021-10542-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-021-10542-7