Skip to main content
Springer Nature Link
Log in

Eye state recognition based on deep integrated neural network and transfer learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Eye state recognition is widely used in many fields, such as driver drowsiness recognition, facial expression classification, and human–computer interface technology. This study proposes a novel framework based on the deep learning method to classify eye states in still facial images. The proposed method combines a deep neural network and a deep convolutional neural network to construct a deep integrated neural network for characterizing useful information in the eye region by use of the joint optimization method. A transfer learning strategy is applied to extract effective abstract eye features and improve the classification capability of the proposed model on small sample datasets. Experimental results on the Closed Eyes in the Wild (CEW) and Zhejiang University Eyeblink datasets show that the proposed approach outperforms other state-of-the-art methods. In addition, the effects of transfer learning methods with different pretraining datasets on classification accuracy are investigated with the CEW dataset. A driver drowsiness recognition dataset is constructed and used in an experiment to evaluate the effectiveness of the proposed method in driving environments. Experimental results demonstrate that the proposed method performs more stably and robustly than do other methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Asthana A, Zafeiriou S, Cheng S, Pantic M (2014) Incremental face alignment in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1859–1866

  2. Bhaskar T, Keat FT, Ranganath S, Venkatesh Y (2003) Blink detection and eye tracking for eye localization. In: Proceedings of the IEEE Conference on Convergent Technologies for the Asia-Pacific Region, pp 821–824

  3. Bhimani J, Mi N, Leeser M, Yang Z (2017) FiM: Performance Prediction Model for Parallel Computation in Iterative Data Processing Applications. In: IEEE International Conference on Cloud Computing, published online. https://doi.org/10.1109/CLOUD.2017.53

  4. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  MATH  Google Scholar 

  5. Dehnavi M, Eshghi M (2012) Design and implementation of a real time and train less eye state recognition system. EURASIP J Adv Signal Process 2012(1):30

    Article  Google Scholar 

  6. Dong Y, Zhang Y, Yue J, Hu Z (2016) Comparison of random forest, random ferns and support vector machine for eye state classification. Multimed Tools Appl 75(19):11763–11783

    Article  Google Scholar 

  7. Eddine BD, dos Santos FN, Boulebtateche B et al (2017) EyeLSD a robust approach for eye localization and state detection. J Signal Process Syst. https://doi.org/10.1007/s11265-016-1219-1

  8. Ejbali R, Zaied M (2017) A dyadic multi-resolution deep convolutional neural wavelet network for image classification. Multimed Tools Appl. https://doi.org/10.1007/s11042- 017-4523-2

  9. Flores M, Armingol J, de la Escalera A (2010) Driver drowsiness warning system using visual information for both diurnal and nocturnal illumination conditions. EURASIP J Adv Signal Process 2010(1):438205

    Article  Google Scholar 

  10. Freund Y, Schapire RE (1995) A desicion-theoretic generalization of on-line learning and an application to boosting. In: Proceedings of the European conference on computational learning theory. Springer, pp 23–37

  11. Gao H, Yang Z, Bhimani J, Wang T, Wang J, Sheng B, Mi N (2017) AutoPath: Harnessing Parallel Execution Paths for Efficient Resource Allocation in Multi-stage Big Data Frameworks. In: Proceedings of the 26th International Conference on Computer Communications and Networks

  12. González-Ortega D, Díaz-Pernas F, Antón-Rodríguez M, Martínez-Zarzuela M, Díez-Higuera J (2013) Real-time vision-based eye state detection for driver alertness monitoring. Pattern Anal Applic 16(3):285–306

    Article  MathSciNet  Google Scholar 

  13. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee D-H (2013) Challenges in representation learning: A report on three machine learning contests. In: Proceedings of the International Conference on Neural Information Processing. Springer, pp 117–124

  14. Hassan H, Yaacob S, Radman A, Suandi SA (2016) Eye state detection for driver inattention based on Lucas Kanade optical flow algorithm. In: Proceedings of the 6th IEEE International Conference on Intelligent and Advanced Systems, pp 1–6

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  16. Hearst MA, Dumais ST, Osuna E, Platt J, Scholkopf B (1998) Support vector machines. IEEE Intell Syst Appl 13(4):18–28

    Article  Google Scholar 

  17. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  MATH  Google Scholar 

  18. Hu Q, Zhang R, Zhou Y (2016) Transfer learning for short-term wind speed prediction with deep neural networks. Renew Energy 85:83–95

    Article  Google Scholar 

  19. Huang GB, Ramesh M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07–49, University of Massachusetts, Amherst

  20. Huang J-T, Li J, Yu D, Deng L, Gong Y (2013) Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp 7304–7308

  21. Huang Z, Siniscalchi SM, Lee C-H (2016) A unified approach to transfer learning of deep neural networks with applications to speaker adaptation in automatic speech recognition. Neurocomputing 218:448–459

    Article  Google Scholar 

  22. Jung H, Lee S, Yim J, Park S, Kim J (2015) Joint Finetuning in Deep Neural Networks for Facial Expression Recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2983–2991

  23. Krizhevsky A (2009) Learning Multiple Layers of Features from Tiny Images. MS thesis, Dept Comput Sci, Univ Toronto, Toronto, ON, Canada

  24. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proceedings of the Advances in neural information processing systems, pp 1097–1105

  25. Królak A, Strumiłło P (2012) Eye-blink detection system for human–computer interaction. Univ Access Inf Soc 11(4):409–419

    Article  Google Scholar 

  26. Kurylyak Y, Lamonaca F, Mirabelli G (2012) Detection of the eye blinks for human's fatigue monitoring. In: Proceedings of the IEEE International Symposium on Medical Measurements and Applications Proceedings, pp 1–4

  27. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  28. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  29. Lin X, Cai L, Ji R (2015) An effective eye states detection method based on the projection of the gray interval distribution. In: Proceedings of the IEEE International Conference on Image Processing, pp 1875–1879

  30. Liu Z, Ai H (2008) Automatic eye state recognition and closed-eye photo correction. In: Proceedings of the 19th IEEE International Conference on Pattern Recognition, pp 1–4

  31. Liu A, Li Z, Wang L, Zhao Y (2010) A practical driver fatigue detection algorithm based on eye state. In: Proceedings of the IEEE Asia Pacific Conference on Postgraduate Research in Microelectronics and Electronics, pp 235–238

  32. Liu X, Tan X, Chen S (2012) Eyes closeness detection using appearance based methods. In: Proceedings of the International Conference on Intelligent Information Processing, pp 398–408

  33. Mandal B, Li L, Wang GS, Lin J (2016) Towards Detection of Bus Driver Fatigue Based on Robust Visual Analysis of Eye State. IEEE Trans Intell Transp Syst 99:1–13

    Google Scholar 

  34. Pan G, Sun L, Wu Z, Lao S (2007) Eyeblink-based anti-spoofing in face recognition from a generic webcamera. In: Proceedings of the 11th IEEE International Conference on Computer Vision, pp 1–8

  35. Punitha A, Geetha MK (2015) Driver eye state detection based on Minimum Intensity Projection using Tree Based Classifiers. Intell Syst Technol Appl 384:103–111

    Google Scholar 

  36. Radlak K, Smolka B (2012) A novel approach to the eye movement analysis using a high speed camera. In: Proceedings of the 2nd IEEE International Conference on Advances in Computational Tools for Engineering Applications, pp 145–150

  37. Singh R, Om H (2017) Newborn face recognition using deep convolutional neural network. Multimed Tools Appl, published online. https://doi.org/10.1007/s11042-016-4342-x

  38. Song F, Tan X, Chen S, Zhou Z-H (2013) A literature survey on robust and efficient eye localization in real-life scenarios. Pattern Recogn 46(12):3157–3173

    Article  Google Scholar 

  39. Song F, Tan X, Liu X, Chen S (2014) Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients. Pattern Recogn 47(9):2825–2838

    Article  Google Scholar 

  40. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1–9

  41. Tian Y-l, Kanade T, Cohn JF (2000) Eye-state action unit detection by gabor wavelets. In: Proceedings of the Advances in Multimodal Interfaces. Springer, pp 143–150

  42. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol P-A (2010) Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11:3371–3408

    MathSciNet  MATH  Google Scholar 

  43. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  44. Wu Y-S, Lee T-W, Wu Q-Z, Liu H-S (2010) An eye state recognition method for drowsiness detection. In: Proceedings of the IEEE Vehicular Technology Conference, pp 1–5

  45. Xu Q (2013) A Novel Machine Learning Strategy Based on Two-Dimensional Numerical Models in Financial Engineering. Math Probl Eng 2013:2013

    Google Scholar 

  46. Xu Q, Wu J, Chen Q (2014) A Novel Mobile Personalized Recommended Method Based on Money Flow Model for Stock Exchange. Math Probl Eng 2014:2014

    MathSciNet  Google Scholar 

  47. Xu Q, Wang Z, Wang F, Li J (2017) Thermal comfort research on human CT data modeling. Multimed Tools Appl, published online. https://doi.org/10.1007/s11042-017-4537-9

  48. Yan C, Zhang Y, Dai F, Li L (2013) Highly parallel framework for HEVC motion estimation on many-core platform. In: Proceedings of the IEEE Data Compression Conference, pp 63–72

  49. Yan C, Zhang Y, Xu J, Dai F, Zhang J, Dai Q, Wu F (2014) Efficient parallel framework for HEVC motion estimation on many-core processors. IEEE Trans Circuits Syst Video Technol 24(12):2077–2089

    Article  Google Scholar 

  50. Yan C, Zhang Y, Dai F, Zhang J, Li L, Dai Q (2014) Efficient parallel HEVC intra-prediction on many-core processor. Electron Lett 50(11):805–806

    Article  Google Scholar 

  51. Yan C, Zhang Y, Dai F, Wang X, Li L, Dai Q (2014) Parallel deblocking filter for HEVC on many-core processor. Electron Lett 50(5):367–368

    Article  Google Scholar 

  52. Yan C, Zhang Y, Xu J, Dai F, Li L, Dai Q, Wu F (2014) A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors. IEEE Signal Process Lett 21(5):573–576

    Article  Google Scholar 

  53. Yang HY, Jiang XH, Wang L, Zhang YH (2012) Eye Statement Recognition for Driver Fatigue Detection Based on Gabor Wavelet and HMM. In: Applied Mechanics and Materials. Trans Tech Publ, pp 123–129

  54. Yang Z, Awasthi M, Ghosh M, Mi N (2016) A fresh perspective on total cost of ownership models for flash storage. In: Proceedings of the 8th IEEE International Conference on Cloud Computing Technology and Science, pp 245–252

  55. Yang Z, Tai J, Bhimani J, Wang J, Mi N, Sheng B (2016) GReM: Dynamic SSD resource allocation in virtualized storage systems with heterogeneous IO workloads. In: Proceedings of the 35th IEEE International Performance Computing and Communications Conference, pp 1–8

  56. Yang Z, Wang J, Evans D, Mi N (2016) AutoReplica: Automatic data replica manager in distributed caching and data processing systems. In: Proceedings of the 35th International Performance Computing and Communications Conference, pp 1–6

  57. Yang J, Li J, Liu S (2017) A novel technique applied to the economic investigation of recommender system. Multimed Tools Appl, published online. https://doi.org/10.1007/s11042-017-4752-4

  58. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: Proceedings of the Advances in neural information processing systems, pp 3320–3328

  59. Yu S, Cheng Y, Su S, Cai G, Li S (2016) Stratified pooling based deep convolutional neural networks for human action recognition. Multimed Tools Appl 76(11):13367–13382

    Article  Google Scholar 

  60. Yutian F, Dexuan H, Pingqiang N (2009) A combined eye states identification method for detection of driver fatigue. In: Proceedings of the IET International Communication Conference on Wireless Mobile and Computing, pp 217–220

  61. Zhang W, Li R, Zeng T, Sun Q, Kumar S, Ye J, Ji S (2016) Deep Model Based Transfer and Multi-Task Learning for Biological Image Analysis. IEEE Trans Big Data, published online. https://doi.org/10.1109/tbdata.2016.2573280

  62. Zhao L, Wang Z, Wang X, Qi Y, Liu Q, Zhang G (2016) Human fatigue expression recognition through image-based dynamic multi-information and bimodal deep learning. J Electron Imaging 25(5):053024

    Article  Google Scholar 

  63. Zhou L, Wang H (2011) Open/closed eye recognition by local binary increasing intensity patterns. In: Proceedings of the IEEE Conference on Robotics, Automation and Mechatronics, pp 7–11

Download references

Acknowledgments

This work was supported by the Open Foundation of State Key Laboratory of Automotive Simulation and Control (China, Grant no. 20161105).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Shandong University, Jinan, China

    Lei Zhao, Zengcai Wang, Guoxin Zhang, Yazhou Qi & Xiaojin Wang

  2. Key Laboratory of High-efficiency and Clean Mechanical Manufacture (Shandong University), Ministry of Education, Jinan, China

    Zengcai Wang

Authors
  1. Lei Zhao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Zengcai Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Guoxin Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Yazhou Qi
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Xiaojin Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Zengcai Wang.

Ethics declarations

Conflict of interest

None.

Additional information

In general, convolutional neural network is a type of neural network. However, a neural network is defined as a multi-layer perceptron using back propagation for training in this study to distinguish between the two models.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, L., Wang, Z., Zhang, G. et al. Eye state recognition based on deep integrated neural network and transfer learning. Multimed Tools Appl 77, 19415–19438 (2018). https://doi.org/10.1007/s11042-017-5380-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5380-8

Keywords