Skip to main content
Log in

An AVOA-LSTM with MRCNN for segmenting and classifying the sunglass image-based eye region identification

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The recognition of the eye region from images is a challenging task, particularly when dealing with dark or thick sunglasses that cause reflections and interfere with accurate identification. To address this issue, a novel system called AVOA-MRCNN-OLSTM has been proposed. This system combines Optimization-driven Long Short-Term Memory (LSTM) with Mask RCNN to achieve precise eye recognition even in the presence of eyeglass frame interference. A mean histogram equalization approach is used in the system's first stage to eliminate noise, which improves the image quality. The system then uses Mask RCNN for segmentation and localization. A potent deep learning model called Mask RCNN can precisely recognize and isolate particular items inside an image. It is used in this instance to identify and divide the eye region. The AVOA-MRCNN-OLSTM framework makes use of LSTM, a recurrent neural network variety that can retain patterns for longer periods. It can efficiently acquire and use temporal information to increase eye recognition accuracy by integrating LSTM into the system. The proposed AVOA-MRCNN-OLSTM system's effectiveness is shown by experimental findings. It outperforms the performance of existing algorithms, achieving a remarkable accuracy of 99% in just 0.02 seconds of computing time. The potential uses of this development include biometric identity, surveillance systems, and human-computer interfaces, all of which need precise eye recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Yu M, Zheng H, Peng Z, Dong J, Du H (2020) Facial expression recognition based on a multi-task global-local network. Pattern Recog Lett 131:166–171

    Article  Google Scholar 

  2. Turan C, Lam KM (2018) Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study. J Vis Commun Image Represent 55:331–341

    Article  Google Scholar 

  3. Gou C, Wu Y, Wang K, Wang K, Wang FY, Ji Q (2017) A joint cascaded framework for simultaneous eye detection and eye state estimation. Pattern Recog 67:23–31

    Article  Google Scholar 

  4. Yiu YH, Aboulatta M, Raiser T, Ophey L, Flanagin VL, Zu Eulenburg P, Ahmadi SA (2019) DeepVOG: Open-source pupil segmentation and gaze estimation in neuroscience using deep learning. J Neurosci Methods 324:108307

    Article  Google Scholar 

  5. Shi Y, Zhang Z, Huang K, Ma W, Tu S (2020) Human-computer interaction based on face feature localization. J Vis Commun Image Represent 70:102740

    Article  Google Scholar 

  6. Fuhl W, Tonsen M, Bulling A, Kasneci E (2016) Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art. Machine Vis Appl 27(8):1275–1288

    Article  Google Scholar 

  7. Akhtar Z, Rattani A (2017) A face in any form: new challenges and opportunities for face recognition technology. Computer 50(4):80–90

    Article  Google Scholar 

  8. Pires R, Avila S, Wainer J, Valle E, Abramoff MD, Rocha A (2019) A data-driven approach to referable diabetic retinopathy detection. Artificial intell Med 96:93–106

    Article  Google Scholar 

  9. Funes-Mora KA, Odobez JM (2016) Gaze estimation in the 3d space using rgb-d sensors. Int J Comput Vis 118(2):194–216

    Article  MathSciNet  Google Scholar 

  10. Lu F, Sugano Y, Okabe T, Sato Y (2015) Gaze estimation from eye appearance: A head pose-free method via eye image synthesis. IEEE Trans Image Process 24(11):3680–3693

    Article  MathSciNet  Google Scholar 

  11. Cristina S, Camilleri KP (2018) Unobtrusive and pervasive video-based eye-gaze tracking. Image Vision Comput 74:21–40

    Article  Google Scholar 

  12. Skodras E, Fakotakis N (2015) Precise localization of eye centers in low resolution color images. Image Vis Comput 36:51–60

    Article  Google Scholar 

  13. Yang S, Tan J, Chen B (2022) Robust spike-based continual meta-learning improved by restricted minimum error entropy criterion. Entropy 24(4):455

    Article  MathSciNet  Google Scholar 

  14. Yang S, Linares-Barranco B, Chen B (2022) Heterogeneous ensemble-based spike-driven few-shot online learning. Front Neurosci, 16

  15. Yang S, Gao T, Wang J, Deng B, Azghadi MR, Lei T, Linares-Barranco B (2022) SAM: a unified self-adaptive multicompartmental spiking neuron model for learning with working memory. Front Neurosci, 16

  16. Vu HN, Nguyen MH, Pham C (2022) Masked face recognition with convolutional neural networks and local binary patterns. Appl Intell 52(5):5497–5512

    Article  Google Scholar 

  17. Skodras E, Kanas VG, Fakotakis N (2015) On visual gaze tracking based on a single low cost camera. Signal Process: Image Commun 36:29–42

    Google Scholar 

  18. Chellappa R, Chen JC, Ranjan R, Sankaranarayanan S, Kumar A, Patel VM, Castillo CD (2016) Towards the design of an end-to-end automated system for image and video-based recognition. In 2016 Information Theory and Applications Workshop (ITA) (pp. 1-7). IEEE

  19. Jamshidi S, Azmi R, Sharghi M, Soryani M (2021) Hierarchical deep neural networks to detect driver drowsiness. Multimed Tools Appl 80(10):16045–16058

    Article  Google Scholar 

  20. Priya GN, Wahida Banu RSD (2014) Occlusion invariant face recognition using mean based weight matrix and support vector machine. Sadhana 39(2):303–315

    Article  Google Scholar 

  21. Savaş BK, Becerikli Y (2020) Real time driver fatigue detection system based on multi-task ConNN. Ieee Access 8:12491–12498

    Article  Google Scholar 

  22. Kuo HF, Deng BS, Fang JY (2016) Fabrication of pn Junction With an n-Type Silicon Nanoparticle Layer by Using Infrared Fiber Laser Illumination. IEEE Access 4:6225–6230

    Article  Google Scholar 

  23. Egger B, Schönborn S, Schneider A, Kortylewski A, Morel-Forster A, Blumer C, Vetter T (2018) Occlusion-aware 3d morphable models and an illumination prior for face image analysis. Int J Comput Vis 126(12):1269–1287

    Article  Google Scholar 

  24. Hernandez-Matamoros A, Bonarini A, Escamilla-Hernandez E, Nakano-Miyatake M, Perez-Meana H (2016) Facial expression recognition with automatic segmentation of face regions using a fuzzy based classification approach. Knowledge-Based Syst 110:1–14

    Article  Google Scholar 

  25. Cyganek B, Gruszczyński S (2014) Hybrid computer vision system for drivers' eye recognition and fatigue monitoring. Neurocomputing 126:78–94

    Article  Google Scholar 

  26. Anitha C (2019) Detection and analysis of drowsiness in human beings using multimodal signals. In: Digital Business. Springer, Cham, pp 157–174

    Chapter  Google Scholar 

  27. Myllyneva A, Hietanen JK (2015) There is more to eye contact than meets the eye. Cognition 134:100–109

    Article  Google Scholar 

  28. Ibrahim LF, Abulkhair M, AlShomrani AD, Al-Garni M, Al-Mutiry A, Al-Gamdi F, Kalenen R (2014) Using Haar classifiers to detect driver fatigue and provide alerts. Multimed Tools Appl 71(3):1857–1877

    Article  Google Scholar 

Download references

Data availability statement

All the data is collected from the simulation reports of the software and tools used by the authors. Authors are working on implementing the same using real world data with appropriate permissions.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Divya A. K.

Ethics declarations

Conflicts of interest

The authors declare that we have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

K, D.A., Keshaveni N An AVOA-LSTM with MRCNN for segmenting and classifying the sunglass image-based eye region identification. Multimed Tools Appl 83, 35073–35095 (2024). https://doi.org/10.1007/s11042-023-16800-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16800-0

Keywords