Skip to main content
SpringerLink
Log in

Eye pupil localization algorithm using convolutional neural networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Eye pupil localization is one of the indispensable technologies in various computer vision applications such as virtual reality and augmented reality. In general, the algorithm consists of finding the approximate eye region and finding the pupil position by extracting the semantic feature from each eye region. However, the performance of the eye pupil location is affected not only by illumination and image resolution but also by glasses wear. Therefore, this paper proposes an eye pupil localization algorithm which is robust against the above disturbance conditions and provides high accuracy. First, a face is detected from an input image and it is determined whether to wear glasses using the detected face. If glasses are present, the glasses are removed to find the correct eye region. Then, facial landmarks are extracted, and eye regions are detected based on facial landmarks. Next, the pupil region is segmented using fully convolutional networks. Finally, the position of the segmented pupil is calculated. Experimental results show that the proposed algorithm outperforms the state-of-the-art algorithms for public databases such as BioID and GI4E by up to 3.44% 0.5%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Abbasi M, Khosravi M (2019) A robust and accurate particle filter-based pupil detection method for big datasets of eye video. J Grid Comput 1–21

  2. Araujo GM, Ribeiro FM, Silva EA, Goldenstein SK (2014) Fast eye localization without a face model using inner product detectors, IEEE International Conference on Image Processing (ICIP), pp. 1366–1370

  3. Baek S, Choi K, Ma C, Kim Y, Ko S (2013) Eyeball model-based iris center localization for visible image-based eye-gaze tracking systems. IEEE Trans Consum Electron 59:415–421

    Article  Google Scholar 

  4. Baldi P, Hornik K (1989) Neural networks and principal component analysis: learning from examples without local minima. Neural Netw 2:53–58

    Article  Google Scholar 

  5. Cai H, Liu B, Ju Z, Thill S, Belpaeme T, Vanderborght B, Liu H (2018) Accurate eye center localization via hierarchical adaptive convolution, British Machine Vision Conference

  6. George A, Routray A (2016) Fast and accurate algorithm for eye localisation for gaze tracking in low-resolution images. IET Comput Vis 10:660–669

    Article  Google Scholar 

  7. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets, Adv Neural Inf Proces Syst, pp. 2672–2680

  8. Gou C, Wu Y, Wang K, Wang K, Wang F, Ji Q (2017) A joint cascaded framework for simultaneous eye detection and eye state estimation. Pattern Recogn 67:23–31

    Article  Google Scholar 

  9. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition, Proc IEEE Conf Comput Vis Pattern Recognit, pp. 770–778

  10. Isola P, Zhu J, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976

  11. O. Jesorsky, K.J. Kirchberg and R.W. Frischholz (2001) Robust face detection using the hausdorff distance, International conference on audio-and video-based biometric person authentication, pp. 90–95

  12. Kassner M, Patera W, Bulling A (2014) Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction, Proceedings of the ACM international joint conference on pervasive and ubiquitous computing: Adjunct publication, pp. 1151–1160

  13. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees, Proc IEEE Conf Comput Vis Pattern Recognit, pp. 1867–1874

  14. Levinshtein A, Phung E, Aarabi P (2018) Hybrid eye center localization using cascaded regression and hand-crafted model fitting. Image Vis Comput 71:17–24

    Article  Google Scholar 

  15. Li B, Fu H (2018) Real time eye detector with cascaded convolutional neural networks, Applied Computational Intelligence and Soft Computing, 2018

  16. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C, Berg AC (2016) Ssd: single shot multibox detector, European conference on computer vision, pp. 21–37

  17. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. Proc IEEE Conf Comput Vis Pattern Recognit:3431–3440

  18. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation, International Conference on Medical image computing and computer-assisted intervention, pp. 234–241

  19. Świrski L, Bulling A, Dodgson N (2012) Robust real-time pupil tracking in highly off-axis images, Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 173–176

  20. Timm F, Barth E (2011) Accurate eye Centre localisation by means of gradients. Visapp 11:125–130

    Google Scholar 

  21. Tonsen M, Zhang X, Sugano Y, Bulling A (2016) Labelled pupils in the wild, Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, pp. 139–142

  22. Valenti R, Gevers T (2012) Accurate eye center location through invariant isocentric patterns. IEEE Trans Pattern Anal Mach Intell 34:1785–1798

    Article  Google Scholar 

  23. Villanueva A, Ponz V, Sesma-Sanchez L, Ariz M, Porta S, Cabeza R (2013) Hybrid method based on topography for robust detection of iris center and eye corners. ACM Transact Multimed Comput Commun Appl (TOMM) 9:25

    Google Scholar 

  24. Yifan X, Hui Y, Yue F (2019) Accurate and robust eye center localization via fully convolutional networks. IEEE/CAA J Automatic Sin 6:1127–1138

    Article  Google Scholar 

  25. Zhu J, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks, IEEE International Conference on Computer Vision (ICCV)

Download references

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Inha University, Inha-ro 100, Michuhol-gu, Incheon, 22212, Republic of Korea

    Jun Ho Choi, Kang Il Lee & Byung Cheol Song

Authors
  1. Jun Ho Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kang Il Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Byung Cheol Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Byung Cheol Song.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Choi, J.H., Lee, K.I. & Song, B.C. Eye pupil localization algorithm using convolutional neural networks. Multimed Tools Appl 79, 32563–32574 (2020). https://doi.org/10.1007/s11042-020-09711-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09711-x

Keywords

Search

Navigation