Skip to main content
Springer Nature Link
Log in

500,000 Images Closer to Eyelid and Pupil Segmentation

  • Conference paper
  • First Online:
Computer Analysis of Images and Patterns (CAIP 2019)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11678))

Included in the following conference series:

Abstract

Human gaze behavior is not the only important aspect about eye tracking. The eyelids reveal additional important information; such as fatigue as well as the pupil size holds indications of the workload. The current state-of-the-art datasets focus on challenges in pupil center detection, whereas other aspects, such as the lid closure and pupil size, are neglected. Therefore, we propose a fully convolutional neural network for pupil and eyelid segmentation as well as eyelid landmark and pupil ellipsis regression. The network is jointly trained using the Log loss for segmentation and L1 loss for landmark and ellipsis regression. The application of the proposed network is the offline processing and creation of datasets. Which can be used to train resource-saving and real-time machine learning algorithms such as random forests. In addition, we will provide the worlds largest eye images dataset with more than 500,000 images DOWNLOAD.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Adam, M., Rossant, F., Amiel, F., Mikovikova, B., Ea, T.: Eyelid localization for iris identification. Radioengineering 17(4), 82–85 (2008)

    Google Scholar 

  2. Anas, E.R., Henríquez, P., Matuszewski, B.J.: Online eye status detection in the wild with convolutional neural networks. In: VISIGRAPP (6: VISAPP), pp. 88–95 (2017)

    Google Scholar 

  3. Badrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv preprint arXiv:1511.00561 (2015)

  4. Benitezy, J.T.: Eye-tracking and optokinetic tests: diagnostic significance in peripheral and central vestibular disorders. Laryngoscope 80(6), 834–848 (1970)

    Article  Google Scholar 

  5. Boraston, Z., Blakemore, S.J.: The application of eye-tracking technology in the study of autism. J. Physiol. 581(3), 893–898 (2007)

    Article  Google Scholar 

  6. Braunagel, C., Rosenstiel, W., Kasneci, E.: Ready for take-over? A new driver assistance system for an automated classification of driver take-over readiness. IEEE Intell. Transp. Syst. Mag. 9(4), 10–22 (2017)

    Article  Google Scholar 

  7. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)

    Article  Google Scholar 

  8. Dai, J., He, K., Li, Y., Ren, S., Sun, J.: Instance-sensitive fully convolutional networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 534–549. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_32

    Chapter  Google Scholar 

  9. Daugman, J.: How Iris recognition works. In: The Essential Guide to Image Processing, pp. 715–739. Elsevier (2009)

    Google Scholar 

  10. Dong, W., Qu, P.: Eye state classification based on multi-feature fusion. In: Chinese Control and Decision Conference, CCDC 2009, pp. 231–234. IEEE (2009)

    Google Scholar 

  11. Duchowski, A.T.: A breadth-first survey of eye-tracking applications. Behav. Res. Methods Instrum. Comput. 34(4), 455–470 (2002)

    Article  Google Scholar 

  12. Duchowski, A.T., Shivashankaraiah, V., Rawls, T., Gramopadhye, A.K., Melloy, B.J., Kanki, B.: Binocular eye tracking in virtual reality for inspection training. In: Proceedings of the 2000 Symposium on Eye Tracking Research & Applications, pp. 89–96. ACM (2000)

    Google Scholar 

  13. Eivazi, S., Bednarik, R., Leinonen, V., von und zu Fraunberg, M., Jääskeläinen, J.E.: Embedding an eye tracker into a surgical microscope: requirements, design, and implementation. IEEE Sens. J. 16(7), 2070–2078 (2016)

    Article  Google Scholar 

  14. Eivazi, S., Bednarik, R., Tukiainen, M., von und zu Fraunberg, M., Leinonen, V., Jääskeläinen, J.E.: Gaze behaviour of expert and novice microneurosurgeons differs during observations of tumor removal recordings. In: Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 377–380. ACM (2012)

    Google Scholar 

  15. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The PASCAL visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303 (2010)

    Article  Google Scholar 

  16. Fuhl, W., Santini, T., Geisler, D., Kübler, T., Rosenstiel, W., Kasneci, E.: Eyes wide open? Eyelid location and eye aperture estimation for pervasive eye tracking in real-world scenarios. In: PETMEI, September 2016

    Google Scholar 

  17. Fuhl, W., Castner, N., Zhuang, L., Holzer, M., Rosenstiel, W., Kasneci, E.: MAM: transfer learning for fully automatic video annotation and specialized detector creation. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 375–388. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_23

    Chapter  Google Scholar 

  18. Fuhl, W., Eivazi, S., Hosp, B., Eivazi, A., Rosenstiel, W., Kasneci, E.: BORE: boosted-oriented edge optimization for robust, real time remote pupil center detection. In: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, p. 48. ACM (2018)

    Google Scholar 

  19. Fuhl, W., Geisler, D., Santini, T., Appel, T., Rosenstiel, W., Kasneci, E.: CBF: circular binary features for robust and real-time pupil center detection. In: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, p. 8. ACM (2018)

    Google Scholar 

  20. Fuhl, W., Geisler, D., Santini, T., Rosenstiel, W., Kasneci, E.: Evaluation of state-of-the-art pupil detection algorithms on remote eye images. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, pp. 1716–1725. ACM (2016)

    Google Scholar 

  21. Fuhl, W., Kübler, T., Sippel, K., Rosenstiel, W., Kasneci, E.: ExCuSe: robust pupil detection in real-world scenarios. In: Azzopardi, G., Petkov, N. (eds.) CAIP 2015. LNCS, vol. 9256, pp. 39–51. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23192-1_4

    Chapter  Google Scholar 

  22. Fuhl, W., Santini, T., Geisler, D., Kübler, T., Kasneci, E.: EyeLad: remote eye tracking image labeling tool. In: 12th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), February 2017

    Google Scholar 

  23. Fuhl, W., Santini, T., Kasneci, E.: Fast and robust eyelid outline and aperture detection in real-world scenarios. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1089–1097. IEEE (2017)

    Google Scholar 

  24. Fuhl, W., Santini, T., Kasneci, G., Kasneci, E.: PupilNet: convolutional neural networks for robust pupil detection. arXiv preprint arXiv:1601.04902 (2016)

  25. Fuhl, W., Santini, T.C., Kübler, T., Kasneci, E.: ElSe: ellipse selection for robust pupil detection in real-world environments. In: Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, pp. 123–130. ACM (2016)

    Google Scholar 

  26. Fuhl, W., Tonsen, M., Bulling, A., Kasneci, E.: Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art. Mach. Vis. Appl. 27(8), 1275–1288 (2016)

    Article  Google Scholar 

  27. Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Garcia-Rodriguez, J.: A review on deep learning techniques applied to semantic segmentation. arXiv preprint arXiv:1704.06857 (2017)

  28. Gegenfurtner, A., Lehtinen, E., Säljö, R.: Expertise differences in the comprehension of visualizations: a meta-analysis of eye-tracking research in professional domains. Educ. Psychol. Rev. 23(4), 523–552 (2011)

    Article  Google Scholar 

  29. Gilzenrat, M.S., Nieuwenhuis, S., Jepma, M., Cohen, J.D.: Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cogn. Affect. Behav. Neurosci. 10(2), 252–269 (2010)

    Article  Google Scholar 

  30. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  31. Guenter, B., Finch, M., Drucker, S., Tan, D., Snyder, J.: Foveated 3D graphics. ACM Trans. Graph. (TOG) 31(6), 164 (2012)

    Article  Google Scholar 

  32. Haro, A., Flickner, M., Essa, I.: Detecting and tracking eyes by using their physiological properties, dynamics, and appearance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 163–168. IEEE (2000)

    Google Scholar 

  33. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988. IEEE (2017)

    Google Scholar 

  34. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  35. Holzman, P.S., Proctor, L.R., Levy, D.L., Yasillo, N.J., Meltzer, H.Y., Hurt, S.W.: Eye-tracking dysfunctions in schizophrenic patients and their relatives. Arch. Gen. Psychiatry 31(2), 143–151 (1974)

    Article  Google Scholar 

  36. Javadi, A.H., Hakimi, Z., Barati, M., Walsh, V., Tcheang, L.: SET: a pupil detection method using sinusoidal approximation. Front. Neuroeng. 8, 4 (2015)

    Article  Google Scholar 

  37. Kasneci, E., et al.: Driving with binocular visual field loss? A study on a supervised on-road parcours with simultaneous eye and head tracking. PLoS ONE 9(2), e87470 (2014)

    Article  Google Scholar 

  38. Kassner, M., Patera, W., Bulling, A.: Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, pp. 1151–1160. ACM (2014)

    Google Scholar 

  39. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)

    Google Scholar 

  40. Krumpe, T., Scharinger, C., Gerjets, P., Rosenstiel, W., Spüler, M.: Disentangeling working memory load—finding inhibition and updating components in EEG data. In: Proceedings of the 6th International Brain-Computer Interface Meeting: BCI Past, Present, and Future, p. 174 (2016)

    Google Scholar 

  41. Lappi, O.: Eye movements in the wild: oculomotor control, gaze behavior & frames of reference. Neurosci. Biobehav. Rev. 69, 49–68 (2016)

    Article  Google Scholar 

  42. LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)

    Article  Google Scholar 

  43. Lee, Y., Micheals, R.J., Filliben, J.J., Phillips, P.J.: VASIR: an open-source research platform for advanced iris recognition technologies. J. Res. Nat. Inst. Stand. Technol. 118, 218 (2013)

    Article  Google Scholar 

  44. Liu, X., Xu, F., Fujimura, K.: Real-time eye detection and tracking for driver observation under various light conditions. In: IEEE Intelligent Vehicle Symposium, vol. 2, pp. 344–351. IEEE (2002)

    Google Scholar 

  45. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

    Google Scholar 

  46. Marshall, S.P.: Identifying cognitive state from eye metrics. Aviat. Space Environ. Med. 78(5), B165–B175 (2007)

    Google Scholar 

  47. Matsushita, M.: Iris identification system and Iris identification method, US Patent 5,901,238, 4 May 1999

    Google Scholar 

  48. Palinko, O., Kun, A.L., Shyrokov, A., Heeman, P.: Estimating cognitive load using remote eye tracking in a driving simulator. In: Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications, pp. 141–144. ACM (2010)

    Google Scholar 

  49. Park, S., Zhang, X., Bulling, A., Hilliges, O.: Learning to find eye region landmarks for remote gaze estimation in unconstrained settings. In: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, p. 21. ACM (2018)

    Google Scholar 

  50. Patney, A., et al.: Towards foveated rendering for gaze-tracked virtual reality. ACM Trans. Graph. (TOG) 35(6), 179 (2016)

    Article  MathSciNet  Google Scholar 

  51. Pinheiro, P.O., Lin, T.-Y., Collobert, R., Dollár, P.: Learning to refine object segments. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 75–91. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_5

    Chapter  Google Scholar 

  52. Prasad, D.K., Leung, M.K., Quek, C.: ElliFit: an unconstrained, non-iterative, least squares based geometric ellipse fitting method. Pattern Recogn. 46(5), 1449–1465 (2013)

    Article  Google Scholar 

  53. Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 FPS via regressing local binary features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1692 (2014)

    Google Scholar 

  54. Santini, T., Fuhl, W., Kasneci, E.: PuRe: robust pupil detection for real-time pervasive eye tracking. Comput. Vis. Image Underst. 170, 40–50 (2018)

    Article  Google Scholar 

  55. Suzuki, M., Yamamoto, N., Yamamoto, O., Nakano, T., Yamamoto, S.: Measurement of driver’s consciousness by image processing-a method for presuming driver’s drowsiness by eye-blinks coping with individual differences. In: SMC, vol. 4, pp. 2891–2896. IEEE (2006)

    Google Scholar 

  56. Świrski, L., Bulling, A., Dodgson, N.: Robust real-time pupil tracking in highly off-axis images. In: Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 173–176. ACM (2012)

    Google Scholar 

  57. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, Inception-ResNet and the impact of residual connections on learning. In: AAAI, vol. 4, p. 12 (2017)

    Google Scholar 

  58. Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  59. Tonsen, M., Zhang, X., Sugano, Y., Bulling, A.: Labelled pupils in the wild: a dataset for studying pupil detection in unconstrained environments. In: Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, pp. 139–142. ACM (2016)

    Google Scholar 

  60. Vera-Olmos, F.J., Malpica, N.: Deconvolutional neural network for pupil detection in real-world environments. In: Ferrández Vicente, J.M., Álvarez-Sánchez, J.R., de la Paz López, F., Toledo Moreo, J., Adeli, H. (eds.) IWINAC 2017. LNCS, vol. 10338, pp. 223–231. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59773-7_23

    Chapter  Google Scholar 

  61. Wan, R., Shi, B., Duan, L.Y., Tan, A.H., Kot, A.C.: Benchmarking single-image reflection removal algorithms. In: Proceedings of ICCV (2017)

    Google Scholar 

  62. Wildes, R.P.: Iris recognition: an emerging biometric technology. Proc. IEEE 85(9), 1348–1363 (1997)

    Article  Google Scholar 

  63. Yang, F., Yu, X., Huang, J., Yang, P., Metaxas, D.: Robust eyelid tracking for fatigue detection. In: ICIP, pp. 1829–1832, September 2012

    Google Scholar 

  64. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  65. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint (2017)

    Google Scholar 

Download references

Acknowledgments

Work of the authors is supported by the Institutional Strategy of the University of Tübingen (Deutsche Forschungsgemeinschaft, ZUK 63). This research was supported by an IBM Shared University Research Grant including an IBM PowerAI environment. We especially thank our partners Benedikt Rombach, Martin Mähler and Hildegard Gerhardy from IBM for their expertise and support.

Author information

Authors and Affiliations

  1. Eberhard Karls University Tübingen, 72076, Tübingen, Germany

    Wolfgang Fuhl, Wolfgang Rosenstiel & Enkelejda Kasneci

Authors
  1. Wolfgang Fuhl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wolfgang Rosenstiel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Enkelejda Kasneci
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wolfgang Fuhl .

Editor information

Editors and Affiliations

  1. Department of Computer and Electrical Engineering and Applied Mathematics, University of Salerno, Fisciano (SA), Italy

    Mario Vento

  2. Department of Computer and Electrical Engineering and Applied Mathematics, University of Salerno, Fisciano (SA), Italy

    Gennaro Percannella

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fuhl, W., Rosenstiel, W., Kasneci, E. (2019). 500,000 Images Closer to Eyelid and Pupil Segmentation. In: Vento, M., Percannella, G. (eds) Computer Analysis of Images and Patterns. CAIP 2019. Lecture Notes in Computer Science(), vol 11678. Springer, Cham. https://doi.org/10.1007/978-3-030-29888-3_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-29888-3_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-29887-6

  • Online ISBN: 978-3-030-29888-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics