short-paper

Tiny convolution, decision tree, and binary neuronal networks for robust and real time pupil outline estimation

Authors:

Enkelejda KasneciAuthors Info & Claims

ETRA '20 Short Papers: ACM Symposium on Eye Tracking Research and Applications

Article No.: 5, Pages 1 - 5

https://doi.org/10.1145/3379156.3391347

Published: 02 June 2020 Publication History

Abstract

In this work, we compare the use of convolution, binary, and decision tree layers in neural networks for the estimation of pupil landmarks. These landmarks are used for the computation of the pupil ellipse and have proven to be effective in previous research. The evaluated structure of the neural networks is the same for all layers and as small as possible to ensure a real-time application. The evaluations include the accuracy of the ellipse determination based on the Jaccard Index and the pupil center. Furthermore, the CPU runtime is considered to make statements about the real-time usability. The trained models are also optimized using pruning to improve the runtime. These optimized nets are also evaluated with respect to the Jaccard index and the accuracy of the pupil center estimation. Link to the framework and models.

References

[1]

Christa J Anderson and John Colombo. 2009. Larger tonic pupil size in young children with autism spectrum disorder. Developmental Psychobiology: The Journal of the International Society for Developmental Psychobiology 51, 2 (2009), 207–211.

[2]

Christian Braunagel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2017. Ready for take-over? A new driver assistance system for an automated classification of driver take-over readiness. IEEE Intelligent Transportation Systems Magazine 9, 4 (2017), 10–22.

[3]

Andreas Bulling. 2016. Pervasive attentive user interfaces. Computer1(2016), 94–98.

[4]

Andreas Bulling and Hans Gellersen. 2010. Toward mobile eye-based human-computer interaction. IEEE Pervasive Computing 9, 4 (2010), 8–12.

Digital Library

[5]

Giovanna Castellano, Anna Maria Fanelli, and Marcello Pelillo. 1997. An iterative pruning algorithm for feedforward neural networks. IEEE transactions on Neural networks 8, 3 (1997), 519–531.

[6]

Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2014. Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024(2014).

[7]

Andrew T Duchowski. 2002. A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments, & Computers 34, 4 (2002), 455–470.

[8]

Andrew T Duchowski, Vinay Shivashankaraiah, Tim Rawls, Anand K Gramopadhye, Brian J Melloy, and Barbara Kanki. 2000. Binocular eye tracking in virtual reality for inspection training. In Proceedings of the 2000 symposium on Eye tracking research & applications. ACM, 89–96.

Digital Library

[9]

Shahram Eivazi, Roman Bednarik, Ville Leinonen, Mikael von und zu Fraunberg, and Juha E Jääskeläinen. 2016. Embedding an eye tracker into a surgical microscope: requirements, design, and implementation. IEEE Sensors Journal 16, 7 (2016), 2070–2078.

[10]

Wolfgang Fuhl, Nora Castner, Lin Zhuang, Markus Holzer, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2018a. MAM: Transfer learning for fully automatic video annotation and specialized detector creation. In Egocentric Perception, Interaction and Computing Workshop (EPIC@ECCV).

[11]

Wolfgang Fuhl, Shahram Eivazi, Benedikt Hosp, Anna Eivazi, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2018b. BORE: boosted-oriented edge optimization for robust, real time remote pupil center detection. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications. ACM, 48.

Digital Library

[12]

Wolfgang Fuhl, David Geisler, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2019a. The applicability of Cycle GANs for pupil and eyelid segmentation, data generation and image refinement. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 0–0.

[13]

Wolfgang Fuhl, David Geisler, Thiago Santini, Tobias Appel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2018c. CBF: circular binary features for robust and real-time pupil center detection. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications. ACM, 8.

Digital Library

[14]

Wolfgang Fuhl, David Geisler, Thiago Santini, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2016a. Evaluation of state-of-the-art pupil detection algorithms on remote eye images. In International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 1716–1725.

Digital Library

[15]

Wolfgang Fuhl and Enkelejda Kasneci. 2019. Learning to validate the quality of detected landmarks. In International Conference of Machine Vision (ICMV). SPIE.

[16]

Wolfgang Fuhl, G. Kasneci, W. Rosenstiel, and Enkelejda Kasneci. 2020. Training Decision Trees as Replacement for Convolution Layers. In Conference on Artificial Intelligence (AAAI). AAAI.

[17]

Wolfgang Fuhl, Thomas Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2015. Excuse: Robust pupil detection in real-world scenarios. In International Conference on Computer Analysis of Images and Patterns. Springer, 39–51.

[18]

Wolfgang Fuhl, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2019b. 500,000 images closer to eyelid and pupil segmentation. In International Conference on Computer Analysis of Images and Patterns. Springer, 336–347.

[19]

Wolfgang Fuhl, Thiago Santini, David Geisler, Thomas Kübler, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2016b. Eyes wide open? eyelid location and eye aperture estimation for pervasive eye tracking in real-world scenarios. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. 1656–1665.

Digital Library

[20]

Wolfgang Fuhl, Thiago Santini, and Enkelejda Kasneci. 2017a. Fast and robust eyelid outline and aperture detection in real-world scenarios. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1089–1097.

[21]

Wolfgang Fuhl, Thiago Santini, Gjergji Kasneci, and Enkelejda Kasneci. 2016c. PupilNet: convolutional neural networks for robust pupil detection. arXiv preprint arXiv:1601.04902(2016).

[22]

Wolfgang Fuhl, Thiago Santini, Gjergji Kasneci, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2017b. PupilNet v2. 0: Convolutional Neural Networks for CPU based real time Robust Pupil Detection. arXiv preprint arXiv:1711.00112(2017).

[23]

Wolfgang Fuhl, Thiago Santini, Carsten Reichert, Daniel Claus, Alois Herkommer, Hamed Bahmani, Katharina Rifai, Siegfried Wahl, and Enkelejda Kasneci. 2016e. Non-intrusive practitioner pupil detection for unmodified microscope oculars. Computers in biology and medicine 79 (2016), 36–44.

[24]

Wolfgang Fuhl, Thiago C Santini, Thomas Kübler, and Enkelejda Kasneci. 2016d. Else: Ellipse selection for robust pupil detection in real-world environments. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications. ACM, 123–130.

Digital Library

[25]

Wolfgang Fuhl, Marc Tonsen, Andreas Bulling, and Enkelejda Kasneci. 2016f. Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art. Machine Vision and Applications 27, 8 (2016), 1275–1288.

Digital Library

[26]

Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Transactions on Graphics (TOG) 31, 6 (2012), 164.

Digital Library

[27]

Antonio Haro, Myron Flickner, and Irfan Essa. 2000. Detecting and tracking eyes by using their physiological properties, dynamics, and appearance. In Computer Vision and Pattern Recognition, Vol. 1. IEEE, 163–168.

[28]

Guillaume Hervet, Katherine Guérard, Sébastien Tremblay, and Mohamed Saber Chtourou. 2011. Is banner blindness genuine? Eye tracking internet text advertising. Applied cognitive psychology 25, 5 (2011), 708–716.

[29]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531(2015).

[30]

Shoya Ishimaru, Kai Kunze, Koichi Kise, Jens Weppner, Andreas Dengel, Paul Lukowicz, and Andreas Bulling. 2014. In the blink of an eye: combining head motion and eye blink frequency for activity recognition with Google Glass. In Proceedings of the 5th augmented human international conference. ACM, 15.

Digital Library

[31]

Amir-Homayoun Javadi, Zahra Hakimi, Morteza Barati, Vincent Walsh, and Lili Tcheang. 2015. SET: a pupil detection method using sinusoidal approximation. Frontiers in neuroengineering 8 (2015), 4.

[32]

Eyyup Karahan, Omer Karti, Aslı Koskderelioglu, Dilek Top Karti, Murat Uyar, Murat Yildirim Kale, and Muhtesem Gedizlioglu. 2017. Pupil cycle time: as indicator of visual pathway dysfunction in multiple sclerosis. Acta Neurologica Belgica 117, 1 (2017), 75–81.

[33]

Enkelejda Kasneci, Katrin Sippel, Kathrin Aehling, Martin Heister, Wolfgang Rosenstiel, Ulrich Schiefer, and Elena Papageorgiou. 2014. Driving with binocular visual field loss? A study on a supervised on-road parcours with simultaneous eye and head tracking. PloS one 9, 2 (2014).

[34]

Thomas Kübler, Shahram Eivazi, and Enkelejda Kasneci. 2015. Automated visual scanpath analysis reveals the expertise level of micro-neurosurgeons. In MICCAI workshop on interventional microscopy. 1–8.

[35]

Xia Liu, Fengliang Xu, and Kikuo Fujimura. 2002. Real-time eye detection and tracking for driver observation under various light conditions. In Intelligent Vehicle Symposium, 2002. IEEE, Vol. 2. IEEE, 344–351.

[36]

Ioan T Oltean, John K Shimmick, and Terrance N Clapham. 2001. Eye tracking device for laser eye surgery using corneal margin detection. US Patent 6,299,307.

[37]

Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics (TOG) 35, 6 (2016), 179.

Digital Library

[38]

Thies Pfeiffer and Patrick Renner. 2014. EyeSee3D: a low-cost approach for analysing mobile 3D eye tracking data using augmented reality technology. In Proceedings of the Symposium on Eye Tracking Research and Applications.

Digital Library

[39]

Eyal M Reingold and Heather Sheridan. 2011. Eye movements and visual expertise in chess and medicine. Oxford handbook on eye movements 528 (2011).

[40]

Thiago Santini, Wolfgang Fuhl, and Enkelejda Kasneci. 2018. PuRe: Robust pupil detection for real-time pervasive eye tracking. Computer Vision and Image Understanding(2018).

[41]

Katrin Sippel, Enkelejda Kasneci, Kathrin Aehling, Martin Heister, Wolfgang Rosenstiel, Ulrich Schiefer, and Elena Papageorgiou. 2014. Binocular glaucomatous visual field loss and its impact on visual exploration-a supermarket study. PloS one 9, 8 (2014).

[42]

Lech Świrski, Andreas Bulling, and Neil Dodgson. 2012. Robust real-time pupil tracking in highly off-axis images. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 173–176.

Digital Library

[43]

Forest Tennant. 1988. The rapid eye test to detect drug abuse. Postgraduate medicine 84, 1 (1988), 108–114.

[44]

Marc Tonsen, Xucong Zhang, Yusuke Sugano, and Andreas Bulling. 2016. Labelled pupils in the wild: a dataset for studying pupil detection in unconstrained environments. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications. ACM, 139–142.

Digital Library

[45]

FJ Vera-Olmos and Norberto Malpica. 2017. Deconvolutional neural network for pupil detection in real-world environments. In International Work-Conference on the Interplay Between Natural and Artificial Computation. Springer, 223–231.

[46]

FJ Vera-Olmos, Esteban Pardo, H Melero, and Norberto Malpica. 2019. DeepEye: Deep convolutional network for pupil detection in real environments. Integrated Computer-Aided Engineering 26, 1 (2019), 85–95.

[47]

Michel Wedel and Rik Pieters. 2008. A review of eye-tracking research in marketing. In Review of marketing research. Emerald Group Publishing Limited, 123–147.

[48]

Yuk-Hoi Yiu, Moustafa Aboulatta, Theresa Raiser, Leoni Ophey, Virginia L Flanagin, Peter zu Eulenburg, and Seyed-Ahmad Ahmadi. 2019. DeepVOG: Open-source Pupil Segmentation and Gaze Estimation in Neuroscience using Deep Learning. Journal of neuroscience methods(2019).

Cited By

Gundler CTemmen MGulberti APötter-Nerger MÜckert F(2024)Improving Eye-Tracking Data Quality: A Framework for Reproducible Evaluation of Detection AlgorithmsSensors10.3390/s2409268824:9(2688)Online publication date: 24-Apr-2024
https://doi.org/10.3390/s24092688
Maquiling VByrne SNiehorster DNyström MKasneci E(2024)Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)Proceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36547047:2(1-16)Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1145/3654704
Chugh SYe JFu YEizenman M(2024)CSA-CNN: A Contrastive Self-Attention Neural Network for Pupil Segmentation in Eye Gaze TrackingProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3653351(1-7)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3653351
Show More Cited By

Tiny convolution, decision tree, and binary neuronal networks for robust and real time pupil outline estimation
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches

Recommendations

CSA-CNN: A Contrastive Self-Attention Neural Network for Pupil Segmentation in Eye Gaze Tracking
ETRA '24: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications

This paper presents a novel Contrastive Self-Attention Convolutional Neural Network (CSA-CNN) model with enhanced Difficulty Aware (DA) loss function to improve the segmentation of pupils in eye images. The incorporation of transformer-style self-...
Pupil vs. Eyelid: Evaluating the Accuracy of Blink Detection in Pupil-based Eye Tracking Devices
Abstract
Spontaneous eye blinks as a means to tap into human cognitive and affective processes have been demonstrated to offer valuable insight in various fields of application, such as human-computer-interaction and driver drowsiness detection. ...
Pistol: Pupil Invisible Supportive Tool in the Wild
Abstract
This paper is an in the wild evaluation of the eye tracking tool Pistol. Pistol supports Pupil Invisible projects and other eye trackers (Dikablis, Emke GmbH, Look, Pupil, and many more) in offline mode. For all eye tracking recordings, Pistol is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ETRA '20 Short Papers: ACM Symposium on Eye Tracking Research and Applications

June 2020

305 pages

ISBN:9781450371346

DOI:10.1145/3379156

Editors:
Andreas Bulling
University of Stuttgart, Germany
,
Anke Huckauf
Ulm University, Germany
,
Eakta Jain
University of Florida, USA
,
Ralph Radach
University of Wuppertal, Germany
,
Daniel Weiskopf
University of Stuttgart, Germany

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

ETRA '20

Sponsor:

ETRA '20: 2020 Symposium on Eye Tracking Research and Applications

June 2 - 5, 2020

Stuttgart, Germany

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

ETRA '25

Sponsor:
sigchi
sigchi

2025 Symposium on Eye Tracking Research and Applications

May 26 - 29, 2025

Tokyo , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
237
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gundler CTemmen MGulberti APötter-Nerger MÜckert F(2024)Improving Eye-Tracking Data Quality: A Framework for Reproducible Evaluation of Detection AlgorithmsSensors10.3390/s2409268824:9(2688)Online publication date: 24-Apr-2024
https://doi.org/10.3390/s24092688
Maquiling VByrne SNiehorster DNyström MKasneci E(2024)Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)Proceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36547047:2(1-16)Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1145/3654704
Chugh SYe JFu YEizenman M(2024)CSA-CNN: A Contrastive Self-Attention Neural Network for Pupil Segmentation in Eye Gaze TrackingProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3653351(1-7)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649902.3653351
Fuhl WWeber DEivazi S(2024)Pistol: Pupil Invisible Supportive Tool in the WildSN Computer Science10.1007/s42979-024-02606-w5:3Online publication date: 21-Feb-2024
https://doi.org/10.1007/s42979-024-02606-w
Moreno-Arjonilla JLópez-Ruiz AJiménez-Pérez JCallejas-Aguilera JJurado J(2024)Eye-tracking on virtual reality: a surveyVirtual Reality10.1007/s10055-023-00903-y28:1Online publication date: 5-Feb-2024
https://dl.acm.org/doi/10.1007/s10055-023-00903-y
Meyer JGering SKasneci E(2023)Static Laser Feedback Interferometry-Based Gaze Estimation for Wearable GlassesIEEE Sensors Journal10.1109/JSEN.2023.325071423:7(7558-7569)Online publication date: 1-Apr-2023
https://doi.org/10.1109/JSEN.2023.3250714
Meyer JSchlebusch TKasneci E(2022)A Highly Integrated Ambient Light Robust Eye-Tracking Sensor for Retinal Projection AR Glasses Based on Laser Feedback InterferometryProceedings of the ACM on Human-Computer Interaction10.1145/35308816:ETRA(1-18)Online publication date: 13-May-2022
https://dl.acm.org/doi/10.1145/3530881
Dai LLiu JJu ZGao Y(2022)Attention-Mechanism-Based Real-Time Gaze Tracking in Natural Scenes With Residual BlocksIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2021.306428014:2(696-707)Online publication date: Jun-2022
https://doi.org/10.1109/TCDS.2021.3064280
Huang KYang QHan YZhang YWang ZWang LWei P(2022)An Easily Compatible Eye-tracking System for Freely-moving Small AnimalsNeuroscience Bulletin10.1007/s12264-022-00834-938:6(661-676)Online publication date: 24-Mar-2022
https://doi.org/10.1007/s12264-022-00834-9
Martinez-Marquez DPingali SPanuwatwanich KStewart RMohamed S(2021)Application of Eye Tracking Technology in Aviation, Maritime, and Construction Industries: A Systematic ReviewSensors10.3390/s2113428921:13(4289)Online publication date: 23-Jun-2021
https://doi.org/10.3390/s21134289
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten