Anuj Harisinghani
Skip Abstract Section
Skip Supplemental Material SectionAbstract
There has been increasing interest in non-invasive predictors of Alzheimer's disease (AD) as an initial screen for this condition. Previously, successful attempts leveraged eye-tracking and language data generated during picture narration and reading tasks. These results were obtained with high-end, expensive eye-trackers. Instead, we explore classification using eye-tracking data collected with a webcam, where our classifiers are built using a deep-learning approach. Our results show that the webcam gaze classifier is not as good as the classifier based on high-end eye-tracking data. However, the webcam-based classifier still beats the majority-class baseline classifier in terms of AU-ROC, indicating that predictive signals can be extracted from webcam gaze tracking. Hence, although our results indicate that there is still a long way to go before webcam gaze tracking can reach practical relevance, they still provide an encouraging proof of concept that this technology should be further explored as an affordable alternative to high-end eye-trackers for the detection of AD.
Supplemental Material
- Baltrusaitis, Tadas, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. "OpenFace 2.0: Facial Behavior Analysis Toolkit." Pp. 59--66 in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).Google Scholar
- Barnes, Deborah E., and Kristine Yaffe. 2011. "The Projected Effect of Risk Factor Reduction on Alzheimer's Disease Prevalence." The Lancet. Neurology 10(9):819--28. doi: 10.1016/S1474--4422(11)70072--2.Google ScholarCross Ref
- Barral, Oswald, Hyeju Jang, Sally Newton-Mason, Sheetal Shajan, Thomas Soroski, Giuseppe Carenini, Cristina Conati, and Thalia Field. 2020. "Non-Invasive Classification of Alzheimer's Disease Using Eye Tracking and Language." Pp. 813--41 in Proceedings of the 5th Machine Learning for Healthcare Conference. PMLR.Google Scholar
- Biondi, Juan, Gerardo Fernandez, Silvia Castro, and Osvaldo Agamennoni. 2018. "Eye-Movement Behavior Identification for AD Diagnosis."Google Scholar
- Bixler, Robert, and Sidney D'Mello. 2016. "Automatic Gaze-Based User-Independent Detection of Mind Wandering during Computerized Reading." User Modeling and User-Adapted Interaction 26(1):33--68. doi: 10.1007/s11257-015--9167--1.Google ScholarCross Ref
- Boote, Bikram, Mansi Agarwal, and Jack Mostow. 2021. "Early Prediction of Children's Disengagement in a Tablet Tutor Using Visual Features." Pp. 98--103 in Artificial Intelligence in Education, Lecture Notes in Computer Science, edited by I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova. Cham: Springer International Publishing.Google Scholar
- Cho, Kyunghyun, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. "On the Properties of Neural Machine Translation: Encoder-Decoder Approaches."Google Scholar
- Chung, Junyoung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling."Google Scholar
- Cordell, Cyndy B., Soo Borson, Malaz Boustani, Joshua Chodosh, David Reuben, Joe Verghese, William Thies, and Leslie B. Fried. 2013. "Alzheimer's Association Recommendations for Operationalizing the Detection of Cognitive Impairment during the Medicare Annual Wellness Visit in a Primary Care Setting." Alzheimer's & Dementia 9(2):141--50. doi: 10.1016/j.jalz.2012.09.011.Google ScholarCross Ref
- Cummings, Louise. 2019. "Describing the Cookie Theft Picture: Sources of Breakdown in Alzheimer's Dementia." Pragmatics and Society 10(2):153--76. doi: 10.1075/ps.17011.cum.Google ScholarCross Ref
- D'Mello, Sidney K., Caitlin Mills, Robert Bixler, and Nigel Bosch. 2017. Zone out No More: Mitigating Mind Wandering during Computerized Reading. International Educational Data Mining Society.Google Scholar
- Fernández, Gerardo, Jochen Laubrock, Pablo Mandolesi, Oscar Colombo, and Osvaldo Agamennoni. 2014. "Registering Eye Movements during Reading in Alzheimer's Disease: Difficulties in Predicting Upcoming Words." Journal of Clinical and Experimental Neuropsychology 36(3):302--16. doi: 10.1080/13803395.2014.892060.Google ScholarCross Ref
- Fraser, Kathleen C., Jed A. Meltzer, and Frank Rudzicz. 2016. "Linguistic Features Identify Alzheimer's Disease in Narrative Speech." Journal of Alzheimer's Disease 49(2):407--22. doi: 10.3233/JAD-150520.Google Scholar
- Futoma, Joseph, Sanjay Hariharan, Katherine Heller, Mark Sendak, Nathan Brajer, Meredith Clement, Armando Bedoya, and Cara O'Brien. 2017. "An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection." Pp. 243--54 in Proceedings of the 2nd Machine Learning for Healthcare Conference. PMLR.Google Scholar
- Garbutt, Siobhan, Alisa Matlin, Joanna Hellmuth, Ana K. Schenk, Julene K. Johnson, Howard Rosen, David Dean, Joel Kramer, John Neuhaus, Bruce L. Miller, Stephen G. Lisberger, and Adam L. Boxer. 2008. "Oculomotor Function in Frontotemporal Lobar Degeneration, Related Disorders and Alzheimer's Disease." Brain 131(5):1268--81. doi: 10.1093/brain/awn047.Google ScholarCross Ref
- Goodglass, Harold, and Edith Kaplan. 1972. The Assessment of Aphasia and Related Disorders. Lea & Febiger.Google Scholar
- Grnarova, Paulina, Florian Schmidt, Stephanie L. Hyland, and Carsten Eickhoff. 2016. "Neural Document Embeddings for Intensive Care Patient Mortality Prediction." doi: 10.48550/arXiv.1612.00467.Google Scholar
- Hutt, Stephen, and Sidney K. D'Mello. 2022. "Evaluating Calibration-Free Webcam-Based Eye Tracking for Gaze-Based User Modeling." Pp. 224--35 in Proceedings of the 2022 International Conference on Multimodal Interaction, ICMI '22. New York, NY, USA: Association for Computing Machinery.Google Scholar
- Jang, Hyeju, Thomas Soroski, Matteo Rizzo, Oswald Barral, Anuj Harisinghani, Sally Newton-Mason, Saffrin Granby, Thiago Monnerat Stutz da Cunha Vasco, Caitlin Lewis, and Pavan Tutt. 2021. "Classification of Alzheimer's Disease Leveraging Multi-Task Machine Learning Analysis of Speech and Eye-Movement Data." Frontiers in Human Neuroscience 512.Google Scholar
- Kar, Anuradha, and Peter Corcoran. 2017. "A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms." IEEE Access 5:16495--519. doi: 10.1109/ACCESS.2017.2735633.Google ScholarCross Ref
- Karlekar, Sweta, Tong Niu, and Mohit Bansal. 2018. "Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models."Google Scholar
- Kong, Weirui, Hyeju Jang, Giuseppe Carenini, and Thalia Field. 2019. "A Neural Model for Predicting Dementia from Language." Pp. 270--86 in Proceedings of the 4th Machine Learning for Healthcare Conference. PMLR.Google Scholar
- Krafka, Kyle, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, and Antonio Torralba. 2016. "Eye Tracking for Everyone." Pp. 2176--84 in.Google Scholar
- Lallé, Sébastien, Cristina Conati, and Giuseppe Carenini. 2016. "Predicting Confusion in Information Visualization from Eye Tracking and Interaction Data." Pp. 2529--35 in IJCAI.Google Scholar
- Li, Shuai, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. 2018. "Independently Recurrent Neural Network (Indrnn): Building a Longer and Deeper Rnn." Pp. 5457--66 in Proceedings of the IEEE conference on computer vision and pattern recognition.Google Scholar
- Lipton, Zachary C., David C. Kale, Charles Elkan, and Randall Wetzel. 2015. "Learning to Diagnose with LSTM Recurrent Neural Networks." doi: 10.48550/arXiv.1511.03677.Google Scholar
- MacAskill, Michael R., and Tim J. Anderson. 2016. "Eye Movements in Neurodegenerative Diseases." Current Opinion in Neurology 29(1):61--68. doi: 10.1097/WCO.0000000000000274.Google ScholarCross Ref
- Molitor, Robert J., Philip C. Ko, and Brandon A. Ally. 2015. "Eye Movements in Alzheimer's Disease." Journal of Alzheimer's Disease": JAD 44(1):1--12. doi: 10.3233/JAD-141173.Google ScholarCross Ref
- Müller, Philipp, Michael Xuelin Huang, Xucong Zhang, and Andreas Bulling. 2018. "Robust Eye Contact Detection in Natural Multi-Person Interactions Using Gaze and Speaking Behaviour." Pp. 1--10 in Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ETRA '18. New York, NY, USA: Association for Computing Machinery.Google Scholar
- Nasreddine, Ziad S., Natalie A. Phillips, Valérie Bédirian, Simon Charbonneau, Victor Whitehead, Isabelle Collin, Jeffrey L. Cummings, and Howard Chertkow. 2005. "The Montreal Cognitive Assessment, MoCA: A Brief Screening Tool For Mild Cognitive Impairment." Journal of the American Geriatrics Society 53(4):695--99. doi: 10.1111/j.1532--5415.2005.53221.x.Google ScholarCross Ref
- Pavisic, Ivanna M., Nicholas C. Firth, Samuel Parsons, David Martinez Rego, Timothy J. Shakespeare, Keir X. X. Yong, Catherine F. Slattery, Ross W. Paterson, Alexander J. M. Foulkes, Kirsty Macpherson, Amelia M. Carton, Daniel C. Alexander, John Shawe-Taylor, Nick C. Fox, Jonathan M. Schott, Sebastian J. Crutch, and Silvia Primativo. 2017. "Eyetracking Metrics in Young Onset Alzheimer's Disease: A Window into Cognitive Visual Functions." Frontiers in Neurology 8.Google Scholar
- Prince, Martin, Dr Renata Bryce, and Dr Cleusa Ferri. n.d. "World Alzheimer Report 2011: The Benefits of Early Diagnosis and Intervention."Google Scholar
- Pusiol, Guido, Andre Esteva, Scott S. Hall, Michael Frank, Arnold Milstein, and Li Fei-Fei. 2016. "Vision-Based Classification of Developmental Disorders Using Eye-Movements." Pp. 317--25 in Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2016, Lecture Notes in Computer Science, edited by S. Ourselin, L. Joskowicz, M. R. Sabuncu, G. Unal, and W. Wells. Cham: Springer International Publishing.Google Scholar
- Sims, Shane D., and Cristina Conati. 2020. "A Neural Architecture for Detecting User Confusion in Eye-Tracking Data." Pp. 15--23 in Proceedings of the 2020 International Conference on Multimodal Interaction, ICMI '20. New York, NY, USA: Association for Computing Machinery.Google Scholar
- Tay, Yi, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. "Efficient Transformers: A Survey." ACM Computing Surveys. doi: 10.1145/3530811.Google ScholarDigital Library
- Tran, Minh, Taylan Sen, Kurtis Haut, Mohammad Rafayet Ali, and Ehsan Hoque. 2022. "Are You Really Looking at Me" A Feature-Extraction Framework for Estimating Interpersonal Eye Gaze From Conventional Video." IEEE Transactions on Affective Computing 13(2):912--25. doi: 10.1109/TAFFC.2020.2979440.Google Scholar
- Trauzettel-Klosinski, Susanne, Klaus Dietz, and the IReST Study Group. 2012. "Standardized Assessment of Reading Performance: The New International Reading Speed Texts IReST." Investigative Ophthalmology & Visual Science 53(9):5452--61. doi: 10.1167/iovs.11--8284.Google ScholarCross Ref
- Tripathi, Subarna, Zachary C. Lipton, Serge Belongie, and Truong Nguyen. 2016. "Context Matters: Refining Object Detection in Video with Recurrent Neural Networks." doi: 10.48550/arXiv.1607.04648.Google Scholar
- Valliappan, Nachiappan, Na Dai, Ethan Steinberg, Junfeng He, Kantwon Rogers, Venky Ramachandran, Pingmei Xu, Mina Shojaeizadeh, Li Guo, Kai Kohlhoff, and Vidhya Navalpakkam. 2020. "Accelerating Eye Movement Research via Accurate and Affordable Smartphone Eye Tracking." Nature Communications 11(1):4553. doi: 10.1038/s41467-020--18360--5.Google ScholarCross Ref
- Yue-Hei Ng, Joe, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. 2015. "Beyond Short Snippets: Deep Networks for Video Classification." Pp. 4694--4702 in.Google Scholar
- Zhang, Xucong, Yusuke Sugano, and Andreas Bulling. 2017. "Everyday Eye Contact Detection Using Unsupervised Gaze Target Discovery." Pp. 193--203 in Proceedings of the 30th annual ACM symposium on user interface software and technology.Google Scholar
Index Terms
- Classification of Alzheimer's using Deep-learning Methods on Webcam-based Gaze Data
Recommendations
Evaluating Calibration-free Webcam-based Eye Tracking for Gaze-based User Modeling
ICMI '22: Proceedings of the 2022 International Conference on Multimodal InteractionEye tracking has been a research tool for decades, providing insights into interactions, usability, and, more recently, gaze-enabled interfaces. Recent work has utilized consumer-grade and webcam-based eye tracking, but is limited by the need to ...
Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing SystemsAn eye-gaze-guided computer interface could enable computer use by the seriously disabled but existing systems cost tens of thousands of dollars or have cumbersome setups. This paper presents a methodology for real-time eye gaze tracking using a ...
Scalable Webcam Eye Tracking by Learning from User Interactions
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing SystemsEye tracking systems are commonly used in a variety of research domains, but cost thousands of dollars. In my thesis I investigate a new approach to enable eye tracking for common webcams. The aim is to provide a natural experience to everyday users ...
Comments