skip to main content
10.1145/3379156.3391335acmconferencesArticle/Chapter ViewAbstractPublication PagesetraConference Proceedingsconference-collections
short-paper

Towards Predicting Reading Comprehension From Gaze Behavior

Published: 02 June 2020 Publication History

Abstract

As readers of a language, we all agree to move our eyes in roughly the same way. Yet might there be hidden within this self-similar behavior subtle clues as to how a reader is understanding the material being read? Here we attempt to decode a reader’s eye movements to predict their level of text comprehension and related states. Eye movements were recorded from 95 people reading 4 published SAT passages, each followed by corresponding SAT questions and self-evaluation questionnaires. A sequence of 21 fixation-location (x,y), fixation-duration, and pupil-size features were extracted from the reading behavior and input to two deep networks (CNN/RNN), which were used to predict the reader’s comprehension level and other comprehension-related variables. The best overall comprehension prediction accuracy was 65% (cf. null accuracy = 54%) obtained by CNN. This prediction generalized well to fixations on new passages (64%) from the same readers, but did not generalize to fixations from new readers (41%), implying substantial individual differences in reading behavior. Our work is the first attempt to predict comprehension from fixations using deep networks, where we hope that our large reading dataset and our protocol for evaluation will benefit the development of new methods for predicting reading comprehension by decoding gaze behavior.

References

[1]
Hosam Al-Samarraie, Atef Eldenfria, and Husameddin Dawoud. 2017. The impact of personality traits on users’ information-seeking behavior. Information Processing & Management 53, 1 (2017), 237–247.
[2]
Hosam Al-Samarraie, Samer Muthana Sarsam, Ahmed Ibrahim Alzahrani, and Nasser Alalwan. 2018. Personality and individual differences: the potential of using preferences for visual stimuli to predict the Big Five traits. Cognition, Technology & Work(2018), 1–13.
[3]
Olivier Augereau, Hiroki Fujiyoshi, and Koichi Kise. 2016. Towards an automated estimation of English skill via TOEIC score based on reading analysis. In 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, 1285–1290.
[4]
Ralf Biedert, Jörn Hees, Andreas Dengel, and Georg Buscher. 2012. A robust realtime reading-skimming classifier. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 123–130.
[5]
Jonathan FG Boisvert and Neil DB Bruce. 2016. Predicting task from eye movements: On the importance of spatial distribution, dynamics, and image features. Neurocomputing 207(2016), 653–668.
[6]
Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks 18, 5-6 (2005), 602–610.
[7]
John M Henderson. 2003. Human gaze control during real-world scene perception. Trends in cognitive sciences 7, 11 (2003), 498–504.
[8]
John M Henderson, Svetlana V Shinkareva, Jing Wang, Steven G Luke, and Jenn Olejarczyk. 2013. Predicting cognitive state from eye movements. PloS one 8, 5 (2013), e64937.
[9]
Shoya Ishimaru, Kensuke Hoshika, Kai Kunze, Koichi Kise, and Andreas Dengel. 2017. Towards reading trackers in the wild: detecting reading activities by EOG glasses and deep neural networks. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 704–711.
[10]
Conor Kelton, Zijun Wei, Seoyoung Ahn, Aruna Balasubramanian, Samir Das, Dimitris Samaras, and Gregory Zelinsky. 2019. Reading Detection in Real-time. In In 2019 Symposium on Eye Tracking Research and Applications (ETRA ’19).
[11]
Ya Lou, Yanping Liu, Johanna K Kaakinen, and Xingshan Li. 2017. Using support vector machines to identify literacy skills: Evidence from eye movements. Behavior research methods 49, 3 (2017), 887–895.
[12]
Silvia Makowski, Lena A Jäger, Ahmed Abdelwahab, Niels Landwehr, and Tobias Scheffer. 2018. A Discriminative Model for Identifying Readers and Assessing Text Comprehension from Eye Movements. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 209–225.
[13]
Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research.Psychological bulletin 124, 3 (1998), 372.
[14]
Keith Rayner. 2009. Eye movements and attention in reading, scene perception, and visual search. The quarterly journal of experimental psychology 62, 8(2009), 1457–1506.
[15]
Geoffrey Underwood, Alison Hubbard, and Howard Wilkinson. 1990. Eye fixations predict reading comprehension: The relationships between reading skill, reading speed, and visual inspection. Language and speech 33, 1 (1990), 69–81.
[16]
Zijun Wei, Hossein Adeli, Minh Hoai Nguyen, Greg Zelinsky, and Dimitris Samaras. 2016. Learned Region Sparsity and Diversity Also Predicts Visual Attention. In Advances in Neural Information Processing Systems. 1894–1902.

Cited By

View all
  • (2025)Automatic Classification of Difficulty of Texts From Eye Gaze and Physiological Measures of L2 English SpeakersIEEE Access10.1109/ACCESS.2025.353715613(24555-24575)Online publication date: 2025
  • (2024)A review of machine learning in scanpath analysis for passive gaze-based interactionFrontiers in Artificial Intelligence10.3389/frai.2024.13917457Online publication date: 5-Jun-2024
  • (2024)Using Eye Movement to Determine Whether Closed-Frame Shots Attract Viewers’ AttentionSage Open10.1177/2158244024129062914:4Online publication date: 29-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ETRA '20 Short Papers: ACM Symposium on Eye Tracking Research and Applications
June 2020
305 pages
ISBN:9781450371346
DOI:10.1145/3379156
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 June 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Eye tracking
  2. Machine learning
  3. Reading dataset
  4. Text comprehension prediction

Qualifiers

  • Short-paper
  • Research
  • Refereed limited

Funding Sources

Conference

ETRA '20

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

ETRA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)150
  • Downloads (Last 6 weeks)12
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Automatic Classification of Difficulty of Texts From Eye Gaze and Physiological Measures of L2 English SpeakersIEEE Access10.1109/ACCESS.2025.353715613(24555-24575)Online publication date: 2025
  • (2024)A review of machine learning in scanpath analysis for passive gaze-based interactionFrontiers in Artificial Intelligence10.3389/frai.2024.13917457Online publication date: 5-Jun-2024
  • (2024)Using Eye Movement to Determine Whether Closed-Frame Shots Attract Viewers’ AttentionSage Open10.1177/2158244024129062914:4Online publication date: 29-Oct-2024
  • (2024)A Multimodal Understanding of the Eye-Mind LinkProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3688617(632-636)Online publication date: 4-Nov-2024
  • (2024)Putting the “Brain” Back in the Eye-Mind Link: Aligning Eye Movements and Brain Activations During Naturalistic ReadingProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685745(407-417)Online publication date: 4-Nov-2024
  • (2024)VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze BehaviourProceedings of the ACM on Human-Computer Interaction10.1145/36556138:ETRA(1-18)Online publication date: 28-May-2024
  • (2024)EyeLiveMetrics: Real-time Analysis of Online Reading with Eye TrackingProceedings of the 2024 Symposium on Eye Tracking Research and Applications10.1145/3649902.3656495(1-7)Online publication date: 4-Jun-2024
  • (2024)GazePrompt: Enhancing Low Vision People's Reading Experience with Gaze-Aware AugmentationsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642878(1-17)Online publication date: 11-May-2024
  • (2024)Dual Input Stream Transformer for Vertical Drift Correction in Eye-Tracking Reading DataIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.341193846:12(8715-8726)Online publication date: Dec-2024
  • (2024)Predicting Chinese reading proficiency based on eye movement features and machine learningReading and Writing10.1007/s11145-024-10563-2Online publication date: 12-Jun-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media