Keep Eyes on the Sentence: An Interactive Sentence Simplification System for English Learners Based on Eye Tracking and Large Language Models
Article No.: 211, Pages 1 - 7
Abstract
Language learners should read challenging texts regularly. However, using dictionaries or search engines to look up difficult expressions can be time-consuming and distracting. To address this, we have developed a system combining eye tracking with Large Language Models (LLMs) to simplify sentences automatically, allowing learners to focus on the content. The system incorporates user-tailored models that estimate users’ comprehension of sentences using gaze data and sentence information. The system also features user-triggered simplification, resulting from iterative design improvements. We conducted a user study with 17 English learners where they read English text using either our system or a baseline involving online dictionaries and search engines. Our system significantly improved both reading speed and comprehension, especially for complex sentences. The gaze-based simplification improved concentration on the content, allowing for an interruption-free reading experience. It could assist in daily reading practice, particularly for extensive reading focused on large volumes of text at a rapid pace.
Supplemental Material
- Download
- 24.35 MB
- Transcript
References
[1]
David W. Aha and Richard L. Bankert. 1996. A Comparative Evaluation of Sequential Feature Selection Algorithms. Springer New York, New York, NY, 199–206. https://doi.org/10.1007/978-1-4612-2404-4_19
[2]
Olivier Augereau, Kai Kunze, Hiroki Fujiyoshi, and Koichi Kise. 2016. Estimation of English Skill with a Mobile Eye Tracker. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 1777–1781. https://doi.org/10.1145/2968219.2968275
[3]
Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018. Sequence Classification with Human Attention. In Proceedings of the 22nd Conference on Computational Natural Language Learning. Association for Computational Linguistics, Brussels, Belgium, 302–312. https://doi.org/10.18653/v1/K18-1030
[4]
R. Biedert, G. Buscher, and A. Dengel. 2010. The eyeBook – Using Eye Tracking to Enhance the Reading Experience. Informatik Spektrum 33 (2010), 272–281. https://doi.org/10.1007/s00287-009-0381-2
[5]
Ralf Biedert, Georg Buscher, Sven Schwarz, Jörn Hees, and Andreas Dengel. 2010. Text 2.0. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI EA ’10). Association for Computing Machinery, New York, NY, USA, 4003–4008. https://doi.org/10.1145/1753846.1754093
[6]
Marc Brysbaert. 2015. SUBTLEX_US Word Frequency Database. https://osf.io/djpqz.
[7]
Charles Clifton, Adrian Staub, and Keith Rayner. 2007. Chapter 15 - Eye movements in reading words and sentences. In Eye Movements, Roger P.G. Van Gompel, Martin H. Fischer, Wayne S. Murray, and Robin L. Hill (Eds.). Elsevier, Oxford, 341–371. https://doi.org/10.1016/B978-008044980-7/50017-3
[8]
Kevyn Collins-Thompson. 2014. Computational assessment of text readability: A survey of current and future research. ITL - International Journal of Applied Linguistics 165 (12 2014), 97–135. https://doi.org/10.1075/itl.165.2.01col
[9]
Jiexin Ding, Bowen Zhao, Yuqi Huang, Yuntao Wang, and Yuanchun Shi. 2023. GazeReader: Detecting Unknown Word Using Webcam for English as a Second Language (ESL) Learners. arxiv:2303.10443 [cs.HC]
[10]
Yutao Feng, Jipeng Qiang, Yun Li, Yunhao Yuan, and Yi Zhu. 2023. Sentence Simplification via Large Language Models. arxiv:2302.11957 [cs.CL]
[11]
Katsuya Fujii and Jun Rekimoto. 2019. SubMe: An Interactive Subtitle System with English Skill Estimation Using Eye Tracking. In Proceedings of the 10th Augmented Human International Conference 2019 (Reims, France) (AH2019). Association for Computing Machinery, New York, NY, USA, Article 23, 9 pages. https://doi.org/10.1145/3311823.3311865
[12]
Ana Valeria González-Garduño and Anders Søgaard. 2017. Using Gaze to Predict Text Readability. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, Copenhagen, Denmark, 438–443. https://doi.org/10.18653/v1/W17-5050
[13]
Ana González-Garduño and Anders Søgaard. 2018. Learning to Predict Readability Using Eye-Movement Data From Natives and Learners. Proceedings of the AAAI Conference on Artificial Intelligence 32, 1 (Apr. 2018). https://doi.org/10.1609/aaai.v32i1.11978
[14]
Taichi Higasa, Keitaro Tanaka, Qi Feng, and Shigeo Morishima. 2023 (to appear). Gaze-Driven Sentence Simplification for Language Learners: Enhancing Comprehension and Readability. In International Conference on Multimodal Interaction (ICMI ’23 Companion). ACM, Paris, France, 5. https://doi.org/10.1145/3610661.3616177
[15]
Rui Hiraoka, Hiroki Tanaka, Sakriani Sakti, Graham Neubig, and Satoshi Nakamura. 2016. Personalized Unknown Word Detection in Non-Native Language Reading Using Eye Gaze. In Proceedings of the 18th ACM International Conference on Multimodal Interaction (Tokyo, Japan) (ICMI ’16). Association for Computing Machinery, New York, NY, USA, 66–70. https://doi.org/10.1145/2993148.2993167
[16]
Aulikki Hyrskykari, Päivi Majaranta, Antti Aaltonen, and Kari-Jouko Räihä. 2000. Design Issues of IDICT: A Gaze-Assisted Translation Aid. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (Palm Beach Gardens, Florida, USA) (ETRA ’00). Association for Computing Machinery, New York, NY, USA, 9–14. https://doi.org/10.1145/355017.355019
[17]
Joseph Marvin Imperial. 2021. BERT Embeddings for Automatic Readability Assessment. arxiv:2106.07935 [cs.CL]
[18]
Syeda Iqbal. 2017. The Impact of Extensive Reading on Learning and Increasing Vocabulary at Elementary Level. Studies in English Language Teaching 5 (07 2017), 481. https://doi.org/10.22158/selt.v5n3p481
[19]
Shoya Ishimaru, Syed Saqib Bukhari, Carina Heisel, Nicolas Großmann, Pascal Klein, Jochen Kuhn, and Andreas Dengel. 2018. Augmented Learning on Anticipating Textbooks with Eye Tracking. Springer Fachmedien Wiesbaden, Wiesbaden, 387–398. https://doi.org/10.1007/978-3-658-19567-0_23
[20]
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2021. Neural CRF Model for Sentence Alignment in Text Simplification. arxiv:2005.02324 [cs.CL]
[21]
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical Report. Institute for Simulation and Training, University of Central Florida.
[22]
Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods 44, 4 (2012), 978–990. https://doi.org/10.3758/s13428-012-0210-4
[23]
Shazia Maqsood, Abdul Shahid, Muhammad Afzal, Muhammad Roman, Zahid Khan, Zubair Nawaz, and Muhammad Aziz. 2022. Assessing English language sentences readability using machine learning models. PeerJ Computer Science 7 (01 2022), e818. https://doi.org/10.7717/peerj-cs.818
[24]
Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2021. MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases. arxiv:2005.00352 [cs.CL]
[25]
Eugene W. Myers. 1986. An O(ND) difference algorithm and its variations. Algorithmica 1 (1986), 251–266. https://doi.org/10.1007/BF01840446
[26]
Alireza Niazifar and Gholamreza Shakibaei. 2019. Effects of different text difficulty levels on Iranian EFL learners’ foreign language Reading motivation and Reading comprehension. Asian J. Second. Foreign. Lang. Educ. 4 (2019), 1–18. Issue 7. https://doi.org/10.1186/s40862-019-0070-x
[27]
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2019. CatBoost: unbiased boosting with categorical features. arxiv:1706.09516 [cs.LG]
[28]
Keith Rayner. 1998. Eye Movements in Reading and Information Processing: 20 Years of Research. Psychological Bulletin 124, 3 (November 1998), 372–422. https://doi.org/10.1037/0033-2909.124.3.372
[29]
Dario D. Salvucci and Joseph H. Goldberg. 2000. Identifying Fixations and Saccades in Eye-Tracking Protocols. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (Palm Beach Gardens, Florida, USA) (ETRA ’00). Association for Computing Machinery, New York, NY, USA, 71–78. https://doi.org/10.1145/355017.355028
[30]
RJ Senter and Edgar A Smith. 1967. Automated readability index. Technical Report. Technical report, DTIC document.
[31]
Garain Utpal, Pandit Onkar, Augereau Olivier, Okoso Ayano, and Kise Koichi. 2017. Identification of Reader Specific Difficult Words by Analyzing Eye Gaze and Document Content. 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) 01 (11 2017), 1346–1351. https://doi.org/10.1109/icdar.2017.221
Index Terms
- Keep Eyes on the Sentence: An Interactive Sentence Simplification System for English Learners Based on Eye Tracking and Large Language Models
Recommendations
Gaze-Driven Sentence Simplification for Language Learners: Enhancing Comprehension and Readability
ICMI '23 Companion: Companion Publication of the 25th International Conference on Multimodal InteractionLanguage learners should regularly engage in reading challenging materials as part of their study routine. Nevertheless, constantly referring to dictionaries is time-consuming and distracting. This paper presents a novel gaze-driven sentence ...
Combining sentence similarities measures to identify paraphrases
It proposes a new paraphrase identification system based on lexical, syntactic, semantic analysis.It uses different machine learning algorithms to classify the paraphrase.The measure was evaluated using state-of-art dataset: Microsoft Paraphrase Corpus. ...
4th international workshop on pervasive eye tracking and mobile eye-based interaction
UbiComp '14 Adjunct: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct PublicationPrevious work on eye tracking and eye-based human-computer interfaces mainly concentrated on making use of the eyes in traditional desktop settings. With the recent growth of interest in smart glass devices and low-cost eye trackers, however, gaze-based ...
Comments
Information & Contributors
Information
Published In
May 2024
4761 pages
ISBN:9798400703317
DOI:10.1145/3613905
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Work in progress
- Research
- Refereed limited
Funding Sources
- JSPS KAKENHI Grant
Conference
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Upcoming Conference
CHI 2025
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 445Total Downloads
- Downloads (Last 12 months)445
- Downloads (Last 6 weeks)90
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format