Abstract
This paper proposes a method for estimating speech balloons with unknown words for English learners based on speech and eye gaze information during reading aloud Japanese comics translated into English. In this method, a headset and eye tracker are used to record speech and eye gaze information. Then we extract 47 features from them together with text information. The features are used to train a support vector machine to estimate unknown words for each speech balloon. We evaluated the proposed method by measuring data from 20 Japanese university students. As a result, we confirmed that the proposed method performs better than the estimation using only text information and that the speech and eye gaze information are effective for estimating unknown words.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Buscher, G., Dengel, A., van Elst, L.: Eye movements as implicit relevance feedback. In: CHI 2008 Extended Abstracts on Human Factors in Computing Systems, pp. 2991–2996 (2008)
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)
Daiku, Y.: Estimation of understanding and interest based on reading behavior in extensive reading with Japanese comics translated in English. In: Master Thesis of Osaka Prefecture University (2019)
Doukhan, D., Carrive, J., Vallet, F., Larcher, A., Meignier, S.: An open-source speaker gender detection framework for monitoring gender equality. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5214–5218 (2018). https://doi.org/10.1109/ICASSP.2018.8461471
Garain, U., Pandit, O., Augereau, O., Okoso, A., Kise, K.: Identification of reader specific difficult words by analyzing eye gaze and document content. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 1346–1351. IEEE (2017)
Ishimaru, S., Bukhari, S.S., Heisel, C., Kuhn, J., Dengel, A.: Towards an intelligent textbook: eye gaze based attention extraction on materials for learning and instruction in physics. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, pp. 1041–1045 (2016)
Martínez-Gómez, P., Aizawa, A.: Recognition of understanding level and language skill using measurements of reading behavior. In: Proceedings of the 19th International Conference on Intelligent User Interfaces, pp. 95–104 (2014)
Pudil, P., Novovičová, J., Kittler, J.: Floating search methods in feature selection. Pattern Recogn. Lett. 15(11), 1119–1125 (1994)
Sabu, K., Rao, P.: Automatic prediction of confidence level from children’s oral reading recordings. In: Interspeech, pp. 3141–3145 (2020)
Takahashi, R.: Estimation of understanding based on eye information in extensive reading with Japanese comics translated in English. In: Graduation Thesis of Osaka Prefecture University (2021)
TheEnglishClub: Reading aloud in English—scientific method and tips and recommended materials for the best results (2019). https://english-club.jp/blog/english-reading-aloud/. Accessed 5 June 2022
Zhang, T., Hasegawa-Johnson, M., Levinson, S.E.: Children’s emotion recognition in an intelligent tutoring scenario. In: Proceedings of Eighth European Conference on Speech Communication and Technology (INTERSPEECH) (2004)
Acknowledgements
This work was supported in part by grants from JST Trilateral AI Research (Grant No. JPMJCR20G3), JSPS Grant-in-Aid for Scientific Research (B) (Grant No. 20H04213), and JSPS Fund for the Promotion of Joint International Research (Fostering Joint International Research (B)) (Grant No. 20KK0235).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this paper
Cite this paper
Takaike, T., Iwata, M., Kise, K. (2023). Estimation of Unknown Words Using Speech and Eye Gaze When Reading Aloud Comics. In: Rousseau, JJ., Kapralos, B. (eds) Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges. ICPR 2022. Lecture Notes in Computer Science, vol 13644. Springer, Cham. https://doi.org/10.1007/978-3-031-37742-6_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-37742-6_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-37741-9
Online ISBN: 978-3-031-37742-6
eBook Packages: Computer ScienceComputer Science (R0)
