Abstract
Various language services exist to support the listening comprehension of non-native speakers (NNSs). One important service is to provide NNSs with real-time transcripts generated by automatic speech recognition (ASR) technologies. The goal of our research is to explore the effects of ASR transcripts on the listening comprehension of NNSs and consider how to support NNSs with ASR transcripts more effectively. To reach our goal, we ran three studies. The first study investigates the comprehension problems faced by NNSs, and the second study examines how ASR transcripts impact their listening comprehension, e.g., what types of comprehension problems could and could not be solved by reading ASR transcripts. Finally, the third study explores the potential of using eye-tracking data to detect their comprehension problems. Our data analysis identified thirteen types of listening comprehension problems. ASR transcripts helped the NNSs solve certain problems, e.g., “failed to recognize words they know.” However, the transcripts did not solve problems such as “lack of vocabulary,” and indeed NNS burden was increased. Results also show that from eye-tracking data we can make reasonably accurate predictions (83.8%) about the types of problems encountered by NNSs. Our findings provide insight into ways of designing real-time adaptive support systems for NNSs.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aaltonen, A., Hyrskykari, A., Räihä, K.J.: 101 spots, or how do users read menus? In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 132–139. ACM Press/Addison-Wesley Publishing Co. (1998)
Anderson, J.R.: Cognitive Psychology and Its Implications. WH Freeman/Times Books/Henry Holt & Co., New York (1990)
Bloomfield, A., Wayland, S.C., Rhoades, E., Blodgett, A., Linck, J., Ross, S.: What makes listening difficult? Factors affecting second language listening comprehension. Technical Report, DTIC Document (2010)
Bondareva, D., Conati, C., Feyzi-Behnagh, R., Harley, J.M., Azevedo, R., Bouchet, F.: Inferring learning from gaze data during interaction with an environment to support self-regulated learning. In: International Conference on Artificial Intelligence in Education, pp. 229–238. Springer (2013)
Cao, X., Yamashita, N., Ishida, T.: How non-native speakers perceive listening comprehension problems: Implications for adaptive support technologies. In: International Conference on Collaboration Technologies, pp. 89–104. Springer (2016)
Cao, X., Yamashita, N., Ishida, T.: Investigating the impact of automated transcripts on non-native speakers’ listening comprehension. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 121–128. ACM (2016)
Conati, C., Jaques, N., Muir, M.: Understanding attention to adaptive hints in educational games: an eye-tracking study. Int. J. Artif. Intell. Educ. 23(1–4), 136–161 (2013)
Gao, G., Yamashita, N., Hautasaari, A.M., Fussell, S.R.: Improving multilingual collaboration by displaying how non-native speakers use automated transcripts and bilingual dictionaries. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 3463–3472. ACM (2015)
Goh, C.C.: A cognitive perspective on language learners’ listening comprehension problems. System 28(1), 55–75 (2000)
Hautasaari, A., Yamashita, N.: Do automated transcripts help non-native speakers catch up on missed conversation in audio conferences? In: Proceedings of the 5th ACM International Conference on Collaboration Across Boundaries: Culture, Distance and Technology, pp. 65–72. ACM (2014)
Ishida, T. (ed.): The Language Grid: Service-Oriented Collective Intelligence for Language Resource Interoperability. Springer, Heidelberg (2011)
Ishida, T.: Intercultural Collaboration and Support Systems: a brief history. In: International Conference on Principle and Practices in Multi-Agent Systems (PRIMA 2016). Invited Paper, pp. 3–19 (2016)
Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting affect from gaze data during interaction with an intelligent tutoring system. In: International Conference on Intelligent Tutoring Systems, pp. 29–38. Springer (2014)
Kalnikaitė, V., Ehlen, P., Whittaker, S.: Markup as you talk: establishing effective memory cues while still contributing to a meeting. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, pp. 349–358. ACM (2012)
Kardan, S., Conati, C.: Exploring gaze data for determining user learning with an interactive simulation. In: International Conference on User Modeling, Adaptation, and Personalization, pp. 126–138. Springer (2012)
Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artif. Intell. 97(1–2), 273–324 (1997)
Kohavi, R., et al.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai, vol. 14, pp. 1137–1145. Stanford, CA (1995)
Murakami, Y., Lin, D., Ishida, T.: Service-oriented architecture for interoperability of multilanguage services. In: Towards the Multilingual Semantic Web, pp. 313–328. Springer (2014)
Otten, M., Van Berkum, J.J.: Does working memory capacity affect the ability to predict upcoming words in discourse? Brain Res. 1291, 92–101 (2009)
Pan, Y., Jiang, D., Yao, L., Picheny, M., Qin, Y.: Effects of automated transcription quality on non-native speakers’ comprehension in real-time computer-mediated communication. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1725–1734. ACM (2010)
Rayner, K., Chace, K.H., Slattery, T.J., Ashby, J.: Eye movements as reflections of comprehension processes in reading. Sci. Stud. Reading 10(3), 241–255 (2006)
Rubin, J.: A review of second language listening comprehension research. The Mod. Lang. J. 78(2), 199–221 (1994)
Steichen, B., Carenini, G., Conati, C.: User-adaptive information visualization: using eye gaze data to infer visualization tasks and user cognitive abilities. In: Proceedings of the 2013 International Conference on Intelligent User Interfaces, pp. 317–328. ACM (2013)
Van Petten, C., Luka, B.J.: Prediction during language comprehension: benefits, costs, and ERP components. Int. J. Psychophysiol. 83(2), 176–190 (2012)
Yao, L., Pan, Y.X., Jiang, D.N.: Effects of automated transcription delay on non-native speakers’ comprehension in real-time computer-mediated communication. In: IFIP Conference on Human-Computer Interaction, pp. 207–214. Springer (2011)
Acknowledgements
This research was partially supported by a Grant-in-Aid for Scientific Research (A) (17H00759, 2017–2020) from Japan Society for the Promotion of Science (JSPS).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Cao, X., Yamashita, N., Ishida, T. (2018). Supporting Non-native Speakers’ Listening Comprehension with Automated Transcripts. In: Murakami, Y., Lin, D., Ishida, T. (eds) Services Computing for Language Resources . Cognitive Technologies. Springer, Singapore. https://doi.org/10.1007/978-981-10-7793-7_10
Download citation
DOI: https://doi.org/10.1007/978-981-10-7793-7_10
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-7792-0
Online ISBN: 978-981-10-7793-7
eBook Packages: Computer ScienceComputer Science (R0)