Skip to main content
SpringerLink
Log in

Supporting Non-native Speakers’ Listening Comprehension with Automated Transcripts

  • Chapter
  • First Online:
Services Computing for Language Resources

Part of the book series: Cognitive Technologies ((COGTECH))

  • 417 Accesses

Abstract

Various language services exist to support the listening comprehension of non-native speakers (NNSs). One important service is to provide NNSs with real-time transcripts generated by automatic speech recognition (ASR) technologies. The goal of our research is to explore the effects of ASR transcripts on the listening comprehension of NNSs and consider how to support NNSs with ASR transcripts more effectively. To reach our goal, we ran three studies. The first study investigates the comprehension problems faced by NNSs, and the second study examines how ASR transcripts impact their listening comprehension, e.g., what types of comprehension problems could and could not be solved by reading ASR transcripts. Finally, the third study explores the potential of using eye-tracking data to detect their comprehension problems. Our data analysis identified thirteen types of listening comprehension problems. ASR transcripts helped the NNSs solve certain problems, e.g., “failed to recognize words they know.” However, the transcripts did not solve problems such as “lack of vocabulary,” and indeed NNS burden was increased. Results also show that from eye-tracking data we can make reasonably accurate predictions (83.8%) about the types of problems encountered by NNSs. Our findings provide insight into ways of designing real-time adaptive support systems for NNSs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aaltonen, A., Hyrskykari, A., Räihä, K.J.: 101 spots, or how do users read menus? In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 132–139. ACM Press/Addison-Wesley Publishing Co. (1998)

    Google Scholar 

  2. Anderson, J.R.: Cognitive Psychology and Its Implications. WH Freeman/Times Books/Henry Holt & Co., New York (1990)

    Google Scholar 

  3. Bloomfield, A., Wayland, S.C., Rhoades, E., Blodgett, A., Linck, J., Ross, S.: What makes listening difficult? Factors affecting second language listening comprehension. Technical Report, DTIC Document (2010)

    Google Scholar 

  4. Bondareva, D., Conati, C., Feyzi-Behnagh, R., Harley, J.M., Azevedo, R., Bouchet, F.: Inferring learning from gaze data during interaction with an environment to support self-regulated learning. In: International Conference on Artificial Intelligence in Education, pp. 229–238. Springer (2013)

    Google Scholar 

  5. Cao, X., Yamashita, N., Ishida, T.: How non-native speakers perceive listening comprehension problems: Implications for adaptive support technologies. In: International Conference on Collaboration Technologies, pp. 89–104. Springer (2016)

    Google Scholar 

  6. Cao, X., Yamashita, N., Ishida, T.: Investigating the impact of automated transcripts on non-native speakers’ listening comprehension. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 121–128. ACM (2016)

    Google Scholar 

  7. Conati, C., Jaques, N., Muir, M.: Understanding attention to adaptive hints in educational games: an eye-tracking study. Int. J. Artif. Intell. Educ. 23(1–4), 136–161 (2013)

    Article  Google Scholar 

  8. Gao, G., Yamashita, N., Hautasaari, A.M., Fussell, S.R.: Improving multilingual collaboration by displaying how non-native speakers use automated transcripts and bilingual dictionaries. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 3463–3472. ACM (2015)

    Google Scholar 

  9. Goh, C.C.: A cognitive perspective on language learners’ listening comprehension problems. System 28(1), 55–75 (2000)

    Article  Google Scholar 

  10. Hautasaari, A., Yamashita, N.: Do automated transcripts help non-native speakers catch up on missed conversation in audio conferences? In: Proceedings of the 5th ACM International Conference on Collaboration Across Boundaries: Culture, Distance and Technology, pp. 65–72. ACM (2014)

    Google Scholar 

  11. Ishida, T. (ed.): The Language Grid: Service-Oriented Collective Intelligence for Language Resource Interoperability. Springer, Heidelberg (2011)

    Google Scholar 

  12. Ishida, T.: Intercultural Collaboration and Support Systems: a brief history. In: International Conference on Principle and Practices in Multi-Agent Systems (PRIMA 2016). Invited Paper, pp. 3–19 (2016)

    Google Scholar 

  13. Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting affect from gaze data during interaction with an intelligent tutoring system. In: International Conference on Intelligent Tutoring Systems, pp. 29–38. Springer (2014)

    Google Scholar 

  14. Kalnikaitė, V., Ehlen, P., Whittaker, S.: Markup as you talk: establishing effective memory cues while still contributing to a meeting. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, pp. 349–358. ACM (2012)

    Google Scholar 

  15. Kardan, S., Conati, C.: Exploring gaze data for determining user learning with an interactive simulation. In: International Conference on User Modeling, Adaptation, and Personalization, pp. 126–138. Springer (2012)

    Google Scholar 

  16. Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artif. Intell. 97(1–2), 273–324 (1997)

    Article  MATH  Google Scholar 

  17. Kohavi, R., et al.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai, vol. 14, pp. 1137–1145. Stanford, CA (1995)

    Google Scholar 

  18. Murakami, Y., Lin, D., Ishida, T.: Service-oriented architecture for interoperability of multilanguage services. In: Towards the Multilingual Semantic Web, pp. 313–328. Springer (2014)

    Google Scholar 

  19. Otten, M., Van Berkum, J.J.: Does working memory capacity affect the ability to predict upcoming words in discourse? Brain Res. 1291, 92–101 (2009)

    Article  Google Scholar 

  20. Pan, Y., Jiang, D., Yao, L., Picheny, M., Qin, Y.: Effects of automated transcription quality on non-native speakers’ comprehension in real-time computer-mediated communication. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1725–1734. ACM (2010)

    Google Scholar 

  21. Rayner, K., Chace, K.H., Slattery, T.J., Ashby, J.: Eye movements as reflections of comprehension processes in reading. Sci. Stud. Reading 10(3), 241–255 (2006)

    Article  Google Scholar 

  22. Rubin, J.: A review of second language listening comprehension research. The Mod. Lang. J. 78(2), 199–221 (1994)

    Article  Google Scholar 

  23. Steichen, B., Carenini, G., Conati, C.: User-adaptive information visualization: using eye gaze data to infer visualization tasks and user cognitive abilities. In: Proceedings of the 2013 International Conference on Intelligent User Interfaces, pp. 317–328. ACM (2013)

    Google Scholar 

  24. Van Petten, C., Luka, B.J.: Prediction during language comprehension: benefits, costs, and ERP components. Int. J. Psychophysiol. 83(2), 176–190 (2012)

    Article  Google Scholar 

  25. Yao, L., Pan, Y.X., Jiang, D.N.: Effects of automated transcription delay on non-native speakers’ comprehension in real-time computer-mediated communication. In: IFIP Conference on Human-Computer Interaction, pp. 207–214. Springer (2011)

    Google Scholar 

Download references

Acknowledgements

This research was partially supported by a Grant-in-Aid for Scientific Research (A) (17H00759, 2017–2020) from Japan Society for the Promotion of Science (JSPS).

Author information

Authors and Affiliations

  1. Department of Social Informatics, Kyoto University, Kyoto, 606-8510, Japan

    Xun Cao & Toru Ishida

  2. NTT Communication Science Labs, 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237, Japan

    Naomi Yamashita

Authors
  1. Xun Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Naomi Yamashita
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Toru Ishida
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xun Cao .

Editor information

Editors and Affiliations

  1. Kyoto University , Kyoto, Japan

    Yohei Murakami

  2. Kyoto University , Kyoto, Japan

    Donghui Lin

  3. Kyoto University, Kyoto, Japan

    Toru Ishida

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Cao, X., Yamashita, N., Ishida, T. (2018). Supporting Non-native Speakers’ Listening Comprehension with Automated Transcripts. In: Murakami, Y., Lin, D., Ishida, T. (eds) Services Computing for Language Resources . Cognitive Technologies. Springer, Singapore. https://doi.org/10.1007/978-981-10-7793-7_10

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-7793-7_10

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-7792-0

  • Online ISBN: 978-981-10-7793-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation