Skip to main content

Advertisement

Log in

Effect of chatbot-assisted language learning: A meta-analysis

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Chatbots have shown great potential for language learning. However, previous studies have reported mixed results on the efficiency of chatbot-assisted language learning (CALL). This study integrated the results of previous experimental studies on CALL by using meta-analysis to explore its effectiveness. A total of 61 samples from 18 studies were examined. The results showed that CALL had a moderate average effect (g = .527). In addition, nine potential moderating variables (educational level, target language, language domain, learning outcome, instruction duration, chatbot interface, chatbot development, task dominance, and interaction way) were identified and discussed. The results of this study provided insights into the use and design of chatbots for language learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Source: Complied based on the websites: https://www.prisma-statement.org/PRISMAStatement/FlowDiagram

Fig. 2

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References 

*Studies included in the meta-analysis are marked with an asterisk

  • Abdul-Kader, S. A., Woods, J. C. J. I. J. o. A. C. S., & Applications. (2015). Survey on chatbot design techniques in speech conversation systems. International Journal of Advanced Computer Science and Applications, 6(7).

  • Adamopoulou, E., & Moussiades, L. (2020). An overview of chatbot technology. In Artificial Intelligence Applications and Innovations: 16th IFIP WG 12.5 International Conference, AIAI 2020, Neos Marmaras, Greece, June 5–7, 2020, Proceedings, Part II 16 (pp. 373–383). Springer International Publishing.

  • * Ahn, S. (2022). The Effects of Chatbot on Grammar Competence for Korean EFL College Students. Journal of Digital Convergence, 20(3), 53-61.

  • Atwell, E. (1999). The language machine (pp.1–72). The British Council.

  • Baby, C. J., Khan, F. A., & Swathi, J. (2017). Home automation using IoT and a chatbot using natural language processing. In 2017 Innovations in Power and Advanced Computing Technologies (i-PACT) (pp. 1–6). IEEE.

  • Bii, P. (2013). Chatbot technology: A possible means of unlocking student potential to learn how to learn. Educational Research, 4(2), 218–221.

    Google Scholar 

  • Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2021). Introduction to meta-analysis. John Wiley & Sons.

    Book  MATH  Google Scholar 

  • Bruno, D. (2022). Conversational interfaces supporting the search for content within the juridical field.

  • Cameron, G., Cameron, D., Megaw, G., Bond, R., Mulvenna, M., O’Neill, S., Armour, C., & McTear, M. (2017). Towards a chatbot for digital counselling. In Proceedings of the 31st International BCS Human Computer Interaction Conference (HCI 2017) 31 (pp. 1–7).

  • Chen, H.-L., Vicki Widarso, G., & Sutrisno, H. (2020). A chatbot for learning Chinese: Learning achievement and technology acceptance. Journal of Educational Computing Research, 58(6), 1161–1189.

    Article  Google Scholar 

  • Chen, X., Zou, D., Xie, H., & Cheng, G. (2021). Twenty years of personalized language learning. Educational Technology & Society, 24(1), 205–222.

    Google Scholar 

  • Chopra, S., Gianforte, R., & Sholar, J. J. A. T. o. G. (2016). Meet percy: The CS 221 teaching assistant chatbot. ACM Transactions on Graphics, 1(1), 1–8.

  • Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Academic press.

    Book  MATH  Google Scholar 

  • Durlak, J. A. (2009). How to select, calculate, and interpret effect sizes. Journal of Pediatric Psychology, 34(9), 917–928.

    Article  Google Scholar 

  • Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463.

    Article  MATH  Google Scholar 

  • Flynn, S., Foley, C., & Vinnitskaya, I. J. I. J. o. M. (2004). The cumulative-enhancement model for language acquisition: Comparing adults' and children's patterns of development in first, second and third language acquisition of relative clauses. International Journal of Multilingualism, 1(1), 3–16.

  • Freed, B. F., Segalowitz, N., & Dewey, D. P. (2004). Context of learning and second language fluency in French: Comparing regular classroom, study abroad, and intensive domestic immersion programs. Studies in Second Language Acquisition, 26(2), 275–301.

    Article  Google Scholar 

  • Fryer, L. K., Ainley, M., Thompson, A., Gibson, A., & Sherlock, Z. (2017). Stimulating and sustaining interest in a language course: An experimental comparison of Chatbot and Human task partners. Computers in Human Behavior, 75, 461–468.

    Article  Google Scholar 

  • * Fryer, L. K., Thompson, A., Nakao, K., Howarth, M., Gallacher, A. J. L., & Differences, I. (2020). Supporting self-efficacy beliefs and interest as educational inputs and outcomes: Framing AI and Human partnered task experiences. Learning and Individual Differences, 80, 101850.

  • * Goda, Y., Yamada, M., Matsukawa, H., Hata, K., Yasunami, S. J. T. J. o. I., & Education, S. i. (2014). Conversation with a chatbot before an online EFL group discussion and the effects on critical thinking. The Journal of Information and Systems in Education, 13(1), 1–7.

  • Guan, C., Mou, J., & Jiang, Z. J. I. J. o. I. S. (2020). Artificial intelligence innovation in education: a twenty-year data-driven historical analysis. International Journal of Innovation Studies, 4(4), 134–147.

  • Hallahan, D. P., Keller, C. E., McKinney, J. D., Lloyd, J. W., & Bryan, T. (1988). Examining the research base of the regular education initiative: Efficacy studies and the adaptive learning environments model. Journal of Learning Disabilities, 21(1), 29–35.

    Article  Google Scholar 

  • Han, D.-E. (2020). The Effects of Voice-based AI Chatbots on Korean EFL Middle School Students’ Speaking Competence and Affective Domains. Asia-Pacific Journal of Convergent Research Interchange, 6(7), 71–80.

    Article  Google Scholar 

  • Han, J.-H., Jo, M.-H., Jones, V., & Jo, J.-H. J. J. o. I. P. S. (2008). Comparative study on the educational use of home robots for children. Journal of Information Processing Systems, 4(4), 159–168.

  • Hedges, L. V., & Olkin, I. (2014). Statistical methods for meta-analysis. Academic press.

  • Higgins, J. P., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (2019). Cochrane handbook for systematic reviews of interventions. John Wiley & Sons.

    Book  Google Scholar 

  • * Hsu, M.-H., Chen, P.-S., & Yu, C.-S. (2021). Proposing a task-oriented chatbot system for EFL learners speaking practice. Interactive Learning Environments, 1–12.

  • Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237–257.

    Article  Google Scholar 

  • Hwang, G.-J., & Chang, C.-Y. J. I. L. E. (2021). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 1–14.

  • * Hwang, W.-Y., Guo, B.-C., Hoang, A., Chang, C.-C., & Wu, N.-T. (2022). Facilitating authentic contextual EFL speaking and conversation with smart mechanisms and investigating its influence on learning achievements. Computer Assisted Language Learning, 1–27.

  • Jakonen, T., & Jauni, H. (2021). Mediated learning materials: Visibility checks in telepresence robot mediated classroom interaction [Article]. Classroom Discourse, 12(1–2), 121–145.

    Article  Google Scholar 

  • * Jeon, J. (2021a). Chatbot-assisted dynamic assessment (CA-DA) for L2 vocabulary learning and diagnosis. Computer Assisted Language Learning, 1–27.

  • Jeon, J. (2021b). Exploring AI chatbot affordances in the EFL classroom: Young learners’ experiences and perspectives. Computer Assisted Language Learning, 1–26.

  • Jeong, H., Hmelo-Silver, C. E., & Jo, K. (2019). Ten years of computer-supported collaborative learning: A meta-analysis of CSCL in STEM education during 2005–2014. Educational Research Review, 28, 100284.

    Article  Google Scholar 

  • * Kim, H.-S., Cha, Y., & Kim, N.-Y. (2020). Impact of mobile interactions with AI on writing performance. Modern English Education, 21(2), 1-13.

  • * Kim, N.-Y. (2016). Effects of voice chat on EFL learners’ speaking ability according to proficiency levels. Multimedia-Assisted Language Learning, 19(4), 63-88.

  • * Kim, N.-Y. (2018a). Chatbots and Korean EFL students' English vocabulary learning. Journal of Digital Convergence, 16(2), 1-7.

  • * Kim, N.-Y. (2018b). A study on chatbots for developing Korean college students' English listening and reading skills. Journal of Digital Convergence, 16(8), 19-26.

  • * Kim, N.-Y. (2019). A study on the use of artificial intelligence chatbots for improving English grammar skills. Journal of Digital Convergence, 17(8), 37-46.

  • Krashen, S. (1981). Second language acquisition. Second Language Learning, 3(7), 19–39.

  • Lantolf, J. P. (2000). Introducing sociocultural theory. Sociocultural Theory and Second Language Learning, 1, 1–26.

    Google Scholar 

  • Lantolf, J. P., & Pavlenko, A. (1995). Sociocultural theory and second language acquisition. Annual Review of Applied Linguistics, 15, 108–124.

    Article  Google Scholar 

  • Lee, H., & Lee, J. H. (2022). The effects of robot-assisted language learning: A meta-analysis. Educational Research Review, 35, 100425.

    Article  Google Scholar 

  • Lee, J.-Y., & Hwang, Y. (2022). A meta-analysis of the effects of using AI chatbot in Korean EFL education. Studies in English Language & Literature, 48(1), 213–243.

    Article  Google Scholar 

  • Lin, C.-J., & Mubarok, H. (2021). Learning analytics for investigating the mind map-guided AI Chatbot approach in an EFL flipped speaking classroom. Educational Technology & Society, 24(4), 16–35.

    Google Scholar 

  • * Lin, M. P. C., & Chang, D. (2020). Enhancing post-secondary writers' writing skills with a chatbot: A mixed-method classroom study. Educational Technology & Society, 23(1), 78–92.

  • * Liu, C.-C., Liao, M.-G., Chang, C.-H., & Lin, H.-M. (2022). An analysis of children’interaction with an AI chatbot and its impact on their interest in reading. Computers & education, 189, 104576.

  • Lyster, R., & Ranta, L. (1997). Corrective feedback and learner uptake: Negotiation of form in communicative classrooms. Studies in Second Language Acquisition, 19(1), 37–66.

    Article  Google Scholar 

  • * Mageira, K., Pittou, D., Papasalouros, A., Kotis, K., Zangogianni, P., & Daradoumis, A. (2022). Educational AI chatbots for content and language integrated learning. Applied Sciences-Basel, 12(7), 3239.

  • Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., Group* P. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine, 151(4), 264–269.

    Article  Google Scholar 

  • Moore, M. G. (2013). The theory of transactional distance. In Handbook of distance education (pp. 84–103). Routledge.

  • * Nagata, R., Hashiguchi, T., & Sadoun, D. (2019). Is the simplest chatbot effective in English writing learning assistance? In Computational Linguistics: 16th International Conference of the Pacific Association for Computational Linguistics, PACLING 2019, Hanoi, Vietnam, October 11–13, 2019, Revised Selected Papers 16 (pp. 245–256). Springer Singapore.

  • * Nghi, T. T., Phuc, T. H., & Thang, N. T. (2019). Applying ai chatbot for teaching a foreign language: An empirical research. International Journal of Scientific Research, 8.

  • Oh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018). I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–13).

  • Peng, J. E., & Woodrow, L. (2010). Willingness to communicate in English: A model in the Chinese EFL classroom context. Language Learning, 60(4), 834–876.

    Article  Google Scholar 

  • Pham, X. L., Pham, T., Nguyen, Q. M., Nguyen, T. H., & Cao, T. T. H. (2018). Chatbot as an intelligent personal assistant for mobile language learning. In Proceedings of the 2018 2nd International Conference on Education and E-Learning (pp. 16–21).

  • Polatidis, N. (2014). Chatbot for admissions. arXiv preprint arXiv:1408.6762.

  • Qiu, X. b., Shan, C., & Yao, J. (2023). The effects of virtual reality on EFL learning: A meta-analysis. Education and Information Technologies, 1–27.

  • Saxton, M. (2017). Child language: Acquisition and development. Child Language, 1–408.

  • Schmidt, R. (1992). Awareness and second language acquisition. Annual Review of Applied Linguistics, 13, 206–226.

    Article  Google Scholar 

  • Shakhovska, N., Basystiuk, O., & Shakhovska, K. (2019). Development of the Speech-to-Text Chatbot Interface Based on Google API. In MoMLeT (pp. 212–221).

  • Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151, 103862.

    Article  Google Scholar 

  • Tai, T.-Y., & Chen, H. H.-J. (2020). The impact of Google Assistant on adolescent EFL learners’ willingness to communicate. Interactive Learning Environments, 1–18.

  • * Vázquez-Cano, E., Mengual-Andrés, S., & López-Meneses, E. (2021). Chatbot to improve learning punctuation in Spanish and to enhance open and flexible learning environments. International Journal of Educational Technology in Higher Education, 18(1), 1–20.

  • Von Worde, R. (2003). Students’ Perspectives on Foreign Language Anxiety. Inquiry, 8(1), n1.

    Google Scholar 

  • Wang, Y. F., Petrina, S., & Feng, F. (2017). VILLAGE—Virtual Immersive Language Learning and Gaming Environment: Immersion and presence. British Journal of Educational Technology, 48(2), 431–450.

    Article  Google Scholar 

  • Wei, X., Saab, N., & Admiraal, W. (2021). Assessment of cognitive, behavioral, and affective learning outcomes in massive open online courses: A systematic literature review. Computers & Education, 163, 104097.

    Article  Google Scholar 

  • Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering (pp. 1–10).

  • * Ye, Y., Deng, J., & Liu, X. (2021). The Effect of Oral Practice via Chatbot on Students’ Oral English Accuracy. In Emerging Technologies for Education: 6th International Symposium, SETE 2021, Zhuhai, China, November 11–12, 2021, Revised Selected Papers (pp. 344–354). Cham: Springer International Publishing.

  • Zhang, S., Che, S., Nan, D., & Kim, J. H. (2023). How does online social interaction promote students’ continuous learning intentions? Frontiers in Psychology, 14.

Download references

Funding

This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (NRF-2020R1A2C1014957). The authors declare that they have no competing interests.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jang Hyun Kim.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Shan, C., Lee, J.S.Y. et al. Effect of chatbot-assisted language learning: A meta-analysis. Educ Inf Technol 28, 15223–15243 (2023). https://doi.org/10.1007/s10639-023-11805-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-023-11805-6

Keywords

Navigation