Abstract
Chatbots have shown great potential for language learning. However, previous studies have reported mixed results on the efficiency of chatbot-assisted language learning (CALL). This study integrated the results of previous experimental studies on CALL by using meta-analysis to explore its effectiveness. A total of 61 samples from 18 studies were examined. The results showed that CALL had a moderate average effect (g = .527). In addition, nine potential moderating variables (educational level, target language, language domain, learning outcome, instruction duration, chatbot interface, chatbot development, task dominance, and interaction way) were identified and discussed. The results of this study provided insights into the use and design of chatbots for language learning.

Source: Complied based on the websites: https://www.prisma-statement.org/PRISMAStatement/FlowDiagram

Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
*Studies included in the meta-analysis are marked with an asterisk
Abdul-Kader, S. A., Woods, J. C. J. I. J. o. A. C. S., & Applications. (2015). Survey on chatbot design techniques in speech conversation systems. International Journal of Advanced Computer Science and Applications, 6(7).
Adamopoulou, E., & Moussiades, L. (2020). An overview of chatbot technology. In Artificial Intelligence Applications and Innovations: 16th IFIP WG 12.5 International Conference, AIAI 2020, Neos Marmaras, Greece, June 5–7, 2020, Proceedings, Part II 16 (pp. 373–383). Springer International Publishing.
* Ahn, S. (2022). The Effects of Chatbot on Grammar Competence for Korean EFL College Students. Journal of Digital Convergence, 20(3), 53-61.
Atwell, E. (1999). The language machine (pp.1–72). The British Council.
Baby, C. J., Khan, F. A., & Swathi, J. (2017). Home automation using IoT and a chatbot using natural language processing. In 2017 Innovations in Power and Advanced Computing Technologies (i-PACT) (pp. 1–6). IEEE.
Bii, P. (2013). Chatbot technology: A possible means of unlocking student potential to learn how to learn. Educational Research, 4(2), 218–221.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2021). Introduction to meta-analysis. John Wiley & Sons.
Bruno, D. (2022). Conversational interfaces supporting the search for content within the juridical field.
Cameron, G., Cameron, D., Megaw, G., Bond, R., Mulvenna, M., O’Neill, S., Armour, C., & McTear, M. (2017). Towards a chatbot for digital counselling. In Proceedings of the 31st International BCS Human Computer Interaction Conference (HCI 2017) 31 (pp. 1–7).
Chen, H.-L., Vicki Widarso, G., & Sutrisno, H. (2020). A chatbot for learning Chinese: Learning achievement and technology acceptance. Journal of Educational Computing Research, 58(6), 1161–1189.
Chen, X., Zou, D., Xie, H., & Cheng, G. (2021). Twenty years of personalized language learning. Educational Technology & Society, 24(1), 205–222.
Chopra, S., Gianforte, R., & Sholar, J. J. A. T. o. G. (2016). Meet percy: The CS 221 teaching assistant chatbot. ACM Transactions on Graphics, 1(1), 1–8.
Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Academic press.
Durlak, J. A. (2009). How to select, calculate, and interpret effect sizes. Journal of Pediatric Psychology, 34(9), 917–928.
Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463.
Flynn, S., Foley, C., & Vinnitskaya, I. J. I. J. o. M. (2004). The cumulative-enhancement model for language acquisition: Comparing adults' and children's patterns of development in first, second and third language acquisition of relative clauses. International Journal of Multilingualism, 1(1), 3–16.
Freed, B. F., Segalowitz, N., & Dewey, D. P. (2004). Context of learning and second language fluency in French: Comparing regular classroom, study abroad, and intensive domestic immersion programs. Studies in Second Language Acquisition, 26(2), 275–301.
Fryer, L. K., Ainley, M., Thompson, A., Gibson, A., & Sherlock, Z. (2017). Stimulating and sustaining interest in a language course: An experimental comparison of Chatbot and Human task partners. Computers in Human Behavior, 75, 461–468.
* Fryer, L. K., Thompson, A., Nakao, K., Howarth, M., Gallacher, A. J. L., & Differences, I. (2020). Supporting self-efficacy beliefs and interest as educational inputs and outcomes: Framing AI and Human partnered task experiences. Learning and Individual Differences, 80, 101850.
* Goda, Y., Yamada, M., Matsukawa, H., Hata, K., Yasunami, S. J. T. J. o. I., & Education, S. i. (2014). Conversation with a chatbot before an online EFL group discussion and the effects on critical thinking. The Journal of Information and Systems in Education, 13(1), 1–7.
Guan, C., Mou, J., & Jiang, Z. J. I. J. o. I. S. (2020). Artificial intelligence innovation in education: a twenty-year data-driven historical analysis. International Journal of Innovation Studies, 4(4), 134–147.
Hallahan, D. P., Keller, C. E., McKinney, J. D., Lloyd, J. W., & Bryan, T. (1988). Examining the research base of the regular education initiative: Efficacy studies and the adaptive learning environments model. Journal of Learning Disabilities, 21(1), 29–35.
Han, D.-E. (2020). The Effects of Voice-based AI Chatbots on Korean EFL Middle School Students’ Speaking Competence and Affective Domains. Asia-Pacific Journal of Convergent Research Interchange, 6(7), 71–80.
Han, J.-H., Jo, M.-H., Jones, V., & Jo, J.-H. J. J. o. I. P. S. (2008). Comparative study on the educational use of home robots for children. Journal of Information Processing Systems, 4(4), 159–168.
Hedges, L. V., & Olkin, I. (2014). Statistical methods for meta-analysis. Academic press.
Higgins, J. P., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (2019). Cochrane handbook for systematic reviews of interventions. John Wiley & Sons.
* Hsu, M.-H., Chen, P.-S., & Yu, C.-S. (2021). Proposing a task-oriented chatbot system for EFL learners speaking practice. Interactive Learning Environments, 1–12.
Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237–257.
Hwang, G.-J., & Chang, C.-Y. J. I. L. E. (2021). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 1–14.
* Hwang, W.-Y., Guo, B.-C., Hoang, A., Chang, C.-C., & Wu, N.-T. (2022). Facilitating authentic contextual EFL speaking and conversation with smart mechanisms and investigating its influence on learning achievements. Computer Assisted Language Learning, 1–27.
Jakonen, T., & Jauni, H. (2021). Mediated learning materials: Visibility checks in telepresence robot mediated classroom interaction [Article]. Classroom Discourse, 12(1–2), 121–145.
* Jeon, J. (2021a). Chatbot-assisted dynamic assessment (CA-DA) for L2 vocabulary learning and diagnosis. Computer Assisted Language Learning, 1–27.
Jeon, J. (2021b). Exploring AI chatbot affordances in the EFL classroom: Young learners’ experiences and perspectives. Computer Assisted Language Learning, 1–26.
Jeong, H., Hmelo-Silver, C. E., & Jo, K. (2019). Ten years of computer-supported collaborative learning: A meta-analysis of CSCL in STEM education during 2005–2014. Educational Research Review, 28, 100284.
* Kim, H.-S., Cha, Y., & Kim, N.-Y. (2020). Impact of mobile interactions with AI on writing performance. Modern English Education, 21(2), 1-13.
* Kim, N.-Y. (2016). Effects of voice chat on EFL learners’ speaking ability according to proficiency levels. Multimedia-Assisted Language Learning, 19(4), 63-88.
* Kim, N.-Y. (2018a). Chatbots and Korean EFL students' English vocabulary learning. Journal of Digital Convergence, 16(2), 1-7.
* Kim, N.-Y. (2018b). A study on chatbots for developing Korean college students' English listening and reading skills. Journal of Digital Convergence, 16(8), 19-26.
* Kim, N.-Y. (2019). A study on the use of artificial intelligence chatbots for improving English grammar skills. Journal of Digital Convergence, 17(8), 37-46.
Krashen, S. (1981). Second language acquisition. Second Language Learning, 3(7), 19–39.
Lantolf, J. P. (2000). Introducing sociocultural theory. Sociocultural Theory and Second Language Learning, 1, 1–26.
Lantolf, J. P., & Pavlenko, A. (1995). Sociocultural theory and second language acquisition. Annual Review of Applied Linguistics, 15, 108–124.
Lee, H., & Lee, J. H. (2022). The effects of robot-assisted language learning: A meta-analysis. Educational Research Review, 35, 100425.
Lee, J.-Y., & Hwang, Y. (2022). A meta-analysis of the effects of using AI chatbot in Korean EFL education. Studies in English Language & Literature, 48(1), 213–243.
Lin, C.-J., & Mubarok, H. (2021). Learning analytics for investigating the mind map-guided AI Chatbot approach in an EFL flipped speaking classroom. Educational Technology & Society, 24(4), 16–35.
* Lin, M. P. C., & Chang, D. (2020). Enhancing post-secondary writers' writing skills with a chatbot: A mixed-method classroom study. Educational Technology & Society, 23(1), 78–92.
* Liu, C.-C., Liao, M.-G., Chang, C.-H., & Lin, H.-M. (2022). An analysis of children’interaction with an AI chatbot and its impact on their interest in reading. Computers & education, 189, 104576.
Lyster, R., & Ranta, L. (1997). Corrective feedback and learner uptake: Negotiation of form in communicative classrooms. Studies in Second Language Acquisition, 19(1), 37–66.
* Mageira, K., Pittou, D., Papasalouros, A., Kotis, K., Zangogianni, P., & Daradoumis, A. (2022). Educational AI chatbots for content and language integrated learning. Applied Sciences-Basel, 12(7), 3239.
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., Group* P. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine, 151(4), 264–269.
Moore, M. G. (2013). The theory of transactional distance. In Handbook of distance education (pp. 84–103). Routledge.
* Nagata, R., Hashiguchi, T., & Sadoun, D. (2019). Is the simplest chatbot effective in English writing learning assistance? In Computational Linguistics: 16th International Conference of the Pacific Association for Computational Linguistics, PACLING 2019, Hanoi, Vietnam, October 11–13, 2019, Revised Selected Papers 16 (pp. 245–256). Springer Singapore.
* Nghi, T. T., Phuc, T. H., & Thang, N. T. (2019). Applying ai chatbot for teaching a foreign language: An empirical research. International Journal of Scientific Research, 8.
Oh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018). I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–13).
Peng, J. E., & Woodrow, L. (2010). Willingness to communicate in English: A model in the Chinese EFL classroom context. Language Learning, 60(4), 834–876.
Pham, X. L., Pham, T., Nguyen, Q. M., Nguyen, T. H., & Cao, T. T. H. (2018). Chatbot as an intelligent personal assistant for mobile language learning. In Proceedings of the 2018 2nd International Conference on Education and E-Learning (pp. 16–21).
Polatidis, N. (2014). Chatbot for admissions. arXiv preprint arXiv:1408.6762.
Qiu, X. b., Shan, C., & Yao, J. (2023). The effects of virtual reality on EFL learning: A meta-analysis. Education and Information Technologies, 1–27.
Saxton, M. (2017). Child language: Acquisition and development. Child Language, 1–408.
Schmidt, R. (1992). Awareness and second language acquisition. Annual Review of Applied Linguistics, 13, 206–226.
Shakhovska, N., Basystiuk, O., & Shakhovska, K. (2019). Development of the Speech-to-Text Chatbot Interface Based on Google API. In MoMLeT (pp. 212–221).
Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151, 103862.
Tai, T.-Y., & Chen, H. H.-J. (2020). The impact of Google Assistant on adolescent EFL learners’ willingness to communicate. Interactive Learning Environments, 1–18.
* Vázquez-Cano, E., Mengual-Andrés, S., & López-Meneses, E. (2021). Chatbot to improve learning punctuation in Spanish and to enhance open and flexible learning environments. International Journal of Educational Technology in Higher Education, 18(1), 1–20.
Von Worde, R. (2003). Students’ Perspectives on Foreign Language Anxiety. Inquiry, 8(1), n1.
Wang, Y. F., Petrina, S., & Feng, F. (2017). VILLAGE—Virtual Immersive Language Learning and Gaming Environment: Immersion and presence. British Journal of Educational Technology, 48(2), 431–450.
Wei, X., Saab, N., & Admiraal, W. (2021). Assessment of cognitive, behavioral, and affective learning outcomes in massive open online courses: A systematic literature review. Computers & Education, 163, 104097.
Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering (pp. 1–10).
* Ye, Y., Deng, J., & Liu, X. (2021). The Effect of Oral Practice via Chatbot on Students’ Oral English Accuracy. In Emerging Technologies for Education: 6th International Symposium, SETE 2021, Zhuhai, China, November 11–12, 2021, Revised Selected Papers (pp. 344–354). Cham: Springer International Publishing.
Zhang, S., Che, S., Nan, D., & Kim, J. H. (2023). How does online social interaction promote students’ continuous learning intentions? Frontiers in Psychology, 14.
Funding
This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (NRF-2020R1A2C1014957). The authors declare that they have no competing interests.
Author information
Authors and Affiliations
Corresponding author
About this article
Cite this article
Zhang, S., Shan, C., Lee, J.S.Y. et al. Effect of chatbot-assisted language learning: A meta-analysis. Educ Inf Technol 28, 15223–15243 (2023). https://doi.org/10.1007/s10639-023-11805-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-023-11805-6