Abstract
With the prevalence of Large Language Model-based chatbots, middle school students are increasingly likely to engage with these tools to complete their assignments, raising concerns about its potential to harm students’ learning motivation and learning outcomes. However, we know little about its real impact. Through quasi-experiment research with 127 Chinese middle school students, we examined the impact of completing assignments with a Large Language Model-based chatbot, wisdomBot, on middle school students’ assignment performance, learning outcomes, learning motivation, learning satisfaction, and learning experiences; we also summarized teachers’ reflections on learning design. Compared to control groups, the Large Language Model chatbot-assisted group demonstrated significantly higher assignment submission rates, word counts, and scores in assignment performance. However, they gained significantly lower scores on materials refinement and knowledge tests. No significant differences have been observed in learning motivation, satisfaction, enjoyment, and students’ ability to migrate their knowledge. The majority of students expressed satisfaction and willingness to continue using the tool. We also identified three key gaps in learning designs, including providing scaffolds for the potential prompts, suggesting group collaboration mode, and relinquishing the authoritarian of the teacher. Our findings provide insights regarding with Large Language Model-based chatbots we could better design assignment assessment tools, facilitate students’ autonomous learning, provide emotional support, propose guidelines and instructions about applying Large Language Model-based chatbots in K-12, as well as design specialized educational Large Language Model-based chatbots.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
Data will be made available on request.
References
Adams, N. E. (2015). Bloom’s taxonomy of cognitive learning objectives. Journal of the Medical Library Association: JMLA, 103(3), 152. https://doi.org/10.3163/1536-5050.103.3.010
Adarkwah, M. A., Ying, C., Mustafa, M. Y., & Huang, R. (2023, August). Prediction of Learner Information-Seeking Behavior and Classroom Engagement in the Advent of ChatGPT. In International Conference on Smart Learning Environments (pp. 117–126). Singapore: Springer Nature Singapore https://doi.org/10.1007/978-981-99-5961-7_13
Adeshola, I., & Adepoju, A. P. (2023). The opportunities and challenges of ChatGPT. Education Interactive Learning Environments, 1–14. https://doi.org/10.1080/10494820.2023.2253858
Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152
Agarwal, P. K. (2019). Retrieval practice & Bloom’s taxonomy: Do students need fact knowledge before higher order learning? Journal of Educational Psychology, 111(2), 189–209. https://doi.org/10.1037/edu0000282
Ahmad, Z., Kaiser, W., & Rahim, S. (2023). Hallucinations in ChatGPT: An unreliable tool for learning. Rupkatha Journal on Interdisciplinary Studies in Humanities, 15(4), 1–18. https://doi.org/10.21659/rupkatha.v15n4.17
Ait Baha, T., El Hajji, M., Es-Saady, Y., & Fadili, H. (2023). The impact of educational chatbot on student learning experience. Education and Information Technologies, 1–24. https://doi.org/10.1007/s10639-023-12166-w
Al-Hafdi, F. S., & AlNajdi, S. M. (2024). The effectiveness of using chatbot-based environment on learning process, students’ performances and perceptions: A mixed exploratory study. Education and Information Technologies, 1–32. https://doi.org/10.1007/s10639-024-12671-6
AlAfnan, M. A., Dishari, S., Jovic, M., & Lomidze, K. (2023). Chatgpt as an educational tool: Opportunities, challenges, and recommendations for communication, business writing, and composition courses. Journal of Artificial Intelligence and Technology, 3(2), 60–68. https://doi.org/10.37965/jait.2023.0184
Alemdag, E. (2023). The effect of chatbots on learning: A meta-analysis of empirical research. Journal of Research on Technology in Education, 1–23. https://doi.org/10.1080/15391523.2023.2255698
Ayres, P. (2006). Impact of reducing intrinsic cognitive load on learning in a mathematical domain. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition, 20(3), 287–298. https://doi.org/10.1002/acp.1245
Bencherki, N., Matte, F., & Cooren, F. (2019). Authority and power in social interaction: Methods and analysis. Routledge.
Berger, E., Sæthre, T. H., & Divitini, M. (2019). PrivaCity: A Chatbot Game to Raise Privacy Awareness Among Teenagers. In Informatics in Schools. New Ideas in School Informatics: 12th International Conference on Informatics in Schools: Situation, Evolution, and Perspectives, ISSEP 2019, Larnaca, Cyprus, November 18–20, 2019, Proceedings 12 (pp. 293–304). Springer International Publishing. https://doi.org/10.1007/978-3-030-33759-9_23
Bilgic, K., & Dogusoy, B. (2023). Exploring secondary school students’ computational thinking experiences enriched with block-based programming activities: An action research. Education and Information Technologies, 28(8), 10359–10384. https://doi.org/10.1007/s10639-023-11583-1
Buijs, M., & Admiraal, W. (2013). Homework assignments to enhance student engagement in secondary education. European Journal of Psychology of Education, 28, 767–779. https://doi.org/10.1007/s10212-012-0139-0
Cain, K., Oakhill, J., & Bryant, P. (2004). Children’s reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology, 96(1), 31. https://doi.org/10.1037/0022-0663.96.1.31
Chang, C. Y., Hwang, G. J., & Gau, M. L. (2022). Promoting students’ learning achievement and self-efficacy: A mobile chatbot approach for nursing training. British Journal of Educational Technology, 53(1), 171–188. https://doi.org/10.1111/bjet.13158
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., & Co, M. T. H. (2023). ChatGPT versus human in generating medical graduate exam multiple choice questions—A multinational prospective study (Hong Kong SAR, Singapore, Ireland, and the United Kingdom). PLoS One, 18(8), e0290691. https://doi.org/10.1371/journal.pone.0290691
Chi, S., Wang, Z., & Liu, X. (2023). Assessment of context-based chemistry problem-solving skills: Test design and results from ninth-grade students. Research in Science Education, 53(2), 295–318. https://doi.org/10.1007/s11165-022-10056-8
Chinese Ministry of Education (2022, March 25). Curriculum Standard for Comulsory Education Information Science and Technology (2022). Retrieved April 25, 2024 from: http://www.moe.gov.cn/srcsite/A26/s8001/202204/W020220420582361024968.pdf
de Winter, J. C. (2023). Can ChatGPT pass high school exams on English Language Comprehension? International Journal of Artificial Intelligence in Education, 1–16. https://doi.org/10.1007/s40593-023-00372-z
Ekambaranathan, A., Zhao, J., & Van Kleek, M. (2021, May). Money makes the world go around: Identifying Barriers to Better Privacy in Children’s Apps From Developers’ Perspectives. In Proceedings of the 2021 CHI conference on human factors in computing systems (pp. 1–15). https://doi.org/10.1145/3411764.3445599
Fryer, L. K., Nakao, K., & Thompson, A. (2019). Chatbot learning partners: Connecting learning experiences, interest and competence. Computers in human Behavior, 93, 279–289. https://doi.org/10.1016/j.chb.2018.12.023
Fujimaki, A. (2023). (n.a.). Regarding the Use of Generative AI. Retrieved October 15, from https://en.nagoya-u.ac.jp/academics/ai/index.html
Fuller, A., Morbitzer, K., Zeeman, K. A., Persky, J. M. M., Savage, A. C., A., & McLaughlin, J. E. (2024). Exploring the use of ChatGPT to analyze student course evaluation comments. BMC Medical Education, 24(1), 1–8. https://doi.org/10.1186/s12909-024-05316-2
Gabrielli, S., Rizzi, S., Carbone, S., & Donisi, V. (2020). A chatbot-based coaching intervention for adolescents to promote life skills: Pilot study. JMIR Human Factors, 7(1), e16762. https://humanfactors.jmir.org/2020/1/e16762
Gan, W., Qi, Z., Wu, J., & Lin, J. C. W. (2023, December). Large language models in education: Vision and opportunities. In 2023 IEEE International Conference on Big Data (BigData) (pp. 4776–4785). IEEE.
Grassini, S. (2023). Shaping the future of education: Exploring the potential and consequences of AI and ChatGPT in educational settings. Education Sciences, 13(7), 692. https://doi.org/10.3390/educsci13070692
Grogan, K. A. (2017). Will this be on the test? How exam structure affects perceptions of innovative assignments in a masters of science microeconomics course. International Review of Economics Education, 26, 1–8. https://doi.org/10.1016/j.iree.2017.06.001
Guo, Y., & Lee, D. (2023). Leveraging chatgpt for enhancing critical thinking skills. Journal of Chemical Education, 100(12), 4876–4883. https://doi.org/10.1021/acs.jchemed.3c00505
Guo, K., Zhong, Y., Li, D., & Chu, S. K. W. (2023). Effects of chatbot-assisted in-class debates on students’ argumentation skills and task motivation. Computers & Education, 203, 104862. https://doi.org/10.1016/j.compedu.2023.104862
Henthorn, R., Lowden, K., & McArdle, K. (2024). It gives meaning and purpose to what you do’: Mentors’ interpretations of practitioner action research in education. Educational Action Research, 32(2), 169–185. https://doi.org/10.1080/09650792.2022.2106260
Ho, D. Y. F., Peng, S. Q., & Chan, S. F. F. (2002). Authority and learning in Confucian-heritage education: A relational methodological analysis. In F. Salili, Y. Y. Hong, & C. Y. Chiu, (Eds.), Multiple competencies and self-regulated learning: Implications for multicultural education (pp. 29–47). Greenwich, CT: Information Age Publishing.
Holmes, W., & Miao, F. (2023). Guidance for generative AI in education and research. UNESCO Publishing.
Hong Kong University of Science and Technology. (n.a.). AI Literacy for End-Users: Use AI Wisely. Retrieved October 15 (2023). from https://libguides.hkust.edu.hk/ai-literacy
Hübscher-Younger, T., & Narayanan, N. H. (2003). Designing for divergence. In Designing for change in networked learning environments: Proceedings of the international conference on computer support for collaborative learning 2003 (pp. 461–470). Dordrecht: Springer Netherlands.
Jauhiainen, J. S., & Guerra, A. G. (2023). Generative AI and ChatGPT in School Children’s education: Evidence from a school lesson. Sustainability, 15(18), 14025. https://doi.org/10.3390/su151814025
Jiang, X., Tian, Y., Hua, F., Xu, C., Wang, Y., & Guo, J. (2024). A Survey on Large Language Model Hallucination via a Creativity Perspective. arXiv preprint arXiv:2402.06647. https://doi.org/10.48550/arXiv.2402.06647
Jishnu, D., Srinivasan, M., Dhanunjay, G. S., & Shamala, R. (2023). Unveiling student motivations: A study of ChatGPT usage in education. ShodhKosh: Journal of Visual and Performing Arts, 4(2), 65–73. https://doi.org/10.29121/shodhkosh.v4.i2.2023.503.
Jungherr, A. (2023). Using ChatGPT and other large language model (LLM) applications for academic paper assignments. SocArXiv. https://doi.org/10.31235/osf.io/d84q6
Kaplan-Rakowski, R., Grotewold, K., Hartwick, P., & Papin, K. (2023). Generative AI and teachers’ perspectives on its implementation in Education. Journal of Interactive Learning Research, 34(2), 313–338. https://doi.org/10.21275/SR23219122412
Karthikeyan, C. (2023). Literature Review on pros and cons of ChatGPT implications in Education. International Journal of Science and Research (IJSR), 12(3), 283–291.
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., & Dementieva, D. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274. Fischer, F.Kasneci, G.
Kumar, N. A., & Lan, A. (2024). Using large Language models for Student-Code guided Test Case Generation in Computer Science Education. arXiv Preprint arXiv:2402 07081. https://doi.org/10.48550/arXiv.2402.07081
Lan, Y. J., & Chen, N. S. (2024). Teachers’ agency in the era of LLM and generative AI. Educational Technology & Society, 27(1), I–XVIII. https://www.jstor.org/stable/48754837
Lee, H. (2023). The rise of ChatGPT: Exploring its potential in medical education. Anatomical Sciences Education. https://doi.org/10.1002/ase.2270
Lee, H. Y., Chen, P. H., Wang, W. S., Huang, Y. M., & Wu, T. T. (2024). Empowering ChatGPT with guidance mechanism in blended learning: Effect of self-regulated learning, higher-order thinking skills, and knowledge construction. International Journal of Educational Technology in Higher Education, 21(1), 1–28. https://doi.org/10.1186/s41239-024-00447-4
Lee, I., Ali, S., Zhang, H., DiPaola, D., & Breazeal, C. (2021, March). Developing middle school students’ AI literacy. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (pp. 191–197). https://doi.org/10.1145/3408877.3432513
Levin, D. A. (2021). The state of K-12 cybersecurity: 2020 year in review. K-12 cybersecurity resource center. Retrieved April 20, 2024, from https://www.k12six.org/the-report
Li, J., & Xue, E. (2021). Compulsory Education Policy in China: Concept and Practice. Springer Nature. https://doi.org/10.1007/978-981-33-6358-8
Li, Y., Chen, J., Zhou, H., Yuan, H., & Yang, R. (2023). Research on motivation and behavior of ChatGPT use in middle school students. International Journal of New Developments in Education, 5(19), 69–77. https://doi.org/10.25236/IJNDE.2023.051911
Lin, K. M. (2011). e-Learning continuance intention: Moderating effects of user e-learning experience. Computers & Education, 56(2), 515–526. https://doi.org/10.1016/j.compedu.2010.09.017
Liu, C. C., Liao, M. G., & Chang, C. H. (2022). An analysis of children’s interaction with an AI chatbot and its impact on their interest in reading. Computers & Education, 189, 104576. https://doi.org/10.1016/j.compedu.2022.104576. Lin, H. M.
Liu, M., Ren, Y., Nyagoga, L. M., Stonier, F., Wu, Z., & Yu, L. (2023). Future of education in the era of generative artificial intelligence: Consensus among Chinese scholars on applications of ChatGPT in schools. Future in Educational Research, 1(1), 72–101. https://doi.org/10.1002/fer3.10
Mahajan, V. (2023, October 13). 100 + Incredible ChatGPT Statistics & Facts in 2023. Retrieved November 12, 2023, from https://www.notta.ai/en/blog/chatgpt-statistics
Malik, A., Khan, M. L., & Hussain, K. (2023). How is ChatGPT transforming academia? Examining its impact on teaching, research, assessment, and learning. Examining its Impact on Teaching, Research, Assessment, and Learning (April 9, 2023). https://doi.org/10.2139/ssrn.4413516
Masouleh, N. S., & Jooneghani, R. B. (2012). Autonomous learning: A teacher-less learning! Procedia-Social and Behavioral Sciences, 55, 835–842. https://doi.org/10.1016/j.sbspro.2012.09.570
Meyer, J. G., Urbanowicz, R. J., & Martin, P. C. (2023). ChatGPT and large language models in academia: Opportunities and challenges. Big Data Mining, 16(1), 20. https://doi.org/10.1186/s13040-023-00339-9. O’Connor, K.Li, R., Peng, P. C., … Moore, J. H.
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., & Gao, J. (2024). Large language models: A survey. arXiv Preprint arXiv:2402 06196. https://doi.org/10.48550/arXiv.2402.06196
Moritz, S., Romeike, B., Stosch, C., & Tolks, D. (2023). Generative AI (gAI) in medical education: Chat-GPT and co. GMS Journal for Medical Education, 40(4). https://doi.org/10.3205/zma001636
Morrison, R., & Barton, G. (2018). Search engine use as a literacy in the middle years: The need for explicit instruction and active learners. Literacy Learning: The Middle Years, 26(3), 37–47. https://doi.org/10.3316/informit.846488641829827
Nanyang Technological University. (n.a.). NTU Position on the Use of Generative Artificial Intelligence in Research Retrieved October 15 (2023). from https://www.ntu.edu.sg/research/resources/use-of-gai-in-research
Nee, C. K., Rahman, M. H. A., Yahaya, N., Ibrahim, N. H., Razak, R. A., & Sugino, C. (2023). Exploring the Trend and potential distribution of Chatbot in Education: A systematic review. International Journal of Information and Education Technology, 13(3), 516–525. https://doi.org/10.18178/ijiet.2023.13.3.1834
Nowicki, J. M. (2020). Data Security: Recent K-12 Data Breaches Show That Students Are Vulnerable to Harm. Report to the Republican Leader, Committee on Education and Labor, House of Representatives. GAO-20-644. US Government Accountability Office. Retrieved April 21, 2024, from https://files.eric.ed.gov/fulltext/ED609671.pdf
Ofcom (2023, November 28). Gen Z driving early adoption of Gen AI, our latest research shows Retrieved November 29, 2023, from https://www.ofcom.org.uk/news-centre/2023/gen-z-driving-early-adoption-of-gen-ai
Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033
Olney, A. M. (2023, June). Generating multiple choice questions from a textbook: LLMs match human performance on most metrics. In AIED Workshops. https://ceur-ws.org/Vol-3487/paper7.pdf
Oxford English Dictionary (2023, July). s.v. chatbot, n Retrieved November 20, 2023, from https://doi.org/10.1093/OED/2981785869
Özer, B., Duran, V., & Tekke, M. (2020). Training of trainers: An action-based research for improving the Pedagogical skills of academicians. International Journal of Evaluation and Research in Education, 9(3), 704–715. https://doi.org/10.11591/ijere.v9i3.20327
Park, S. (2023). C. Kulkarni (Ed.), Thinking assistants: LLM-Based conversational assistants that help users think by asking rather than answering. arXiv Preprint arXiv:2312 06024https://doi.org/10.48550/arXiv.2312.06024
Pedaste, M., Mäeots, M., Siiman, L. A., De Jong, T., Van Riesen, S. A., Kamp, E. T., & Tsourlidaki, E. (2015). Phases of inquiry-based learning: Definitions and the inquiry cycle. Educational Research Review, 14, 47–61. https://doi.org/10.1016/j.edurev.2015.02.003
Perkins, M. (2023). Academic Integrity considerations of AI large Language models in the post-pandemic era: ChatGPT and beyond. Journal of University Teaching & Learning Practice, 20(2). https://doi.org/10.53761/1.20.02.07
Playfoot, D., Quigley, M., & Thomas, A. G. (2024). Hey ChatGPT, give me a title for a paper about degree apathy and student use of AI for assignment writing. The Internet and Higher Education, 100950. https://doi.org/10.1016/j.iheduc.2024.100950
Rillig, M. C., Ågerstrand, M., Bi, M., Gould, K. A., & Sauerland, U. (2023). Risks and benefits of large language models for the environment. Environmental Science & Technology, 57(9), 3464–3466. https://doi.org/10.1021/acs.est.3c01106
Ruan, S., Willis, A., Xu, Q., Davis, G. M., & Jiang, L. (2019, June). Brunskill, E., & Landay, J. A. Bookbuddy: Turning digital materials into interactive foreign language lessons through a voice chatbot. In Proceedings of the sixth (2019) ACM conference on learning@ scale (pp.1–4). https://doi.org/10.1145/3330430.3333643
Ruzek, E. A., Hafen, C. A., Allen, J. P., Gregory, A., Mikami, A. Y., & Pianta, R. C. (2016). How teacher emotional support motivates students: The mediating roles of perceived peer relatedness, autonomy support, and competence. Learning and Instruction, 42, 95–103. https://doi.org/10.1016/j.learninstruc.2016.01.004
Shneiderman, B. (1984). Response time and display rate in human performance with computers. ACM Computing Surveys (CSUR), 16(3), 265–285.
Shoufan, A. (2023). Exploring students’ perceptions of ChatGPT: Thematic analysis and follow-up survey. IEEE Access, 11, 38805–38818. https://doi.org/10.1109/ACCESS.2023.3268224.
Sjödahl Hammarlund, C., Nordmark, E., & Gummesson, C. (2013). Integrating theory and practice by self-directed inquiry-based learning? A pilot study. The European Journal of Physiotherapy, 15(4), 225–230. https://doi.org/10.3109/21679169.2013.836565
Sovrano, F., Ashley, K., & Bacchelli, A. (2023, July). Toward eliminating hallucinations: Gpt-based explanatory AI for intelligent textbooks and documentation. In CEUR Workshop Proceedings (pp. 54–65). CEUR-WS. https://ceur-ws.org/Vol-3444/itb23_s3p2.pdf
Strzelecki, A. (2023). To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interactive Learning Environments, 1–14. https://doi.org/10.1080/10494820.2023.2209881
Taasoobshirazi, G., & Carr, M. (2008). A review and critique of context-based physics instruction and assessment. Educational Research Review, 3(2), 155–167. https://doi.org/10.1016/j.edurev.2008.01.002
Tamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021). Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503. https://doi.org/10.48550/arXiv.2103.14411
Tanwar, H., Shrivastva, K., Singh, R., & Kumar, D. (2024). OpineBot: Class Feedback Reimagined Using a Conversational LLM. arXiv preprint arXiv:2401.15589. https://doi.org/10.48550/arXiv.2401.15589
The University of North Carolina at Chapel Hill (n.a.) Teaching About The Use Of Generative AI Guidance For Instructors Retrieved October 19 (2023). from: https://provost.unc.edu/teaching-generative-ai-guidance/
Theophilou, E., Koyutürk, C., Yavari, M., Bursic, S., Donabauer, G., & Telari, A. (2023, November). … Ognibene, D. Learning to Prompt in the Classroom to Understand AI Limits: A pilot study. In International Conference of the Italian Association for Artificial Intelligence (pp. 481–496). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-47546-7_33
University of OXFORD (2023, 6 February). Unauthorised use of AI in exams and assessment Retrieved October 15, 2023, from https://academic.admin.ox.ac.uk/article/unauthorised-use-of-ai-in-exams-and-assessment
van Duijn, M. J., van Dijk, B., Kouwenhoven, T., de Valk, W., Spruit, M. R., & van der Putten, P. (2023). Theory of mind in large language models: Examining performance of 11 state-of-the-art models vs. children aged 7–10 on advanced tests. arXiv preprint arXiv:2310.20320. https://doi.org/10.48550/arXiv.2310.20320
Van Merrienboer, J. J., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17, 147–177. https://doi.org/10.1007/s10648-005-3951-0
Waltzer, T., Cox, R. L., & Heyman, G. D. (2023). Testing the ability of teachers and students to Differentiate between essays generated by ChatGPT and High School Students. Human Behavior and Emerging Technologies, 1923981, 1–9. https://doi.org/10.1155/2023/1923981
Wang, L., Chen, X., Wang, C., Xu, L., Shadiev, R., & Li, Y. (2024). ChatGPT’s capabilities in providing feedback on undergraduate students’ argumentation: A case study. Thinking Skills and Creativity, 51, 101440. https://doi.org/10.1016/j.tsc.2023.101440
Wei, T. (2022). An interpretation of the revised compulsory education curriculum program and standards: A revolution in China’s compulsory education. Science Insights Education Frontiers, 13(1), 1845–1853. https://doi.org/10.15354/sief.22.re065
Welding, L. (2023, March 17). Half of College Students Say Using AI on Schoolwork Is Cheating or Plagiarism. Retrieved October 13, 2023, from https://www.bestcolleges.com/research/college-students-ai-tools-survey/
Wentzel, K. R., Russell, S., & Baker, S. (2016). Emotional support and expectations from parents, teachers, and peers predict adolescent competence at school. Journal of Educational Psychology, 108(2), 242. https://doi.org/10.1037/edu0000049
Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are we there yet?-a systematic literature review on chatbots in education. Frontiers in Artificial Intelligence, 4, 654924. https://doi.org/10.3389/frai.2021.654924
Wu, R., & Yu, Z. (2024). Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis. British Journal of Educational Technology, 55(1), 10–33. https://doi.org/10.1111/bjet.13334
Wu, T. T., Lee, H. Y., Li, P. H., Huang, C. N., & Huang, Y. M. (2024). Promoting self-regulation progress and knowledge construction in blended learning via ChatGPT-based learning aid. Journal of Educational Computing Research, 61(8), 3–31. https://doi.org/10.1177/07356331231191125
Xiao, C., Xu, S. X., Zhang, K., Wang, Y., & Xia, L. (2023, July). Evaluating reading comprehension exercises generated by LLMs: A showcase of ChatGPT in education applications. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) (pp. 610–625). https://aclanthology.org/2023.bea-1.52
Yan, S. (2012). Teachers’ roles in autonomous learning. Journal of Sociological Research, 3(2), 557–562. https://doi.org/10.5296/jsr.v3i2.2860
Yang, R. Research on motivation and behavior of ChatGPT use in middle school students. International Journal of New Developments in Education 5(19): 69–77. https://doi.org/10.25236/IJNDE.2023.051911
You, Y., Kou, Y., Ding, X., & Gui, X. (2021, May). The medical authority of AI: A study of AI-enabled consumer-facing health technology. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–16). https://doi.org/10.1145/3411764.3445657
Youth League (2022, November). 2021 National Research Report on Internet Use by Minors. Retrieved November 1, 2023, from: https://www.cagd.gov.cn/data/uploads//ueditor/php/upload/file/2022/11/1669791740317797.pdf
Yu, H. (2023). Reflection on whether Chat GPT should be banned by academia from the perspective of education and teaching. Frontiers in Psychology, 14, 1181712. https://doi.org/10.3389/fpsyg.2023.1181712
Zha, S., Qiao, Y., Hu, Q., Li, Z., Gong, J., & Xu, Y. (2024). Designing Child-Centric AI Learning Environments: Insights from LLM-Enhanced Creative Project-Based Learning. arXiv preprint arXiv:2403.16159. https://doi.org/10.48550/arXiv.2403.16159
Zhai, X. (2022). ChatGPT user experience: Implications for education. Available at SSRN 4312418. https://doi.org/10.2139/ssrn.4312418
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., & Wen, J. R. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223https://doi.org/10.48550/arXiv.2303.18223
Acknowledgements
The authors would like to acknowledge the funding support of National Science and Technology Major Project (2022ZD0115904).
Author information
Authors and Affiliations
Contributions
Yumeng Zhu: Investigation, Data curation, Writing, Methodology, Visualisation. Caifeng Zhu: Investigation, Data curation, Writing. Tao Wu: Software and Writing-Reviewing. Shulei Wang: Software. Yiyun Zhou: Software. Jingyuan Chen: Writing-Reviewing and Editing. Wei Fu: Writing-Reviewing and Editing. Yan Li: Conceptualization, Investigation, Supervision, Writing- Reviewing and Editing.
Corresponding author
Ethics declarations
Submission declaration and verification
This work described has not been published previously.
Competing interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Full version of assignments scoring rubric.
Dimension\score | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
Completion: Assess the extent to which the task is fully completed. | Significant elements are missing or incomplete. | Several key components are incomplete. | Most elements are complete, but some are missing. | Nearly all elements are complete, with minor omissions. | The task is fully and thoroughly completed. |
Creativity: Evaluate the level of creativity demonstrated in the work. | Little to no evidence of original thinking or creativity. | Limited creativity; ideas are somewhat derivative. | Some original elements, but overall lacking creativity. | Demonstrates creativity in certain aspects. | Exceptional creativity are evident throughout. |
Accuracy: Examine the correctness of the content or information presented. | Numerous inaccuracies or errors in content. | Several factual inaccuracies are present. | Few inaccuracies; some minor errors. | Rare inaccuracies, with only minor errors. | All content is accurate and error-free. |
Material Refinement: Assess the degree of secondary processing or refinement of materials(from wisdomBot or search engine). | Little to no evidence of secondary processing. | Minimal efforts in refining materials. | Some attempts at processing or refining materials. | Effective use of secondary processing in certain areas. | Comprehensive and skillful refinement of materials. |
Depth: Assess the extent to which the content demonstrates thorough understanding. | Content lacks depth and is superficial, with little elaboration or analysis. | Limited depth, with some aspects lacking elaboration or analysis. | Moderate depth, with most aspects adequately elaborated and analyzed. | Substantial depth, with thorough elaboration and analysis of most aspects. | Exceptional depth, demonstrating comprehensive understanding and insightful analysis throughout. |
Logic: Evaluate the logical flow and coherence of the work. | The work lacks a logical structure or flow. | Major inconsistencies disrupt the logical flow. | Overall logical flow, but with some disruptions. | Logical progression with minimal disruptions. | A highly logical and coherent presentation. |
Page Aesthetics: Consider the visual attractiveness and formatting skills in creating the document. | The document lacks visual appeal and proper formatting. | Minimal attention to aesthetics and formatting. | Adequate visual appeal, but some formatting issues. | Well-designed with minor formatting improvements possible. | Visually appealing, with excellent formatting skills demonstrated. |
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhu, Y., Zhu, C., Wu, T. et al. Impact of assignment completion assisted by Large Language Model-based chatbot on middle school students’ learning. Educ Inf Technol 30, 2429–2461 (2025). https://doi.org/10.1007/s10639-024-12898-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-024-12898-3