Abstract
With the rapid development of Artificial Intelligence, automatic writing evaluation (AWE) has received much attention from English Foreign Language (EFL) writing teachers. However, the obstacles and potential problems of integrating AWE in EFL writing instruction have yet to be explored. Scholars have indicated that the effectiveness of AWE in EFL writing instruction depends on the learners' depth of reflection. Hence, this study proposes a learning approach that integrates AWE and peer assessment (PA) based on the knowledge-building theory, with the expectation that learners will be able to strengthen their reflections on AWE feedback through PA, and thereby improve their EFL writing performance. To examine the effectiveness of the proposed approach, a quasi-experiment was conducted in a university EFL writing class. One of the classes (33 students) was the experimental group using the PA-AWE approach, and the other class (31 students) was a control group that studied using the conventional AWE approach (C-AWE approach). Findings revealed that the PA-AWE group outperformed the C-AWE group regarding EFL writing performance, learning motivation, critical thinking, and reduced EFL writing anxiety. In addition, a thematic inductive qualitative analysis of the interview data indicated each approach's benefits and learning conceptions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data and materials are available upon request to the corresponding author.
Code availability
Not applicable
References
Andrade, H., & Du, Y. (2005). Student perspectives on rubric-referenced assessment. Practical Assessment, Research, and Evaluation, 10(1), 3. https://doi.org/10.7275/g367-ye94
Arnold, N., Ducate, L., & Kost, C. (2012). Collaboration or cooperation? Analyzing group dynamics and revision processes in wikis. Calico Journal, 29(3), 431–448. https://doi.org/10.11139/cj.29.3.431-448
Bai, L., & Hu, G. (2017). In the face of fallible AWE feedback: How do students respond? Educational Psychology, 37(1), 67–81. https://doi.org/10.1080/01443410.2016.1223275
Baker, B. A., Homayounzadeh, M., & Arias, A. (2020). Development of a test taker-oriented rubric: Exploring its usefulness for test preparation and writing development. Journal of Second Language Writing, 50, 100771. https://doi.org/10.1016/j.jslw.2020.100771
Barrot, J. S., & Agdeppa, J. Y. (2021). Complexity, accuracy, and fluency as indices of college-level L2 writers’ proficiency. Assessing Writing, 47, 100510. https://doi.org/10.1016/j.asw.2020.100510
Bitchener, J. (2012). A reflection on ‘the language learning potential’ of written. Journal of Second Language Writing, 21(4), 348–363. https://doi.org/10.1016/j.jslw.2012.09.006
Calvo, R. A., & Ellis, R. A. (2010). Students’ conceptions of tutor and automated feedback in professional writing. Journal of Engineering Education, 99(4), 427–438. https://doi.org/10.1002/j.2168-9830.2010.tb01072.x
Challob, A. I. (2021). The effect of flipped learning on EFL students’ writing performance, autonomy, and motivation. Education and Information Technologies, 26(4), 3743–3769.
Chang, C. F. (2012). Peer review via three modes in an EFL writing course. Computers and Composition, 29(1), 63–78. https://doi.org/10.1016/j.compcom.2012.01.001
Chang, S. C., Hsu, T. C., & Jong, M. S. Y. (2020). Integration of the peer assessment approach with a virtual reality design system for learning earth science. Computers & Education, 146, 103758. https://doi.org/10.1016/j.compedu.2019.103758
Chen, C. H. (2010). The implementation and evaluation of a mobile self-and peer-assessment system. Computers & Education, 55(1), 229–236. https://doi.org/10.1016/j.compedu.2010.01.008
Cheng, K. H., Liang, J. C., & Tsai, C. C. (2015). Examining the role of feedback messages in undergraduate students’ writing performance during an online peer assessment activity. The Internet and Higher Education, 25, 78–84. https://doi.org/10.1016/j.iheduc.2015.02.001
Cheng, Y. S. (2017). Development and preliminary validation of four brief measures of L2 language-skill-specific anxiety. System, 68, 15–25. https://doi.org/10.1016/j.system.2017.06.009
Cho, K., Schunn, C. D., & Wilson, R. W. (2006). Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. Journal of Educational Psychology, 98(4), 891. https://doi.org/10.1037/0022-0663.98.4.891
Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving comments. Instructional Science, 39(5), 629–643. https://doi.org/10.1007/s11251-010-9146-1
Cohen, J. (1988). Statistical power analysis for the social sciences (2nd ed.). Lawrence Erlbaum Associates.
Cotos, E. (2015). AWE for writing pedagogy: From healthy tension to tangible prospects. Special issue on assessment for writing and pedagogy. Writing & Pedagogy, 7(2–3), 197–231. https://doi.org/10.1558/wap.v7i2-3.26381
Creswell, J. W., & Poth, C. N. (2016). Qualitative inquiry and research design: Choosing among five approaches. Sage Publications. https://doi.org/10.1177/1524839915580941
Diab, N. M. (2010). Effects of peer-versus self-editing on students’ revision of language errors in revised drafts. System, 38(1), 85–95. https://doi.org/10.1016/j.system.2009.12.008
Dikli, S., & Bleyle, S. (2014). Automated essay scoring feedback for second language writers: How does it compare to instructor feedback? Assessing Writing, 22, 1–17. https://doi.org/10.1016/j.asw.2014.03.006
Ebadi, S., & Rahimi, M. (2017). Exploring the impact of online peer-editing using Google Docs on EFL learners’ academic writing skills: A mixed methods study. Computer Assisted Language Learning, 30(8), 787–815. https://doi.org/10.1080/09588221.2017.1363056
Ferris, D. R. (2010). Second language writing research and written corrective feedback in SLA: Intersections and practical applications. Studies in Second Language Acquisition, 32(2), 181–201. https://doi.org/10.1017/S0272263109990490
Filippou, K., Kallo, J., & Mikkilä-Erdmann, M. (2021). Supervising master’s theses in international master’s degree programmes: Roles, responsibilities and models. Teaching in Higher Education. https://doi.org/10.1080/13562517.2019.1636220
Fu, Q. K., Zou, D., Xie, H., & Cheng, G. (2022). A review of AWE feedback: types, learning outcomes, and implications. Computer Assisted Language Learning, 1–43. https://doi.org/10.1080/09588221.2022.2033787
Grimes, D., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. The Journal of Technology, Learning and Assessment, 8(6). Retrieved from https://ejournals.bc.edu/index.php/jtla/article/view/1625
Guardado, M., & Shi, L. (2007). ESL students’ experiences of online peer feedback. Computers and Composition, 24(4), 443–461. https://doi.org/10.1016/j.compcom.2007.03.002
Guo, Q., Feng, R., & Hua, Y. (2021). How effectively can EFL students use automated written corrective feedback (AWCF) in research writing?. Computer Assisted Language Learning, 1–20. https://doi.org/10.1080/09588221.2021.1879161
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
Ho, M. C., & Savignon, S. J. (2007). Face-to-face and computer-mediated peer review in EFL writing. CALICO Journal, 269–290. https://doi.org/10.1558/cj.v24i2.269-290
Hwang, G. J., Yang, T. C., Tsai, C. C., & Yang, S. J. H. (2009). A context-aware ubiquitous learning environment for conducting complex science experiments. Computers & Education, 53(2), 402–413. https://doi.org/10.1016/j.compedu.2009.02.016
Hyland, K., & Hyland, F. (2006). Feedback on second language students writing. Language Teaching, 39(2), 83–101. https://doi.org/10.1017/S0261444806003399
Jeong, A. C. (2003). The sequential analysis of group interaction and critical thinking in online. The American Journal of Distance Education, 17(1), 25–43. https://doi.org/10.1207/S15389286AJDE1701_3
Jiang, J. P., Hu, J. Y., Zhang, Y. B., & Yin, X. C. (2022). Fostering college students’ critical thinking skills through peer assessment in the knowledge building community. Interactive Learning Environments, 1–17. https://doi.org/10.1080/10494820.2022.2039949
Jin, T., Jiang, Y., Gu, M. M., & Chen, J. (2022). “Their encouragement makes me feel more confident”: Exploring peer effects on learner engagement in collaborative reading of academic texts. Journal of English for Academic Purposes, 101177. https://doi.org/10.1016/j.jeap.2022.101177
Jones, R. H., Garralda, A., Li, D. C., & Lock, G. (2006). Interactional dynamics in on-line and face-to-face peer-tutoring sessions for second language writers. Journal of Second Language Writing, 15(1), 1–23. https://doi.org/10.1016/j.jslw.2005.12.001
Lai, C. L., & Hwang, G. J. (2015). An interactive peer-assessment criteria development approach to improving students’ art design performance using handheld devices. Computers & Education, 85, 149–159. https://doi.org/10.1016/j.compedu.2015.02.011
Latifi, S., & Noroozi, O. (2021). Supporting argumentative essay writing through an online supported peer-review script. Innovations in Education and Teaching International, 58(5), 501–511. https://doi.org/10.1080/14703297.2021.1961097
Latifi, S., Noroozi, O., Hatami, J., & Biemans, H. J. (2021). How does online peer feedback improve argumentative essay writing and learning? Innovations in Education and Teaching International, 58(2), 195–206. https://doi.org/10.1080/14703297.2019.1687005
Li, H., Xiong, Y., Hunter, C. V., Guo, X., & Tywoniw, R. (2020). Does peer assessment promote student learning? A meta-analysis. Assessment & Evaluation in Higher Education, 45(2), 193–211. https://doi.org/10.1080/02602938.2019.1620679
Li, J., Huang, J., & Cheng, S. (2022). The reliability, effectiveness, and benefits of peer assessment in college EFL speaking classrooms: Student and teacher perspectives. Studies in Educational Evaluation, 72, 101120. https://doi.org/10.1016/j.stueduc.2021.101120
Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing, 27, 1–18. https://doi.org/10.1016/j.jslw.2014.10.004
Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536. https://doi.org/10.1111/j.1467-8535.2009.00968.x
Liao, H. C. (2016). Using automated writing evaluation to reduce grammar errors in writing. Elt Journal, 70(3), 308–319. https://doi.org/10.1093/elt/ccv058
Lin, H. C., Hwang, G. J., Chang, S. C., & Hsu, Y. D. (2021). Facilitating critical thinking in decision making-based professional training: An online interactive peer-review approach in a flipped learning context. Computers & Education, 173, 104266. https://doi.org/10.1016/j.compedu.2021.104266
Lin, Y. N., Hsia, L. H., Sung, M. Y., & Hwang, G. H. (2019). Effects of integrating mobile technology-assisted peer assessment into flipped learning on students’ dance skills and self-efficacy. Interactive Learning Environments, 27(8), 995–1010. https://doi.org/10.1080/10494820.2018.1461115
Link, S., Mehrzad, M., & Rahimi, M. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 35(4), 605–634. https://doi.org/10.1080/09588221.2020.1743323
Liou, H. C., & Peng, Z. Y. (2009). Training effects on computer-mediated peer review. System, 37(3), 514–525. https://doi.org/10.1016/j.system.2009.01.005
Liu, C., Hou, J., Tu, Y. F., Wang, Y., & Hwang, G. J. (2021). Incorporating a reflective thinking promoting mechanism into artificial intelligence-supported English writing environments. Interactive Learning Environments, 1–19. https://doi.org/10.1080/10494820.2021.2012812
Ma, Q. (2020). Examining the role of inter-group peer online feedback on wiki writing in an EAP context. Computer Assisted Language Learning, 33(3), 197–216. https://doi.org/10.1080/09588221.2018.1556703
Matsumura, S., & Hann, G. (2004). Computer anxiety and students’ preferred feedback methods in EFL writing. The Modern Language Journal, 88(3), 403–415. https://doi.org/10.1111/j.0026-7902.2004.00237.x
Noroozi, O., & Hatami, J. (2018). The effects of online peer feedback and epistemic beliefs on students’ argumentation-based learning. Innovations in Education and Teaching International. https://doi.org/10.1080/14703297.2018.1431143
Noroozi, O., Hatami, J., Bayat, A., van Ginkel, S., Biemans, H. J., & Mulder, M. (2020). Students’ online argumentative peer feedback, essay writing, and content learning: Does gender matter? Interactive Learning Environments, 28(6), 698–712. https://doi.org/10.1080/10494820.2018.1543200
Panadero, E., & Alqassab, M. (2019). An empirical review of anonymity effects in peer assessment, peer feedback, peer review, peer evaluation and peer grading. Assessment & Evaluation in Higher Education. https://doi.org/10.1080/02602938.2019.1600186
Pham, H. T. P. (2020). Computer-mediated and face-to-face peer feedback: student feedback and revision in EFL writing. Computer Assisted Language Learning, 1–37. https://doi.org/10.1080/09588221.2020.1868530
Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it? Computer Assisted Language Learning, 31(7), 653–674. https://doi.org/10.1080/09588221.2018.1428994
Ranalli, J., Link, S., & Chukharev-Hudilainen, E. (2017). Automated writing evaluation for formative assessment of second language writing: Investigating the accuracy and usefulness of feedback as part of argument-based validation. Educational Psychology, 37(1), 8–25. https://doi.org/10.1080/01443410.2015.1136407
Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3(3), 265–283. https://doi.org/10.1207/s15327809jls0303_3
Schunn, C., Godley, A., & DeMartino, S. (2016). The reliability and validity of peer review of writing in high school AP English classes. Journal of Adolescent & Adult Literacy, 60(1), 13–23. https://doi.org/10.1002/jaal.525
Seifert, T., & Feliks, O. (2019). Online self-assessment and peer-assessment as a tool to enhance student-teachers’ assessment skills. Assessment & Evaluation in Higher Education, 44(2), 169–185. https://doi.org/10.1080/02602938.2018.1487023
Shang, H. F. (2022). Exploring online peer feedback and automated corrective feedback on EFL writing performance. Interactive Learning Environments, 30(1), 4–16. https://doi.org/10.1080/10494820.2019.1629601
Shih, R. C. (2011). Can Web 2.0 technology assist college students in learning English writing? Integrating Facebook and peer assessment with blended learning. Australasian Journal of Educational Technology, 27(5). https://doi.org/10.14742/ajet.934
Shintani, N. (2016). The effects of computer-mediated synchronous and asynchronous direct corrective feedback on writing: A case study. Computer Assisted Language Learning, 29(3), 517–538. https://doi.org/10.1080/09588221.2014.993400
Stevenson, M., & Phakiti, A. (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing, 19, 51–65. https://doi.org/10.1016/j.asw.2013.11.007
Tan, S., Cho, Y. W., & Xu, W. (2022). Exploring the effects of automated written corrective feedback, computer-mediated peer feedback and their combination mode on EFL learner’s writing performance. Interactive Learning Environments, 1–11. https://doi.org/10.1080/10494820.2022.2066137
Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249–276. https://doi.org/10.3102/00346543068003249
Tsivitanidou, O. E., Zacharia, Z. C., & Hovardas, T. (2011). Investigating secondary school students’ unmediated peer assessment skills. Learning and Instruction, 21(4), 506–519. https://doi.org/10.1016/j.learninstruc.2010.08.002
Tuzi, F. (2004). The impact of e-feedback on the revisions of L2 writers in an academic writing course. Computers and Composition, 21(2), 217–235. https://doi.org/10.1016/j.compcom.2004.02.003
Van Beuningen, C. G., De Jong, N. H., & Kuiken, F. (2012). Evidence on the effectiveness of comprehensive error correction in second language writing. Language Learning, 62(1), 1–41. https://doi.org/10.1111/j.1467-9922.2011.00674.x
Wang, L. C., & Chen, M. P. (2010). The effects of game strategy and preference-matching on flow experience and programming performance in game-based learning. Innovations in Education and Teaching International, 47(1), 39–52. https://doi.org/10.1080/14703290903525838
Wang, Y. J., Shang, H. F., & Briody, P. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing. Computer Assisted Language Learning, 26(3), 234–257. https://doi.org/10.1080/09588221.2012.655300
Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal, 3(1), 22–36. https://doi.org/10.1080/15544800701771580
Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 157–180. https://doi.org/10.1191/1362168806lr190oa
Wu, W. C. V., Petit, E., & Chen, C. H. (2015). EFL writing revision with blind expert and peer review using a CMC open forum. Computer Assisted Language Learning, 28(1), 58–80. https://doi.org/10.1080/09588221.2014.937442
Yang, Y. F. (2011). A reciprocal peer review system to support college students’ writing. British Journal of Educational Technology, 42(4), 687–700. https://doi.org/10.1111/j.1467-8535.2010.01059.x
Zhai, N., & Ma, X. (2021). Automated writing evaluation (AWE) feedback: A systematic investigation of college students’ acceptance. Computer Assisted Language Learning, 1–26. https://doi.org/10.1080/09588221.2021.1897019
Zhang, Z. V., & Hyland, K. (2018). Student engagement with teacher and automated feedback on L2 writing. Assessing Writing, 36, 90–102. https://doi.org/10.1016/j.asw.2018.02.004
Funding
This research received grants from the National Office for Educational Sciences Planning (NO. BCA210086).
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Chen-Chen Liu, Shi-Jie Liu and Naini Wang. Project administration were performed by Gwo-Jen Hwang, Yun-Fang Tu and Youmei Wang. Methodology and supervision were performed Gwo-Jen Hwang and Yun-Fang Tu. The first draft of the manuscript was written by Chen-Chen Liu and Shi-Jie Liu. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval
The ethical requirements for research in this selected university were followed.
Consent to participate
The participants all agreed to take part in this study.
Consent for publication
The publication of this study has been approved by all authors.
Conflicts of interest/Competing interests
There is no potential conflict of interest in this study.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Liu, CC., Liu, SJ., Hwang, GJ. et al. Engaging EFL students’ critical thinking tendency and in-depth reflection in technology-based writing contexts: A peer assessment-incorporated automatic evaluation approach. Educ Inf Technol 28, 13027–13052 (2023). https://doi.org/10.1007/s10639-023-11697-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-023-11697-6