M Parvez Rashid
ABSTRACT
Engaging students in the peer review process has been recognized as a valuable educational tool. It not only nurtures a collaborative learning environment where reviewees receive timely and rich feedback but also enhances the reviewer’s critical thinking skills and encourages reflective self-evaluation. However, a common concern arises when students encounter misaligned or conflicting feedback. Not only can such feedback confuse students; but it can also make it difficult for the instructor to rely on the reviews when assigning a score to the work. Addressing this pressing issue, our paper introduces an innovative, AI-assisted approach that is designed to detect and highlight disagreements within formative feedback. We’ve harnessed extensive data from 170 students, analyzing 15,500 instances of peer feedback from a software development course. By utilizing clustering techniques coupled with sophisticated natural language processing (NLP) models, we transform feedback into distinct feature vectors to pinpoint disagreements. The findings from our study underscore the effectiveness of our approach in enhancing text representations to significantly boost the capability of clustering algorithms in discerning disagreements in feedback. These insights bear implications for educators and software development courses, offering a promising route to streamline and refine the peer review process for the betterment of student learning outcomes.
- Ali Darvishi, Hassan Khosravi, Solmaz Abdi, Shazia Sadiq, and Dragan Gašević. 2022. Incorporating training, self-monitoring and AI-assistance to improve peer feedback quality. In Proceedings of the Ninth ACM Conference on Learning@ Scale. 35–47.Google Scholar
Digital Library
- Ali Darvishi, Hassan Khosravi, Afshin Rahimi, Shazia Sadiq, and Dragan Gašević. 2022. Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models. IEEE Transactions on Learning Technologies 16, 1 (2022), 106–120.Google ScholarDigital Library
- Ali Darvishi, Hassan Khosravi, and Shazia Sadiq. 2021. Employing peer review to evaluate the quality of student generated content at scale: A trust propagation approach. In Proceedings of the eighth ACM conference on learning@ scale. 139–150.Google ScholarDigital Library
- Ali Darvishi, Hassan Khosravi, Shazia Sadiq, and Dragan Gašević. 2022. Incorporating AI and learning analytics to build trustworthy peer assessment systems. British Journal of Educational Technology 53, 4 (2022), 844–875.Google ScholarCross Ref
- Michel Galley, Kathleen McKeown, Julia Bell Hirschberg, and Elizabeth Shriberg. 2004. Identifying agreement and disagreement in conversational speech: Use of bayesian networks to model pragmatic dependencies. (2004).Google Scholar
- Edward F Gehringer. 2014. A survey of methods for improving review quality. In International Conference on Web-Based Learning. Springer, 92–97.Google ScholarCross Ref
- Renchu Guan, Hao Zhang, Yanchun Liang, Fausto Giunchiglia, Lan Huang, and Xiaoyue Feng. 2020. Deep feature-based text clustering and its explanation. IEEE Transactions on Knowledge and Data Engineering (2020).Google ScholarCross Ref
- Dustin Hillard, Mari Ostendorf, and Elizabeth Shriberg. 2003. Detection of agreement vs. disagreement in meetings: Training with unlabeled data. In Companion Volume of the Proceedings of HLT-NAACL 2003-Short Papers. 34–36.Google ScholarDigital Library
- Sushant Hiray and Venkatesh Duppada. 2017. Agree to disagree: Improving disagreement detection with dual GRUs. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 147–152.Google ScholarCross Ref
- Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, 2003. The ICSI meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., Vol. 1. IEEE, I–I.Google Scholar
- Supakpong Jinarat, Bundit Manaskasemsak, and Arnon Rungsawang. 2018. Short text clustering based on word semantic graph with word embedding model. In 2018 Joint 10th International Conference on Soft Computing and Intelligent Systems (SCIS) and 19th International Symposium on Advanced Intelligent Systems (ISIS). IEEE, 1427–1432.Google ScholarCross Ref
- Hassan Khosravi, Simon Buckingham Shum, Guanliang Chen, Cristina Conati, Yi-Shan Tsai, Judy Kay, Simon Knight, Roberto Martinez-Maldonado, Shazia Sadiq, and Dragan Gašević. 2022. Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence 3 (2022), 100074.Google ScholarCross Ref
- Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications.Google Scholar
- Heng Luo, Anthony Robinson, and Jae-Young Park. 2014. Peer grading in a MOOC: Reliability, validity, and perceived effects. Online Learning Journal 18, 2 (2014).Google Scholar
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26 (2013).Google Scholar
- Marie-Francine Moens, Erik Boiy, Raquel Mochales Palau, and Chris Reed. 2007. Automatic detection of arguments in legal texts. In Proceedings of the 11th international conference on Artificial intelligence and law. 225–230.Google ScholarDigital Library
- Saif M Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Transactions on Internet Technology (TOIT) 17, 3 (2017), 1–23.Google ScholarDigital Library
- Raquel Mochales Palau and Marie-Francine Moens. 2009. Argumentation mining: the detection, classification and structure of arguments in text. In Proceedings of the 12th international conference on artificial intelligence and law. 98–107.Google ScholarDigital Library
- Ernesto Panadero and Maryam Alqassab. 2019. An empirical review of anonymity effects in peer assessment, peer feedback, peer review, peer evaluation and peer grading. Assessment & Evaluation in Higher Education 44, 8 (2019), 1253–1278.Google ScholarCross Ref
- M Parvez Rashid, Edward F Gehringer, Mitchell Young, Divyang Doshi, Qinjin Jia, and Yunkai Xiao. 2021. Peer Assessment Rubric Analyzer: An NLP approach to analyzing rubric items for better peer-review. In 2021 19th International Conference on Information Technology Based Higher Education and Training (ITHET). IEEE, 1–9.Google Scholar
- M Parvez Rashid, Yunkai Xiao, and Edward F Gehringer. 2022. Going beyond" Good Job": Analyzing Helpful Feedback from the Student’s Perspective.International Educational Data Mining Society (2022).Google Scholar
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).Google Scholar
- Sara Rosenthal and Kathleen McKeown. 2015. I couldn’t agree more: The role of conversational structure in agreement and disagreement detection in online discussions. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 168–177.Google ScholarCross Ref
- Burr Settles. 2012. Active learning. Synthesis lectures on artificial intelligence and machine learning 6, 1 (2012), 1–114.Google Scholar
- Nan Sun, Dongsheng Liu, Anding Zhu, Yahui Chen, and Yufei Yuan. 2019. Do Airbnb’s “Superhosts” deserve the badge? An empirical study from China. Asia Pacific Journal of Tourism Research 24, 4 (2019), 296–313.Google ScholarCross Ref
- Keith Topping. 2017. Peer assessment: Learning by judging and discussing the work of other learners. Interdisciplinary Education and Psychology 1, 1 (2017), 1–17.Google ScholarCross Ref
- Jie Yin, Nalin Narang, Paul Thomas, and Cecile Paris. 2012. Unifying local and global agreement and disagreement classification in online debates. In Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis. 61–69.Google ScholarDigital Library
Index Terms
- Navigating (Dis)agreement: AI Assistance to Uncover Peer Feedback Discrepancies
Recommendations
Strengthening dialogic peer feedback aiming for deep learning in SPOCs
AbstractThis study is focused on how peer feedback in SPOCs (Small Private Online Courses) can effectively lead to deep learning. Promoting deep learning in online courses, such as SPOCs, is often a challenge. We aimed for deep learning by ...
Highlights- This study shows how peer feedback in SPOCs effectively leads to deep learning.
Enhance Students' Computing Skills via Web-Mediated Self-Regulated Learning with Feedback in Blended Environment
The vocational colleges in Taiwan regard professional certifications as a badge of skills achievement. To enhance student learning in this specific context, the authors conducted a quasi-experiment to explore effects of web-mediated self-regulated ...
Paradoxical effects of feedback in international online reciprocal peer tutoring
This paper reports an online reciprocal peer tutoring project for improving language competence in Spanish and English. Students aged 9-12 years from Scotland and Catalonia were matched to act as tutors in their own language and as tutees in a modern ...
Comments