Abstract
This study reports on the application of text mining and machine learning methods in the context of asynchronous peer instruction, with the objective of automatically identifying high quality student explanations. Our study compares the performance of state-of-the-art methods across different reference datasets and validation schemes. We demonstrate that when we extend the task of argument quality assessment along the dimensions of convincingness, from curated datasets, to data from a real learning environment, new challenges arise, and simpler vector space models can perform as well as a state-of-the-art neural approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Modified from the run_glue.py script provided by the tranformers package, built by company hugging face. All code for this study provided at https://github.com/sameerbhatnagar/ectel2020.
References
Bhatnagar, S., Lasry, N., Desmarais, M., Charles, E.: DALITE: asynchronous peer instruction for MOOCs. In: Verbert, K., Sharples, M., Klobučar, T. (eds.) EC-TEL 2016. LNCS, vol. 9891, pp. 505–508. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45153-4_50
Univeristy of British Columbia, T.L.T.: ubc/ubcpi, August 2019. https://github.com/ubc/ubcpi. original-date: 2015–02-17T21:37:02Z
Cambre, J., Klemmer, S., Kulkarni, C.: Juxtapeer: comparative peer review yields higher quality feedback and promotes deeper reflection. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI 2018, pp. 1–13. ACM Press, Montreal QC, Canada (2018). https://doi.org/10.1145/3173574.3173868
Charles, E.S., et al.: Harnessing peer instruction in- and out- of class with myDALITE. In: Fifteenth Conference on Education and Training in Optics and Photonics: ETOP 2019. pp. 11143–11189. Optical Society of America (2019). http://www.osapublishing.org/abstract.cfm?URI=ETOP-2019-11143_89
Chen, X., Bennett, P.N., Collins-Thompson, K., Horvitz, E.: Pairwise ranking aggregation in a crowdsourced setting. In: Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, pp. 193–202 (2013)
Crouch, C.H., Mazur, E.: Peer instruction: ten years of experience and results. Am. J. Phys. 69(9), 970–977 (2001)
Denny, P.: The effect of virtual achievements on student engagement. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2013, Paris, France, pp. 763–772. ACM, New York (2013). https://doi.org/10.1145/2470654.2470763. http://doi.acm.org/10.1145/2470654.2470763
Denny, P., Hamer, J., Luxton-Reilly, A., Purchase, H.: PeerWise: students sharing their multiple choice questions. In: Proceedings of the Fourth International Workshop on Computing Education Research, ICER 2008, Sydney, Australia, pp. 51–58. ACM, New York (2008). https://doi.org/10.1145/1404520.1404526. http://doi.acm.org/10.1145/1404520.1404526
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Gagnon, V., Labrie, A., Desmarais, M., Bhatnagar, S.: Filtering non-relevant short answers in peer learning applications. In: Proceedings Conference on Educational Data Mining (EDM) (2019)
Gleize, M., et al.: Are you convinced? choosing the more convincing evidence with a Siamese network. arXiv preprint arXiv:1907.08971 (2019). https://www.aclweb.org/anthology/P19-1093/
Habernal, I., Gurevych, I.: Which argument is more convincing? analyzing and predicting convincingness of web arguments using bidirectional LSTM. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1589–1599. Association for Computational Linguistics, Berlin, Germany, August 2016). https://doi.org/10.18653/v1/P16-1150. https://www.aclweb.org/anthology/P16-1150
Herbrich, R., Graepel, T., Obermayer, K.: Support Vector Learning For Ordinal Regression. IET (1999)
Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 133–142 (2002)
Khosravi, H., Kitto, K., Williams, J.J.: Ripple: a crowdsourced adaptive platform for recommendation of learning activities. arXiv preprint arXiv:1910.05522 (2019)
Lippi, M., Torroni, P.: Argumentation mining: state of the art and emerging trends. ACM Trans. Internet Technol. (TOIT) 16(2), 10 (2016)
Louis, A., Nenkova, A.: What makes writing great? first experiments on article quality prediction in the science journalism domain. Trans. Assoc. Comput. Linguist. 1, 341–352 (2013)
Moens, M.F., Boiy, E., Palau, R.M., Reed, C.: Automatic detection of arguments in legal texts. In: Proceedings of the 11th International Conference on Artificial Intelligence and Law, pp. 225–230 (2007)
Nguyen, D., Doğruöz, A.S., Rosé, C.P., de Jong, F.: Computational sociolinguistics: a survey. arXiv preprint arXiv:1508.07544 (2015)
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. EMNLP 14, 1532–1543 (2014)
Persing, I., Ng, V.: End-to-end argumentation mining in student essays. In: HLT-NAACL, pp. 1384–1394 (2016)
Pollitt, A.: The method of adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, vol. 19, no. 3, pp. 281–300. Routledge (2012). https://doi.org/10.1080/0969594X.2012.665354
Potter, T., Englund, L., Charbonneau, J., MacLean, M.T., Newell, J., Roll, I.: ComPAIR: a new online tool using adaptive comparative judgement to support learning with peer feedback. Teach. Learn. Inquiry 5(2), 89–113 (2017)
Simpson, E., Gurevych, I.: Finding convincing arguments using scalable Bayesian preference learning. Trans. Assoc. Comput. Linguist. 6, 357–371 (2018). https://www.aclweb.org/anthology/Q18-1026
Tan, C., Niculae, V., Danescu-Niculescu-Mizil, C., Lee, L.: Winning arguments: interaction dynamics and persuasion strategies in good-faith online discussions. arXiv:1602.01103 [physics], pp. 613–624 (2016). https://doi.org/10.1145/2872427.2883081
Toledo, A., et al.: Automatic argument quality assessment-new datasets and methods. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5629–5639 (2019). https://www.aclweb.org/anthology/D19-1564.pdf
Weir, S., Kim, J., Gajos, K.Z., Miller, R.C.: Learnersourcing subgoal labels for how-to videos. In: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pp. 405–416. ACM (2015)
Williams, J.J., et al.: AXIS: generating explanations at scale with learnersourcing and machine learning. In: Proceedings of the Third (2016) ACM Conference on Learning @ Scale - L@S 2016, Edinburgh, Scotland, UK, pp. 379–388. ACM Press (2016). https://doi.org/10.1145/2876034.2876042
Acknowledgements
Funding for the development of myDALITE.org is made possible by Entente-Canada-Quebec, and the Ministère de l’Éducation et Enseignment Supérieure du Québec. Funding for this research was made possible by the support of the Canadian Social Sciences and Humanities Research Council Insight Grant. This project would not have been possible without the SALTISE/S4 network of researcher practitioners, and the students using myDALITE.org who consented to share their learning traces with the research community.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Bhatnagar, S., Zouaq, A., Desmarais, M.C., Charles, E. (2020). Learnersourcing Quality Assessment of Explanations for Peer Instruction. In: Alario-Hoyos, C., Rodríguez-Triana, M.J., Scheffel, M., Arnedillo-Sánchez, I., Dennerlein, S.M. (eds) Addressing Global Challenges and Quality Education. EC-TEL 2020. Lecture Notes in Computer Science(), vol 12315. Springer, Cham. https://doi.org/10.1007/978-3-030-57717-9_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-57717-9_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-57716-2
Online ISBN: 978-3-030-57717-9
eBook Packages: Computer ScienceComputer Science (R0)