Abstract
Automated Essay Scoring (AES) efforts have recently made it possible for platforms to provide real-time feedback and grades for student essays. With the growing importance of addressing usability issues that arise from integrating artificial intelligence (AI) into educational-based platforms, there have been significant efforts to improve the visual elements of User Interfaces (UI) for these types of platforms. However, little research has been done on how AI explainability and algorithm transparency affect the usability of AES platforms. To address this gap, a qualitative study was conducted using an AI-driven essay writing and grading platform. The study involved participants of students and instructors, and utilized surveys, semi-structured interviews, and a focus group to collect data on users’ experiences and perspectives. Results show that user understanding of the system, quality of feedback, error handling, and creating trust are the main usability concerns related to explainability and transparency. Understanding these challenges can help guide the development of effective grading tools that prioritize explainability and transparency, ultimately improving their usability.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3, 77–101 (2006). https://doi.org/10.1191/1478088706qp063oa
Braun, V., Clarke, V.: Reflecting on reflexive thematic analysis. Qual. Res. Sport Exerc. Health 11(4), 589–597 (2019)
Haque, A.K.M.B., Islam, A.K.M.N., Mikalef, P.: Explainable artificial intelligence (XAI) from a user perspective: a synthesis of prior literature and problematizing avenues for future research. Technol. Forecasting Soc. Change 186, 122120 (2023)
Kumar, V., Boulanger, D.: Explainable automated essay scoring: deep learning really has pedagogical value. Front. Educ. 5 (2020). https://doi.org/10.3389/feduc.2020.572367, https://www.frontiersin.org/article/10.3389/feduc.2020.572367
Long, D., Magerko, B.: What is AI literacy? Competencies and design considerations, pp. 1–16. Association for Computing Machinery (2020). https://doi.org/10.1145/3313831.3376727
Lyons, J.B., Wynne, K.T., Mahoney, S., Roebke, M.A.: Chapter 6 - Trust and Human-Machine Teaming: A Qualitative Study, pp. 101–116. Academic Press (2019). https://doi.org/10.1016/B978-0-12-817636-8.00006-5, https://www.sciencedirect.com/science/article/pii/B9780128176368000065
Petch, J., Di, S., Nelson, W.: Opening the black box: the promise and limitations of explainable machine learning in cardiology. Can. J. Cardiol. 38(2), 204–213 (2022)
Rader, E., Cotter, K., Cho, J.: Explanations as mechanisms for supporting algorithmic transparency (2018)
Semire, D.: An overview of automated scoring of essays. J. Technol. Learn. Assess. 5(1) (2006). https://ejournals.bc.edu/index.php/jtla/article/view/1640
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hall, E., Seyam, M., Dunlap, D. (2023). Identifying Usability Challenges in AI-Based Essay Grading Tools. In: Wang, N., Rebolledo-Mendez, G., Dimitrova, V., Matsuda, N., Santos, O.C. (eds) Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky. AIED 2023. Communications in Computer and Information Science, vol 1831. Springer, Cham. https://doi.org/10.1007/978-3-031-36336-8_104
Download citation
DOI: https://doi.org/10.1007/978-3-031-36336-8_104
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36335-1
Online ISBN: 978-3-031-36336-8
eBook Packages: Computer ScienceComputer Science (R0)