Evaluating the Rater Bias in Response Scoring in Digital Learning Platform: Analysis of Student Writing Styles

Abstract

Automated essay scoring and short-answer scoring have shown tremendous potential for enhancing and promoting large-scale assessments. Challenges still remain in the equity and the implicit bias in scoring that is ingrained in the scoring system. One promising solution to mitigate the problem is the introduction of a measurement model that quantifies and evaluates the degree to which the raters are showing biases in students’ written responses. This paper presents an adoption of a generalized many-facet Rasch model (GMFRM, [8]) to evaluate the rater biases regarding students’ writing styles. We modeled students’ writing styles using a rich set of computational linguistic indices from LingFeat [7] that are empirically and theoretically associated with the text difficulty. The findings showed that the rater bias exists in scoring responses that are explicitly covering a variety of topics with less elaborative descriptions compared to the other writing styles captured in students’ responses. The rater severity was the only type of bias characteristic that was statistically significant. We discussed the implications and future applications of our findings to advance automated essay scoring and teacher professional development.