AI-assisted automated scoring of picture-cued writing tasks for language assessment

Abstract

Grading assignments is inherently subjective and time-consuming; automatic scoring tools can greatly reduce teacher workload and shorten the time needed for providing feedback to learners. The purpose of this paper is to propose a novel method for automatically scoring student responses to picture-cued writing tasks. As a popular paradigm for language instruction and assessment, a picture-cued writing task typically requires students to describe a picture or pictures. Correspondingly, the automatic scoring methods must measure the link(s) between visual pictures and their textual descriptions. For this purpose, we first designed a picture-cued writing test and collected nearly 4 k responses from 279 K12 students. Based on these responses, we then developed an AI scoring model by incorporating the emerging cross-modal matching technology and some NLP algorithms. The performance of the model was evaluated carefully with six popular measures and was found to demonstrate accurate scoring results with a small mean absolute error of 0.479 and a high adjacent-agreement rate of 90.64%. We believe this method could reduce the subjective elements inherent in human grading and save teachers’ time from the mundane task of grading to other valuable endeavors such as designing teaching plans based on AI-generated diagnosis of student progress.

Appendices

Appendix 1

All 15 pictures were used in our writing test, and they were grouped in three categories. Each picture of first category includes one dominant character, e.g., a person or an animal, each picture of second category covers two characters having some interaction, and each picture third category often includes a group of people in an activates.

Note. The numbers are the ids of the pictures in our research and this paper.

Appendix 2

Table 5

Table 5 The scoring rubric designed for the picture-cued writing test in this study, and all the example responses were written by students for the 8^th picture in Appendix A

Appendix 3. Machine learning model training.

Considering the problem of imbalanced data, that there are only few responses assigned with a low and high score. Serious imbalance of data can even cause the trained model to directly ignore categories with a small sample size. Some optimization tricks can be used to make the model pay more attention to the classes with fewer samples. This allows the model to focus on learning such features, rather than focusing too much on categories with a large number of samples. A simple way is to copy the same data directly to achieve the purpose of expanding the data, that is, oversampling. Another method is SMOTE. First, each sample ${x}_{i}$ is selected from the minority class samples as the root sample for synthesizing new samples; secondly, the k nearest neighbor samples of the same class of ${x}_{i}$ are used as the reference for synthesizing new samples, and new samples are generated by interpolation, repeat this process until the required number of samples is reached. Here, k is generally an odd number, we have tried k = 5, 15, and 25 on the training set, and found that k = 5 works better. Two sampling methods are compared, and finally we choose oversampling as the method for balancing the data based on their performance on the training set.

Next, we trained several machine learning models and used grid search to tune the parameters of each model by tenfold cross-validation on the training set. These models are: K-nearest neighbors algorithm (k-NN) (Coomans & Massart, 1982), which classify samples by finding the k most similar, nearest neighbors. The searching space of hyperparameters is {Number of neighbors: 5, 10, 20, 50. Weight function: “uniform”, “distance”}. The second model we trained is Random Forest (Breiman, 2001), which builds multiple decision trees in order to combine all predictions for a more robust behavior. The searching space of hyperparameters is {Number of decision trees in the forest: 50, 100, 200, 400. Criterion: “gini”, “entropy”. Maximum depth of the tree: 4,5,6,8,10. Minimum number of samples to split: 20,40,80,100. Number of features to consider when split: “sqrt”, “log2″}. Support-vector machines (SVM) (Cortes & Vladimir, 1995) is also considered, it maps training samples to point in space for classification and regression analysis. The searching space of hyperparameters is {Regularization parameter: 1, 10, 100. Kernel: “linear”, “poly”, “rbf”, “sigmoid”. Kernel coefficient gamma: 1e-2, 1e-3, 1e-4,” auto”}. Another model we trained is XGBoost (Chen & Guestrin, 2016), which implements machine learning algorithms under the Gradient Boosting framework. The searching space of hyperparameters is {Learning rate: 0.05, 0.1, 0.2, 0.3. Maximum depth: 4, 6, 8. Subsample: 0.6, 0.8, 0.9, 1. Scale_pos_weight: 1, 5, 10. Alpha: 0, 1, 2, 5, 10}. Based on the model performance on the tenfold CV, the models with the least MAE were selected for each component. We trained models to predict 6 components first, and the performance are list in the following table. By adding scores of these components, we get the final scores. Compared with the human scores, the total MAE is 0.479.

Components	MAE
Grammar $\in \left[\mathrm{0,3}\right]$	0.230
Spelling $\in \left[0, 1\right]$	0.056
Convention $\in \left[0, 1\right]$	0.054
Comprehensiveness $\in \left[0, 3\right]$	0.198
Vividness $\in \left[0, 1\right]$	0.083
Sentence structure $\in \left[0, 1\right]$	0.018

Appendix 4

Main indices used in our automated scoring model. In the figure, the values represent the Gini importance (Steinberg & Colla, 2009) of the indices in evaluating the grammar, spelling, convention, comprehensiveness, vividness, and sentence structure of student response, as well as in predicting the final score for the responses. The larger the values, the more important the indices are.

The six variables in the upper part are the features generated by three cross-modal matching methods (i.e., ALBEF, GSMN and SGRAF), and they are mainly used in the prediction of “Comprehensiveness”. Each method estimated a set of similarities with their models trained on different datasets or with multiple scales of parameters, and we chose the two most effective similarities in our experiments. The variables in the lower part are the features generated in natural language processing. This graph shows the importance of the variables by which the machine predicts the score, and they all fit our conjectures. For example, the vividness of a sentence mainly depends on whether a lot of adjectives and adverbs are used, and when scoring the sentences structure, more attention is paid to the number of clauses and pronouns.