Prompt-Independent Automated Scoring of L2 Oral Fluency by Capturing Prompt Effects

Abstract

We propose a prompt-independent automated scoring method of second language (L2) oral fluency, which is robust to different cognitive demands of speaking prompts. When human examiners assess L2 learners’ oral fluency, they can consider the effects of different task prompts on speaking performance, systematically adjusting their evaluation criteria across prompts. However, conventional automated scoring methods tend to ignore such variability in speaking performance caused by prompt design and use prompt-specific features of speech. Their robustness is thus arguably limited to a specific prompt used in model training. To address this challenge, we operationalize prompt effects in terms of conceptual, linguistic and phonological features of speech and embed them, as well as a set of temporal features of speech, into a scoring model. We examined the agreement between true and predicted fluency scores in four different L2 English monologue prompts. The proposed method outperformed a conventional method which used only temporal features ($\kappa = 0.863 \text { vs. } 0.797$). The detailed analysis showed that the conceptual and phonological features improved the performance of automated scoring. Meanwhile, the effectiveness of the linguistic features was not confirmed possibly because it may largely reflect redundant information to capture the prompt demands. These results suggest that the robustness of the automated fluency scoring should be achieved by careful consideration of what characteristics of L2 speech reflect the prompt effects.