A three-stage approach to the automated scoring of spontaneous spoken responses

Abstract

This paper presents a description and evaluation of $SpeechRate{r}^{SM}$, a system for automated scoring of non-native speakers’ spoken English proficiency, based on tasks which elicit spontaneous monologues on particular topics. This system builds on much previous work in the automated scoring of test responses, but differs from previous work in that the highly unpredictable nature of the responses to this task type makes the challenge of accurate scoring much more difficult.

SpeechRater uses a three-stage architecture. Responses are first processed by a filtering model to ensure that no exceptional conditions exist which might prevent them from being scored by SpeechRater. Responses not filtered out at this stage are then processed by the scoring model to estimate the proficiency rating which a human might assign to them, on the basis of features related to fluency, pronunciation, vocabulary diversity, and grammar. Finally, an aggregation model combines an examinee’s scores for multiple items to calculate a total score, as well as an interval in which the examinee’s score is predicted to reside with high confidence.

SpeechRater’s current level of accuracy and construct representation have been deemed sufficient for low-stakes practice exercises, and it has been used in a practice exam for the TOEFL since late 2006. In such a practice environment, it offers a number of advantages compared to human raters, including system load management, and the facilitation of immediate feedback to students. However, it must be acknowledged that SpeechRater presently fails to measure many important aspects of speaking proficiency (such as intonation and appropriateness of topic development), and its agreement with human ratings of proficiency does not yet approach the level of agreement between two human raters.

Introduction

The last decade has seen a proliferation of new applications of speech technology in the educational domain, in particular in support of English language learner (ELL) populations. Computer-based learning tools for spoken English allow learners who may have limited access to native or high-proficiency English speakers an opportunity to practice their English and receive feedback on their performance. Automated systems also have the potential to make educational materials for English speaking accessible to a wider range of learners, by reducing their cost (compared to a human tutor), and making them available on a more flexible schedule.

Speech-enabled dialogue systems allow learners to practice their speaking and listening in an exchange with a virtual interlocutor—e.g., SpeakESL and Auralog’s Tell Me More system. Automated tutoring systems for speaking practice provide feedback to learners on their pronunciation, vocabulary, and grammar—e.g., Carnegie Speech NativeAccent (Eskenazi et al., 2007), Saybot (Chevalier, 2007) and SRI’s EduSpeak (Franco et al., 2000). The contribution of the work described in this paper is to the field of automated scoring of spoken responses: assessing the level of English speaking proficiency demonstrated in some speaking task used in a testing context.

In particular, this paper describes SpeechRater, an automated system for scoring spontaneous spoken responses. SpeechRater uses automatically derived measures of fluency, pronunciation, vocabulary diversity, and grammar to score spoken responses to tasks similar to those found on the Test of English as a Foreign Language (TOEFL^®), a test of English proficiency for academic settings, used internationally for college admissions. Only the internet-delivered version of the TOEFL test (TOEFL iBT) includes tasks to assess speaking proficiency; the paper-based version of the test assesses only reading, listening, and writing. SpeechRater is currently used operationally to score spoken responses to the low-stakes TOEFL Practice Online (TPO) test, but not for the high-stakes TOEFL iBT test.

Where other spoken response scoring systems had previously relied on restricted speaking tasks such as reading a passage aloud, or answering questions to which the range of responses is narrowly circumscribed (Bernstein, 1999, Bernstein et al., 2000, Balogh et al., 2007), SpeechRater addresses the more challenging task of scoring responses which are relatively unstructured, unrestricted, and spontaneous (for a set of speakers varying considerably in English proficiency and first language). While much work remains to be done to improve SpeechRater’s agreement with human raters and the coverage of important aspects of speaking proficiency in its feature set, the current capability has advanced sufficiently to allow it to be used for the scoring of practice tests used in preparing for the operational TOEFL test.

Section 2 surveys previous work in automated scoring of test items, especially speaking tasks. Section 3 then provides a brief overview of the SpeechRater system architecture, and the speech recognition system used as the basis for feature computation. After an overview of the spoken response data is provided in Section 4, Section 5 describes the first component of SpeechRater, which filters out non-scoreable responses. Section 6 introduces the scoring model itself, and Section 7 describes the way in which scores are aggregated across tasks, and bounds on prediction error are calculated. Finally, a brief summary of the major findings of the paper is presented in Section 8.