research-article

Automatically assessing the ABCs: Verification of children's spoken letter-names and letter-sounds

Authors:

Matthew P. Black,

Abe Kazemzadeh,

Joseph Tepperman,

Shrikanth S. NarayananAuthors Info & Claims

ACM Transactions on Speech and Language Processing (TSLP), Volume 7, Issue 4

Article No.: 15, Pages 1 - 17

https://doi.org/10.1145/1998384.1998389

Published: 18 August 2011 Publication History

Abstract

Automatic literacy assessment is an area of research that has shown significant progress in recent years. Technology can be used to automatically administer reading tasks and analyze and interpret children's reading skills. It has the potential to transform the classroom dynamic by providing useful information to teachers in a repeatable, consistent, and affordable way. While most previous research has focused on automatically assessing children reading words and sentences, assessments of children's earlier foundational skills is needed. We address this problem in this research by automatically verifying preliterate children's pronunciations of English letter-names and the sounds each letter represents (“letter-sounds”). The children analyzed in this study were from a diverse bilingual background and were recorded in actual kindergarten to second grade classrooms. We first manually verified (accept/reject) the letter-name and letter-sound utterances, which serve as the ground-truth in this study. Next, we investigated four automatic verification methods that were based on automatic speech recognition techniques. We attained percent agreement with human evaluations of 90% and 85% for the letter-name and letter-sound tasks, respectively. Humans agree between themselves an average of 95% of the time for both tasks. We discuss the various confounding factors for this assessment task, such as background noise and the presence of disfluencies, that impact automatic verification performance.

References

[1]

Alwan, A., Bai, Y., Black, M. P., Casey, L., Gerosa, M., Heritage, M., Iseli, M., Jones, B., Kazemzadeh, A., Lee, S., Narayanan, S., Price, P., Tepperman, J., and Wang, S. 2007. A system for technology based assessment of language and literacy in young children: The role of multiple information sources. In Proceedings of the International Workshop on Multimedia Signal Processing (MMSP).

[2]

Blachman, B. A., Ball, E. W., Black, R. S., and Tangel, D. M. 1994. Kindergarten teachers develop phoneme awareness in low-income, inner-city classrooms. Read. Writ. 6, 1, 1--18.

[3]

Black, M. P., Tepperman, J., Kazemzadeh, A., Lee, S., and Narayanan, S. 2008. Pronunciation verification of English letter-sounds in preliterate children. In Proceedings of Interspeech.

[4]

Black, M. P., Tepperman, J., Kazemzadeh, A., Lee, S., and Narayanan, S. 2009. Automatic pronunciation verification of English letter-names for early literacy assessment of preliterate children. In Proceedings of the IEEE International Conference on Accustics, Speech, and Signal Processing (ICASSP).

Digital Library

[5]

Black, M. P., Tepperman, J., and Narayanan, S. 2011. Automatic prediction of children's reading ability for high-level literacy assessment. IEEE Trans. Audio Speech Lang. Process. 19, 4, 1015--1028.

Digital Library

[6]

Black, P. and Wiliam, D. 1998. Assessment and classroom learning. Assess. Educ. Princ. Policy Pract. 5, 1, 7--74.

[7]

Cassell, J. and Ryokai, K. 2001. Making space for voice: Technologies to support children's fantasy and storytelling. Pers. Ubiq. Comput. 5, 3, 169--190.

Digital Library

[8]

Cincarek, T., Gruhn, R., Hacker, C., Nöth, E., and Nakamura, S. 2009. Automatic pronunciation scoring of words and sentences independent from the non-native's first language. Comput. Speech Lang. 23, 1, 65--88.

Digital Library

[9]

Cosi, P., Delmonte, R., Biscetti, S., Cole, R. A., Pellom, B., and Vuren, S. 2004. Italian literacy tutor-tools and technologies for individuals with cognitive disabilities. In Proceedings of the InSTIL/ICALL Symposium on Computer Assisted Learning.

[10]

Duchateau, J., Cleuren, L., Van Hamme, H., and Ghesquière, P. 2007. Automatic assessment of children's reading level. In Proceedings of Interspeech.

[11]

Eskenazi, M. 2009. An overview of spoken language technology for education. Speech Comm. 51, 10, 832--844.

Digital Library

[12]

Hagen, A., Pellom, B., and Cole, R. 2007. Highly accurate children's speech recognition for interactive reading tutors using subword units. Speech Comm. 49, 12, 861--873.

Digital Library

[13]

Heritage, M. 2007. Formative assessment: What do teachers need to know and do&quest; Phi Delta Kappan 89, 2, 140--145.

[14]

Kazemzadeh, A., Tepperman, J., Silva, J., You, H., Lee, S., Alwan, A., and Narayanan, S. 2006. Automatic detection of voice onset time contrasts for use in pronunciation assessment. In Proceedings of Interspeech.

[15]

Kazemzadeh, A., You, H., Iseli, M., Jones, B., Cui, X., Heritage, M., Price, P., Anderson, E., Narayanan, S., and Alwan, A. 2005. TBALL data collection: The making of a young children's speech corpus. In Proceedings of Interspeech.

[16]

Lee, S., Potamianos, A., and Narayanan, S. 1999. Acoustics of children's speech: Developmental changes of temporal and spectral parameters. J. Acoust. Soci. Am. 105, 3, 1455--1468.

[17]

McBride-Chang, C. 1999. The ABCs of the ABCs: The development of letter-name and letter-sound knowledge. Merrill-Palmer Quart. 45, 2, 285--308.

[18]

Mostow, J., Roth, S. F., Hauptmann, E. G., and Kane, M. 1994. A prototype reading coach that listens. In Proceedings of AAAI.

Digital Library

[19]

National Reading Panel. 2000. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Tech. rep. 00-4769, National Institute for Child Health and Human Development, National Institute of Health, Washington, D.C.

[20]

Paratore, J. and McCormack, R. 2007. Classroom Literacy Assessment: Making Sense of What Students Know and Do. Guilford Press, New York, NY.

[21]

Price, P., Tepperman, J., Iseli, M., Duong, T., Black, M. P., Wang, S., Boscardin, C. K., Heritage, M., David Pearson, P., Narayanan, S., and Alwan, A. 2009. Assessment of emerging reading skills in young native speakers and language learners. Speech Comm. 51, 10, 968--984.

Digital Library

[22]

Rasmussen, M. H., Tan, Z. H., Lindberg, B., and Jensen, S. H. 2009. A system for detecting miscues in dyslexic read speech. In Proceedings of Interspeech.

[23]

Tepperman, J., Lee, S., Alwan, A., and Narayanan, S. 2011. A generative student model for scoring word reading skills. IEEE Trans. Audio Speech Lang. Process. 19, 2, 348--360.

Digital Library

[24]

van Doremalen, J. J. H. C., Cucchiarini, C., and Strik, H. 2010. Optimizing automatic speech recognition for low-proficient non-native speakers. EURASIP J. Audio Speech Music Process. 1--13.

Digital Library

[25]

Witt, S. M. and Young, S. J. 2000. Phone-level pronunciation scoring and assessment for interactive language learning. Speech Comm. 30, 2-3, 95--108.

Digital Library

[26]

Yildirim, S., Narayanan, S., and Potamianos, A. 2011. Detecting emotional state of a child in a conversational computer game. Comput. Speech Lang. 25, 1, 29--44.

Digital Library

[27]

You, H., Alwan, A., Kazemzadeh, A., and Narayanan, S. 2005. Pronunciation variations of Spanish-accented English spoken by young children. In Proceedings of Interspeech.

[28]

Young, S., Evermann, G., Gales, M., Hain, T., Kershaw, D., Liu, X., Moore, G., Odell, J., Ollason, D., Povey, D., Valtchev, V., and Woodland, P. C. 2006. The HTK Book (version 3.4). Cambridge University Engineering Department.

[29]

Zheng, J., Huang, C., Chu, M., Soong, F. K., and Ye, W. 2007. Generalized segment posterior probability for automatic Mandarin pronunciation evaluation. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP).

Cited By

Papakyriakopoulos OChoi AThong WZhao DAndrews JBourke RXiang AKoenecke A(2023)Augmented Datasheets for Speech Datasets and Ethical Decision-MakingProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594049(881-904)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594049
Chen LJang J(2015)Automatic pronunciation scoring with score combination by learning to rank and class-normalized DP-based quantizationIEEE/ACM Transactions on Audio, Speech and Language Processing10.1109/TASLP.2015.244908923:11(1737-1749)Online publication date: 1-Nov-2015
https://dl.acm.org/doi/10.1109/TASLP.2015.2449089
Black MNarayanan S(2012)Improvements in predicting children's overall reading ability by modeling variability in evaluators' subjective judgments2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2012.6289060(5069-5072)Online publication date: Mar-2012
https://doi.org/10.1109/ICASSP.2012.6289060

Index Terms

Automatically assessing the ABCs: Verification of children's spoken letter-names and letter-sounds
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Signal processing systems

Recommendations

Automatic Prediction of Children's Reading Ability for High-Level Literacy Assessment

Automatic literacy assessment technology can help children acquire reading skills by providing teachers valuable feedback in a repeatable, consistent manner. Recent research efforts have concentrated on detecting mispronunciations during word-reading and ...
Prosodic characterisation of children's Filipino read speech for oral reading fluency assessment

This paper explores the extraction and analysis of prosodic features in children's Filipino speech for application in automated oral reading fluency assessment. Automatic syllabication was optimised in the context of children's Filipino read speech. Using ...
Tabby Talks

Automated tool to assess productions from children with apraxia of speech.Consists of clinician interface, mobile application and speech processing engine.Automatically detects groping errors, articulation errors and prosodic errors.Lattice-based ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Speech and Language Processing

ACM Transactions on Speech and Language Processing Volume 7, Issue 4

August 2011

143 pages

ISSN:1550-4875

EISSN:1550-4883

DOI:10.1145/1998384

Issue’s Table of Contents

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 August 2011

Accepted: 01 January 2011

Revised: 01 October 2010

Received: 01 June 2010

Published in TSLP Volume 7, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
238
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Papakyriakopoulos OChoi AThong WZhao DAndrews JBourke RXiang AKoenecke A(2023)Augmented Datasheets for Speech Datasets and Ethical Decision-MakingProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594049(881-904)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594049
Chen LJang J(2015)Automatic pronunciation scoring with score combination by learning to rank and class-normalized DP-based quantizationIEEE/ACM Transactions on Audio, Speech and Language Processing10.1109/TASLP.2015.244908923:11(1737-1749)Online publication date: 1-Nov-2015
https://dl.acm.org/doi/10.1109/TASLP.2015.2449089
Black MNarayanan S(2012)Improvements in predicting children's overall reading ability by modeling variability in evaluators' subjective judgments2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2012.6289060(5069-5072)Online publication date: Mar-2012
https://doi.org/10.1109/ICASSP.2012.6289060

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents