Assessment for automatic speech recognition: I. Comparison of assessment methods

doi:10.1016/0167-6393(93)90094-2

Speech Communication

Volume 12, Issue 3, July 1993, Pages 241-246

https://doi.org/10.1016/0167-6393(93)90094-2 Get rights and content

Abstract

The use of three types of vocabularies (cockpit-control words, digits, and initial consonants) was compared for the assessment of five speech recognizers. The goal of this study is to compare various assessment methods from application oriented to carefully controlled laboratory situations. It was found that the discrimination between various recognizer (input) conditions is improved for more difficult vocabularies. Confusions between stimuli and responses of testwords can be used as a diagnostic tool for prediction of performance and developments.

Zusammenfassung

Es wurden drei Wortgruppen (Kontrollworte in einem Cockpit, Zahlen und Anfangskonsonanten) verglichen, um die systeme zur Spracherkennung zu bewerten. Ziel dieser Untersuchung ist der Vergleich mehrerer applikationsorientierter Bewertungsmethoden, um Laborsituationen zu kontrollieren. Wir haben beobachtet daß die Unterscheidung zwischen mehreren Erkennungsbedingungen durch Verwendung von schwierigen Vokabeln verbessert wird. Die Verwechslung zwischen den Stimuli und den Antworten der Testworte können als Diagnosemittel verwendet werden, um Leistungen und Entwicklungen vorauszusagen.

Résumé

Trois vocabulaires (mots de contrôle d'un cockpit, chiffres, et consonnes et initiales) ont été comparés en vue d'évaluer des systèmes de reconnaissance de la parole. Le but de cette étude est de comparer plusieurs méthodes d'évaluation orientées vers des applications pour contrôler des situations de laboratoire. Nous avons observé que la discrimination entre plusieurs conditions de reconnaissance est améliorée par l'utilisation de vocabulaires difficiles. Les confusions entre les stimuli et les réponses des mots de tests peuvent être utilisées comme un outil de diagnostic pour prédire les performances et les développements.

References (9)

R.K. moore
Speech recognition: Available assessment methods and needs for standardisation
H.J.M. Steeneken
On measuring and predicting speech intelligibility
H.J.M. Steeneken et al.
Objective and diagnostic assessment of (isolated) word recognizers
H.J.M. Steeneken et al.
RAMOS — Recognizer assessment by means of manipulation of speech

There are more references available in the full text version of this article.

Cited by (7)

Cost-Effectiveness of a Technology-Facilitated Depression Care Management Adoption Model in Safety-Net Primary Care Patients with Type 2 Diabetes
2018, Value in Health
Citation Excerpt :
Harnessing advanced information and communication technologies (ICTs) to automate key aspects of depression care is a promising approach to facilitating adoption of the team-based collaborative depression care model [8]. For example, automated speech recognition telephonic assessment technology [20,21] combined with an electronic decision rules and priorities management system [22] can automate depression assessments, patient self-management behavior prompting, optimization of treatment follow-up, and ongoing monitoring and management. This approach can fill gaps in the current implementation of depression care to facilitate optimal adaptive depression management in primary care that could simultaneously improve health care outcomes at reduced costs, reduce the physical or economic burden on patients, and be responsive to patient choice.
The Diabetes-Depression Care-Management Adoption Trial is a translational study of safety-net primary care predominantly Hispanic/Latino patients with type 2 diabetes in collaboration with the Los Angeles County Department of Health Services.
To evaluate the cost-effectiveness of an information and communication technology (ICT)-facilitated depression care management program.
Cost-effectiveness of the ICT-facilitated care (TC) delivery model was evaluated relative to a usual care (UC) and a supported care (SC) model. TC added automated low-intensity periodic depression assessment calls to patients. Patient-reported outcomes included the 12-Item Short Form Health Survey converted into quality-adjusted life-years (QALYs) and the 9-Item Patient Health Questionnaire–calculated depression-free days (DFDs). Costs and outcomes data were collected over a 24-month period (−6 to 0 months baseline, 0 to 18 months study intervention).
A sample of 1406 patients (484 in UC, 480 in SC, and 442 in TC) was enrolled in the nonrandomized trial. TC had a significant improvement in DFDs (17.3; P = 0.011) and significantly greater 12-Item Short Form Health Survey utility improvement (2.1%; P = 0.031) compared with UC. Medical costs were statistically significantly lower for TC (−$2328; P = 0.001) relative to UC but not significantly lower than for SC. TC had more than a 50% probability of being cost-effective relative to SC at willingness-to-pay thresholds of more than $50,000/QALY.
An ICT-facilitated depression care (TC) delivery model improved QALYs, DFDs, and medical costs. It was cost-effective compared with SC and dominant compared with UC.
Speech enhancement of instantaneous amplitude and phase for applications in noisy reverberant environments
2016, Speech Communication
We previously proved that restoring the instantaneous amplitude as well as instantaneous phase on the output from Gammatone filterbank plays a significant role in speech enhancement. However, dereverberation is still a challenge since the previously proposed scheme can only work in noisy environments. In this paper, we extend our previously proposed scheme to general speech enhancement for removing both the effects of noise and reverberation by restoring instantaneous amplitude and phase simultaneously. Objective and subjective experiments were conducted under various noisy reverberant conditions to evaluate the effectiveness of the extension of the proposed scheme. The signal to error ratio (SER), correlation, PESQ, and SNR loss were used in objective evaluations. The normalized mean preference score and correctness in modified rhyme test (MRT) were used in subjective evaluations. We also tested how effective our proposed scheme is as a front-end for an automatic speech recognition (ASR) system in realistic noisy reverberant environments. The results of all evaluations revealed that the proposed scheme could effectively improve quality and intelligibility of speech signals under noisy reverberant conditions.
Assessment for automatic speech recognition: II. NOISEX-92: A database and an experiment to study the effect of additive noise on speech recognition systems
1993, Speech Communication
The NOISEX-92 experiment and database is described and discussed. NOISEX-92 specifies a carefully controlled experiment on artificially noisy speech data, examining performance for a limited digit recognition task but with a relatively wide range of noises and signal-to-noise ratios. Example recognition results are given.
Es werden die Datenbank und die Tests NOISEX-92 beschrieben und kommentiert. NOISEX-92 spezifiziert Tests über künstliche Lärmdaten, wobei die Leistungen im Rahmen der Erjennung von Zahlen und ein relativ breiter Rauschabstand bewertet werden. Es werden Beispiele der Ergebnisse für Erkennungen beschrieben.
La base de données et les tests NOISEX-92 sont décrits et commentés. NOISEX-92 spécifie des tests sur des données bruitées artificiellement en évaluant les performances dans le cadre de la reconnaissance de chiffres et une gamme relativement large de rapports signal sur bruit. Des examples de scores de reconnaissances sont présentés.
Robust front-end for speech recognition by human and machine in noisy reverberant environments: The effect of phase information
2017, Proceedings of 2016 10th International Symposium on Chinese Spoken Language Processing, ISCSLP 2016
Diagnostics for debugging speech recognition systems
2010, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
A comparison of decision tree classifiers for automatic diagnosis of speech recognition errors
2010, Computing and Informatics

View all citing articles on Scopus

View full text

Assessment for automatic speech recognition: I. Comparison of assessment methods

Abstract

Zusammenfassung

Résumé

Speech recognition: Available assessment methods and needs for standardisation

On measuring and predicting speech intelligibility

Objective and diagnostic assessment of (isolated) word recognizers

RAMOS — Recognizer assessment by means of manipulation of speech