Modelling error recovery and repair in automatic speech recognition

doi:10.1006/imms.1993.1070

International Journal of Man-Machine Studies

Volume 39, Issue 3, September 1993, Pages 495-515

https://doi.org/10.1006/imms.1993.1070 Get rights and content

Abstract

While automatic speech recognition (ASR) has achieved some level of success, it often fails to live up to its hype. One of the principal reasons for this apparent failure is the prevalence of "recognition errors". This makes error correction a topic of increasing importance to ASR system development, with a growing awareness that, by designing for error, a number of problems can be overcome. Currently, there is a wide range of possible techniques which could be used for correcting recognition errors, and it is often difficult to compare the techniques objectively because their performance is closely related to their implementation. Furthermore, different techniques may be more suited to different applications and domains. It would be useful to have some means of defining the requirements of an error correction dialogue, based on characteristics of the dialogue and ASR system in which it is to be used, in order to develop design specifications for appropriate error correction. This paper reports an approach, based on task-network modelling, which could be used to this end.

References (0)

Cited by (30)

Handling uncertainty in multimodal pervasive computing applications
2008, Computer Communications
Multimodal interaction can improve accessibility to pervasive computing applications. However, recognition-based interaction techniques used in multimodal interfaces (e.g. speech and gesture recognition) are still error prone. Recognition errors and misinterpretations can compromise the security, robustness, and efficiency of pervasive computing applications. In this paper we briefly review the various error handling strategies that can be found in the multimodal literature. We then discuss the new challenges arising from novel affective and context-aware applications for error correction. We show that traditional multimodal error handling strategies are ill adapted to pervasive computing applications, where the computing devices have become invisible, and when users may not be aware of their own behaviour. Finally, we present an original experimental study into users’ synchronisation of speech and pen inputs in error correction. The results of the study suggest that users are likely to modify their synchronisation patterns in the belief that it can help error resolution. This study is a first step towards a better understanding of spontaneous user strategies for error correction in multimodal interfaces and pervasive environments.
Towards a taxonomy of error-handling strategies in recognition-based multi-modal human-computer interfaces
2006, Signal Processing
Citation Excerpt :
Deciding on a threshold is in many cases an ad hoc process, and unfortunately, thresholding sometimes provokes false rejection errors, and does not contribute to the automatic correction of the rejected input. A more sophisticated method has been suggested in [47], which consists in using a rule base to determine when errors may have occurred. Rules may be based on semantic, pragmatic or common sense knowledge.
In this paper, we survey the different types of error-handling strategies that have been described in the literature on recognition-based human–computer interfaces. A wide range of strategies can be found in spoken human–machine dialogues, handwriting systems, and multi-modal natural interfaces. We then propose a taxonomy for classifying error-handling strategies that has the following three dimensions: the main actor in the error-handling process (machine versus user), the purpose of the strategy (error prevention, discovery, or correction), and the use of different modalities of interaction. The requirements that different error-handling strategies have on different sets of interaction modalities are also discussed. The main aim of this work is to establish a classification that can serve as a tool for understanding how to develop more efficient and more robust multi-modal human–machine interfaces.
A longitudinal evaluation of hands-free speech-based navigation during dictation
2006, International Journal of Human Computer Studies
Despite a reported recognition accuracy rate of 98%, speech recognition technologies have yet to be widely adopted by computer users. When considering hands-free use of speech-based solutions, as is the case for individuals with physical impairments that interfere with the use of traditional solutions such as a mouse, the considerable time required to complete basic navigation tasks presents a significant barrier to adoption. Several solutions were proposed to improve navigation efficiency based on the results of a previous study. In the current study, a longitudinal experiment was conducted to investigate the process by which users learn to use hands-free speech-based navigation in the context of large vocabulary, continuous dictation tasks as well the efficacy of the proposed solutions. Due to the influence initial interactions have on the adoption of speech-based solutions, the current study focused on these critical, initial, interactions of individuals with no prior experience using speech-based dictation solutions. Our results confirm the efficacy of the solutions proposed earlier while providing valuable insights into the strategies users employ when using speech-based navigation commands as well as design decisions that can influence these patterns.
How productivity improves in hands-free continuous dictation tasks: Lessons learned from a longitudinal study
2005, Interacting with Computers
Speech recognition technology continues to improve, but users still experience significant difficulty using the software to create and edit documents. The reported composition speed using speech software is only between 8 and 15 words per minute [Proc CHI 99 (1999) 568; Universal Access Inform Soc 1 (2001) 4], much lower than people's normal speaking speed of 125–150 words per minute. What causes the huge gap between natural speaking and composing using speech recognition? Is it possible to narrow the gap and make speech recognition more promising to users? In this paper we discuss users' learning processes and the difficulties they experience as related to continuous dictation tasks using state of the art Automatic Speech Recognition (ASR) software. Detailed data was collected for the first time on various aspects of the three activities involved in document composition tasks: dictation, navigation, and correction. The results indicate that navigation and error correction accounted for big chunk of the dictation task during the early stages of interaction. As users gained more experience, they became more efficient at dictation, navigation and error correction. However, the major improvements in productivity were due to dictation quality and the usage of navigation commands. These results provide insights regarding the factors that cause the gap between user expectation with speech recognition software and the reality of use, and how those factors changed with experience. Specific advice is given to researchers as to the most critical issues that must be addressed.
Using critical path analysis to model multimodal human-computer interaction
2001, International Journal of Human Computer Studies
In this paper, the concept of multimodal human-computer interaction is explored. It is proposed that multimodality can be defined from human or technology perspectives, which place emphasis on different attributes of the system. Furthermore, in this paper it is argued that the most effective definition of multimodality concentrates on task and goal dependencies. Not only does this permit consideration over and above the technology/human distinction, but also allows consideration of multiple tasks. In order to explore this notion, critical path analysis is used to develop models of multimodal systems. The model describes multimodal HCI, and allows consideration of the effects of modality dependency. The models allow prediction of transaction time under various conditions. Predictions arising from these models are shown to be good fits with data obtained from user trials. Thus, it is proposed that one can develop and evaluate preliminary versions of multimodal systems prior to prototype development.
OOPS: A toolkit supporting mediation techniques for resolving ambiguity in recognition-based interfaces
2000, Computers and Graphics (Pergamon)
Recognition technologies are being used extensively in both commercial and research systems. Recognizers are still error-prone however, and this results in performance problems and brittle dialogues, creating barriers to the acceptance and usefulness of recognition systems. Better interfaces to systems using recognition, which can help to reduce the burden of recognition errors, are difficult to build because of lack of knowledge about the ambiguity inherent in recognition. We present a survey of the design of correction techniques in interfaces which make use of recognizers. Based on this survey, we have created a user interface toolkit, OOPS (organized option pruning system) (Mankoff et al., Proceedings of ACM CHI 2000 Conference on Human Factors in Computing Systems, 2000, pp. 368–375). OOPS consists of a library of reusable error correction, or mediation, techniques drawn from the survey, combined with necessary architectural extensions to model and to provide structured support for ambiguity at the input event level of a GUI toolkit. The resulting infrastructure makes it easier for application developers to support error handling, thus helping to reduce the negative effects of recognition errors, and allowing us to explore new types of interfaces for dealing with ambiguity.

View all citing articles on Scopus

View full text

Regular ArticleModelling error recovery and repair in automatic speech recognition

Abstract

Regular Article
Modelling error recovery and repair in automatic speech recognition