Automatic prosodic event detection using a novel labeling and selection method in co-training

Abstract

Most previous approaches to automatic prosodic event detection are based on supervised learning, relying on the availability of a corpus that is annotated with the prosodic labels of interest in order to train the classification models. However, creating such resources is an expensive and time-consuming task. In this paper, we exploit semi-supervised learning with the co-training algorithm for automatic detection of coarse-level representation of prosodic events such as pitch accent, intonational phrase boundaries, and break indices. Since co-training works on the condition that the views are compatible and uncorrelated, and real data often do not satisfy these conditions, we propose a method to label and select examples in co-training. In our experiments on the Boston University radio news corpus, when using only a small amount of the labeled data as the initial training set, our proposed labeling method can effectively use unlabeled data to improve performance and finally reach performance close to the results of the supervised method using more labeled data. We perform a thorough analysis of various factors impacting the learning curves, including labeling error rate and informativeness of added examples, performance of the individual classifiers and their difference, and the initial and added data size.

Introduction

Prosody represents suprasegmental information in speech since the acoustic correlates of a prosodic event extend over more than one phoneme segment. Prosodic phenomena manifest themselves in different ways, including changes in relative intensity to emphasize specific syllables or words, variations of the fundamental frequency (pitch) range and contour, and subtle timing variations, such as syllable lengthening and insertion of pause. In spoken language, prosody conveys linguistic and paralinguistic information such as emphasis, intent, attitude, and emotion of a speaker. Listeners rely on prosodic information to help interpretation of speech.

In many spoken language processing tasks, prosody also plays an important role because it includes aspects of higher level information that is not completely revealed by segmental acoustics or lexical information. Below we list a few applications where prosody can contribute to augmenting the abilities of spoken language systems.

• Speech recognition: the correlation of pitch accent patterns between acoustic and lexical-prosodic evidence can be used to reduce word error rate in speech recognition (Chen and Hasegawa-Johnson, 2006, Ananthakrishnan and Narayanan, 2007, Jeon et al., 2011).

• Dialog act detection: intonation patterns at the end of a sentence are useful indications of specific dialog acts or sentence categories (question, statement, exclamation, etc.) (Shriberg et al., 1998).

• Lexical and syntactic disambiguation: knowledge of syllable pitch accent patterns and boundary information can help resolve lexical or syntactic ambiguity (Price et al., 1991).

• Natural speech synthesis: one of the challenges in natural sounding speech synthesis systems is to generate human-like prosody to accompany the segmental acoustic properties. This includes local effects (such as syllable accent), properly timed boundaries used to reflect the syntactic structure of the sentence, as well as modulation of pitch at a global level to produce appropriate intonation patterns.

For some applications such as speech recognition, raw prosodic features capturing pitch, energy, and duration might be used in statistical classifiers. However, these low level prosodic features are not appropriate for some applications (e.g., they cannot be directly combined with other information sources in the system), rather, symbolic representations of prosody are more useful, for example, knowing whether there is a pitch accent in a syllable, or whether there is a phrase boundary in a place. Automatic labeling of such prosodic events has received a lot of attention over the past decades because it is important for some speech understanding tasks mentioned above, as well as for scientific understanding of prosody and its relationship with lexical, syntactic, and semantic structure of sentences. Many previous efforts on prosodic event detection adopt supervised learning approaches using acoustic, lexical, and syntactic cues. However, the major drawback with these methods is that they require a large hand-labeled training corpus, and system performance is highly dependent on the specific corpus used for model training. It is very expensive and time-consuming to create a reasonable amount of training data annotated with prosodic information.

Limited research has been conducted using unsupervised and semi-supervised methods for automatic prosodic event labeling. In this paper, we exploit semi-supervised learning with the co-training algorithm (Blum and Mitchell, 1998) for this task. Two different views corresponding to the acoustic and lexical/syntactic knowledge sources are used in the co-training framework. We propose a novel labeling and selection scheme to address the problem that natural data do not meet the compatible and uncorrelated conditions required in the co-training algorithm. Our experiments on the Boston Radio News corpus (Ostendorf et al., 1995) show that our proposed approach works well – using unlabeled data can lead to significant improvement for prosodic event detection compared to using the original small training set, yielding results comparable to those from supervised learning with a similar amount of labeled training data. Our analysis shows that reducing the labeling error rate of the added examples for the next iteration does not always improve performance, and that increasing the informativeness of examples is more useful for performance gain, even though some erroneous examples are included. We also find that the performance gain is affected by the status of the two classifiers, such as the similarity of their views.

The remainder of this paper is organized as follows. In the next section, we provide details of the corpus and the prosodic event detection tasks. Section 3 reviews previous work briefly. In Section 4, we describe the basic classification method for prosodic event detection, including the acoustic and lexical/syntactic prosodic models, and the features used. Section 5 introduces the co-training algorithm we use. Section 6 presents our experimental results and analysis. The final section gives a brief summary along with future directions.