Parameter reduction schemes for loosely coupled HMMs

Abstract

While Hidden Markov Models (HMMs) have been successful in many speech recognition tasks, performance on conversational speech is somewhat less successful, arguably due in part to the greater variation in timing of articulatory events. Loosely Coupled or Factorial HMMs (FHMMs) represent a family of models that have more flexibility for modeling such variation in speech, but there are tradeoffs to be studied in terms of computation and potential added confusability. This paper investigates two specific instances – Mixed-Memory and Parameter-Tied FHMMs – that can both be thought of as loosely coupled HMMs for modelling multiple time series. The Parameter-Tied FHMM, introduced here, has a potential advantage for speech modelling since it allows a left-to-right topology across the product state space. Experimental results on the ISOLET task show both models are feasible for speech recognition; TI-DIGITS recognition results show the Parameter-Tied FHMM is competitive with Multiband Models. State occupancy and pruning analyses show trends related to asynchrony that hold across the different models.

Introduction

Hidden Markov Model (HMM)-based automatic speech recognition (ASR) has been successfully applied to dictated speech tasks, but the approach is less successful when confronted with more conversational speech (Weintraub, Stolcke, & Sankar, 1995). The experiment in Saraclar, Nock, and Khudanpur (2000) suggests that at least part of the problem may be related to the increased pronunciation variability in conversational speech. It is often hypothesised that the lack of robustness to pronunciation variability is related to the construction of word models by concatenating sequences of phoneme models as specified by the pronunciation dictionary.

One hypothesised source of difficulty is the limited nature of existing pronunciation dictionaries: typical pronunciation dictionaries contain only a few pronunciations for each word. For example, both the LIMSI (Lamel & Adda, 1996; Hain, Woodland, Evermann, & Povey, 2000) and Pronlex (Kingsbury, Strassel, & McLemore, 1997) recognition dictionaries have a single pronunciation for over 90% of the words and less than 1% of words have more than two pronunciations. Most recognition dictionaries contain even fewer pronunciations per word. The source of dictionary pronunciations varies, but is rarely conversational speech. Given such a pronunciation dictionary, it is assumed that the subword acoustic modelling scheme represents all remaining pronunciation variability. Context conditioning does model the influence of context on the realisation of sounds, and mixture of Gaussian output distributions in HMMs can capture variability in segment realisations. However, it can be argued that neither technique is an efficient model of these types of pronunciation change. Whilst reliance on the acoustic models to characterise pronunciation variability proved sufficient for dictated speech, it may not be adequate for capturing the increased range of pronunciations in conversational speech (Keating, 1997; Ostendorf, 2000; Weintraub et al., 1996). Firstly, inappropriate or inadequate pronunciations can lead to recognition errors. Secondly, broad variance models (resulting from a training scheme in which each subword model is potentially trained on data from other subword classes) can increase error rates due to increased acoustic confusability and also tend to increase decoding costs.

Inadequacy of the dictionary motivates explicit pronunciation modelling schemes: these augment the dictionary with one or more pronunciations per word which are more representative of the target style or accent (e.g. Humphries & Woodland, 1997; Riley, 1991). Whilst there is evidence to support research into explicit pronunciation modelling (Saraclar, 2000; Saraclar et al., 2000), the gains achieved in practice have been less than spectacular. Use of an expanded recognition-time dictionary yields improvements of 1–2% absolute on Switchboard (e.g. Byrne et al., 1998; Finke & Waibel, 1997); use of an expanded dictionary during acoustic model training as well as in recognition has not resulted in performance gains (Saraclar et al., 2000; Saraclar, 2000). Difficulties arise from lexical confusability – many new word pronunciations overlap with those of other words, increasing the difficulty of mapping back to word strings from phone sequences – and because pronunciation change often occurs at levels below the segment, rather than simply complete changes of phoneme identity.

The latter observation brings us to a second weakness of current designs: the assumption that speech can be segmented into a linear sequence of (usually phone-like) segments, which is sometimes referred to as the “beads-on-a-string” model. Speech scientists, linguists and engineers agree the notion of a speech segment is not a realistic one (e.g. Huckvale, 1994; Deng & Erler, 1992; King & Taylor, 2000). Speech is produced by loosely coupled articulators and speech production studies show the amplitude and phase between these gestures varies with changes in speaking rate, manner and style (e.g. Vaxelaire, Sock, & Perrier, 2000). The changes in relative timing can have extreme effects on the resulting acoustic signal: it often appears that there has been colouring and merging of the underlying ‘segments’ or even ‘segment-like’ insertions due to interaction between articulatory gestures. Examples include feature spreading e.g. CAN’T /k ae n t/ → [ k ae_n t ], where vowel /ae/ becomes nasalised due to “deleted” segment /n/, and asynchronous articulatory gestures causing stop insertions e.g. WARMTH /w ao m th/ → [ w ao m p th ]. The beads-on-a-string scheme was adequate for dictated speech recognition since amplitude and timing of gestures are fairly consistent. But as speech becomes more conversational, relative timing effects become more significant (e.g. Vaxelaire et al., 2000), and this type of variability may not be sufficiently well modelled by the beads-on-a-string approach.

Attempts to better model relative timing effects are mostly implicit schemes, incorporated at the level of the acoustic model. One such approach introduces more flexible state-level parameter sharing schemes, perhaps incorporating more knowledge of phonology or measures of speaking rate and style (e.g. Hain & Woodland, 1999; Hain & Woodland, 2000; Ostendorf, 2000; Finke, Fritsch, Koll, & Waibel, 1999; Saraclar, 2000). A more speculative direction of research investigates schemes for extracting and modelling intermediate articulatory or phonetic representations of speech, which may be a simpler domain in which to model the phonological effects in conversational speech. Rather than model speech as a linear sequence of segments, it is represented as a structured arrangement of phonetic or articulatory features between which there may be some degree of variation in the relative timing of phonetic events. Thus, for example, when nasality from phoneme /n/ partially colours a neighbouring vowel /ae/, this is modelled by asynchrony in the feature changes. There is considerable work about extracting appropriate intermediate representations of speech (e.g. Richmond, 2001; Kirchhoff, 1999). Fewer papers consider schemes for incorporating these ideas within a statistical framework; exceptions include (e.g. Kirchhoff, 1999; King, Stephenson, Isard, Taylor, & Strachan, 1998). The latter problem is considered in this paper.

The problem of modelling asynchronous articulatory, phonological or acoustic feature streams is a problem of modelling multiple, loosely coupled time series. Section 2 reviews conventional speech models that have been applied to modelling loosely coupled time series, particularly with respect to the degree of asynchrony allowed. Section 3 outlines the theory of Factorial Hidden Markov Models (FHMMs), a more general family of models which is potentially applicable to this modelling problem. Two specific instances of the FHMM are then described: one from the machine learning literatures (Saul & Jordan, 1999) and another designed to reflect the left-to-right nature of speech. Section 4 presents some experimental results; Section 5 summarises key results and discusses some open questions.