Noise-tolerant speech recognition: the SNN-TA approach

Abstract

Neural network learning theory draws a relationship between “learning with noise” and applying a regularization term in the cost function that is minimized during the training process on clean (non-noisy) data. Application of regularizers and other robust training techniques are aimed at improving the generalization capabilities of connectionist models, reducing overfitting. In spite of that, the generalization problem is usually overlooked by automatic speech recognition (ASR) practioners who use hidden Markov models (HMM) or other standard ASR paradigms. Nonetheless, it is reasonable to expect that an adequate neural network model (due to its universal approximation property and generalization capability) along with a suitable regularizer can exhibit good recognition performance whenever noise is added to the test data, although training is accomplished on clean data. This paper presents applications of a variant of the so called segmental neural network (SNN), introduced at BBN by Zavaliagkos et al. for rescoring the N-best hypothesis yielded by a standard continuous density HMM (CDHMM). An enhanced connectionist model, called SNN with trainable amplitude of activation functions (SNN-TA) is first used in this paper instead of the CDHMM to perform the recognition of isolated words. Viterbi-based segmentation is then introduced, relying on the level-building algorithm, that can be combined with the SNN-TA to obtain a hybrid framework for continuous speech recognition. The proposed paradigm is applied to the recognition of isolated and connected Italian digits under several noisy conditions, outperforming the CDHMMs.

Introduction

Increasing robustness to noise in an automatic speech recognition (ASR) system can be described as a generalization problem [15]: the recognizer is trained on a given training corpus (possibly collected in a laboratory, with clean acoustic conditions) and then applied on different, noisy signals, featuring only partially predictable environmental conditions. Techniques that allow for good recognition performance in spite of differences in conditions between training and test datasets are sought. State-of-the-art ASR systems usually rely on hidden Markov models (HMM) [6], [9]. HMMs offer good performance in laboratory tests, but robustness toward noise and variability to changes in acoustic conditions are problems far from being solved. In particular, no regularization theory [4] for HMMs has been developed so far. On the contrary, generalization properties of artificial neural networks (ANNs) are much more understood and exploited in ANNs regularized training algorithms.

In [11], [13] we emphasized that ANNs were intensively applied to ASR throughout a whole decade, but they basically failed as a general paradigm for ASR, especially with long sequences of acoustic observations (e.g. whole words from a dictionary or whole sentences), mostly due to the fact that learning long-term time dependencies with “conventional” connectionist architectures (including recurrent nets) is difficult [3]. A variety of different hybrid ANN/HMM systems [2], [5], [13], [14] was indeed introduced in recent years to tackle this problem, attempting to combine the desirable properties of both connectionist and markovian models.

The research described in the present paper starts by considering one such hybrid system, modifying the scope of the ANN within the paradigm, and applying a novel regularization technique to the connectionist model in order to increase its robustness to a significant extent. The hybrid under consideration is presented in [16]. An ANN called segmental neural network (SNN) is used therein for rescoring the N-best hypothesis of a HMM. The network computes scores on whole segments (sub-sequences) of frames, corresponding to phonemes, according to the segmentation provided by the underlying HMM. In so doing, correlations between nearby frames––belonging to the same phoneme––are exploited, thus overcoming the usual limitations following the assumption of independence made in standard HMMs. In addition, segmental information is expected to reduce noise sensitivity.

When an acoustic segment is fed into the SNN, the latter produces an estimation of the posterior probabilities of phonemes given that segment. Since the segments are made up of a variable number of frames, a technique of “normalization” of the length of the input window is needed in order to feed the fixed-sized input layer of the network. A discrete cosine transform (DCT) is applied to each segment, retaining as many parameters as it is necessary to fill the input layer.

In this paper the SNN is enhanced to provide robustness against noise by improving its generalization ability via a soft “self-regularization” technique based on the introduction of trainable amplitudes of activation functions (see Section 2). In addition, it is used as a speech recognizer itself, instead of as a mere rescoring tool for a HMM. We call such a model SNN-TA (segmental neural network with trainable amplitudes). To summarize, differences between the standard SNN and SNN-TA are the following: (a) SNN yields scores for individual phonemes over acoustic segments provided by the HMM. SNN-TA provides a posterior probability estimation for each “word” of the dictionary to be recognized over the whole input acoustic sequence; (b) in [16] SNNs are trained on a relative entropy criterion. Instead, backpropagation of squared errors between target and actual outputs is used herein, along with a gradient descent algorithm to train amplitudes; (c) the SNN-TA is said to be “regularized”, since a technique that increases its noise-tolerance (i.e., its generalization capabilities) is induced by the amplitude training algorithm.