Clustering of triphones using phoneme similarity estimation for the definition of a multilingual set of triphones

Abstract

This paper addresses the problem of multilingual acoustic modelling for the design of multilingual speech recognisers. An agglomerative clustering algorithm for the definition of multilingual set of triphones is proposed. This clustering algorithm is based on the definition of an indirect distance measure for triphones defined as a weighted sum of the explicit estimates of the context similarity on a monophone level. The monophone similarity estimation method is based on the algorithm of Houtgast. The new clustering algorithm was tested in a multilingual speech recognition experiment for three languages. The algorithm was applied on monolingual triphone sets of language specific recognisers for all languages. In order to evaluate the clustering algorithm, the performance of the multilingual set of triphones was compared to the performance of the reference system composed of all three language specific recognisers operating in parallel, and to the performance of the multilingual set of triphones produced by the tree-based clustering algorithm. All experiments were based on the 1000 FDB SpeechDat(II) databases (Slovenian, Spanish and German). Experiments have shown that the use of the clustering algorithm results in a significant reduction of the number of triphones with minor degradation of recognition rate.

Zusammenfassung

Dieser Beitrag befasst sich mit dem Problem der multilingualen akustischen Modellierung für die automatische Spracherkennung. Die Verwendung eines agglomerativen Cluster-Algorithmus zur Defintion einer Menge multilingualer kontextabhängiger phonetischer Einheiten (Triphone) wird eingeführt. Der Algorithmus basiert auf einem indirekten Abstandsmaß für Triphone, das als eine gewichtete Summe der geschätzten Ähnlichkeiten der zu den Triphonen gehörenden Monophone definiert ist. Die Ähnlichkeitsschätzung der Monophone basiert auf dem Algorithmus von Houtgast. Der neue Cluster-Algorithmus wurde auf multilinguale Spracherkennungsexperimente für drei verschiedene Sprachen angewendet. Dazu wurden sprachspezifische Erkennungssysteme mit monolingualen Triphonen für alle drei Sprachen eingesetzt. Um den Cluster-Algorithmus bewerten zu können, wurde die Leistungsfähigkeit des auf den multilingualen Triphonen basierenden Systems mit zwei Referenzsystemen verglichen. Während in dem ersten Referenzsystem die sprachenspezifischen Modelle gleichzeitig (parallel) eingesetzt wurden, fanden im zweiten Referenzsystem multilinguale Modelle Verwendung, die mithilfe eines entscheidungsbaumbasierten Cluster-Algorithmus erstellt wurden. Alle Experimente basieren auf der 1000 FDB SpeechDat(II) Datenbasis (slowenisch, spanisch und deutsch). Die Untersuchungen haben gezeigt, dass die Verwendung des vorgeschlagenen agglomerativen Cluster-Algorithmus bei einer geringgradigen Abnahme der Erkennungsrate zu einer deutlichen Verringerung der Anzahl der Triphonparameter führt.

Introduction

During the last few years the development of speech technology raised an interest in the research of multilingual speech recognition. The goal of research in this area is twofold. On the one hand, the multilingual speech recognition system should extend the functionality of language specific recognisers to a number of languages without degrading the recognition accuracy or considerably increasing the computational complexity. On the other hand, the multilingual speech recognition system should serve as a tool for porting the speech technology from one language, or more languages, to another language with little or no spoken language resources. The work presented in this paper focuses on the first goal––the development of methods for definition of the multilingual phonetic inventories that reduces the complexity of multilingual speech recognisers.

The definition of multilingual phonetic inventories by exploiting similarities amongst the sounds of different languages is a promising approach. One of the first attempts was reported in (Andersen et al., 1993) where multilingual phonetic inventory, consisting of language-dependent and language-independent speech units, was defined by using the data-driven clustering technique. Other attempts based on different distance measures and clustering techniques also followed (Koehler, 1996; Berkling, 1996; Bonaventura et al., 1997; Weng et al., 1997), however, most of the work so far has been focused on context-independent phoneme modelling (monophone). These experiments show that the transition from language-dependent monophone set to multilingual inventory of monophones may result in a degradation of recognition accuracy due to the lack of acoustic resolution of the multilingual monophone set.

Transition from context-independent to context-dependent phoneme modelling seems inevitable in order to improve the performance of multilingual speech recognition systems. Context-dependent modelling has already proved it can enhance the performance of monolingual recognisers (Young, 1996; Bourlard, 1995). Such an improvement can also be expected in the multilingual case, however, the crucial problem is how to define the multilingual set of context-dependent phoneme models, that is, how to define the clustering procedure.

Even though the introduction of context-dependent phoneme modelling is a self-evident step in the evolution of multilingual speech recognition research, only a few research reports have been given so far. Most of these reports were given by Schultz and Waibel, 1998, Schultz and Waibel, 1999 who applied a decision tree-based clustering procedure to generate a multilingual set of context-dependent phoneme models. The purpose of the work reported in this paper is to complement the previous work on “top-down” clustering procedures by proposing a “bottom-up” approach. A new technique for the definition of the multilingual set of context-dependent phoneme models based on agglomerative clustering is introduced in this paper. A clustering algorithm has been implemented for the triphones and is based on an indirect similarity estimation of two triphones defined as the weighted sum of explicit estimation of the similarity of the phonemes of left and right contexts, and the center phonemes.

This paper is organized as follows. The formulation of the triphone distance measure and the clustering algorithm as well as the description of the reference tree-based clustering algorithm are given in Section 2. The experimental multilingual recognition systems and the speech databases used for the experiments are described in 3 Baseline recogniser, 4 Speech databases, respectively. Experimental results are presented in Section 5, while the discussion and the conclusion are made in 6 Discussion, 7 Conclusion.