A method of combining multiple probabilistic classifiers through soft competition on different feature sets

Abstract

A novel method is proposed for combining multiple probabilistic classifiers on different feature sets. In order to achieve the improved classification performance, a generalized finite mixture model is proposed as a linear combination scheme and implemented based on radial basis function networks. In the linear combination scheme, soft competition on different feature sets is adopted as an automatic feature rank mechanism so that different feature sets can be always simultaneously used in an optimal way to determine linear combination weights. For training the linear combination scheme, a learning algorithm is developed based on Expectation–Maximization (EM) algorithm. The proposed method has been applied to a typical real-world problem, viz., speaker identification, in which different feature sets often need consideration simultaneously for robustness. Simulation results show that the proposed method yields good performance in speaker identification.

Introduction

The problem of pattern classification can be stated as follows: Given a set of training data, each with an associated label, find a classification system that will produce the correct label for any data drawn from the same source as the training data. As illustrated in Fig. 1a, in general, such a classification system is composed of three stages: preprocessing, feature extraction, and classification. In particular, feature extraction is necessary to avoid a so-called curse of dimensionality problem [18] which may lead to prohibitively expensive computation in the stage of classification. Therefore, the performance of a classification system highly depends upon a feature set used. For a classification task, numerous types of features can be extracted from the same raw data by means of different methods. A selection technique is often adopted to find an optimal feature set for use in classification [6]. Sometimes, however, it is impossible to find such an optimal feature set. Instead several different feature sets can result in similar classification performance so that none of them can be optimal or robust for a specific classification task. Because different feature sets represent raw data from different viewpoints, the simultaneous use of different feature sets can lead to a better or robust classification result. As illustrated in Fig. 1b, this kind of problems are called pattern classification on different feature sets in this paper. There are many real-world problems belonging to this category. A typical example is speaker identification that classifies an unlabeled voice token as belonging to one of reference speakers. For this problem, several different spectrum feature sets have been turned out useful, but none of them can be regarded as an optimal or robust one. It has been suggested that multiple spectrum feature sets need consideration simultaneously for robustness in speaker identification 20, 21. As a result, a technique that efficiently utilizes different feature sets becomes a solution to pattern classification on different feature sets.

For simultaneous use of different feature sets, a traditional method is to lump different feature vectors together into a single composite feature vector. Although there are several methods to form a composite feature vector, the use of a composite feature set may result in the following problems: (1) Curse of dimensionality; the dimension of a composite feature vector becomes much higher than any of component feature vectors. (2) Difficulty in formation; it is often difficult to lump several different feature vectors together due to their diversified forms. (3) Redundancy; the component feature vectors are usually not independent of each other. Therefore, the composite-feature based method can simply achieve limited success. Recently, combination of multiple classifiers has been viewed as a new direction for the development of highly reliable pattern recognition systems. Preliminary results indicate that combination of several complementary classifiers may lead to the improved performance 26, 25, 3, 38, 33, 32, 35, 23, 46. There are at least two reasons for necessity of combining multiple classifiers. On the one hand, there are a number of classification algorithms developed from different theories and methodologies in almost all the current pattern recognition application areas. Each of these classifiers may reach a certain degree of success, but none of them is totally perfect or so good as expected in practical applications. On the other hand, the demand for a solution to pattern classification on different feature sets becomes the other reason to do so. A combination technique allows multiple classifiers to work on different feature sets so that different feature sets can be utilized simultaneously. Therefore, a method of combining multiple classifiers on different feature sets provides an alternative way to solve the problem of pattern classification on different feature sets.

From the viewpoint of statistics, combination of multiple classifiers can be viewed as combination of multiple probability distributions if each classifier is interpreted as an estimator of probability distribution. In general, there are two frameworks to perform such a combination [27]. One is that of a decision maker who consults several classifiers regarding some events. The classifiers express their opinions in the form of probability distributions. The decision maker must aggregate the classifiers’ distributions into a single distribution that can be used to make the final decision. The other is the framework of linear opinion pools in which the decision maker forms a linear combination of classifiers’ opinions. Under the two frameworks, there have been extensive studies in combination of multiple probabilistic classifiers 27, 22, 44, 46, 10, 1. In terms of pattern classification on different feature sets, combination techniques under the framework of a decision maker, e.g. a supra Bayesian procedure 27, 46, can be directly used for this problem since outputs of the classifiers are merely considered for combination regardless of their inputs. On the other hand, several combination techniques under the framework of linear opinion pools can be also directly applied for combination of multiple classifiers on different feature sets 33, 32, 35, 23, 7, 43. These techniques are based on constrained or unconstrained least-squares regression with model selection to make a system achieve good generalization properties. However, the existing techniques of linear opinion pools with weights as veridical probabilities 27, 45, 44 cannot be used to deal with a problem of pattern classification on different feature sets unless a single composite feature set is used since the process of generating weights used for linear combination highly depends upon inputs of multiple classifiers.

In this paper, we propose a new linear combination scheme to extend the existing techniques of linear opinion pools with weights as veridical probabilities for pattern classification on different feature sets. In contrast to the winner-take-all mechanism, soft competition is a concept that a competitor and its rivals can work for the same task together, but the winner plays a more important role than losers. In the linear combination scheme, we adopt such a soft competition mechanism on different feature sets to determine weights in an optimal way for linear combination. An EM learning algorithm is also proposed for parameter estimation in the linear combination scheme. To demonstrate its effectiveness, we have applied the proposed method to a real-world problem, viz., speaker identification, in which diversified feature sets need consideration simultaneously for robustness. Simulation results show that the proposed method yields satisfactory performance in speaker identification.

The remainder of this paper is organized as follows. Section 2presents the methodology on the linear combination scheme through soft competition. Section 3describes an EM learning algorithm for parameter estimation in the linear combination scheme. Section 4reports simulation results on speaker identification. Conclusions are drawn in the final section.