Similarity-based classification of sequences using hidden Markov models

Abstract

Hidden Markov models (HMM) are a widely used tool for sequence modelling. In the sequence classification case, the standard approach consists of training one HMM for each class and then using a standard Bayesian classification rule. In this paper, we introduce a novel classification scheme for sequences based on HMMs, which is obtained by extending the recently proposed similarity-based classification paradigm to HMM-based classification. In this approach, each object is described by the vector of its similarities with respect to a predetermined set of other objects, where these similarities are supported by HMMs. A central problem is the high dimensionality of resulting space, and, to deal with it, three alternatives are investigated. Synthetic and real experiments show that the similarity-based approach outperforms standard HMM classification schemes.

Introduction

The analysis of sequential data is an interesting and important research area. Probabilistic modelling and classification is intrinsically more difficult when each observation is a sequence, compared to the standard scenario where each observation is a set (vector) of features. In fact, since the length of the sequences may vary, it is not possible to directly use standard pattern recognition techniques. Moreover, sequence classification problems usually involve very large data sets.

Hidden Markov models (HMMs) are commonly employed probabilistic models of sequential data [1]. HMMs can be viewed as stochastic generalizations of finite-state automata, when both the transitions between states and the generation of output symbols are governed by probabilistic mechanisms [1]. Although the basic theory and inference tools were developed in the late 1960s [2], [3], HMMs have only been extensively applied in the last decade. Speech recognition [1], DNA and protein modelling [4], [5], handwritten character recognition [6], gesture recognition [7], and behavior analysis and synthesis [8] are examples of problems for which HMMs have been exploited.

The standard HMM-based approach to sequence classification consists in training one HMM for each class, which are subsequently used as class-conditional densities in a standard Bayes classification paradigm. For example, assuming a priori equiprobable classes, an unknown sequence is classified into the class whose model shows the highest probability (likelihood) of having generated this sequence (this is the well-known maximum-likelihood (ML) classification rule).

In this paper, an alternative classification scheme is proposed, by extending the similarity-based paradigm [9], [10], [11], [12], [13], [14] to HMM-based classification. This paradigm, which has been introduced recently, differs from typical pattern recognition approaches where objects to be classified are represented by sets (vectors) of features. In the similarity-based paradigm, objects are described using pairwise (dis)similarities, i.e. distances from other objects in the data set. In this way, objects are not constrained to be explicitly represented in a feature space, and all that is necessary is a way to compute (dis)similarities between pairs of objects. The goal is then to learn a classifier only from these relational data.

The literature on similarity-based classification is not vast [9], [10], [11], [12], [13], [14] (a brief review is given in Section 2.1). The general idea behind all these approaches is basically the same: given a set of pairwise dissimilarity values, a new representation space can be built, in which each object is described by these values. In Ref. [13], a simple synthetic experiment shows that a complex problem in a 2D space (requiring a quadratic classifier to achieve almost correct separation), becomes a linearly separable problem in a dissimilarity space.

In this paper, we extend this dissimilarity-based classification paradigm to HMM-based sequences classification problems. We propose to build a similarity¹ space, representing each object (sequence) by the vector of its similarities with respect to a predetermined set of objects (this can be the whole data set, in the simplest approach), called the representatives set; the classification is then performed in this new representation space. The similarities are derived by considering the likelihood $P(\mathbf{O}|\mathit{\lambda })$ as a measure of the similarity between the sequence $\mathbf{O}$ and the HMM specified by the set of parameters $\mathit{\lambda }$. This similarity measure was previously used in sequence clustering applications [15], [16].

The similarity-based classification paradigm seems to be particularly well suited to HMMs, as it can be seen as a natural extension of the standard HMM classification scheme. Specifically, the standard ML approach assigns an unknown sequence $\mathbf{O}$ to the class whose model shows the highest likelihood. To do so, the likelihoods of $\mathbf{O}$ with respect to the HMMs of all classes are evaluated, each stating a likelihood-based measure of the similarity between that class and the observed sequence. In other words, HMMs are used to compute similarities between sequences and classes, with each class being represented by a single HMM. Subsequently, only the maximum of these values is used to take the classification decision. In the similarity-based approach, the classification decision is taken using the whole set of similarities between each observed sequence and all the other sequences. We will show that this strategy results in a substantial improvement in the classification performance, compared to standard HMM-based approaches. Moreover, with the use of HMMs and the similarity representation, the problem of classification of sequences is reduced to a more standard classification task (where each object is described by a fixed-length feature vector), for which arbitrarily sophisticated techniques can be used, allowing to increase even more the classification performance.

The proposed approach was successfully tested on both synthetic and real data, involving 2D shape recognition and face recognition problems. In comparison with the standard HMM-based ML classification approach, our method showed a significant performance improvement, confirming all the potential of the similarity-based classification approach.

The main problem of the similarity-based approach, of particular relevance in practical applications, is the high dimensionality of the resulting similarity space. Actually, in the basic approach, this dimensionality is equal to the cardinality of the whole training data set, possibly leading to a huge computational burden. In the literature, two types of solutions of this problem could be identified, summarized in Section 2.2. In this paper, three methods to face this problem are proposed. The first one aims at removing redundancy from the data by applying linear dimensionality reduction techniques, such as Fisher discriminant analysis (FDA) [17] and principal component analysis (PCA) [19]. The second proposed method is based on a greedy strategy known as matching pursuit [20], which selects a subset of representatives based on which the similarity values are computed. These two approaches are very general, and can be applied in all distance-based classification contexts. The third proposed approach is more specific to the HMM case, and is based on a simple adaptation of the similarity-based classification approach to the standard HMM learning procedure. All these approaches were experimentally evaluated, confirming the discriminative power of the similarity space, even when the dimensionality is reduced to more manageable numbers.

Summarizing, the main contribution of this paper is the introduction of the similarity-based recognition paradigm in an HMM context, resulting in a significant performance improvement with respect to standard HMM-based classification. The mapping to the similarity space proposed in this paper allows us to reduce complex problems of sequence classification to a more standard point classification problem, for which arbitrarily sophisticated techniques could be used. From the point of view of similarity-based recognition, we propose two different approaches for dealing with the high dimensionality of the similarity space, which is one of the main problems of the method. First, the potential of linear reduction techniques, as PCA and FDA, is exploited, showing that they are able to reduce the curse of dimensionality impact on the classification process. Second, we address the choice of a set of appropriate representatives using the matching pursuit algorithm, which proves to be a robust and effective approach.

The rest of the paper is organized as follows. In Section 2, the state of the art related to the similarity-based classification and to the dimensionality issue is summarized. In Section 3, HMMs are introduced, together with the standard classification scheme. The proposed strategy is described in Section 4, and Section 5 reports the experiments and the related results are discussed. Finally, in Section 6, conclusions are drawn and future perspectives are envisaged.