A two-stage methodology for sequence classification based on sequential pattern mining and optimization

Abstract

We present a methodology for sequence classification, which employs sequential pattern mining and optimization, in a two-stage process. In the first stage, a sequence classification model is defined, based on a set of sequential patterns and two sets of weights are introduced, one for the patterns and one for classes. In the second stage, an optimization technique is employed to estimate the weight values and achieve optimal classification accuracy. Extensive evaluation of the methodology is carried out, by varying the number of sequences, the number of patterns and the number of classes and it is compared with similar sequence classification approaches.

Introduction

Sequence classification is an important problem which arises in many real-world applications, such as protein function prediction, text classification or speech recognition [16]. Sequential data are sequences of ordered “events” representing a situation, where each event might be described by a set of predicates. Examples of sequential data include text, biosequences (DNA, proteins), web-usage data, multiplayer games, plan-execution traces, etc. Classification is the procedure in which given a collection of training records, each one containing a set of attributes and a class, to find a model that maps the features of each record to a class attribute. Subsequently, this model can be used in order to provide predictions for new records. Based on that, given a sequence (constructed from letters drawn from a finite alphabet; i.e. 20-letter alphabet of amino acids in the case of protein classification; a vocabulary of English words in text classification), a sequence classifier assigns a class label (typically drawn from a finite set of mutually exclusive class labels) to this sequence. Data mining and machine learning algorithms offer a number of effective approaches to design sequence classifiers, when a training set of labeled sequences is available [18].

A sequential pattern is a sequence of itemsets that frequently occur in a specific order. An itemset is a non-empty subset of elements, called items, from a set which is called alphabet. In this manner, an itemset represents the set of items that occur together. Sequential pattern mining is a procedure that discovers sequential patterns existing in databases of sequences. Sequential pattern mining is widely used in a variety of domains, ranging from text to proteins and DNA sequences. The problem was first introduced by Agrawal and Srikant [2], and since then the goal of sequential pattern mining is to discover all frequent sequences of itemsets in a dataset.

The problem of sequence classification has been addressed in the literature in many ways; the earliest approaches employed finite automata and entropy based approaches [29]. Several methodologies have also been proposed which use either hidden Markov models [35], [40] or support vector machines [9], [27]. Also, several sequence classification methods have been proposed, as applications in specific domains, such as protein classification [11], [24], [30], [36], text classification [20], [22], speech [35] and handwriting [19] recognition. A different category of techniques treat the problem of sequence classification as a feature mining problem [25], [26], [38], i.e. they mine features from a set of training sequences and then use these features as input in a standard classification algorithm. The FeatureMine algorithm uses these features with the naïve Bayes and Winnow algorithm [25], [26]. The Classify By Sequences (CBS) algorithm uses a simple scoring function [38]. Tseng and Lee [38] proposed two different approaches, the CBS_ALL and the CBS_CLASS. Experimental results showed that CBS_CLASS outperforms CBS_ALL [38].

Recently, data mining techniques like association rule mining, sequential pattern mining, clustering and classification, emerged in various research topics [1], [2], [6], [34], [41]. However, most of the existing data mining methods are designed for solving a specific problem. On the other hand, some few compound methods integrate two or more types of data mining techniques to solve complex problems. These compound methods can effectively utilize the advantages of each individual mining technique to improve the overall performance in data mining tasks. For example, the Classification Based on Associations (CBA) [28] method provides higher accuracy than traditional classification methods such as C4.5 [34]. Hence, it is a promising direction to integrate different types of data mining methods to form a new methodology for solving complex data mining problems.

In this work, we propose a novel methodology for the generation of sequence classification models, that consists of two stages. In the first stage, a sequence classification model based on sequential patterns is created. This first stage is similar to the CBS_CLASS algorithm, which also builds a sequence classification model from the extracted sequential patterns. The innovation of the proposed methodology is the introduction of weights, which are applied to the sequential patterns and to the classes, and their tuning through optimization, during the second stage, which is an extension of the previously reported CBS_CLASS algorithm. The methodology can be considered as a compound data mining method that uses sequential pattern mining for sequence classification. The input to our methodology is a set of labeled training sequences, and the output is a function mapping a new, unknown sequence, to a class. The classification of an unknown sequence is realized automatically. The methodology employs a sequential pattern mining algorithm, a scoring function that uses the sequential patterns for classification and an optimization technique, in order to automatically assign weights to the sequential patterns and to the classes for improving the classification accuracy. The proposed methodology is evaluated using both artificial and real data. Artificial data are employed in order to present a working example of the proposed methodology, while real data correspond to two biological problems of high importance: protein fold recognition and class prediction.

The pattern weights that are assigned to the sequential patterns, after the optimization stage, identify the relative significance of each pattern. The motivation for the use of class weights is that sequential patterns extracted from sequences do not describe all classes with the same adequacy; some classes are over described from the sequential patterns while for others this description is rather poor. Thus, the class weights are introduced to equalize this preference, and subsequently, an optimization technique is used to automatically calculate optimal values for them. To our knowledge, there is no other work in the literature which assigns weights to the extracted sequential patterns and to the classes for sequence classification. Our results indicate that this integration leads to high classification accuracy, superior to previously reported sequence classification methods. Furthermore, the weights assigned to the patterns can provide to the experts additional knowledge on the domain of application, through the identification of the most important patterns. Finally, the proposed methodology is generic and can incorporate different algorithms/approaches in any of its stages.