Fuzzy Passive–Aggressive classification: A robust and efficient algorithm for online classification problems

Abstract

Fuzzy weighting, which is designed to reduce the effects of outliers for batch classification problems, might generate unreasonable membership grades especially for the samples following an input outlier, when incorporated into online classification algorithms directly. In this paper, a generalized framework for online fuzzy weighting is presented, which incrementally calculates the membership of each incoming sample by taking into account the membership grades of previous samples in a pairwise manner. The advocated pairwise-distance based scheme can not only identify possible outliers, but also show good adaptation to the sequentially received samples in the online setting. We apply it to online Passive–Aggressive (PA) algorithm in a direct way. The resulting Fuzzy Passive–Aggressive (FPA) algorithm achieves comparable classification accuracy with benchmark incremental SVM, while still enjoying the time efficiency of simple PA, which is a Perceptron-like algorithm. Besides, FPA exhibits the best performance among PA family, which makes it a robust and efficient alternative to PA, in order to deal with unavoidable outliers in large-scale or high-dimensional real datasets. The study is supported by a series of experiments with IDA benchmark repository, as well as two real-world problems namely place recognition and radar emitter recognition.

Introduction

Nowadays machine learning techniques, such as K-NNs [16], Bayes classifiers [18], [47], decision trees [11], [36], and support vector machines (SVMs) [8], [13], [42], have been widely used for classification problems [19]. However, in practical applications, when the data arrive over long periods of time or the storage capacities are very limited (processing of data at once is impossible), these techniques fail to work well. Therefore, online learning is of increasing importance to deal with endless stream of received data such as stock market indexes, sensor data, and video streams [6]. For a system involved in online learning, it should allow to incorporate additional training data, when it is available, without re-training from scratch.

In the online setting, samples are received in a sequential manner. The model makes a prediction label for the current sample on each round. Then it may select to update the prediction mechanism or not, upon receiving the correct label. Perceptron [39] is a simple but efficient online learning algorithm, which has been extensively studied to enhance its generalization power. Crammer et al. [15] presented an online Passive–Aggressive (PA) algorithm, which is as fast as but more accurate than Perceptron. It updates the model to have a low loss on the new sample, as well as to ensure that the new model is close to the current one. Orabona et al. [34] proposed the Projectron algorithm, which is also based on Perceptron. But the number of online hypothesis in this method is bounded by projecting the samples onto the space spanned by the previous online hypothesis, instead of discarding them. In [20], ALMA_p was presented, which approximates the maximal margin hyperplane for a set of linearly separable data. Additionally, SVMs have been modified for many incremental versions [9], [17], [24], which formulate an online optimization problem in general. The motivation of these works is to maintain the KKT conditions by varying the margin vector coefficients, in response to the perturbation imparted by the incremented new coefficient. To sum up, quite a few online machine learning algorithms have been developed to deal with the huge or stream data.

For real-world problems, the application of fuzzy set theory (FST) based model has also become attractive and visible, especially in the realms of automatic control and pattern recognition [29], [38]. The vague and imprecise expressions that are used by human beings to describe practical processes could be modeled gracefully by FST [35]. Linguistic rules based fuzzy classifiers (FCs) [4], [5], [37], [40] provide a comprehensive way to illustrate underlying concepts of complicated systems. In [1], an efficient method was discussed for extracting fuzzy rules directly from numerical input–output data for pattern classification. Mencar et al. [31] designed fuzzy rule-based classifiers from data with the constraint of semantic cointension. Moreover, fuzzy rules can be also extracted from the trained SVM models [10]. Generally speaking, these classifiers, supported by FST, are mainly structured on fuzzy-rules.

It is of great interests to realize the incremental application of a fuzzy-classification method, aiming to update fuzzy rules or generate a new one in an online way. In [2], a recursive approach using online clustering for adaptation of fuzzy-rule based model structure was discussed. This algorithm is instrumental for online identification of T–S models. A new approach was proposed in [7] for learning fuzzy rules in an incremental way. As new data arrive, new rules may be discovered while the existing ones may be updated or partially removed. Lughofer [28] presented two families of online evolving classifiers for image classification. The developed classifiers are initially trained on some pre-labeled training data and further updated according to newly recorded samples. As a whole, this research line has shown substantial potential for incremental adaptation of the model parameters and high understandability of fuzzy system in dynamically changing environments.

However, the research community should not neglect another direction in which FST is applied to machine learning directly, in order to deal with outliers and noise in real-world classification problems. Traditional machine learning approaches assumed that each sample has equal importance in the training procedure. However, in realistic applications, different samples should make different contributions to the decision boundary of involved classifiers [46]. In order to increase the robustness of the classifiers, fuzzy weighting is introduced naturally to score each sample. Different from above fuzzy-classification methods, which assign membership degrees for the outputs generated by classifiers together with the output class or extract fuzzy rules from the trained models, the methods in this category simply apply a membership value to define the importance of each training pattern and train the models on fuzzified input data [3], [30]. In [41], a robust support vector machine was proposed to solve the over-fitting problem. Lin and Wang [25] proposed a fuzzy SVM, which associates each sample with a membership value such that different samples have different effects on the learning of the separating hyperplane. Another similar effort can be found in [22]. Chen and Chen [14] introduced a membership function to kernel Perceptron for solving the linearly non-separable problems. Then in [21], a robust membership calculation scheme was presented, which performs well on noisy data. However, these methods of membership generation are only designed for specific data distributions or assume that training samples are received batchwise. Direct extension of such strategies to online learning might cause new problems because the knowledge of distribution information in the initial stages is not correct. More importantly, new decision boundary needs to be recalculated using entire current training set upon receiving a new sample, which is time consuming. To the best of our knowledge, few literatures have been devoted to this subject. Therefore a robust and efficient framework of incremental fuzzy weighting is imperative for online classification.

Our work follows the second direction. We consider the online membership generation only in the context of the fuzzified input space. It is worthwhile to highlight the main contributions of this paper here:

• A generalized framework of membership calculation is presented, which can be properly integrated into many online classification algorithms. Compared with the center-distance or margin-distance based schemes, the advocated pairwise-distance based membership generation scheme can not only identify possible outliers, but also show better adaptation to the sequentially received samples in the online setting.

• The proposed scheme for membership generation is incorporated into the online Passive–Aggressive algorithm in a direct way. The resulting Fuzzy Passive–Aggressive (FPA) approach achieves competitive classification performance with benchmark incremental SVM, especially when received training samples are insufficient. Besides, FPA exhibits the best robustness to outliers and label noise among PA family, which is demonstrated by IDA benchmark tests and two realistic recognition tasks.

• A detailed theoretical analysis on the computational complexity of related PA algorithms is provided. In particular, we show that FPA shares almost the same time efficiency as standard PA, which is a Perceptron-like algorithm. The slight increase in training time introduced by pairwise-distance based fuzzy weighting is acceptable in the online setting, compared to another two possible fuzzy extensions. The above merits make the proposed FPA a robust and efficient alternative to PA, in order to deal with unavoidable outliers in large-scale or high-dimensional real datasets.

This rest of paper is organized as follows. In section 2 we set up the problem and describe our generalized online weighting framework. Section 3 presents the fuzzy PA family, along with the time complexity analysis. A variety of experimental results are presented in Section 4. Finally, we conclude in Section 5 with a discussion of future work.