Clustering-based ensembles for one-class classification
Introduction
Well-known and reliable classifiers tend to fail when faced with new problems such as an atypical class distribution, non-stationary environments, or massive data. Therefore, new methods must be developed to deal with the challenges arising and improve the quality of real-life decision support systems.
One of these newly introduced methodologies is known as one-class classification (OCC) [31], which assumes that during the training stage only objects originating from a single class are available. These are called the target concept and are denoted by . The purpose of OCC is to calculate a decision boundary that encloses all available data samples, thereby describing the concept [53]. During the execution phase, new objects, unseen during training, may appear. These may originate from one or more distributions and represent data outside the target concept. Such objects, denoted by , are referred to as outliers.
For a single OCC classifier it may be difficult or even impossible to find a good model owing to limited training data, high feature space dimensionality, and/or the properties of the particular classifier. To avoid a too complex model and overfitting of the training target data, a simpler model with a lower number of features or one that has been trained with smaller chunks of data, can be created. Although the complexity of such a model is reduced, the quality thereof also declines significantly. However, it has been shown that a group of individual OCC models can help alleviate the aforementioned problems.
Here one may use an approach known as multiple classifier systems (MCSs), which is considered to be one of the fastest growing fields in machine learning [26]. MCSs are based on the idea of combining several classifiers into a compound recognition system that can exploit the strengths of individual predictors [60]. Each classifier may output a different decision boundary, and so have different competence areas over the analyzed dataset [7]. When combined, the collective decision accuracy can outperform any of the individual predictors. However, several important issues, such as selecting the individual classifiers, as well as choosing a fusion method to establish a group decision, must be considered when designing an MCS. Classifiers used to create the ensemble in an ideal situation should be highly accurate and complement each other (i.e., the ensemble should display high diversity). Adding classifiers that are not diverse with respect to those already in the pool will not improve the accuracy of the compound classifier, but will only increase the overall computational cost [5]. It is worth noting that combination rules, for example, majority voting, could even lead to a deterioration in performance of the ensemble of classifiers [36]. On the other hand, building an MCS with highly diverse but poor quality classifiers will result in a weak committee. Therefore, classifier selection is a critical step in the ensemble design process [15].
MCSs are an attractive yet still largely unexplored, alternative for OCC problems. Most of the works concentrate on practical applications of OCC ensembles. Much still needs to be done to gain insight into the theoretical background to this problem, as well as to draw conclusions on how to build efficient OCC ensembles regardless of the intended application [35].
We propose an approach based on the idea of data clustering in the feature space. OCC models are built based on each of the clusters. In this way we ensure that the pool of predictors is highly diverse and mutually complementary (owing to training on different inputs, i.e., clusters of training objects). This can be seen as an extension of the popular family of ensembles derived from the idea of clustering and selection proposed by Kuncheva [37]. So far, two other research teams have worked on this topic, proposing very simple hybrid methods for combining clustering and OCC [38], [45].
The contributions of this work are as follows:
- •
We propose building an ensemble of one-class classifiers based on clustering of the target class. This ensures initial diversity among the classifiers in the pool (as they are based on different inputs) and the correct handling of possible issues embedded in the nature of data, such as a rare distribution or chunks of objects.
- •
We propose an elastic and efficient framework for this task, which requires only the selection of several components, namely, the clustering algorithm, individual classifier model, and fusion method. These can easily be chosen by the user, as there are practically no limitations on their nature. All other parameters for the method are selected automatically.
- •
We discuss the possibility of extending our one-class ensemble to an efficient tool for multi-class problems.
- •
We carry out extensive computational tests on a diverse set of benchmarks that highlight the influence of component selection on the overall method quality and show that the proposed approach outperforms the standard OCC methods as well as a single multi-class support vector machine (SVM) in multi-class classification problems.
Our ensemble is easy to use in many practical applications where it is difficult or even impossible to obtain counter-examples (e.g., machine fault diagnosis), or where, owing to a complex data distribution, the class decomposition approach can lead to a significant improvement in recognition quality over the well known multi-class approaches (e.g., imbalanced classification).
This paper is organized as follows. In the next section the idea of OCC is presented. In Section 3 the architecture of the proposed compound recognition system is explained. The components that must be selected as input for the system are also presented. In Section 4 the experimental results are presented and discussed. The paper ends with the presentation of our conclusions in Section 5.
Section snippets
One-class classification
OCC aims to distinguish the target concept objects from possible outliers, and hence it is often referred to as learning in the absence of counter-examples. Although OCC is quite similar to binary classification, the primary difference lies in how the one-class classifier is trained. In standard dichotomy problems it is expected that objects from the other classes tend to come from one direction. Here the available class must be separated from all the possible outliers, which leads to a
Architectures for the proposed method
In this paper we propose a new architecture for creating ensembles of one-class classifiers based on the clustering of a feature space into smaller partitions. Additionally, we incorporate our new compound classifier into an architecture that allows both one-class and multi-class problems to be solved. Therefore, in this section we describe our algorithm from two different perspectives – a local perspective (the details of the introduced one-class clustering based ensemble) and a global
Experimental investigation
In this section, we present the results of thorough experimental investigation examining the behavior of the proposed one-class ensemble approach. The aim of the experiments was to assess the quality of the OCClustE components tested (clustering methods, classification algorithms, and fusers) and to compare the proposed method with known approaches for multi-class decomposition using one-class classifiers, i.e., where a single one-class classifier is assigned to each of the classes.
Our aim is
Conclusion and future work
This paper presented a method for creating a one-class classifier ensemble based on feature space partitioning. We proposed a two-level architecture for the design of such a classification system. The main advantage of the proposed method is that the combined classifiers trained on the basis of clusters allow us to exploit individual classifier strengths. As a result, these usually outperform traditional methods for one-class classifier combinations for multi-class classification problems
Acknowledgments
The work was supported by the Polish National Science Centre under Grant No. N519 576638 for the years 2010–2013, as well as by the Polish National Science Centre Grant No. DEC-2011/01/B/ST6/01994.
References (65)
- et al.
Fcm: the fuzzy c-means clustering algorithm
Comput. Geosci.
(1984) The impact of diversity on the accuracy of evidential classifier ensembles
Int. J. Approx. Reason.
(2012)- et al.
Soft clustering using weighted one-class support vector machines
Pattern Recogn.
(2009) - et al.
Dynamic fusion method using localized generalization error model
Inform. Sci.
(2012) A competitive ensemble pruning approach based on cross-validation technique
Knowl.-Based Syst.
(2013)- et al.
An overview of ensemble methods for binary classifiers in multi-class problems: experimental study on one-vs-one and one-vs-all schemes
Pattern Recogn.
(2011) - et al.
Supervised subspace projections for constructing ensembles of classifiers
Inform. Sci.
(2012) - et al.
Intrusion detection in computer networks by a modular ensemble of one-class classifiers
Inf. Fusion
(January 2008) - et al.
Minimum spanning tree based one-class classifier
Neurocomputing
(2009) - et al.
Cueing, feature discovery, and one-class learning for synthetic aperture radar automatic target recognition
Neural Netw.
(1995)
One-class document classification via neural networks
Neurocomputing
Vocs classification based on the committee of classifiers coupled with single sensor signals
Chemometr. Intell. Lab. Syst.
Improving fuzzy c-means clustering based on feature-weight learning
Pattern Recogn. Lett.
Soft computing methods applied to combination of one-class classifiers
Neurocomputing
Coding and decoding strategies for multi-class learning problems
Inform. Fusion
A survey of multiple classifier systems as hybrid systems
Inform. Fusion
Combined 5 × 2 cv f test for comparing supervised classification learning algorithms
Neural Comput.
Outliers analysis and one class classification approach for planetary gearbox diagnosis
J. Phys.: Conf. Ser.
Pattern Recognition With Fuzzy Objective Function Algorithms
LIBSVM: a library for support vector machines
ACM Trans. Intell. Syst. Technol.
One-cluster clustering based data description
Jisuanji Xuebao/Chinese J. Comput.
One-class svm for learning in image retrieval
IEEE Int. Conf. Image Process.
Fuzzy one-mean algorithm: Formulation, convergence analysis, and applications
J. Intell. Fuzzy Syst.
Novelty detection using one-class parzen density estimator: an application to surveillance of nosocomial infections
Stud. Health Technol. Inform.
One-class support vector ensembles for image segmentation and classification
J. Math. Imag. Vis.
Solving multiclass learning problems via error-correcting output codes
J. Artif. Int. Res.
A fast k-means implementation using coresets
Int. J. Comput. Geometry Appl.
One-class novelty detection for seizure analysis from intracranial eeg
J. Machine Learn. Res.
The random subspace method for constructing decision forests
IEEE Trans. Pattern Anal. Mach. Intell.
Cited by (126)
Time series clustering to improve one-class classifier performance[Formula presented]
2024, Expert Systems with ApplicationsBounded exponential loss function based AdaBoost ensemble of OCSVMs
2024, Pattern RecognitionPrivileged multi-view one-class support vector machine
2024, NeurocomputingProximity-based density description with regularized reconstruction algorithm for anomaly detection
2024, Information SciencesPredicting the risky encounters without distance knowledge between the ships via machine learning algorithms
2023, Expert Systems with ApplicationsClustering ensemble-based novelty score for outlier detection
2023, Engineering Applications of Artificial Intelligence