Dynamic classifier selection: Recent advances and perspectives

Highlights

• An updated taxonomy of Dynamic Selection techniques is proposed.

• A review of the state-of-the-art dynamic selection techniques is presented.

• Empirical comparison between 18 dynamic selection techniques is conducted.

• We discuss about the recent findings and open research question in this field.

Abstract

Multiple Classifier Systems (MCS) have been widely studied as an alternative for increasing accuracy in pattern recognition. One of the most promising MCS approaches is Dynamic Selection (DS), in which the base classifiers are selected on the fly, according to each new sample to be classified. This paper provides a review of the DS techniques proposed in the literature from a theoretical and empirical point of view. We propose an updated taxonomy based on the main characteristics found in a dynamic selection system: (1) The methodology used to define a local region for the estimation of the local competence of the base classifiers; (2) The source of information used to estimate the level of competence of the base classifiers, such as local accuracy, oracle, ranking and probabilistic models, and (3) The selection approach, which determines whether a single or an ensemble of classifiers is selected. We categorize the main dynamic selection techniques in the DS literature based on the proposed taxonomy. We also conduct an extensive experimental analysis, considering a total of 18 state-of-the-art dynamic selection techniques, as well as static ensemble combination and single classification models. To date, this is the first analysis comparing all the key DS techniques under the same experimental protocol. Furthermore, we also present several perspectives and open research questions that can be used as a guide for future works in this domain.

Introduction

Multiple Classifier System (MCS) is a very active area of research in machine learning and pattern recognition. In recent years, several studies have been published demonstrating its advantages over individual classifier models based on theoretical [1], [2], [3] and empirical [4], [5], [6] evaluations. They are widely used to solve many real-world problems, such as face recognition [7], music genre classification [8], credit scoring [9], [10], class imbalance [11], recommender system [12], [13], software bug prediction [14], [15], intrusion detection [16], [17], and for dealing with changing environments [18], [19], [20].

Several approaches are currently used to construct an MCS, and they have been presented in many excellent reviews covering different aspects of MCS [3], [6], [21], [22], [23]. One of the most promising MCS approaches is Dynamic Selection (DS), in which the base classifiers¹ are selected on the fly, according to each new sample to be classified. DS has become an active research topic in the multiple classifier systems literature in past years. This has been due to the fact that more and more works are reporting the superior performance of such techniques over traditional combination approaches, such as majority voting and Boosting [24], [25], [26], [27]. DS techniques work by estimating the competence level of each classifier from a pool of classifiers. Only the most competent, or an ensemble containing the most competent classifiers is selected to predict the label of a specific test sample. The rationale for such techniques is that not every classifier in the pool is an expert in classifying all unknown samples; rather, each base classifier is an expert in a different local region of the feature space [28].

In dynamic selection, the key is how to select the most competent classifiers for any given query sample. Usually, the competence of the classifiers is estimated based on a local region of the feature space where the query sample is located. This region can be defined by different methods, such as applying the K-Nearest Neighbors technique, to find the neighborhood of this query sample, or by using clustering techniques [29], [30]. Then, the competence level of the base classifiers is estimated, considering only the samples belonging to this local region according to any selection criteria; these include the accuracy of the base classifiers in this local region [30], [31], [32] or ranking [33] and probabilistic models [25], [34]. At this point, the classifier(s) that attained a certain competence level is (are) selected.

In this paper, we present an updated taxonomy of dynamic classifier and ensemble selection techniques, taking into account the following three aspects: (1) The selection approach, which considers, whether a single classifier is selected (this is known as Dynamic Classifier Selection (DCS)) or an ensemble is selected (this for its part is known as Dynamic Ensemble Selection (DES)); (2) The method used to define the local region in which the local competences of the base classifiers are estimated, and (3) The selection criteria used to estimate the competence level of the classifier. We review and categorize the state-of-the-art dynamic classifier and ensemble selection techniques based on the proposed taxonomy.

We also discuss the increasing use of dynamic selection techniques, considering different classification contexts, such as One-Class Classification (OCC) [35], concept drift [36], [37], [38], One-Versus-One (OVO) decomposition problems [39], [40], [41], as well as the application of DS techniques to solve complex real-world problems such as signature verification [42], face recognition [7], [43], [44], [45], music classification [8] and credit scoring [10]. In particular, we describe how the properties of dynamic selection techniques can be used to handle the intrinsic properties of each problem.

An experimental analysis is conducted comparing the performance of 18 state-of-the-art dynamic classifier and ensemble selection techniques over multiple classification datasets. The DS techniques are also compared against the baseline methods, namely, (1) Static Selection (SS), i.e., the selection of an ensemble of classifiers during the training stage of the system [46]; (2) Single Best (SB), which corresponds to the performance of the best classifier in the pool according to the validation data, and majority voting (MV), which corresponds to the majority voting combination of all classifiers in the pool without any pre-selection of classifiers. To allow a fair comparison of the techniques, all the DS and static techniques were evaluated using the same experimental protocol, i.e., the same division of datasets, as well as the same pool of classifiers. The performance of the DS techniques was also compared with those of the best classification models according to [4], including Support Vector Machine (SVM) and Random Forests.

The contributions of this paper in relation to other reviews in classifiers ensembles are:

• It proposes an updated taxonomy of dynamic selection techniques.

• It discusses the use of dynamic selection techniques on different contexts, including One-Versus-One decomposition (OVO), and One-Class Classification (OCC).

• It reviews the use of DS techniques to solve complex real-world problems such as image classification and biomedical applications.

• It presents an empirical comparison between several state-of-the-art dynamic selection techniques over several classification datasets under the same experimental protocol.

• It discusses the most recent findings in this field, and examines the open questions that can be addressed in future works.

This work is organized as follows: Section 2 presents an overview of multiple classifier system approaches. In Section 3, we propose an updated dynamic selection taxonomy, and discuss each component of a DS technique. In Section 4, we describe the most relevant dynamic selection methods and categorize them according to the proposed taxonomy. Section 5 presents a review of several real-world applications that use dynamic selection techniques to achieve a higher classification accuracy. An empirical comparison between the state-of-the-art DS techniques is conducted in Section 6. The conclusion and perspectives for future research in dynamic selection are given in the last section.