Multiobjective genetic classifier selection for random oracles fuzzy rule-based classifier ensembles: How beneficial is the additional diversity?
Introduction
Classifier ensembles (CEs), also called multiclassifiers, are well-recognized tools in the machine learning community and more recently in the soft computing community. They are able not only to outperform a single classifier but also to deal with complex and high dimensional classification problems [1].
In a preceding contribution [2], we incorporated Random Linear Oracles (RLOs) [3], a classical CE design methodology, into a previously proposed CE framework [4] to derive fuzzy rule-based classifier ensembles (FRBCEs). Thanks to the additional diversity introduced by RLOs into the robust FURIA-based fuzzy classifiers [5], [6], the obtained FRBCEs were able to achieve an outstanding performance in terms of accuracy, outperforming RLO combined with the classical base classifiers.
Nevertheless, the performance of FRBCEs can still be improved. It has been theoretically and empirically shown that smaller ensembles can outperform larger ones [7], [8], [9]. Thus, selecting a subset of classifiers is a natural way to follow. In our previous contributions, we used the well known overproduce-and-choose strategy [10] (OCS) to reduce the CE dimensionality, while improving its accuracy. OCS is a classifier selection method based on the generation of a large number of component classifiers and a subsequent selection of the best cooperating subset of them.
Therefore, OCS helps to obtain a good accuracy-complexity trade-off in the CE design as well as in many cases it also improves the accuracy of the final CE. In fact, these characteristics were exhibited in [11] for FRBCEs using an OCS strategy based on NSGA-II [12]. NSGA-II, which is a state-of-the-art evolutionary multi-objective (EMO) algorithm [13], generated a set of CE designs with different accuracy-complexity trade-offs in a single run.
In this contribution, we introduce two novel aspects to our FRBCE design methodology in [2] in order to improve the CE accuracy, while reducing its complexity:
- 1.
To keep a high diversity in the set of classifiers as well as high performance, we incorporate a new Random Oracle (RO) approach, namely the Random Spherical Oracle (RSO) [14], into the FRBCE framework. Opposite to RLO, RSO uses an oracle based on a random hypersphere to divide the feature space into two regions in order to feed two subclassifiers, which both compose the final RSO. We expect to improve the performance of the FRBCEs by combining the RSO randomness and its oracle shapes with the “soft boundaries” provided by the FURIA-based component classifier.
- 2.
To reduce the complexity, we design a specific EMO-based OCS strategy for RO-based FRBCEs from our previous proposal in [11]. Since RO is composed of two base classifiers, this approach offers a tremendous advantage over bagging FURIA-based component classifiers because each classifier can be independently selected within each pair component. A higher degree of freedom is achieved during the selection procedure, while still having the potential of drastically reducing the complexity.
On the one hand, we aim to obtain a good accuracy-complexity trade-off when dealing with high complexity datasets. While the main goal in the design of CEs is to obtain an accurate system, the complexity is an interesting secondary objective allowing us to obtain simpler and quicker CEs. On the other hand, we aim to analyze whether the additional diversity induced by ROs is beneficial for the EMO OCS-based FRBCEs. That is, our goal is to check if the OCS-based selection leads to more accurate results when applied on RO-based FURIA fuzzy CEs than on bagging fuzzy CEs thanks to the additional freedom degrees resulting from the RO design. For that purpose, we use a novel NSGA-II design with a three-objective fitness function including an advanced accuracy measure as well as complexity and diversity indices for the component classifier selection. Specifically, we propose a special binary coding for NSGA-II in order to take advantage of the additional degrees of freedom offered by the RO base classifiers, and test two different mutation operator settings to look for the best performance.
To perform the experimental analysis, we carry out exhaustive experiments on 29 high complexity datasets from the UCI machine learning [15] and the KEEL dataset [16] repositories.
This paper is set up as follows. In the next section, the preliminaries required for a good understanding of our work are reviewed. Section 3 presents RLOs, RSOs, both RLO- and RSO-based FRBCEs, and a set of experiments focused on the comparison of different RO-based strategies for the combination of the component classifiers. Then, Section 4 introduces our NSGA-II proposal for RSO component fuzzy classifier selection incorporating a three-objective fitness function and the analysis of the experiments performed. Finally, Section 5 concludes this contribution with some future research lines.
Section snippets
Preliminaries
This section explores the current literature related to the generation of a FRBCE. The techniques used to generate CEs and fuzzy CEs are described in Sections 2.1 Classifier ensembles design methodologies, 2.2 Related work on fuzzy classifier ensembles, respectively. Some ways to reduce the size of the ensembles are described in Section 2.3. The use of genetic algorithms (GAs) within the OCS strategy is explored in Section 2.4. Finally, we briefly introduce evolutionary fuzzy systems in Section
Using random oracles to design fuzzy rule-based classifier ensembles
An RO [3], [14] is a structured classifier, also defined as a “mini-ensemble”, encapsulating the base classifier of the CE. It is composed of two classifiers and an oracle that decides which one to use in each case. Basically, the oracle is a random function whose objective is to randomly split the dataset into two subsets by dividing the feature space into two regions. Each of the two generated regions and the corresponding data subset is assigned to one classifier. Any shape for the decision
EMO OCS classifier selection for RSO-based FRBCEs
The second part of this contribution introduces the use of the OCS strategy for classifier selection in RO-based fuzzy CEs. On the one hand, the aim is to refine the accuracy-complexity trade-off in the RO-based bagging FRBCEs (both in RLO and RSO) when dealing with high complexity classification problems. On the other hand, an interesting objective is to study whether the additional diversity induced by ROs is beneficial for the EMO OCS-based component fuzzy classifier selection. Thus, we
Conclusions and future works
In this contribution, we focused on two aims. Firstly, we introduced a new RSO approach into our previous FRBCE design framework. Secondly, we incorporated the EMO-based OCS strategy to these kinds of classifiers analyzing the influence of the additional RO diversity in the final FRBCEs performance. We used an advanced accuracy measure and proposed a specific binary coding for the RO-based classifier selection. A three-objective fitness function using three different optimization criteria such
Acknowledgment
We would like to thank Ludmila I. Kuncheva and Juan José Rodríguez for their kind help with answering all the questions regarding ROs and sharing their source code with us.
This work was supported by the Spanish Ministerio de Economía y Competitividad under Projects TIN2012-38525-C02-01 and TIN2011-24302, including funding from the European Regional Development Fund.
References (62)
- et al.
Ensembling neural networks: many could be better than all
Artificial Intelligence
(2002) - et al.
Diversity in search strategies for ensemble feature selection
Information Fusion
(2005) - et al.
A fuzzy random forest
International Journal of Approximate Reasoning
(2010) - et al.
A dynamic overproduce-and-choose strategy for the selection of classifier ensembles
Pattern Recognition
(2008) - et al.
Classifier selection for majority voting
Information Fusion
(2005) - et al.
Looking for a good fuzzy system interpretability index: an experimental approach
International Journal of Approximate Reasoning
(2009) - et al.
Ten years of genetic fuzzy systems: current framework and new trends
Fuzzy Sets and Systems
(2004) A historical review of evolutionary learning methods for mamdani-type fuzzy rule-based systems: designing interpretable genetic fuzzy systems
International Journal of Approximate Reasoning
(2011)- et al.
Voting in fuzzy rule-based systems for pattern classification problems
Fuzzy Sets and Systems
(1999) - et al.
A proposal on reasoning methods in fuzzy rule-based classification systems
International Journal of Approximate Reasoning
(1999)
Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power
Information Sciences
A hierarchical genetic fuzzy system based on genetic programming for addressing classification with highly imbalanced and borderline data-sets
Knowledge-Based Systems
Combining Pattern Classifiers: Methods and Algorithms
Classifier ensembles with a random linear oracle
IEEE Transactions on Knowledge and Data Engineering
On designing fuzzy rule-based multiclassification systems by combining FURIA with bagging and feature selection
International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems
FURIA: an algorithm for unordered fuzzy rule induction
Data Mining and Knowledge Discovery
Bagging predictors
Machine Learning
Making use of population information in evolutionary artificial neural networks
IEEE Transactions on Systems Man and Cybernetics Part B: Cybernetics
Engineering multiversion neural-net systems
Neural Computation
A study on the use of multi-objective genetic algorithms for classifier selection in FURIA-based fuzzy multiclassifiers
International Journal of Computational Intelligence Systems
A fast and elitist multiobjective genetic algorithm: NSGA-II
IEEE Transactions on Evolutionary Computation
Evolutionary Algorithms for Solving Multi-Objective Problems
Naïve bayes ensembles with a random oracle
Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework
Journal of Multiple-Valued Logic and Soft Computing
An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization
Machine Learning
A comparison of decision tree ensemble creation techniques
IEEE Transactions on Pattern Analysis and Machine Intelligence
Popular ensemble methods: an empirical study
Journal of Artificial Intelligence Research
The strength of weak learnability
Machine Learning
Cited by (23)
Evolutionary fuzzification of RIPPER for regression: Case study of stock prediction
2019, NeurocomputingCitation Excerpt :In addition, this MF consists of simple straight line segments, so it is easy to be implemented. Moreover, triangular MF is simple and linear and is capable to model nonlinear and complex treatments, when it is employed in creation of a fuzzy system [13,26,38,53,54]. The intervals in the crisp classification rules of RIPPER are replaced with corresponding triangular fuzzy numbers.
A comparative assessment of GIS-based data mining models and a novel ensemble model in groundwater well potential mapping
2017, Journal of HydrologyCitation Excerpt :In other investigations, Tien Bui et al. (2012), and Pham et al. (2015) reported higher performance of the Bagging model than AdaBoost model in landslide susceptibility mapping which is consistent with the results of the current study. Bagging is sensitive to variations in training dataset (Tien Bui et al., 2014), and has superior variance mitigation versus AdaBoost (Ganjisaffar et al., 2011), which is capable for dealing with multi-dimensional and complex classification problems (Tien Bui et al., 2014; Trawiński et al., 2013), and is more immune to noise in dataset (Dietterich, 1997). These strong features could have caused for its better performance among the data mining models.
Complexity-based parallel rule induction for multiclass classification
2017, Information SciencesRipMC: RIPPER for Multiclass Classification
2016, NeurocomputingCitation Excerpt :This algorithm, which extends its predecessors including IREP, is the first to effectively tackle the overfitting problem. It is still considered state-of-the-art in inductive rule learning [1,28,29]. It is a powerful classification rule learning algorithm that can deal with a very common problem of rule-based classification problems, the so-called curse of dimensionality.
ACORI: A novel ACO algorithm for rule induction
2016, Knowledge-Based Systems