Elsevier

Knowledge-Based Systems

Volume 54, December 2013, Pages 3-21
Knowledge-Based Systems

Multiobjective genetic classifier selection for random oracles fuzzy rule-based classifier ensembles: How beneficial is the additional diversity?

https://doi.org/10.1016/j.knosys.2013.08.006Get rights and content

Abstract

Recently we proposed the use of the Random Linear Oracles classical classifier ensemble (CE) design methodology in a fuzzy environment. It derived fuzzy rule-based CEs obtaining an outstanding performance. Random Oracles introduce an additional diversity into the base classifiers improving the accuracy of the entire CE. Meanwhile, the overproduce-and-choose strategy leads to a good accuracy-complexity trade-off. It is based on the generation of a large number of component classifiers and a subsequent selection of the best cooperating subset of them. The current contribution has a twofold aim: (1) Introduce a new Random Oracles approach into the fuzzy rule-based CEs design; (2) Incorporate an evolutionary multi-objective overproduce-and-choose strategy to our approach analyzing the influence of this additional diversity in the final CE performance (focusing on the accuracy). To do so, firstly, we incorporate the two Random Oracle variants into the fuzzy rule-based CE framework. Then, we use NSGA-II to provide a specific component classifier selection driven by three different criteria. Exhaustive experiments are carried out over 29 UCI and KEEL datasets with high complexity (considering both the number of attributes as well as the number of examples) showing the good performance of the proposed approach.

Introduction

Classifier ensembles (CEs), also called multiclassifiers, are well-recognized tools in the machine learning community and more recently in the soft computing community. They are able not only to outperform a single classifier but also to deal with complex and high dimensional classification problems [1].

In a preceding contribution [2], we incorporated Random Linear Oracles (RLOs) [3], a classical CE design methodology, into a previously proposed CE framework [4] to derive fuzzy rule-based classifier ensembles (FRBCEs). Thanks to the additional diversity introduced by RLOs into the robust FURIA-based fuzzy classifiers [5], [6], the obtained FRBCEs were able to achieve an outstanding performance in terms of accuracy, outperforming RLO combined with the classical base classifiers.

Nevertheless, the performance of FRBCEs can still be improved. It has been theoretically and empirically shown that smaller ensembles can outperform larger ones [7], [8], [9]. Thus, selecting a subset of classifiers is a natural way to follow. In our previous contributions, we used the well known overproduce-and-choose strategy [10] (OCS) to reduce the CE dimensionality, while improving its accuracy. OCS is a classifier selection method based on the generation of a large number of component classifiers and a subsequent selection of the best cooperating subset of them.

Therefore, OCS helps to obtain a good accuracy-complexity trade-off in the CE design as well as in many cases it also improves the accuracy of the final CE. In fact, these characteristics were exhibited in [11] for FRBCEs using an OCS strategy based on NSGA-II [12]. NSGA-II, which is a state-of-the-art evolutionary multi-objective (EMO) algorithm [13], generated a set of CE designs with different accuracy-complexity trade-offs in a single run.

In this contribution, we introduce two novel aspects to our FRBCE design methodology in [2] in order to improve the CE accuracy, while reducing its complexity:

  • 1.

    To keep a high diversity in the set of classifiers as well as high performance, we incorporate a new Random Oracle (RO) approach, namely the Random Spherical Oracle (RSO) [14], into the FRBCE framework. Opposite to RLO, RSO uses an oracle based on a random hypersphere to divide the feature space into two regions in order to feed two subclassifiers, which both compose the final RSO. We expect to improve the performance of the FRBCEs by combining the RSO randomness and its oracle shapes with the “soft boundaries” provided by the FURIA-based component classifier.

  • 2.

    To reduce the complexity, we design a specific EMO-based OCS strategy for RO-based FRBCEs from our previous proposal in [11]. Since RO is composed of two base classifiers, this approach offers a tremendous advantage over bagging FURIA-based component classifiers because each classifier can be independently selected within each pair component. A higher degree of freedom is achieved during the selection procedure, while still having the potential of drastically reducing the complexity.

On the one hand, we aim to obtain a good accuracy-complexity trade-off when dealing with high complexity datasets. While the main goal in the design of CEs is to obtain an accurate system, the complexity is an interesting secondary objective allowing us to obtain simpler and quicker CEs. On the other hand, we aim to analyze whether the additional diversity induced by ROs is beneficial for the EMO OCS-based FRBCEs. That is, our goal is to check if the OCS-based selection leads to more accurate results when applied on RO-based FURIA fuzzy CEs than on bagging fuzzy CEs thanks to the additional freedom degrees resulting from the RO design. For that purpose, we use a novel NSGA-II design with a three-objective fitness function including an advanced accuracy measure as well as complexity and diversity indices for the component classifier selection. Specifically, we propose a special binary coding for NSGA-II in order to take advantage of the additional degrees of freedom offered by the RO base classifiers, and test two different mutation operator settings to look for the best performance.

To perform the experimental analysis, we carry out exhaustive experiments on 29 high complexity datasets from the UCI machine learning [15] and the KEEL dataset [16] repositories.

This paper is set up as follows. In the next section, the preliminaries required for a good understanding of our work are reviewed. Section 3 presents RLOs, RSOs, both RLO- and RSO-based FRBCEs, and a set of experiments focused on the comparison of different RO-based strategies for the combination of the component classifiers. Then, Section 4 introduces our NSGA-II proposal for RSO component fuzzy classifier selection incorporating a three-objective fitness function and the analysis of the experiments performed. Finally, Section 5 concludes this contribution with some future research lines.

Section snippets

Preliminaries

This section explores the current literature related to the generation of a FRBCE. The techniques used to generate CEs and fuzzy CEs are described in Sections 2.1 Classifier ensembles design methodologies, 2.2 Related work on fuzzy classifier ensembles, respectively. Some ways to reduce the size of the ensembles are described in Section 2.3. The use of genetic algorithms (GAs) within the OCS strategy is explored in Section 2.4. Finally, we briefly introduce evolutionary fuzzy systems in Section

Using random oracles to design fuzzy rule-based classifier ensembles

An RO [3], [14] is a structured classifier, also defined as a “mini-ensemble”, encapsulating the base classifier of the CE. It is composed of two classifiers and an oracle that decides which one to use in each case. Basically, the oracle is a random function whose objective is to randomly split the dataset into two subsets by dividing the feature space into two regions. Each of the two generated regions and the corresponding data subset is assigned to one classifier. Any shape for the decision

EMO OCS classifier selection for RSO-based FRBCEs

The second part of this contribution introduces the use of the OCS strategy for classifier selection in RO-based fuzzy CEs. On the one hand, the aim is to refine the accuracy-complexity trade-off in the RO-based bagging FRBCEs (both in RLO and RSO) when dealing with high complexity classification problems. On the other hand, an interesting objective is to study whether the additional diversity induced by ROs is beneficial for the EMO OCS-based component fuzzy classifier selection. Thus, we

Conclusions and future works

In this contribution, we focused on two aims. Firstly, we introduced a new RSO approach into our previous FRBCE design framework. Secondly, we incorporated the EMO-based OCS strategy to these kinds of classifiers analyzing the influence of the additional RO diversity in the final FRBCEs performance. We used an advanced accuracy measure and proposed a specific binary coding for the RO-based classifier selection. A three-objective fitness function using three different optimization criteria such

Acknowledgment

We would like to thank Ludmila I. Kuncheva and Juan José Rodríguez for their kind help with answering all the questions regarding ROs and sharing their source code with us.

This work was supported by the Spanish Ministerio de Economía y Competitividad under Projects TIN2012-38525-C02-01 and TIN2011-24302, including funding from the European Regional Development Fund.

References (62)

  • S. García et al.

    Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power

    Information Sciences

    (2010)
  • V. López et al.

    A hierarchical genetic fuzzy system based on genetic programming for addressing classification with highly imbalanced and borderline data-sets

    Knowledge-Based Systems

    (2013)
  • L.I. Kuncheva

    Combining Pattern Classifiers: Methods and Algorithms

    (2004)
  • K. Trawiński, O. Cordón, A. Quirin, L. Sánchez, Random oracles fuzzy rule-based multiclassifiers for high complexity...
  • L.I. Kuncheva et al.

    Classifier ensembles with a random linear oracle

    IEEE Transactions on Knowledge and Data Engineering

    (2007)
  • K. Trawiński et al.

    On designing fuzzy rule-based multiclassification systems by combining FURIA with bagging and feature selection

    International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems

    (2011)
  • J.C. Hühn et al.

    FURIA: an algorithm for unordered fuzzy rule induction

    Data Mining and Knowledge Discovery

    (2009)
  • J.C. Hühn, E. Hüllermeier, An analysis of the FURIA algorithm for fuzzy rule induction, in: Advances in Machine...
  • L. Breiman

    Bagging predictors

    Machine Learning

    (1996)
  • X. Yao et al.

    Making use of population information in evolutionary artificial neural networks

    IEEE Transactions on Systems Man and Cybernetics Part B: Cybernetics

    (1998)
  • D. Partridge et al.

    Engineering multiversion neural-net systems

    Neural Computation

    (1996)
  • K. Trawiński et al.

    A study on the use of multi-objective genetic algorithms for classifier selection in FURIA-based fuzzy multiclassifiers

    International Journal of Computational Intelligence Systems

    (2012)
  • K. Deb et al.

    A fast and elitist multiobjective genetic algorithm: NSGA-II

    IEEE Transactions on Evolutionary Computation

    (2002)
  • C. Coello et al.

    Evolutionary Algorithms for Solving Multi-Objective Problems

    (2007)
  • J.J. Rodríguez et al.

    Naïve bayes ensembles with a random oracle

  • C.L. Blake, C.J. Merz, UCI repository of machine learning databases, 1998....
  • J. Alcalá-Fdez et al.

    Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework

    Journal of Multiple-Valued Logic and Soft Computing

    (2011)
  • T. Dietterich

    An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization

    Machine Learning

    (2000)
  • R. Banfield et al.

    A comparison of decision tree ensemble creation techniques

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2007)
  • D. Optiz et al.

    Popular ensemble methods: an empirical study

    Journal of Artificial Intelligence Research

    (1999)
  • R. Schapire

    The strength of weak learnability

    Machine Learning

    (1990)
  • Cited by (23)

    • Evolutionary fuzzification of RIPPER for regression: Case study of stock prediction

      2019, Neurocomputing
      Citation Excerpt :

      In addition, this MF consists of simple straight line segments, so it is easy to be implemented. Moreover, triangular MF is simple and linear and is capable to model nonlinear and complex treatments, when it is employed in creation of a fuzzy system [13,26,38,53,54]. The intervals in the crisp classification rules of RIPPER are replaced with corresponding triangular fuzzy numbers.

    • A comparative assessment of GIS-based data mining models and a novel ensemble model in groundwater well potential mapping

      2017, Journal of Hydrology
      Citation Excerpt :

      In other investigations, Tien Bui et al. (2012), and Pham et al. (2015) reported higher performance of the Bagging model than AdaBoost model in landslide susceptibility mapping which is consistent with the results of the current study. Bagging is sensitive to variations in training dataset (Tien Bui et al., 2014), and has superior variance mitigation versus AdaBoost (Ganjisaffar et al., 2011), which is capable for dealing with multi-dimensional and complex classification problems (Tien Bui et al., 2014; Trawiński et al., 2013), and is more immune to noise in dataset (Dietterich, 1997). These strong features could have caused for its better performance among the data mining models.

    • RipMC: RIPPER for Multiclass Classification

      2016, Neurocomputing
      Citation Excerpt :

      This algorithm, which extends its predecessors including IREP, is the first to effectively tackle the overfitting problem. It is still considered state-of-the-art in inductive rule learning [1,28,29]. It is a powerful classification rule learning algorithm that can deal with a very common problem of rule-based classification problems, the so-called curse of dimensionality.

    View all citing articles on Scopus
    View full text