Elsevier

Applied Soft Computing

Volume 13, Issue 4, April 2013, Pages 1800-1812
Applied Soft Computing

A fuzzy evolutionary framework for combining ensembles

https://doi.org/10.1016/j.asoc.2012.12.027Get rights and content

Abstract

We propose an evolutionary framework for the production of fuzzy rule bases where each rule executes an ensemble of predictors. The architecture, the rule base and the composition of the ensembles are evolved over time. To achieve this, we employ a context-free grammar within a hybrid genetic programming system using a multi-population model. As base predictors, multilayer perceptron neural networks and support vector machines are available. We apply the system to several function approximation and regression tasks and compare the results with recent research and state-of-the-art models. We conclude that the proposed architecture is competitive and has a number of very desirable features supporting automation of predictive model building and their adaptation over time. Finally, we suggest further potential research directions.

Highlights

► We propose an integrated framework for generating fuzzy rule-based systems that combine ensembles. ► We use neural networks and support vector machines as learners and genetic programming to evolve combinations. ► We test the proposed methodologies over a series of artificial and real-world datasets.

Introduction

Ensemble systems is a term describing combinations of individual learners [36]. They consist effective and efficient approaches for a wide range of problems, where individual learners may demonstrate overfitting or weak learning [45], [12], [29], [38], [10]. Reasons to prefer a combined solution can be statistical, computational and representational [16]. From the statistical perspective, an ensemble approach can deal with problems having few available training data in a large hypothesis space. In that case, the deviation of performance by individual learners can be high. Ensembles can solve the issue by averaging the system output and thus reducing the risk of selecting the wrong learner. Computational issues concern the optimal training of an individual algorithm which can be computationally a NP-hard task [8], [15]. Combined systems can assist in such problems, by supporting a divide-and-conquer strategy [60]. Finally, an ensemble can increase the expressibility of a system, by providing for example weighted combinations that allow for more complex representations of the desired solution.

Two-level ensembles, also known as two-tier ensembles, are a class of combined systems that consist of a meta level and a base level [40], [4]. Each of the levels can implement different learner classes and the output of the base level ensemble provides the input for the meta level system. Typical methods to train two-level ensembles include cascading, stacking and grading [40]. Two-level ensembles can be preferred in order to combine the strength of different ensemble classes [22], [62], to deal with problems where a specific ensemble class is unable to deal with nominal data [40], or to address issues arising from incremental learning [64]. Two-level ensembles further extend the concept of the divide-and-conquer strategy by employing a population of smaller subsystems, each being easier to train than single-level ensemble approaches.

Since evolutionary algorithms inherit mechanisms to implicitly promote diversity among competitive population members [9], their use within ensemble building has been investigated. Evolutionary training has been applied to both predictor and combiner levels. A successful paradigm of the first case is the production of ensembles, where the predictors are genetically trained with the aim to provide diverse neural networks [11], [61]. Belonging to the second case, the use of genetic programming (GP) has been proposed by the authors for the generation of ensembles for function approximation and regression tasks in a model, named GRADIENT [58]. In that approach, multilayer perceptron neural networks and support vector machines (SVMs) are combined hierarchically, and the output is shown to perform better than standard techniques, such as individual neural networks, for a series of regression tasks.

Two-level architectures have also been considered for ensembles that are built by evolutionary methods [31]. In that work, an evolved system of network ensembles demonstrated competitive performance over single-level ensembles and individual learners, while still maintaining smaller-sized solutions. In addition, the two-level framework was considered advantageous in providing insight into how and why the resulting solution has improved predictive accuracy.

Another class of ensemble systems considers the incorporation of fuzzy interference, either at the predictor level or at the combiner level. Fuzzy systems simulate the human brain behaviour when dealing with imprecise information, by using linguistic variables and fuzzy inference [27]. Their application can often increase the readability of the resulted prediction model, while still providing a high precision result. Typical methods to create and tune these fuzzy systems include evolutionary, neural and heuristic approaches. Using an evolutionary approach to configure and tune fuzzy systems can have attractive properties, such as maximizing accuracy and producing interpretable fuzzy systems at the same time often achieved within a multi-objective framework [26].

Taking into consideration the aforementioned advances, this work proposes an evolutionary framework for building two-level prediction systems consisting of evolved fuzzy rule bases and the locally applicable regression ensembles. The proposed model, named hereinafter fG*, is designed to evolve variable-sized, arbitrarily composed ensembles which are then competing through a fuzzy evaluation scheme. The versatile GRADIENT framework for combined regression systems [58] has been used to evolve the proposed combining architecture. The idea behind this model is to synthesize the capability of a fuzzy inference with the generalization ability of an ensemble. With this configuration, the methodological scheme can be thought as utilizing a divide-and-conquer approach, as compared to the single combining hierarchical engine of GRADIENT. In the proposed two-level system of fG*, ensembles are created at the base level, and they are assigned to consequent parts of fuzzy rules at the meta level of the ensemble (Fig. 1).

To realize the proposed approach, a descriptive context-free grammar is developed in GRADIENT, effectively expressing the required functional complexity into GP individuals. Furthermore, evolution is organized in sub-populations, to allow for better exploration and high diversity in the solution pool. We then evaluate this system in a series of experiments using synthetic and real-world data and extract conclusions on its effectiveness. The proposed approach manages to produce competitive two-level ensembles, internally represented as single individuals in an integrated evolutionary framework, with the aid of expressive grammar representation. The following summarizes the contribution of this paper:

  • A hybrid evolutionary framework for the production of two-level ensemble systems with co-evolved locally applicable regression ensembles.

  • An analysis of the system properties with respect to the base predictor types and different data quality.

  • A comparative study with state-of-the-art techniques and other ensemble building methods for a number of regression tasks.

The rest of the paper is organized as follows. Next section describes the recent related research. Section 3 presents the details of the system design. In Section 4, our results from synthetic and real-world data problems are included, together with a discussion. Finally, in Section 5, we present conclusions and propose further research directions.

Section snippets

Background

Fuzzy sets are an advance of the classic (crisp) sets, where the transition for a value from belonging to a set and not belonging to the set is gradual and quantified by a membership function (MF) [47]. A fuzzy set is defined as:A={(v,μA(v))|vV}where μA(v) is the MF for the fuzzy set. The V is called universe of discourse. We commonly use linguistic terms for the fuzzy sets such as small or medium (hence the term linguistic variables). We then apply the linguistic variables within fuzzy rules

Overview

The fG* environment is shown in Fig. 2. The basic elements of fG* are the following.

  • A pool of base predictors. These can be simple regression learners, such as multilayer perceptrons and support vector machines. A description of the available base predictors is given in Section 3.2.

  • A context-free grammar. The grammar is used to describe the hierarchy of the GP functions and to restrict the evolutionary operations. We discuss the grammar and the GP functions that compose the GP individuals in

Results and discussion

This section includes our experimentation with the proposed system. We first examine a toy example, which allows to demonstrate the fuzzy inference and the functionality of the combined system. We then compare fG* with GRADIENT, which is a non-fuzzy, ensemble-building approach [58]. This experiment enables comparative assessment of the two models using different levels and types of noise and different function complexity. A comparison between fG* and five state-of-the-art regression models then

Conclusion and further research

This paper presented a system for the generation of ensembles for function approximation and regression tasks. This system incorporates fuzzy inference for the production of competitive rule-bases with each rule corresponding to a separate combination of predictors. The overall system is using an integrated evolutionary framework, further enhancing the robustness and the versatility of the ensembles. We performed a series of tests using synthetic and real-world data to compare the proposed

Acknowledgement

The research leading to these results has received funding from the European Commission within the Marie Curie Industry and Academia Partnerships and Pathways (IAPP) programme under grant agreement no. 251617.

References (64)

  • J. Mendel

    Advances in type-2 fuzzy sets and systems

    Information Sciences

    (2007)
  • F. Minku et al.

    Clustering and co-evolution to construct neural network ensembles: an experimental study

    Neural Networks

    (2008)
  • M. Rucinski et al.

    On the impact of the migration topology on the Island model

    Parallel Computing

    (2010)
  • A. Sala et al.

    Perspectives of fuzzy systems and control

    Fuzzy Sets and Systems

    (2005)
  • M. Sugeno et al.

    Structure identification of fuzzy model

    Fuzzy Sets and Systems

    (1988)
  • A. Tsakonas

    A comparison of classification accuracy of four genetic programming evolved intelligent structures

    Information Sciences, Elsevier

    (2006)
  • X. Yao

    Automatic divide-and-conquer using populations and ensembles

  • Y. Zhang et al.

    Genetic programming in classifying large-scale data: an ensemble method

    Information Sciences

    (2004)
  • Z.-H. Zhou et al.

    Ensembling neural networks: many could be better than all

    Artificial Intelligence

    (2002)
  • E. Alba et al.

    Evolutionary design of fuzzy logic controllers using strongly-typed GP

  • E. Alba et al.

    Evolutionary design of fuzzy logic controllers using strongly-typed GP

    Mathware and Soft Computing

    (1999)
  • J. Alcala-Fdez et al.

    KEEL data-mining software tool: data set repository, integration of algorithms and experimental analysis framework

    Journal of Multiple-Valued Logic and Soft Computing

    (2011)
  • M. Analoui et al.

    Hierarchical two-tier ensemble learning: a new paradigm for network intrusion detection

  • L. Araujo et al.

    Genotypic differences and migration policies in an island

  • M. Berthold et al.

    KNIME: the Konstanz information miner

  • C. Bishop

    Neural networks and their applications

    Review of Scientific Instruments

    (1994)
  • A. Blum et al.

    Training a 3-node neural network is NP-Complete (ext. abs.)

  • A. Chandra et al.

    DIVACE: diverse and accurate ensemble learning algorithm

  • H. Chen et al.

    Multiobjective neural network ensembles based on regularized negative correlation learning

    IEEE Transactions on Knowledge and Data Engineering

    (2010)
  • N. Chomsky

    Syntactic Structures

    (1957)
  • T. Dietterich

    Ensemble methods in machine learning

  • R.P.W. Duin

    The combining classifier: to train or not to train?

  • Cited by (12)

    • An approximation to solve regression problems with a genetic fuzzy rule ordinal algorithm

      2019, Applied Soft Computing Journal
      Citation Excerpt :

      The fuzzy rule models commonly used to solve this type of problem are Mamdani [1] or TSK [2,3] since they have been shown to obtain good approximations in terms of precision. Thus, we can find interesting proposals for regression models based on the use of Mamdani rules such as [4,5], and models based on the use of TSK rules such as [6–8]. However, both present problems in terms of the interpretability of the solution, since neither of these models has the capacity to express the irrelevance of the input variables with respect to the variable to be approximated.

    • Robust ensemble learning framework for day-ahead forecasting of household based energy consumption

      2018, Applied Energy
      Citation Excerpt :

      This suggests that such instability can negatively affect the generalization ability of machine learning models if certain inferior conditions exist in the variables considered or in the data available for training and validation. To this extent, ensemble learning is a recent advancement to common machine learning techniques, and has been suggested over a wide spectrum of applications in the literature [47–50]. The main advantages of ensemble models, compared to single models, are depicted in their improved generalization ability and flexible functional mapping between the system’s variables [51,52].

    • Regional frequency analysis at ungauged sites using a two-stage resampling generalized ensemble framework

      2015, Advances in Water Resources
      Citation Excerpt :

      Ensemble learning has received a significant amount of interest. Intensive research in ensemble techniques is continuously conveyed in the literature with good results [5,8,27,41,44]. Mendes-Moreira et al. [27] provided a systematic survey of ensemble approaches for regression.

    • An ensemble neuro-fuzzy radial basis network with self-adaptive swarm based supervisor and negative correlation for modeling automotive engine coldstart hydrocarbon emissions: A soft solution to a crucial automotive problem

      2015, Applied Soft Computing Journal
      Citation Excerpt :

      Besides, to find a set of competent ensemble networks, they introduce preference to topology design problem. Tsakonas and Gabrys [46] proposed an evolutionary framework for production of fuzzy rules, which in turn delegate an ensemble of predictors. Their method proved its applicability for automotive design of predictive models.

    • A unified evolutionary training scheme for single and ensemble of feedforward neural network

      2014, Neurocomputing
      Citation Excerpt :

      For complicated problems, it may not be easy to train the ANN. Therefore, instead of a single ANN, combination of multiple simple neural networks is adopted to tackle the complex problems [9,10,29–36]. While applying evolutionary algorithms to train the ANNs, the best individual is always chosen as the solution.

    View all citing articles on Scopus
    View full text