A fuzzy evolutionary framework for combining ensembles
Graphical abstract
Highlights
► We propose an integrated framework for generating fuzzy rule-based systems that combine ensembles. ► We use neural networks and support vector machines as learners and genetic programming to evolve combinations. ► We test the proposed methodologies over a series of artificial and real-world datasets.
Introduction
Ensemble systems is a term describing combinations of individual learners [36]. They consist effective and efficient approaches for a wide range of problems, where individual learners may demonstrate overfitting or weak learning [45], [12], [29], [38], [10]. Reasons to prefer a combined solution can be statistical, computational and representational [16]. From the statistical perspective, an ensemble approach can deal with problems having few available training data in a large hypothesis space. In that case, the deviation of performance by individual learners can be high. Ensembles can solve the issue by averaging the system output and thus reducing the risk of selecting the wrong learner. Computational issues concern the optimal training of an individual algorithm which can be computationally a NP-hard task [8], [15]. Combined systems can assist in such problems, by supporting a divide-and-conquer strategy [60]. Finally, an ensemble can increase the expressibility of a system, by providing for example weighted combinations that allow for more complex representations of the desired solution.
Two-level ensembles, also known as two-tier ensembles, are a class of combined systems that consist of a meta level and a base level [40], [4]. Each of the levels can implement different learner classes and the output of the base level ensemble provides the input for the meta level system. Typical methods to train two-level ensembles include cascading, stacking and grading [40]. Two-level ensembles can be preferred in order to combine the strength of different ensemble classes [22], [62], to deal with problems where a specific ensemble class is unable to deal with nominal data [40], or to address issues arising from incremental learning [64]. Two-level ensembles further extend the concept of the divide-and-conquer strategy by employing a population of smaller subsystems, each being easier to train than single-level ensemble approaches.
Since evolutionary algorithms inherit mechanisms to implicitly promote diversity among competitive population members [9], their use within ensemble building has been investigated. Evolutionary training has been applied to both predictor and combiner levels. A successful paradigm of the first case is the production of ensembles, where the predictors are genetically trained with the aim to provide diverse neural networks [11], [61]. Belonging to the second case, the use of genetic programming (GP) has been proposed by the authors for the generation of ensembles for function approximation and regression tasks in a model, named GRADIENT [58]. In that approach, multilayer perceptron neural networks and support vector machines (SVMs) are combined hierarchically, and the output is shown to perform better than standard techniques, such as individual neural networks, for a series of regression tasks.
Two-level architectures have also been considered for ensembles that are built by evolutionary methods [31]. In that work, an evolved system of network ensembles demonstrated competitive performance over single-level ensembles and individual learners, while still maintaining smaller-sized solutions. In addition, the two-level framework was considered advantageous in providing insight into how and why the resulting solution has improved predictive accuracy.
Another class of ensemble systems considers the incorporation of fuzzy interference, either at the predictor level or at the combiner level. Fuzzy systems simulate the human brain behaviour when dealing with imprecise information, by using linguistic variables and fuzzy inference [27]. Their application can often increase the readability of the resulted prediction model, while still providing a high precision result. Typical methods to create and tune these fuzzy systems include evolutionary, neural and heuristic approaches. Using an evolutionary approach to configure and tune fuzzy systems can have attractive properties, such as maximizing accuracy and producing interpretable fuzzy systems at the same time often achieved within a multi-objective framework [26].
Taking into consideration the aforementioned advances, this work proposes an evolutionary framework for building two-level prediction systems consisting of evolved fuzzy rule bases and the locally applicable regression ensembles. The proposed model, named hereinafter fG*, is designed to evolve variable-sized, arbitrarily composed ensembles which are then competing through a fuzzy evaluation scheme. The versatile GRADIENT framework for combined regression systems [58] has been used to evolve the proposed combining architecture. The idea behind this model is to synthesize the capability of a fuzzy inference with the generalization ability of an ensemble. With this configuration, the methodological scheme can be thought as utilizing a divide-and-conquer approach, as compared to the single combining hierarchical engine of GRADIENT. In the proposed two-level system of fG*, ensembles are created at the base level, and they are assigned to consequent parts of fuzzy rules at the meta level of the ensemble (Fig. 1).
To realize the proposed approach, a descriptive context-free grammar is developed in GRADIENT, effectively expressing the required functional complexity into GP individuals. Furthermore, evolution is organized in sub-populations, to allow for better exploration and high diversity in the solution pool. We then evaluate this system in a series of experiments using synthetic and real-world data and extract conclusions on its effectiveness. The proposed approach manages to produce competitive two-level ensembles, internally represented as single individuals in an integrated evolutionary framework, with the aid of expressive grammar representation. The following summarizes the contribution of this paper:
- •
A hybrid evolutionary framework for the production of two-level ensemble systems with co-evolved locally applicable regression ensembles.
- •
An analysis of the system properties with respect to the base predictor types and different data quality.
- •
A comparative study with state-of-the-art techniques and other ensemble building methods for a number of regression tasks.
The rest of the paper is organized as follows. Next section describes the recent related research. Section 3 presents the details of the system design. In Section 4, our results from synthetic and real-world data problems are included, together with a discussion. Finally, in Section 5, we present conclusions and propose further research directions.
Section snippets
Background
Fuzzy sets are an advance of the classic (crisp) sets, where the transition for a value from belonging to a set and not belonging to the set is gradual and quantified by a membership function (MF) [47]. A fuzzy set is defined as:where is the MF for the fuzzy set. The V is called universe of discourse. We commonly use linguistic terms for the fuzzy sets such as small or medium (hence the term linguistic variables). We then apply the linguistic variables within fuzzy rules
Overview
The fG* environment is shown in Fig. 2. The basic elements of fG* are the following.
- •
A pool of base predictors. These can be simple regression learners, such as multilayer perceptrons and support vector machines. A description of the available base predictors is given in Section 3.2.
- •
A context-free grammar. The grammar is used to describe the hierarchy of the GP functions and to restrict the evolutionary operations. We discuss the grammar and the GP functions that compose the GP individuals in
Results and discussion
This section includes our experimentation with the proposed system. We first examine a toy example, which allows to demonstrate the fuzzy inference and the functionality of the combined system. We then compare fG* with GRADIENT, which is a non-fuzzy, ensemble-building approach [58]. This experiment enables comparative assessment of the two models using different levels and types of noise and different function complexity. A comparison between fG* and five state-of-the-art regression models then
Conclusion and further research
This paper presented a system for the generation of ensembles for function approximation and regression tasks. This system incorporates fuzzy inference for the production of competitive rule-bases with each rule corresponding to a separate combination of predictors. The overall system is using an integrated evolutionary framework, further enhancing the robustness and the versatility of the ensembles. We performed a series of tests using synthetic and real-world data to compare the proposed
Acknowledgement
The research leading to these results has received funding from the European Commission within the Marie Curie Industry and Academia Partnerships and Pathways (IAPP) programme under grant agreement no. 251617.
References (64)
- et al.
Diversity creation methods: a survey and categorisation
Journal of Information Fusion
(2005) - et al.
Robust predictive modelling of water pollution using biomarker data
Water Research
(2010) - et al.
Evolving hybrid ensembles of learning machines for better generalisation
Neurocomputing
(2006) - et al.
On approximate learning by multi-layered feedforward circuits
Journal of Theoretical Computer Science – TCS
(2005) - et al.
The classification of cancer based on DNA microarray data that uses diverse ensemble genetic programming
Artificial Intelligence in Medicine
(2006) - et al.
Constructing support vector machine ensemble
Pattern Recognition
(2003) - et al.
Optimal ensemble construction via meta-evolutionary ensembles
Expert Systems with Applications
(2006) - et al.
Efficient multi-objective higher order mutation testing with genetic programming
Journal of Systems and Software
(2010) - et al.
Meta-learning for time series forecasting and forecast combination
Neurocomputing
(2010) - et al.
Novel algorithm for constructing support vector machine regression ensemble
Journal of Systems Engineering and Electronics
(2006)
Advances in type-2 fuzzy sets and systems
Information Sciences
Clustering and co-evolution to construct neural network ensembles: an experimental study
Neural Networks
On the impact of the migration topology on the Island model
Parallel Computing
Perspectives of fuzzy systems and control
Fuzzy Sets and Systems
Structure identification of fuzzy model
Fuzzy Sets and Systems
A comparison of classification accuracy of four genetic programming evolved intelligent structures
Information Sciences, Elsevier
Automatic divide-and-conquer using populations and ensembles
Genetic programming in classifying large-scale data: an ensemble method
Information Sciences
Ensembling neural networks: many could be better than all
Artificial Intelligence
Evolutionary design of fuzzy logic controllers using strongly-typed GP
Evolutionary design of fuzzy logic controllers using strongly-typed GP
Mathware and Soft Computing
KEEL data-mining software tool: data set repository, integration of algorithms and experimental analysis framework
Journal of Multiple-Valued Logic and Soft Computing
Hierarchical two-tier ensemble learning: a new paradigm for network intrusion detection
Genotypic differences and migration policies in an island
KNIME: the Konstanz information miner
Neural networks and their applications
Review of Scientific Instruments
Training a 3-node neural network is NP-Complete (ext. abs.)
DIVACE: diverse and accurate ensemble learning algorithm
Multiobjective neural network ensembles based on regularized negative correlation learning
IEEE Transactions on Knowledge and Data Engineering
Syntactic Structures
Ensemble methods in machine learning
The combining classifier: to train or not to train?
Cited by (12)
An approximation to solve regression problems with a genetic fuzzy rule ordinal algorithm
2019, Applied Soft Computing JournalCitation Excerpt :The fuzzy rule models commonly used to solve this type of problem are Mamdani [1] or TSK [2,3] since they have been shown to obtain good approximations in terms of precision. Thus, we can find interesting proposals for regression models based on the use of Mamdani rules such as [4,5], and models based on the use of TSK rules such as [6–8]. However, both present problems in terms of the interpretability of the solution, since neither of these models has the capacity to express the irrelevance of the input variables with respect to the variable to be approximated.
Robust ensemble learning framework for day-ahead forecasting of household based energy consumption
2018, Applied EnergyCitation Excerpt :This suggests that such instability can negatively affect the generalization ability of machine learning models if certain inferior conditions exist in the variables considered or in the data available for training and validation. To this extent, ensemble learning is a recent advancement to common machine learning techniques, and has been suggested over a wide spectrum of applications in the literature [47–50]. The main advantages of ensemble models, compared to single models, are depicted in their improved generalization ability and flexible functional mapping between the system’s variables [51,52].
Regional frequency analysis at ungauged sites using a two-stage resampling generalized ensemble framework
2015, Advances in Water ResourcesCitation Excerpt :Ensemble learning has received a significant amount of interest. Intensive research in ensemble techniques is continuously conveyed in the literature with good results [5,8,27,41,44]. Mendes-Moreira et al. [27] provided a systematic survey of ensemble approaches for regression.
An ensemble neuro-fuzzy radial basis network with self-adaptive swarm based supervisor and negative correlation for modeling automotive engine coldstart hydrocarbon emissions: A soft solution to a crucial automotive problem
2015, Applied Soft Computing JournalCitation Excerpt :Besides, to find a set of competent ensemble networks, they introduce preference to topology design problem. Tsakonas and Gabrys [46] proposed an evolutionary framework for production of fuzzy rules, which in turn delegate an ensemble of predictors. Their method proved its applicability for automotive design of predictive models.
A unified evolutionary training scheme for single and ensemble of feedforward neural network
2014, NeurocomputingCitation Excerpt :For complicated problems, it may not be easy to train the ANN. Therefore, instead of a single ANN, combination of multiple simple neural networks is adopted to tackle the complex problems [9,10,29–36]. While applying evolutionary algorithms to train the ANNs, the best individual is always chosen as the solution.
A survey of evolutionary algorithms for supervised ensemble learning
2023, Knowledge Engineering Review