Design of stabilized polynomial-based ensemble fuzzy neural networks based on heterogeneous neurons and synergy of multiple techniques

Abstract

In this study, a novel category of polynomial-based ensemble fuzzy neural networks (PEFNNs) are proposed. The study is focused on the development of advanced design methodologies to improve the performance (prediction accuracy) of the model when dealing with nonlinear regression problems. In contrast to the conventional fuzzy polynomial-based models, we adopt a hybrid network structure composed of heterogeneous neurons. The first layer of PEFNNs consists of fuzzy regular polynomial neurons optimized by clustering method. In the consecutive layers, we engage two types of polynomial neurons, which are selected and optimized by evolutionary algorithms. Moreover, an enhanced topology based on fuzzy module and enhanced interconnection (FM&EI) is designed to strengthen the characteristics of fuzzy feature information as well as increase the number and diversity of neurons. Multiple techniques are used synergistically to reinforce the performance of PEFNNs. First, a coefficient-based performance compromise algorithm (CPC) is designed to select neurons by considering the performance and complexity of the neuron. Second, L₂-norm regularization is considered to improve the performance of the model. Third, evolutionary algorithm is employed to adjust the structural parameters of PEFNNs. Furthermore, FM&EI and hybrid network structure which consist of heterogeneous neurons are considered as one of the multiple approaches to construct the ensemble model. The performance and stability of PEFNNs are evaluated with a diversity of datasets. A thorough comparative analysis also is covered.

Introduction

With the increasing requirement for complex system models to cope with inherent nonlinearities, higher-order dynamics, time-varying behavior, and inaccurate measurements, a prudently established modeling environment needs to be formed. In response to these problems, conventional machine learning methods are used to build logic-based architectures to support reasoning mechanisms. They exhibit some limitations to adopt to real-world systems working in dynamic and unpredictable domains. Computational intelligence (CI), which is widely used in science and engineering, can effectively solve these problems [1], [2], [3]. A lot of research has shown that the collaboration of multiple techniques within the setting of CI can effectively support the design of the model, and its performance (such as accuracy or interpretability) is superior to those models being designed when using a single conventional technique. As a result, the hybrid architecture developed with the use of fuzzy logic, neural networks and evolutionary algorithms have attracted more attention in recent years [4], [5].

Fuzzy neural networks (FNNs) are the result of the synergy between fuzzy logic and neural networks [6], [7]. There are lots of methods for synthesizing neural networks. Oh and Pedrycz combined fuzzy neural networks with polynomial neural networks [8] to propose a category of neuro fuzzy networks, called fuzzy polynomial neural networks (FPNNs) [9], and came up with a series of improvements and optimizations of FPNNs [10], [11].

Park and Pedrycz designed an architecture of genetically oriented fuzzy relation neural networks and proposed a comprehensive design method to support the expansion of its network structure [12]. A fuzzy linear regression estimation method based on the polynomial neural network is proposed by Roh et al., which is a layered cumulative network whose final model is represented by polynomial [13]. FPNNs can also be used to construct the conclusion part of clustering based fuzzy neural network, which is used to strengthen the relationship between input variables and output variables in the local region of the input space of single-layer neural network [14], [15].

Some researchers took advantage of the flexibility of polynomial neural network and mix it with other neural networks to construct a new category of fuzzy polynomial neural networks. Oh and Park et al. proposed a fuzzy polynomial model based on radial basis function and polynomial neural networks, which is composed of radial polynomial neurons [16]. Huang proposed the concept of fuzzy wavelet polynomial neural networks based on the concept and construction of polynomial neural networks and fuzzy wavelet neurons. Each layer of the model consists of fuzzy wavelet neurons [17]. However, similar to the structure of conventional FPNNs [9], such complex neurons are regarded as basic nodes in the successive layer of FPNNs in these improved models, which may lead to increased complexity of the whole model. In addition, topology, node selection criteria, and possible overfitting problems that affect the performance of FPNNs models have received limited attention.

In this study, the proposed polynomial-based ensemble fuzzy neural networks (PEFNNs) combine both hybrid structure and the synergy of multiple techniques. Among them, the hybrid network structure composed of heterogeneous neurons, such as fuzzy regular polynomial neurons and polynomial neurons is helpful to reduce the complexity and improve the flexibility of conventional FPNNs. Moreover, synergistic use of multiple techniques such as enhanced topology based on fuzzy module and enhanced interconnection, coefficient-based performance compromise algorithm, learning mechanisms based on L₂-norm regularization, and evolutionary algorithm that optimizes the overall structure of the model can conducive to improve the generalization ability of PEFNNs. The key issues and the advantages of the proposed PEFNNs could be enumerated as follows:

• Construction of fuzzy regular polynomial neurons (FRPNs) using fuzzy rules optimized by clustering method in the first layer. FRPN can effectively articulate the uncertainty between data. Two kinds of FRPNs including fuzzy set-based regular polynomial neurons (FsRPNs) and fuzzy relation-based regular polynomial neurons (FrRPNs) are applied. Compared with FrRPN, FsRPN can effectively reduce the computational complexity caused by calculating the firing strength of multiple antecedents in fuzzy rules (single antecedent is used in fuzzy rules). According to the type of FRPN used in the model, PEFNNs come from two types: PEFNN based on FrRPN (PEFrNN) and PEFNN based on FsRPN (PEFsNN).

• Construction of two types of polynomial neurons (respectively called regular polynomial neurons, RPNs and exponential polynomial neurons, EPNs) in the second and higher layers, and selected and optimized by evolutionary algorithms. Polynomial neurons can effectively identify the nonlinear relationship between the input and output of a system. Compared with the fuzzy regular polynomial neuron, polynomial neuron can alleviate the complex influence of fuzzy component (viz. conditional part of fuzzy rule), and the learning method is more concise, so it has better flexibility. The construction of EPN needs to undergo the nonlinear transformation, which can improve the prediction performance of the model by the 'shrinkage effect' of the logarithmic loss function. The proposed PEFNNs use the enhanced topology based on fuzzy module and enhanced interconnection (FM&EI). On the one hand, FM&EI is used to reinforce the characteristics of fuzzy feature information, which reflects the uncertainty between the data. On the other hand, FM&EI increases the number and diversity of input variable combinations that build each layer of the network, which increases the chances of improving model prediction capabilities.

• Improvement of generalization capability through the synergetic effect of multiple techniques. Combining the performance and complexity of neurons, we designed a coefficient-based performance compromise algorithm (CPC) to select neurons. For this regard, CPC can not only slow down the impact of complex neurons on the model, but also provides more predictive input for the next layer of the network. L₂-norm regularization is considered to improve the generalization capabilities of the model. Regularization can not only slow down the effects of overfitting, but also improve the predictive power of the model by reducing the deviation between the coefficients. Moreover, FM&EI and hybrid network structure consist of heterogeneous neurons are considered as one of the multiple techniques for constructing the ensemble model. The selected input provided by the CPC and the fuzzy inputs passed by the FM&EI can help to reinforce the reliability and stability of the parameter estimates.

• Evolutionary algorithm was applied to optimize the structure of the proposed ensemble fuzzy neural networks and polynomial neurons. The combination of fuzzy systems and evolutionary algorithms has achieved effective development in terms of interpretability and complexity reduction (e.g., evolutionary fuzzy systems, EFSs and multi-objective evolutionary fuzzy systems, MOEFSs) [28], [35], [38]. However, fuzzy polynomial-based models [10], [17], [39] face complexity issues when combined with evolutionary algorithms, and the optimization of each fuzzy layer of the model takes lots of time. In order to reduce the additional computational complexity of the model caused by optimization and enhance the prediction performance of the model, only non-fuzzy layers (i.e., the second and deeper layers) of PEFNNs are optimized by the evolutionary algorithm. Optimized parameters are considered such as the type of neurons, the type of polynomials, the number of input variables and a group of the specific subset of inputs.

• Statistical analysis of the proposed model according to the Friedman test and Holm test.

In order to complete a thorough statistical comparison between the proposed neural network and the existing model, the Friedman test and the Holm test is used to perform a null-hypothesis test and a post-hoc test, respectively.

The main contributions of our work can be summarized as follows. First, a novel category of hybrid network structure composed of heterogeneous neurons is designed, which can reflect the uncertainty of data and flexibly approximate complex nonlinear problems. Second, an enhanced topology based on fuzzy module and enhanced interconnection is proposed. FM&EI can strengthen the characteristics of fuzzy feature information as well as increase the number of neurons and their diversity, which contributes to the improvement of prediction capabilities. Third, a coefficient-based performance compromise algorithm is designed to select neurons by considering the performance and complexity of the neuron. What is more important, we dwell upon the synergy of multiple techniques to rectify the limitations of a single technique. When multiple techniques are used as an ‘ensemble’, this ensemble can effectively overcome the uncertainty brought by a single method, enhance the stability of the model, and ultimately achieve the preferred performance from the enhanced generalization ability as well as the stabilized network structure of model.

The study is arranged in the following way. First, Section 2 introduces the architecture of PEFNNs. The learning method of PEFNNs is discussed in Section 3. The framework of PEFNNs is designed in Section 4. Analysis of overall structure and algorithmic design methodologies is covered in Section 5. The experimental studies are reported in Section 6. Finally, conclusions are drawn in Section 7.