Nonlinear systems modelling based on self-organizing fuzzy neural network with hierarchical pruning scheme
Introduction
Nonlinear problems have been widely studied by researchers, including engineers, physicists, and mathematicians, because most of the systems involved in practical engineering are essentially nonlinear [1]. Nonlinear system modelling plays an important role in soft computing [2], [3], intelligent control [4], [5], fault diagnosis [6], system optimization [7], [8], [9], and so on. According to the transparency of the modelling process, nonlinear system modelling methods are generally categorized as first-principle modelling, data-driven modelling, and grey-box modelling [10]. Data-driven modelling is widely used in the nonlinear modelling of industrial systems, because there is no need to deeply understand the operating mechanism of the system, and the established model is intuitive, simple and convenient [11].
The core of data-driven modelling is to use the observations to construct a nonlinear model to describe the relationship between the inputs and outputs of the dynamic system. There are many algorithms that can be used for data-driven modelling, such as fuzzy systems (FSs) [12], [13], [14], [15], [16], [17], support vector regression (SVR) [18], [19], and neural networks (NNs) [20], [21], [22], [23], [24]. In [13], a generalized smart evolving fuzzy system (GS-EFS) for online regression or system modelling is proposed and equipped with a new projection concept. In [14], a set of evolving TSK fuzzy models for the identification of finger dynamics is proposed, in which an incremental online identification algorithm is used to adjust the structure and parameters. In [15], a set of evolving fuzzy models with an incremental online identification algorithm is proposed, which are used to design five simple Takagi–Sugeno PI-fuzzy controllers for the control of a prosthetic hand. In [20], a diagonal recurrent neural network (DRNN) for the dynamic identification of motor-driven robotic links is proposed, and it demonstrates a superior learning ability and robustness. In [23], a real implementation of a recurrent neural network is presented to control an industrial paste thickener. In [24], an interactively recurrent self-evolving fuzzy neural network (IRSFNN) is designed for the prediction and identification of dynamic systems. In IRSFNN, the online clustering algorithm and variable-dimensional Kalman filter algorithm are adopted to optimize the structure and parameters simultaneously.
The success of FSs and NNs is mainly due to their universal approximation. Theoretical studies have shown that if there are enough hidden neurons or fuzzy rules, FSs and NNs can approximate any nonlinear system with arbitrary precision [25], [26]. Combining these two paradigms, a robust fuzzy neural network (FNN) is creatively constructed and widely adopted [27]. It not only has the semantic transparency and interpretability of FSs, but also has the parallel processing and adaptive learning ability of NNs. FNNs have attracted wide attention in data mining [28], nonlinear system modelling [29], [30], and intelligent control [31], [32]. When FNNs are utilized to identify the singleton or Takagi–Sugeno–Kang (TSK) fuzzy model of a complex nonlinear system, enough fuzzy rules are used to cover the input and output space of the system. However, it is necessary to strike a balance between the degree of fitting and the complexity to make full use of the generalization ability of FNNs. Therefore, we need to make a great effort to address the following two main issues for designing FNNs [33]: structure identification and parameter estimation. Based on the different learning strategies, nonlinear system identification approaches can be generally divided into two categories [34]: offline identification and online identification.
For offline identification, expert knowledge, trial-and-error methods or clustering algorithms are generally applied to generate suitable fuzzy rules. The well-known adaptive network-based fuzzy inference system (ANFIS) proposed in [35] determines the number of membership functions for each input variable by the trial-and-error method. Some clustering methods, such as the fuzzy C-means (FCM) [36], K-nearest neighbour clustering (KNN) [37], Gustafson–Kessel clustering [38], rule self-splitting strategy [39], and correlated-contours method [40], are used to choose the initial fuzzy rules. Then, local search methods, such as the gradient descent (GD) algorithm [41], linear least square (LLS) algorithm [42], and Levenberg–Marquardt (LM) algorithm [43], are adopted to optimize the network parameters. In addition, some methods apply intelligent optimization algorithms, such as the particle swarm optimization (PSO) [44] algorithm and genetic algorithm (GA) [45], [46], to obtain a compact and excellent FNN at the cost of high computational complexity.
For online identification, geometric growing criteria are usually used to generate fuzzy rules based on each input–output data pair. The resource-allocating network (RAN) proposed in [47] is a typical NN-based online identification model. In addition, some other online algorithms, such as evolving fuzzy neural networks (EFuNNs) [48], dynamic evolving neural fuzzy inference system (DENFIS) [49], self-constructing neural fuzzy inference network (SONFIN) [50], recurrent self-organizing neural fuzzy inference network (RSONFIS) [51], and evolving Takagi–Sugeno (ETS) [52] model, have also attracted much attention from researchers. Nevertheless, if the number of input–output samples is too large, a very large fuzzy rule base will be generated, making practical application difficult. Therefore, when constructing a self-organizing fuzzy neural network (SOFNN), we should not only design the novelty-based growing algorithm to recruit fuzzy rules in the sequence learning process, but also develop the pruning algorithm to delete the redundant rules.
Some typical SOFNNs with different growing and pruning criteria have been proposed in recent years. In [53], a dynamic fuzzy neural network (D-FNN) that is functionally equivalent to a TSK fuzzy system is designed based on an extended radial basis function neural network. In D-FNN, the growth of the fuzzy rules relies on the system error and the accommodation boundary criterion, while the pruning of fuzzy rules depends on the significance of the rules computed by the error reduction ratio (ERR) algorithm. In [54], a generalized dynamic fuzzy neural network (GD-FNN) with ellipsoidal basis functions (EBFs) is presented. Unlike D-FNN and GD-FNN, an SOFNN that recruits neurons by using the firing strength of EBF neurons and deletes inactive neurons by using the optimal brain surgeon (OBS) algorithm, was proposed in [55] and [56]. In addition, some other SOFNNs, such as FAOS-PFNN [57], SOFMLS [58], GEBF-OSFNN [59], GP-FNN [60], SOFNN-ACA [61], SEIT2FNN [62], SIT2FNN [63], and eT2RFNN [64], have also been proposed. Unfortunately, user-defined thresholds are required in these pruning strategies. Inappropriate threshold settings may reduce the generalization performance of the SOFNNs. Simultaneously, the parameters of premises, i.e., the widths of membership functions, are generally modified by a decay factor, and the parameters of consequences, i.e., the weights of the network, are identified online by the LLS algorithm [53], [54], [59] or recursive least square (RLS) algorithm [55], [56], [57], [58].
By analysing the SOFNNs mentioned above, we found that their membership function, rule base, parameters of premises and consequents are worth further study and improvement.
(1) Some of the existing SOFNNs adopt Gaussian function, which makes it difficult to describe the nonlinear characteristics of complex problems comprehensively. For example, in D-FNN and GD-FNN, extended radial basis function and ellipsoidal basis function are shaped by using Gaussian function respectively. However, it is difficult for Gaussian membership function to provide a powerful enough generalization ability of global mapping in the modelling of practical engineering problems, and to deal with local features effectively and flexibly. Therefore, in this paper, the asymmetric Gaussian function is introduced to eliminate the restriction of the uniform width of the original Gaussian function, which can enhance the flexibility of the membership function in identifying complicated nonlinear systems.
(2) Some of the existing SOFNNs adopt a pruning strategy with user-defined thresholds, which has the possibility of removing significant rules or retaining redundant rules by mistake. Taking the pruning threshold settings in D-FNN, GD-FNN, and GEBF-OSFNN as examples, if the threshold values are too large, the significant fuzzy rules may be removed by mistake, thus decreasing the accuracy of the system; on the contrary, if the threshold values are too small, the redundant rules may not be deleted, thus reducing the generalization ability of the network. Therefore, in this paper, a hierarchical pruning strategy, which is developed based on rule density and significance, is utilized to delete the unimportant rules without the setting of the pruning threshold of the ERR method.
(3) Some of the existing SOFNNs adopt the premise adjustment strategy based on the gradient descent algorithm, which makes it difficult to obtain transparent and interpretable fuzzy rule base. For example, GP-FNN and SOFNN-ACA both use an adaptive gradient descent algorithm to adjust the width of the Gaussian membership function, which leads to not only a large number of rules, but also an unclear and incomplete fuzzy rule base. The rule base of the SOFNN is the knowledge extracted from the identified system finally. To evaluate the quality of rule base, not only the number of fuzzy rules, i.e., the number of hidden layer neurons, but also the number and distribution of membership functions should be considered. If too many membership functions are extracted, the input space will be partitioned more densely, and the resulting fuzzy system is difficult to understand, resulting in poor interpretability of the rule base; on the contrary, if too few membership functions are generated, they may not be sufficient to describe the nonlinear system to be identified, resulting in poor model accuracy. To achieve the best balance between the accuracy of the system and the interpretability of the rule base, it is necessary to carefully design the assignment strategy of the antecedent parameters of fuzzy rules. Therefore, in this paper, an adaptive assignment strategy of antecedent parameters is designed, and it can not only adaptively adjust the generalized ellipsoidal basis region to obtain better local approximation, but also balance the accuracy of the system, the speed of rule learning and the transparency of the rule base.
(4) Some of the existing SOFNNs adopt the consequent parameter estimation strategy based on the pseudoinverse algorithm, which makes it difficult to ensure the fast convergence of estimation error and weight parameters. To ensure the convergence of the estimation error, the LLS algorithm is usually used to optimize the consequent parameters of the fuzzy rules. However, the traditional LLS algorithm uses the pseudoinverse technique to derive the consequent weights, which has a large amount of calculation and a slow convergence speed. The traditional RLS algorithm is sensitive to noise and slow to respond to changes in the dynamic characteristics of the input process. Therefore, in this paper, a modified recursive least square algorithm is used for the online estimation of the consequent parameters of fuzzy rules (i.e. the connection weights) to ensure the convergence of the estimation error.
In summary, a self-organizing fuzzy neural network with hierarchical pruning scheme (SOFNN-HPS) is proposed in this paper for nonlinear systems modelling in industrial processes. In addition to improving the performance of SOFNN-HPS from the membership function, self-organizing mechanism, rule base, and optimization algorithm, the convergence of the estimation error and the network linear parameters are also proven and can ensure the successful application of SOFNN-HPS in practical engineering. The performance of the proposed SOFNN-HPS is compared with other state-of-the-art competitors on two benchmark test problems and a key water quality parameter prediction experiment in the wastewater treatment process. Simulation results indicate that the proposed SOFNN-HPS can obtain a network model with compact structure and superior performance.
The major contributions of this paper are summarized as follows: (1) A novel structure learning algorithm based on a novel hierarchical pruning scheme for fuzzy neural networks is proposed. (2) The best balance between system accuracy and rule interpretability using an adaptive allocation strategy with less computational cost is achieved. (3) The optimal values for the free parameters in consequents are developed and obtained by a modified recursive least square algorithm.
The rest of this paper is organized as follows. Section 2 presents the architecture of the SOFNN-HPS. The structure learning and parameter estimation strategies of the SOFNN-HPS are given in Section 3. The convergence of the estimation error and linear parameters for the SOFNN-HPS is discussed in Section 4. In Section 5, the simulation results and some comparative studies with other algorithms are provided. Finally, Section 6 concludes this paper.
Section snippets
Architecture of the SOFNN-HPS
The architecture of the SOFNN-HPS which has four layers, namely, input layer, membership function (MF) layer, rule layer and output layer is shown in Fig. 1. In SOFNN-HPS, the asymmetric Gaussian function (AGF) proposed in [59], [65] is utilized as the membership function that can overcome the symmetry constraint of the standard Gaussian function. Hence, the ellipsoidal basis functions (EBFs) in the GD-FNN are extended into generalized ellipsoidal basis functions (GEBFs), which can improve the
Learning scheme of the SOFNN-HPS
The learning process of SOFNN-HPS includes the structure identification and the parameter estimation. If the number of rules in the FNN is too many, it could increase the complexity of the network structure and reduce the generalization capability; on the contrary, if the number of rules in the FNN is too few, it cannot achieve a satisfactory approximation effect. Therefore, the structure identification of the SOFNN-HPS uses a self-organizing mechanism to recruit or delete GEBF neurons online
Convergence analysis
In this section, the convergence analyses of the estimation error and linear parameters are provided.
Theorem 1 If the nonlinear system to be identified is stable and the weight parameters of SOFNN-HPS are adjusted by Eq. (42), the estimation error of the system can converge to zero.
Proof Define the estimation error e(k) as follows: According to Eq. (42), the following expression is obtained: Substituting (47) into (46) gives:
Simulation studies
In this section, three experimental examples, including Mackey–Glass time-series prediction, electrical energy output prediction in combined cycle power plant, and effluent ammonia nitrogen (NH4-N) prediction in the wastewater treatment process (WWTP), are employed to verify the effectiveness and superiority of the proposed SOFNN-HPS.
To generate or prune the fuzzy rules systematically, many control parameters are introduced into the SOFNN-HPS. Although some recommended values are given for
Conclusion
In this paper, a self-organizing fuzzy neural network with hierarchical pruning scheme (SOFNN-HPS) is proposed for nonlinear system modelling, which is functionally equivalent to a Takagi–Sugeno–Kang fuzzy system. In SOFNN-HPS, the generalized ellipsoidal basis function, hierarchical pruning strategy, adaptive allocation strategy, and modified recursive least square algorithm are combined into an organic whole. In particular, the proposed hierarchical pruning strategy can avoid the setting of
CRediT authorship contribution statement
Hongbiao Zhou: Conceptualization, Methodology, Software, Supervision. Yu Zhang: Visualization, Investigation. Weiping Duan: Software, Validation. Huanyu Zhao: Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This study was supported in part by the General Program of National Natural Science of China under Grant 61873107 and the “333” Project in Jiangsu Province, China under Grant BRA2019285. The authors would like to thank Prof. Ning Wang at Dalian Maritime University for their kind help during the process of this study. The authors would also like to thank the Editor-in-Chief, the Associate Editor and anonymous reviewers for their invaluable suggestions which have been incorporated to improve the
References (77)
- et al.
Volterra-series-based nonlinear system modelling and its engineering applications: A state-of-the-art review
Mech. Syst. Signal Process.
(2017) - et al.
Modelling and optimization of ultimate bearing capacity of strip footing near a slope by soft computing methods
Appl. Soft Comput.
(2018) - et al.
Data-driven soft sensor of downhole pressure for a gas-lift oil well
Control Eng. Pract.
(2014) - et al.
Evolving fuzzy and neuro-fuzzy approaches in clustering, regression, identification, and classification: A survey
Inform. Sci.
(2019) - et al.
CFNN: correlated fuzzy neural network
Neurocomputing
(2015) - et al.
An adaptive second order fuzzy neural network for nonlinear system modelling
Neurocomputing
(2016) - et al.
An on-line algorithm for creating self-organizing fuzzy neural networks
Neural Netw.
(2004) - et al.
An approach for on-line extraction of fuzzy rules using a self-organising fuzzy neural network
Fuzzy Sets and Systems
(2005) - et al.
A fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks
Neurocomputing
(2009) Least square neural network model of the crude oil blending process
Neural Netw.
(2016)
Radial basis function based adaptive fuzzy systems and their applications to system identification and prediction
Fuzzy Sets and Systems
Prediction of full load electrical power output of a base load operated combined cycle power plant using machine learning methods
Int. J. Electr. Power Energy Syst.
A robust least squares support vector machine for regression and classification with noise
Neurocomputing
Influence of operating conditions on the efficiency of domestic wastewater treatment in membrane bioreactors
Desalination
Modelling water quality in an urban river using hydrological factors–data driven approaches
J. Environ. Manag.
Time series prediction using bayesian filtering model and fuzzy neural networks
Optik
Modelling and identification of nonlinear systems: A review of the multimodel approach—Part 1
IEEE Trans. Syst. Man Cybern. A
Neural networks-based adaptive control for nonlinear state constrained systems with input delay
IEEE Trans. Cybern.
Intelligent control of microgrid with virtual inertia using recurrent probabilistic wavelet fuzzy neural network
IEEE Trans. Power Electron.
Parametric fault diagnosis based on fuzzy cerebellar model neural networks
IEEE Trans. Ind. Electron.
Indirect adaptive fuzzy control design with guaranteed tracking error performance for uncertain canonical nonlinear systems
IEEE Trans. Fuzzy Syst.
Neuro-fuzzy-based plant-wide hierarchical coordinating optimization and control: an application to zinc hydrometallurgy plant
IEEE Trans. Ind. Electron.
Multiobjective optimal control for wastewater treatment process using adaptive MOEA/D
Appl. Intell.
Data-driven modelling using improved multi-objective optimization based neural network for coke furnace system
IEEE Trans. Ind. Electron.
Generalized smart evolving fuzzy systems
Evol. Syst.
Experiments in incremental online identification of fuzzy models of finger dynamics
Romanian J. Inf. Sci. Technol.
Evolving fuzzy models for prosthetic hand myoelectric-based control
IEEE Trans. Instrum. Meas.
Fuzzy-affine-model-based memory filter design of nonlinear systems with time-varying delay
IEEE Trans. Fuzzy Syst.
Concise fuzzy system modelling integrating soft subspace clustering and sparse learning
IEEE Trans. Fuzzy Syst.
A novel dynamic-weighted probabilistic support vector regression-based ensemble for prognostics of time series data
IEEE Trans. Reliab.
Evolutionary dynamic multiobjective optimization assisted by a support vector regression predictor
IEEE Trans. Evol. Comput.
Comparative study of neural networks for dynamic nonlinear systems identification
Soft Comput.
Data-driven modelling for UGI gasification processes via an enhanced genetic BP neural network with link switches
IEEE Trans. Neural Netw. Learn. Syst.
An adaptive fuzzy recurrent neural network for solving non-repetitive motion problem of redundant robot manipulators
IEEE Trans. Fuzzy Syst.
Neural network-based model predictive control of a paste thickener over an industrial internet platform
IEEE Trans. Ind. Inf.
Identification and prediction of dynamic systems using an interactively recurrent self-evolving fuzzy neural network
IEEE Trans. Neural Netw. Learn. Syst.
A comparative study on sufficient conditions for Takagi-Sugeno fuzzy systems as universal approximators
IEEE Trans. Fuzzy Syst.
Stability for neural networks with time-varying delays via some new approaches
IEEE Trans. Neural Netw. Learn. Syst.
Cited by (42)
Self-organizing data-driven prediction model of ship maneuvering fast-dynamics
2023, Ocean EngineeringA review on computational intelligence methods for modeling of light weight composite materials
2023, Applied Soft ComputingHesitant hierarchical T–S fuzzy system with fuzzily weighted recursive least square
2023, Engineering Applications of Artificial Intelligence