GSETSK: a generic self-evolving TSK fuzzy neural network with a novel Hebbian-based rule reduction approach
Graphical abstract
The evolution of the fuzzy rules in GSETSK as it learns the DOW JONES index
Introduction
Neuro-fuzzy computing is a popular framework for solving problems with a soft computing approach due to its capability to combine the human-like reasoning style of fuzzy systems with the connectionist structure and learning ability of neural networks [1]. Neuro-fuzzy hybridization is also widely known as fuzzy neural networks (FNN) or neuro-fuzzy systems (NFS). The main strength of the neuro-fuzzy approach is that it can provide insights to the user about the symbolic knowledge embedded within the network [1]. More specifically, the hybrid network can generalize from training data, learn/tune system parameters, and generate the fuzzy rules to create a linguistic model of the problem domain. Extensive reviews of FNNs are discussed in fuzzy rule learning [1] and neuro-fuzzy works [2], [3], [4], [5]. There are two types of fuzzy model; namely the Mamdani model [3] and the Takagi–Sugeno–Kang (TSK) model [6], [7], [8].
The main advantage of the TSK model over the Mamdani model is its ability to achieve higher system modeling accuracy. Hence, there is a continuing trend of using TSK fuzzy models for solving function approximation and regression-centric problems. In practice, these problems are online, meaning that the data are not all available prior to training but are sequentially presented to the learning system. Many dynamic systems such as SONFIN [9], FITSK [10], DENFIS [11], FLEXFIS [12], eTS [13], and others [14], [15], [16], [17], [18], [19], [20], [21] have been developed to provide solutions for such online problems.
Existing dynamic TSK models proposed in the literature can be broadly classified into two classes. Class I TSK models are essentially fuzzy systems that are generally limited to time-invariant environments. In real life, time-variant problems, which most often occurred in many areas of engineering, usually possess non-stationary, temporal data streams which are modified continuously by underlying data-generating processes. Dynamic approaches such as FITSK [10] and DENFIS [11] belong to this class. DENFIS [11] and FLEXFIS [12] implicitly assume prior knowledge of the upper bound and lower bound of the data set to normalize data before learning [22], which is unsuitable for time-variant environments. Kukolj and Levi [16] proposed a heuristic self-organizing network which is based on k-means for structure learning. Quah and Quek [10] proposed a simple process that presupposes an even space partitioning of the linguistic labels for fast learning. However both [10] and [16] require the number of clusters or rules to be specified prior to training, which is an impossible task in time-variant problems.
Class II TSK models are generally evolving systems [13] that can learn in time-variant environments. Many Class II systems such as [9], [17], [18], [19], [22] do not consider the interpretability of the knowledge bases. They generally employ back-propagation or gradient descent algorithms to heuristically tune the widths of their antecedent membership functions, which can result in highly overlapping and indistinguishable fuzzy sets. Thus, the semantic meaning of the derived fuzzy sets is deteriorated. Besides, many systems such as SONFIN [9], FLEXFIS [12], eTS [13], and [15] do not have a rule pruning algorithm, which may lead to the collection of obsolete rules over time and thus degrade the level of human interpretability of the resultant fuzzy rule base.
Some approaches have been proposed to address these issues. In [23], [24], a merging approach using the Jaccard index as a similarity measure to merge strongly overlapping fuzzy sets is employed. However this method does not prevent the fuzzy sets from growing overly large, and thus the fuzzy labels of the fuzzy sets represent may become obscure and poorly defined. In Simpl_eTS [20], a simple rule pruning algorithm which monitors the population of each rule is proposed. If a rule amounts to less than 1% of the total data samples at that current moment, it is considered as obsolete, and it will be pruned. This approach considers the contribution of old data and new data equally in determining the obsolete rules, thus it cannot detect drifts and shifts in online data streams [25]. In systems such as eTS+ [14] and xTS [21], the age of a cluster is used to determine if a rule (cluster) is obsolete. However, the age of the cluster in [14], [21] is determined by a self-driven formula which does not incorporate the membership degrees of the samples forming that cluster. In [25], the authors proposed an enhanced version of the formula to calculate the age of a cluster in [14], [21] by incorporating the membership degrees of the samples. This approach is reasonable, but not biologically plausible. In this paper, an alternative rule pruning method which is simple, computationally efficient and biologically plausible is proposed. This rule pruning algorithm applies a gradual forgetting approach and adopts the Hebbian learning mechanism behind the long-term potentiation phenomenon [26] in the brain. A merging approach which can prevent derived fuzzy sets from expanding too many times to protect their semantic meanings is also proposed.
Ensemble methods have been in hot topics in recent years [27], [28], [29], [30], [31], [32]. Ensemble is a learning paradigm where multiple networks are jointly used to solve a problem. In general, an ensemble consist of multiple individually trained networks that are used to predict a certain problem, and each individual predicted results are then combined using either the average or weighted average of the results. It has been shown that ensemble methods does provide a better accuracy result as compared to an individual predicting network [30]. However, to require multiple networks to be trained requires a longer duration, hence it is not recommended in time critical applications, and interpretability of the result is impossible. Multiple networks each have its own rule base hence an average or weighted average does not summarize a meaningful rule base for interpretation.
This paper focuses on solving the issues of achieving compact, up-to-date fuzzy rule bases and interpretable knowledge bases in TSK models. Thus, this paper presents an evolving TSK neural-fuzzy framework which can work in time-variant environments to address these issues. The framework is termed the generic self-evolving Takagi–Sugeno–Kang fuzzy framework (GSETSK). GSETSK starts with an initial empty rule base. New rules are sequentially added to the rule base by a fuzzy clustering algorithm called as multidimensional-scaling growing clustering which is completely data-driven. Highly overlapping membership functions are merged by a novel merging approach and obsolete rules are constantly pruned by a Hebbian-based rule pruning algorithm to derive a compact fuzzy rule base while maintaining a high level of modeling accuracy. GSETSK does not require prior knowledge of the numbers of clusters or rules present in the training data set. For parameter tuning, GSETSK employs a localized version of the recursive least-square algorithm [33] for high-accuracy online learning performance.
The paper is organized as follows. Section 2 briefly discusses the general structure of the GSETSK and its neural computations. Section 3 presents the novel structural learning phase of GSETSK. Section 4 presents its parameter learning phase. Section 5 evaluates the performance of the GSETSK model using five different simulations and benchmark its performances against other established neural fuzzy systems. Section 6 concludes the paper.
Section snippets
GSETSK – the generic self-evolving TSK fuzzy neural network
This section introduces the structure and functions of GSETSK. The GSETSK model is basically an FNN [3] that consists of six layers of computing nodes as shown in Fig. 1. They are: Layer I (the input layer), Layer II (the input linguistic layer), Layer III (the rule layer), Layer IV (the normalization layer), Layer V (the consequent layer) and Layer VI (the output layer). From Fig. 1, the structure of the proposed GSETSK model defines a set of TSK-type IF-THEN fuzzy rules. The fuzzy rules are
Structure learning of GSETSK
As mentioned, at each arrival of data observations (X(t), d(t)), GSETSK performs its learning process which consists of two phases, namely structural and parameter learning. This section describes the structural learning phase of GSETSK.
GSETSK proposes a clustering technique known as multidimensional-scaling growing clustering (MSGC), initially proposed by the author in [40], to partition the input space from the training data to formulate its fuzzy rules. The multidimensional scaling approach
Parameter learning of GSETSK
In this phase, only the consequent parameters in the consequent nodes at layer V will be tuned. It is shown in Fig. 3 being the last phase of learning before getting another new training tuple into the network. In GSETSK, the output node at layer VI based on the observed data pair (X, D) is shown in
where Bk = [b0k, …, bik, …, bnk]T is the parameter vector of the consequent node Ck; is the normalized firing strength at the
Experimental results
Five different simulations were performed to evaluate the performance of GSETSK, they are, namely: (1) nonlinear dynamic system with non-varying characteristics; (2) nonlinear dynamic system with time-varying characteristics; (3) Mackey–Glass chaotic time series; (4) highway traffic flow density; and (5) Dow Jones index time series. In these experiments, an important parameter that needs to be predefined is λ, the forgetting factor which is used in (14) to determine the fuzzy rule potentials.
Conclusion
This paper presents a self-evolving Takagi–Sugeno–Kang fuzzy framework named GSETSK which can address time-variant problems. It adopts an online data-driven incremental-learning-based approach using the perspective of strict online learning as defined in [50]. GSETSK focuses on addressing the issue of achieving compact and up-to-date fuzzy rule bases in TSK models by using a simple and biologically plausible rule pruning approach. It also attempts to improve the interpretability of derived
References (50)
- et al.
Structure identification of fuzzy model
Fuzzy Sets Syst.
(1988) - et al.
A recurrent self-evolving fuzzy neural network with local feedbacks and its application to dynamic system processing
Fuzzy Sets Syst.
(2010) - et al.
Sequential adaptive fuzzy inference system (SAFIS) for nonlinear system identification and prediction
Fuzzy Sets Syst.
(2006) - et al.
Sequential adaptive fuzzy inference system (SAFIS) for nonlinear system identification and prediction
Appl. Soft Comput.
(2011) - et al.
Ensembling neural networks: many could be better than all
Artif. Intell.
(2002) - et al.
A new approach for time series prediction using ensembles of ANFIS models
Expert Syst. Appl.
(2012) - et al.
Popfnn: a pseudo outer-product based fuzzy neural network
Neural Netw.
(1996) - et al.
Structure and learning algorithms of a nonsingleton input fuzzy neural network based on the approximate analogical reasoning schema
Fuzzy Sets Syst.
(2006) Catastrophic forgetting in connectionist networks
Trends Cogn. Sci.
(1999)- et al.
Neuro-fuzzy rule generation: survey in soft computing framework
IEEE Trans. Neural Netw.
(2000)
Neuro-Fuzzy and Computing
Neural Fuzzy Systems: A Neuro-Fuzzy Synergism to Intelligent Systems
A BCM theory of meta-plasticity for online self-reorganizing fuzzy-associative learning
IEEE Trans. Neural Netw.
EFSM – a novel online neural-fuzzy semantic memory model
IEEE Trans. Neural Netw.
Fuzzy identification of systems and its applications to modeling and control
IEEE Trans. Syst. Man Cybernet.
Deviation of fuzzy control rules from human operator's control action
An online self-constructing neural fuzzy inference network and its applications
IEEE Trans. Fuzzy Syst.
Fitsk: online local learning with generic fuzzy input Takagi–Sugeno–Kang fuzzy framework for nonlinear system estimation
IEEE Trans. Syst. Man Cybernet. B
Denfis: dynamic evolving neural-fuzzy inference system and its application for time-series prediction
IEEE Trans. Syst. Man Cybernet. B
Flexis: a robust incremental learning approach for evolving Takagi–Sugeno fuzzy models
IEEE Trans. Fuzzy Syst.
An approach to online identification of Takagi–Sugeno fuzzy models
IEEE Trans. Syst. Man Cybernet. B
Evolving Takagi–Sugeno fuzzy systems from streaming data (eTS+)
A TSK-type neuro fuzzy network approach to system modeling problems
IEEE Trans. Syst. Man Cybernet. B
Identification of complex systems based on neural and Takagi–Sugeno fuzzy model
IEEE Trans. Syst. Man Cybernet. B
A recurrent self-organizing neural fuzzy inference system
IEEE Trans. Neural Netw.
Cited by (30)
UNFIS: A Novel Neuro-Fuzzy Inference System with Unstructured fuzzy rules
2024, NeurocomputingA novel learning algorithm based on computing the rules’ desired outputs of a TSK fuzzy neural network with non-separable fuzzy rules
2022, NeurocomputingCitation Excerpt :Although backpropagating the output error to adapt premise parts’ parameters in the form of GD or LM method is usual for Mamdani type FNNs (applied generally by removing Normalization layer [1,2]), it is not very common for TSK type FNNs. Generally, in TSK type FNNs, premise parts’ parameters are initialized based on one of the reviewed approaches and remain fixed during learning process [33,29,48,43,9,4]. Indeed, in TSK type FNNs, the backpropagation paradigm can easily get stuck in local optima based on the vanishing gradient problem due to the larger number of parameters and layers in comparison with the Mamdani’s one.
Learning data streams online — An evolving fuzzy system approach with self-learning/adaptive thresholds
2020, Information Sciences