Elsevier

Applied Soft Computing

Volume 35, October 2015, Pages 29-42
Applied Soft Computing

GSETSK: a generic self-evolving TSK fuzzy neural network with a novel Hebbian-based rule reduction approach

https://doi.org/10.1016/j.asoc.2015.06.008Get rights and content

Highlights

  • This paper attempts to achieve a compact, recent, and interpretable fuzzy rule bases.

  • It proposes a novel rule pruning method that is simple and computationally efficient.

  • It proposes a merging approach to improve the interpretability of the knowledge base.

  • GSETSK adopts an online data-driven incremental-learning-based approach.

  • GSETSK derives an up-to-date and interpretable rule base with high level of accuracy.

Abstract

Takagi–Sugeno–Kang (TSK) fuzzy systems have been widely applied for solving function approximation and regression-centric problems. Existing dynamic TSK models proposed in the literature can be broadly classified into two classes. Class I TSK models are essentially fuzzy systems that are limited to time-invariant environments. Class II TSK models are generally evolving systems that can learn in time-variant environments. This paper attempts to address the issues of achieving compact, up-to-date fuzzy rule bases and interpretable knowledge bases in TSK models. It proposes a novel rule pruning method which is simple, computationally efficient and biologically plausible. This rule pruning algorithm applies a gradual forgetting approach and adopts the Hebbian learning mechanism behind the long-term potentiation phenomenon in the brain. It also proposes a merging approach which is used to improve the interpretability of the knowledge bases. This approach can prevent derived fuzzy sets from expanding too many times to protect their semantic meanings. These two approaches are incorporated into a generic self-evolving Takagi–Sugeno–Kang fuzzy framework (GSETSK) which adopts an online data-driven incremental-learning-based approach.

Extensive experiments were conducted to evaluate the performance of the proposed GSETSK against other established evolving TSK systems. GSETSK has also been tested on real world dataset using the high-way traffic flow density and Dow Jones index time series. The results are encouraging. GSETSK demonstrates its fast learning ability in time-variant environments. In addition, GSETSK derives an up-to-date and better interpretable fuzzy rule base while maintaining a high level of modeling accuracy at the same time.

Graphical abstract

The evolution of the fuzzy rules in GSETSK as it learns the DOW JONES index

  1. Download : Download high-res image (95KB)
  2. Download : Download full-size image

Introduction

Neuro-fuzzy computing is a popular framework for solving problems with a soft computing approach due to its capability to combine the human-like reasoning style of fuzzy systems with the connectionist structure and learning ability of neural networks [1]. Neuro-fuzzy hybridization is also widely known as fuzzy neural networks (FNN) or neuro-fuzzy systems (NFS). The main strength of the neuro-fuzzy approach is that it can provide insights to the user about the symbolic knowledge embedded within the network [1]. More specifically, the hybrid network can generalize from training data, learn/tune system parameters, and generate the fuzzy rules to create a linguistic model of the problem domain. Extensive reviews of FNNs are discussed in fuzzy rule learning [1] and neuro-fuzzy works [2], [3], [4], [5]. There are two types of fuzzy model; namely the Mamdani model [3] and the Takagi–Sugeno–Kang (TSK) model [6], [7], [8].

The main advantage of the TSK model over the Mamdani model is its ability to achieve higher system modeling accuracy. Hence, there is a continuing trend of using TSK fuzzy models for solving function approximation and regression-centric problems. In practice, these problems are online, meaning that the data are not all available prior to training but are sequentially presented to the learning system. Many dynamic systems such as SONFIN [9], FITSK [10], DENFIS [11], FLEXFIS [12], eTS [13], and others [14], [15], [16], [17], [18], [19], [20], [21] have been developed to provide solutions for such online problems.

Existing dynamic TSK models proposed in the literature can be broadly classified into two classes. Class I TSK models are essentially fuzzy systems that are generally limited to time-invariant environments. In real life, time-variant problems, which most often occurred in many areas of engineering, usually possess non-stationary, temporal data streams which are modified continuously by underlying data-generating processes. Dynamic approaches such as FITSK [10] and DENFIS [11] belong to this class. DENFIS [11] and FLEXFIS [12] implicitly assume prior knowledge of the upper bound and lower bound of the data set to normalize data before learning [22], which is unsuitable for time-variant environments. Kukolj and Levi [16] proposed a heuristic self-organizing network which is based on k-means for structure learning. Quah and Quek [10] proposed a simple process that presupposes an even space partitioning of the linguistic labels for fast learning. However both [10] and [16] require the number of clusters or rules to be specified prior to training, which is an impossible task in time-variant problems.

Class II TSK models are generally evolving systems [13] that can learn in time-variant environments. Many Class II systems such as [9], [17], [18], [19], [22] do not consider the interpretability of the knowledge bases. They generally employ back-propagation or gradient descent algorithms to heuristically tune the widths of their antecedent membership functions, which can result in highly overlapping and indistinguishable fuzzy sets. Thus, the semantic meaning of the derived fuzzy sets is deteriorated. Besides, many systems such as SONFIN [9], FLEXFIS [12], eTS [13], and [15] do not have a rule pruning algorithm, which may lead to the collection of obsolete rules over time and thus degrade the level of human interpretability of the resultant fuzzy rule base.

Some approaches have been proposed to address these issues. In [23], [24], a merging approach using the Jaccard index as a similarity measure to merge strongly overlapping fuzzy sets is employed. However this method does not prevent the fuzzy sets from growing overly large, and thus the fuzzy labels of the fuzzy sets represent may become obscure and poorly defined. In Simpl_eTS [20], a simple rule pruning algorithm which monitors the population of each rule is proposed. If a rule amounts to less than 1% of the total data samples at that current moment, it is considered as obsolete, and it will be pruned. This approach considers the contribution of old data and new data equally in determining the obsolete rules, thus it cannot detect drifts and shifts in online data streams [25]. In systems such as eTS+ [14] and xTS [21], the age of a cluster is used to determine if a rule (cluster) is obsolete. However, the age of the cluster in [14], [21] is determined by a self-driven formula which does not incorporate the membership degrees of the samples forming that cluster. In [25], the authors proposed an enhanced version of the formula to calculate the age of a cluster in [14], [21] by incorporating the membership degrees of the samples. This approach is reasonable, but not biologically plausible. In this paper, an alternative rule pruning method which is simple, computationally efficient and biologically plausible is proposed. This rule pruning algorithm applies a gradual forgetting approach and adopts the Hebbian learning mechanism behind the long-term potentiation phenomenon [26] in the brain. A merging approach which can prevent derived fuzzy sets from expanding too many times to protect their semantic meanings is also proposed.

Ensemble methods have been in hot topics in recent years [27], [28], [29], [30], [31], [32]. Ensemble is a learning paradigm where multiple networks are jointly used to solve a problem. In general, an ensemble consist of multiple individually trained networks that are used to predict a certain problem, and each individual predicted results are then combined using either the average or weighted average of the results. It has been shown that ensemble methods does provide a better accuracy result as compared to an individual predicting network [30]. However, to require multiple networks to be trained requires a longer duration, hence it is not recommended in time critical applications, and interpretability of the result is impossible. Multiple networks each have its own rule base hence an average or weighted average does not summarize a meaningful rule base for interpretation.

This paper focuses on solving the issues of achieving compact, up-to-date fuzzy rule bases and interpretable knowledge bases in TSK models. Thus, this paper presents an evolving TSK neural-fuzzy framework which can work in time-variant environments to address these issues. The framework is termed the generic self-evolving Takagi–Sugeno–Kang fuzzy framework (GSETSK). GSETSK starts with an initial empty rule base. New rules are sequentially added to the rule base by a fuzzy clustering algorithm called as multidimensional-scaling growing clustering which is completely data-driven. Highly overlapping membership functions are merged by a novel merging approach and obsolete rules are constantly pruned by a Hebbian-based rule pruning algorithm to derive a compact fuzzy rule base while maintaining a high level of modeling accuracy. GSETSK does not require prior knowledge of the numbers of clusters or rules present in the training data set. For parameter tuning, GSETSK employs a localized version of the recursive least-square algorithm [33] for high-accuracy online learning performance.

The paper is organized as follows. Section 2 briefly discusses the general structure of the GSETSK and its neural computations. Section 3 presents the novel structural learning phase of GSETSK. Section 4 presents its parameter learning phase. Section 5 evaluates the performance of the GSETSK model using five different simulations and benchmark its performances against other established neural fuzzy systems. Section 6 concludes the paper.

Section snippets

GSETSK – the generic self-evolving TSK fuzzy neural network

This section introduces the structure and functions of GSETSK. The GSETSK model is basically an FNN [3] that consists of six layers of computing nodes as shown in Fig. 1. They are: Layer I (the input layer), Layer II (the input linguistic layer), Layer III (the rule layer), Layer IV (the normalization layer), Layer V (the consequent layer) and Layer VI (the output layer). From Fig. 1, the structure of the proposed GSETSK model defines a set of TSK-type IF-THEN fuzzy rules. The fuzzy rules are

Structure learning of GSETSK

As mentioned, at each arrival of data observations (X(t), d(t)), GSETSK performs its learning process which consists of two phases, namely structural and parameter learning. This section describes the structural learning phase of GSETSK.

GSETSK proposes a clustering technique known as multidimensional-scaling growing clustering (MSGC), initially proposed by the author in [40], to partition the input space from the training data to formulate its fuzzy rules. The multidimensional scaling approach

Parameter learning of GSETSK

In this phase, only the consequent parameters in the consequent nodes at layer V will be tuned. It is shown in Fig. 3 being the last phase of learning before getting another new training tuple into the network. In GSETSK, the output node at layer VI based on the observed data pair (X, D) is shown in

y=k=1K(t)ZkV=k=1K(t)wkfk(X)=k=1K(t)wk[b0k+b1kx1++bikxi++bnkxn],where Bk = [b0k, …, bik, …, bnk]T is the parameter vector of the consequent node Ck; wk is the normalized firing strength at the

Experimental results

Five different simulations were performed to evaluate the performance of GSETSK, they are, namely: (1) nonlinear dynamic system with non-varying characteristics; (2) nonlinear dynamic system with time-varying characteristics; (3) Mackey–Glass chaotic time series; (4) highway traffic flow density; and (5) Dow Jones index time series. In these experiments, an important parameter that needs to be predefined is λ, the forgetting factor which is used in (14) to determine the fuzzy rule potentials.

Conclusion

This paper presents a self-evolving Takagi–Sugeno–Kang fuzzy framework named GSETSK which can address time-variant problems. It adopts an online data-driven incremental-learning-based approach using the perspective of strict online learning as defined in [50]. GSETSK focuses on addressing the issue of achieving compact and up-to-date fuzzy rule bases in TSK models by using a simple and biologically plausible rule pruning approach. It also attempts to improve the interpretability of derived

References (50)

  • J.S.R. Jang et al.

    Neuro-Fuzzy and Computing

    (1997)
  • C.T. Lin et al.

    Neural Fuzzy Systems: A Neuro-Fuzzy Synergism to Intelligent Systems

    (1996)
  • J. Tan et al.

    A BCM theory of meta-plasticity for online self-reorganizing fuzzy-associative learning

    IEEE Trans. Neural Netw.

    (2010)
  • W.L. Tung et al.

    EFSM – a novel online neural-fuzzy semantic memory model

    IEEE Trans. Neural Netw.

    (2010)
  • T. Takagi et al.

    Fuzzy identification of systems and its applications to modeling and control

    IEEE Trans. Syst. Man Cybernet.

    (1985)
  • T. Takagi et al.

    Deviation of fuzzy control rules from human operator's control action

  • C.F. Juang et al.

    An online self-constructing neural fuzzy inference network and its applications

    IEEE Trans. Fuzzy Syst.

    (1998)
  • K.H. Quah et al.

    Fitsk: online local learning with generic fuzzy input Takagi–Sugeno–Kang fuzzy framework for nonlinear system estimation

    IEEE Trans. Syst. Man Cybernet. B

    (2006)
  • N.K. Kasabov et al.

    Denfis: dynamic evolving neural-fuzzy inference system and its application for time-series prediction

    IEEE Trans. Syst. Man Cybernet. B

    (2002)
  • E.D. Lughofer

    Flexis: a robust incremental learning approach for evolving Takagi–Sugeno fuzzy models

    IEEE Trans. Fuzzy Syst.

    (2008)
  • P. Angelov et al.

    An approach to online identification of Takagi–Sugeno fuzzy models

    IEEE Trans. Syst. Man Cybernet. B

    (2004)
  • P. Angelov

    Evolving Takagi–Sugeno fuzzy systems from streaming data (eTS+)

  • C.S. Ouyang et al.

    A TSK-type neuro fuzzy network approach to system modeling problems

    IEEE Trans. Syst. Man Cybernet. B

    (2005)
  • D. Kukolj et al.

    Identification of complex systems based on neural and Takagi–Sugeno fuzzy model

    IEEE Trans. Syst. Man Cybernet. B

    (2004)
  • C.F. Juang et al.

    A recurrent self-organizing neural fuzzy inference system

    IEEE Trans. Neural Netw.

    (1999)
  • Cited by (30)

    • A novel learning algorithm based on computing the rules’ desired outputs of a TSK fuzzy neural network with non-separable fuzzy rules

      2022, Neurocomputing
      Citation Excerpt :

      Although backpropagating the output error to adapt premise parts’ parameters in the form of GD or LM method is usual for Mamdani type FNNs (applied generally by removing Normalization layer [1,2]), it is not very common for TSK type FNNs. Generally, in TSK type FNNs, premise parts’ parameters are initialized based on one of the reviewed approaches and remain fixed during learning process [33,29,48,43,9,4]. Indeed, in TSK type FNNs, the backpropagation paradigm can easily get stuck in local optima based on the vanishing gradient problem due to the larger number of parameters and layers in comparison with the Mamdani’s one.

    View all citing articles on Scopus
    View full text