INFGMN – Incremental Neuro-Fuzzy Gaussian mixture network

Highlights

• A NFS that learns incrementally using a single scan over the training data.

• The learning process can proceed in perpetuity as new training data become available.

• A Mamdani–Larsen fuzzy rule base is defined automatically and incrementally.

• Attempts to provide the best trade-off between accuracy and interpretability.

• Is unaffected by catastrophic interference (Stability-Plasticity dilemma).

Abstract

Accuracy and interpretability are contradictory objectives that conflict in all machine learning techniques and achieving a satisfactory balance between these two criteria is a major challenge. The objective is not only to maximize interpretability, but also to guarantee a high degree of accuracy. This challenge is even greater when it is considered that the model will have to evolve and adapt itself to the dynamics of the underlying environment, i.e. it will have to learn incrementally. Little research has been published about incremental learning using Mamdani–Larsen (ML) fuzzy models under these conditions. This article presents a novel proposal for a Neuro-Fuzzy System (NFS) with an incremental learning capability, the Incremental Neuro-Fuzzy Gaussian Mixture Network (INFGMN), that attempts to generate incremental models that are highly interpretable and precise. The principal characteristics of the INFGMN are as follows: (i) the INFGMN learns incrementally using a single sweep of the training data (each training pattern can be immediately used and discarded); (ii) it is capable of producing reasonable estimates based on few training data; (iii) the learning process can proceed in perpetuity as new training data become available (learning and recalling phases are not separate); (iv) the INFGMN can deal with the Stability-Plasticity dilemma and is unaffected by catastrophic interference (rules are added or removed whenever necessary); (v) the fuzzy rule base is defined automatically and incrementally (new rules are added whenever necessary); and (vi) the INFGMN maintains an ML-type fuzzy rule base that attempts to provide the best trade-off between accuracy and interpretability, thereby dealing with the Accuracy-Interpretability dilemma. The INFGMNs performance in terms of learning and modelling is assessed using a variety of benchmark applications and the results are promising.

Introduction

Accuracy and interpretability are two contradictory objectives that conflict in all machine learning techniques. In an ideal scenario it would be desirable to meet both these criteria to a high degree, but since they are opposing objectives, this is generally impossible. As a result, many researchers have concentrated on improving the balance between accuracy and interpretability, to fit the nature (prerequisites) of the model. In general, one of these objectives is given priority (Alcalá, Alcalá-Fdez, Casillas, Cordón, Herrera, 2006, Casillas, 2003, Casillas, Cordón, Triguero, Magdalena, 2003, Gacto, Alcalá, Herrera, 2011).

Initially, many machine learning techniques were employed by human specialists to generate models from their specialized knowledge (Takagi, Sugeno, 1985, Zadeh, 1973, Zadeh, 1975). For example, one of the most important tasks in constructing Fuzzy Rule Based Systems (FRBSs) is to derive the knowledge base (fuzzy rule base). This used to be achieved manually, resulting in fuzzy rule bases that were rigidly fixed and could not be adapted or adjusted to achieve better performance after the initial design phase. More recently, researchers have worked on methods to directly construct and adjust fuzzy systems from numerical training data, as a way of dealing with the problem of knowledge acquisition. One popular approach is to use Artificial Neural Networks (ANNs) to derive the structure of a Neural Fuzzy System (or Neuro-Fuzzy System – NFS) (Buckley, Hayashi, 1994, Lin, Lee, 1996).

NFSs have been applied to solve problems in several areas such as in student modeling (Iraji, Aboutalebi, Seyedaghaee, Tosinia, 2012, Sevarac, 2006, Stathacopoulou, Grigoriadou, Samarakou, Mitropoulos, 2007), in medical systems (Agboizebeta, Chukwuyeni, 2012, Khameneh, Arabalibeik, Salehian, Setayeshi, 2012, Sengur, 2008), in economic systems (Atsalakis, Valavanis, 2009, Fang, 2012, Gumus, Guneri, Keles, 2009, Lin, Chen, Peng, 2012), in traffic control (Partouche, Pasquier, Spalanzani, 2007, Sindal, Tokekar, 2009), in image processing and feature extraction (Ja’fari, Kadkhodaie-Ilkhchi, Sharghi, Ghanavati, 2011, Montazer, Saremi, Khatibi, 2010), in forecasting and prediction (Ang, Quek, 2006, Galavi, Shui, 2012, Liu, Liang, Chen, Chen, Shen, 2012), in manufacturing and system modeling (Hsiao, Hwang, Chen, Tsai, 2005, Kayacan, Kayacan, Ramon, Saeys, 2013, Kurnaz, Cetin, Kaynak, 2010), in electrical and electronics system (Coteli, Deniz, Dandil, Tuncer, Ata, 2012, Mohandes, Rehman, Rahman, 2011, Toosi, Kahani, 2007), in NFS enhancement (Cetisli, 2010, Chatterjee, Siarry, 2007, Chen, 2012) and in social sciences (Petrovic-Lazarevic, Coghill, & Abraham, 2004). See Castellano, Castiello, Fanelli, and Jain (2007), Kar, Das, and Ghosh (2014), and Kar et al. (2014) for additional areas of application for real-world problem solving.

One area of focus in machine learning techniques that extract their models from data is that the knowledge extracted should be understandable to a human being. In other words, it is undesirable that the resulting model should be of the type black box. For example, when FRBSs are constructed from specialized knowledge, the result is a very understandable model with satisfactory accuracy. However, when the structure of the fuzzy system is derived using ANNs, the majority of methods concentrate on improving the accuracy of the model (Guillaume & Magdalena, 2006). According to Casillas (2003) and Casillas et al. (2003), the challenge lies in combining knowledge extracted from the data with specialized knowledge, thereby creating compact and robust systems with a good balance between accuracy and interpretability and so dealing with what is generally known as the Accuracy-Interpretability dilemma.

Since the procedures for training or for generating rules used by the majority of these NFSs derived from ANNs presuppose that the characteristics of the underlying processes being modeled do not change over time, they are only applicable in static environments. In the majority of cases, batch mode or pseudo-incremental approaches are used to train and refine the models created and, therefore, the majority of NFSs are not appropriate for modelling more complex processes that vary over time in dynamic environments.

Adaptation to a constantly-changing environment generally requires a retraining phase to construct a new fuzzy model on the basis of an updated training dataset. However, the process of learning new information can potentially modify the fuzzy model. These changes may affect the knowledge originally learned (which remains valid), resulting in it being forgotten or substituted. One result of this is that these NFSs can fall into a trap known as the Stability-Plasticity dilemma (see Grossberg (1982), Carpenter and Grossberg (1988) or Mermillod, Bugaiska, and Bonin (2013) for more detail on this dilemma).

More recently, new NFSs based on the idea of incremental learning have been proposed. Examples include the Artificial Neural Network Based Fuzzy Inference System (ANNBFIS) proposed by Łȩski and Czogała (1999), the Adaptive-Network-based Fuzzy Inference (ANFIS), Evolving Fuzzy Neural Network (EFuNN) and the Dynamic Evolving Neurofuzzy Inference System (DENFIS), based on the Evolving Connectionist System (ECoS), which can be found in work by Kasabov (2001) and Kasabov and Song (2002), respectively, the evolving Takagi-Sugeno (eTS) which is a contribution made by Angelov and Filev (2004), the Self-reorganizing Fuzzy Associative Machine (SeroFAM) studied by Tan and Quek (2010), the Evolving Neural-fuzzy Semantic Memory (eFSM) network, proposed by Tung and Quek (2010), the Self-adaptive Fuzzy Inference Network (SaFIN) from Tung, Quek, and Guan (2011) and the Sequential Probabilistic Learning for Adaptive Fuzzy Inference System (SPLAFIS), developed by Oentaryo, Er, Linn, and Li (2014).

In incremental learning methods, it is assumed that the data patterns are sampled individually. Structural learning (the rule generation procedure) and parameter learning (fitting of parameters) are conducted incrementally, based only on the current sample of training data. As a result, the model that is generated can evolve and adapt its knowledge depending on the dynamics of the underlying environment. However, as pointed out by Tung and Quek (2010), beyond the already well-established models such as the fuzzy adaptive learning control network - Adaptive Resonance Theory (Falcon-ART) (Lin & Lin, 1997), EFuNN, SeroFAM, eFSM and SaFIN, the majority of incremental NFSs proposed in the literature are based on FRBSs of the Takagi-Sugeno (TS) type.

Although FRBSs of the Mamdani–Larsen (ML) type are more interpretable than FRBSs of the TS type (Riid, Rüstern, 2014, Tung, Quek, 2009), there has been little investigation into ML-type NFSs with incremental learning capability (Tung & Quek, 2010). Additionally, as pointed out by Tung and Quek (2010), the Falcon-ART and EFuNN networks have serious defects. Falcon-ART has no mechanism for removing rules. This can lead to a structure containing a large number of out-of-date rules, degrading the level of human interpretability of the resultant fuzzy rule base. In the EFuNN, a separate procedure (generally an offline one) is needed to identify the predefined (and generally static) fuzzy sets of the incremental rules system. The SeroFAM, eFSM and SaFIN demonstrate improvements over their predecessors in terms of incremental learning, dealing better with the Stability-Plasticity and Accuracy-Interpretability dilemmas, but, as will be demonstrated below, there is still considerable scope for further improvements.

Drawing on the equivalence established by Gan, Hanmandlu, and Tan (2005) between a Gaussian mixture model and an ML-type FRBS and on the probabilistic ANN model based on Gaussian mixture models with incremental learning capacity proposed by Engel and Heinen (2010), the objective of this study is to propose the IFNGMM, a neuro-fuzzy network model with incremental learning capability that has the following principal characteristics: (i) the INFGMN learns incrementally using a single sweep of the training data (each training pattern can be immediately used and discarded); (ii) it is capable of producing reasonable estimates based on few training data; (iii) the learning process can continue perpetually as new training data become available (learning and recalling phases are not separate); (iv) the INFGMN can deal with the Stability-Plasticity dilemma and is unaffected by catastrophic interference (rules are added or removed whenever necessary); (v) the fuzzy rule base is defined automatically and incrementally (new rules are added whenever necessary); and (vi) the INFGMN maintains an ML-type fuzzy rule base that attempts to provide the best trade-off between accuracy and interpretability, thereby dealing with the Accuracy-Interpretability dilemma.

The remainder of this paper is organized as follows. Section 2 discusses the importance of interpretability in machine learning in general and, more specifically with relation to FRBSs, demonstrates that under certain conditions there is mathematical equivalence between a Gaussian Mixture model (GMM) and an ML-type fuzzy model. The same section also presents the Incremental GMM (IGMM), which can be viewed as an incremental counterpart to the Expectation-Maximization (EM) algorithm. Section 3 describes the INFGMNs operating modes and configuration parameters. Section 4 analyzes the INFGMNs learning and modelling performance. Section 5 discusses the results of the performed experiments and Section 6 concludes the paper.