Evolving connectionist systems for adaptive learning and knowledge discovery: Trends and directions

doi:10.1016/j.knosys.2014.12.032

Knowledge-Based Systems

Volume 80, May 2015, Pages 24-33

https://doi.org/10.1016/j.knosys.2014.12.032 Get rights and content

Abstract

This paper follows the 25 years of development of methods and systems for knowledge-based neural network systems and more specifically the recent evolving connectionist systems (ECOS). ECOS combine the adaptive/evolving learning ability of neural networks and the approximate reasoning and linguistically meaningful explanation features of symbolic representation, such as fuzzy rules. This review paper presents the classical now hybrid expert systems and evolving neuro-fuzzy systems, along with new developments in spiking neural networks, neurogenetic systems, and quantum inspired systems, all discussed from the point of few of their adaptability, model interpretability and knowledge discovery. The paper discusses new directions for the integration of principles from neural networks, fuzzy systems, bio- and neuroinformatics, and nature in general.

Section snippets

Hybrid connectionist systems

The human brain uniquely combines low level neuronal learning in the neurons and the connections between them and higher level rule abstraction leading to adaptive learning and abstract concept formation. This is the ultimate inspiration for the development of hybrid connectionist systems where specially constructed artificial neural networks (NN) are trained on data so that after training abstract knowledge representation can be derived that explains the data and can be further interpreted as

Fuzzy neurons and fuzzy neural networks. Evolving connectionist systems

A low-level integration of fuzzy rules into a single neuron model and larger neural network structures, tightly coupling learning and fuzzy reasoning rules into connectionists structures, was initiated by Professor Takeshi Yamakawa and other Japanese scientists and promoted at a series of IIZUKA conferences in Japan [39]. Since then, many models of fuzzy neural networks were developed based on these principles [8], [19], [23].

The evolving neuro-fuzzy systems developed these ideas further, where

Current trends in ECOS: Evolving spiking neural networks (eSNN)

A single biological neuron and the associated synapses is a complex information processing machine that involves short term information processing, long term information storage, and evolutionary information stored as genes in the nucleus of the neuron. A spiking neuron model assumes input information represented as trains of spikes over time. When sufficient input information is accumulated in the membrane of the neuron and the neuron’s post synaptic potential exceeds a threshold, the neuron

A current trend and a future direction: Evolving Computational Neuro-Genetic Models (eCNGM)

A neurogenetic model of a neuron is proposed and studied in [4]; Kasabov, 2010). It utilises information about how some proteins and genes affect the spiking activities of a neuron such as fast excitation, fast inhibition, slow excitation, and slow inhibition. An important part of the model is a dynamic gene/protein regulatory network (GRN) model of the dynamic interactions between genes/proteins over time that affect the spiking activity of the neuron – Fig. 9.

New types of neuro-genetic fuzzy

A current trend and a future direction: Quantum inspired eSNN (QeSNN)

QeSNNs use the principle of superposition of states to represent and optimize features (input variables) and gene parameters of an eSNN model [22]. They are optimized through quantum inspired genetic algorithm [6] or QiPSO. Features or genes are represented as qu-bits in a superposition of 1 (selected), with a probability α, and 0 (not selected) with a probability β. When the model has to be calculated, the quantum bits ‘collapse’ in a state of either 1 or 0. Fuzzy rules in QeSNN look like: $IF <$

A current trend and a future direction: The NeuCube eSNN spatio-temporal data machine

The latest development in the direction of eSNN and neurogenetic systems was proposed as a new architecture of a virtual spatio-temporal data machine called NeuCube [27]. It was initially proposed for spatio-temporal brain data modelling, but then it was used for climate data modelling, stroke occurrence prediction and other applications.

The NeuCube framework is depicted in Fig. 11. It consists of the following modules:

–
Input information encoding module.
–
3D SNN module (the Cube).
–
Output module.
–

Conclusion

This paper presents an overview of trends and directions of evolving connectionist systems (ECOS). The main goal of ECOS is to facilitate the creation of computational models and systems for adaptive learning and knowledge discovery from complex data. ECOS principles are derived from the integration of principles from neural networks, fuzzy systems, evolutionary computation, quantum computing and brain information processing. ECOS applications are manifold, but perhaps most welcome in the

Acknowledgement

The work on this paper is supported by the Knowledge Engineering and Discovery Research Institute (KEDRI, http://www.kedri.aut.ac.nz). I was helped with the organization of this paper by Joyce D’Mello. More papers, data and software systems can be found at: http://www.kedri.aut.ac.nz, and: http://ncs.ethz/projects/evospike/.

References (41)

N. Kasabov et al.
FuNN/2-A fuzzy neural network architecture for adaptive learning and knowledge acquisition
Inform. Sci. – Appl.
(1997)
N. Kasabov et al.
Evolving spiking neural networks for personalised modelling of spatio-temporal data and early prediction of events: a case study on stroke
Neurocomputing
(2014)
N. Kasabov et al.
Dynamic evolving spiking neural networks for on-line spatio- and spectro-temporal pattern recognition
Neural Networks
(2013)
Q. Song et al.
TWNFI-a transductive neuro-fuzzy inference system with weighted data normalisation for personalised modelling
Neural Networks
(2006)
S. Thorpe et al.
Spike-based strategies for rapid processing
Neural Networks
(2001)
D. Verstraeten et al.
An experimental unification of reservoir computing methods
Neural Networks
(2007)
S. Wysoski et al.
Evolving spiking neural networks for audiovisual information processing
Neural Networks
(2010)
L. Zadeh
Fuzzy Sets
Inf. Control
(1965)
S. Amari
A theory of adaptive pattern classifiers
IEEE Trans. Electron. Comput.
(1967)
S. Amari
Mathematical foundations of neurocomputing
Proc. IEEEE
(1990)

P. Angelov

Evolving Rule-based Models: A Tool for Design of Flexible Adaptive Systems

(2002)

L. Benuskova et al.

Computational Neuro-genetic Modelling

(2007)

M. Defoin-Platel et al.

Quantum-inspired evolutionary algorithm: a multi-model EDA

IEEE Trans. Evol. Comput.

(2009)

Knowledge-Based Systems Journal, Expert Systems, vol. 1, no. 2, 1988, p....

T. Furuhashi, T. Hasegawa, S. Horikawa, Y. Uchikawa, An Adaptive Fuzzy Controller Using Fuzzy Neural Networks, in:...

M. Futschik, N. Kasabov, Fuzzy clustering in gene expression data analysis, in: Proc. of the World Congress of...

W. Gerstner

Time structure of the activity of neural network models

Phys. Rev.

(1995)

D. Hebb

The Organization of Behavior

(1949)

A.L. Hodgkin et al.

A quantitative description of membrane current and its application to conduction and excitation in nerve

J. Physiol.

(1952)

Cited by (29)

Unsupervised Anomaly Detection in Stream Data with Online Evolving Spiking Neural Networks
2021, Neural Networks
Citation Excerpt :
Depending on the type of input data, the transformation can be carried out by means of the temporal encoding methods such as Step-Forward or Threshold-Based (Maciąg et al., 2019; Petro et al., 2019) or, alternatively, with the use of Gaussian Receptive Fields (Lobo et al., 2018). The distinctive feature of the eSNN is that its repository of output neurons evolves during the training phase based on candidate output neurons that are created for every new input data sample (Kasabov, 2015; Kasabov et al., 2013). More specifically, for each new input value, a new candidate output neuron is created and is either added to the output repository or, based on the provided similarity threshold, is merged with one of the output neurons contained in the repository.
Unsupervised anomaly discovery in stream data is a research topic with many practical applications. However, in many cases, it is not easy to collect enough training data with labeled anomalies for supervised learning of an anomaly detector in order to deploy it later for identification of real anomalies in streaming data. It is thus important to design anomalies detectors that can correctly detect anomalies without access to labeled training data. Our idea is to adapt the Online evolving Spiking Neural Network (OeSNN) classifier to the anomaly detection task. As a result, we offer an Online evolving Spiking Neural Network for Unsupervised Anomaly Detection algorithm (OeSNN-UAD), which, unlike OeSNN, works in an unsupervised way and does not separate output neurons into disjoint decision classes. OeSNN-UAD uses our proposed new two-step anomaly detection method. Also, we derive new theoretical properties of neuronal model and input layer encoding of OeSNN, which enable more effective and efficient detection of anomalies in our OeSNN-UAD approach. The proposed OeSNN-UAD detector was experimentally compared with state-of-the-art unsupervised and semi-supervised detectors of anomalies in stream data from the Numenta Anomaly Benchmark and Yahoo Anomaly Datasets repositories. Our approach outperforms the other solutions provided in the literature in the case of data streams from the Numenta Anomaly Benchmark repository. Also, in the case of real data files of the Yahoo Anomaly Benchmark repository, OeSNN-UAD outperforms other selected algorithms, whereas in the case of Yahoo Anomaly Benchmark synthetic data files, it provides competitive results to the results recently reported in the literature.
IT2-GSETSK: An evolving interval Type-II TSK fuzzy neural system for online modeling of noisy data
2020, Neurocomputing
Citation Excerpt :
However as highlighted in [6], application of IT2FNN in online and evolving modeling is still in the nascent stage and needs more research. Evolving FNNs [7,8] have specific characteristics which are desirable for real-life applications. In non-stationary or evolving environment [9], the distribution of the data related to the behavior of a specific phenomenon changes gradually to abrupt and beyond its historical bounds.
As a core part of a fuzzy neural system, the rule base antecedents and consequents may carry uncertainties because they are trained using noisy data. So, handling the uncertain rule base is an important need in some specific problems such as noisy non-dynamic problems which leads a better data modeling. As a solution, Interval Type-II (IT2) version of GSETSK (Generic Self-Evolving Takagi-Sugeno-Kang), namely IT2-GSETSK, is presented in this paper. This solution uses IT2 membership functions for handling uncertainties, plus having Type-I (GSETSK) capabilities. In this way IT2-GSETSK is a fully-online model able to handle data streams and cope with time-variant data. It also provides up-to-date, relevant and compact rule base that is easily interpretable by human. The IT2-GSETSK is tested over several applications including medical, environmental and financial predictions, which show satisfactory performance of IT2-GSETSK. Moreover, it is observed that while GSETSK performs well enough for dynamic problems with less noise, noisy non-dynamic problems benefit significantly from IT2-GSETSK.
An end-to-end functional spiking model for sequential feature learning
2020, Knowledge-Based Systems
Citation Excerpt :
Inspired by which, the spiking neural network (SNN) and neuromorphic low-power system are proposed and have gained a lot of attention in recent years [3–5]. These SNN based systems have been successfully applied to various fields, such as pattern recognition [6–10], aggregate label learning [11,12], neuromorphic chips [13,14], etc. [15]. One of the important bases for these achievements is to utilize an effective SNN model.
Spiking Neural Network (SNN) has recently gained significant momentum in the neuromorphic low-power systems. However, the existing SNN models have limited use in time-sequential feature learning, and the exhausting spike encoding and decoding make the SNNs not straightforward to use. Inspired by the functional organization in the primate visual system, we propose an end-to-end functional spiking model in this paper to address these issues. Specifically, we propose the functional spike response to make each neuron special, and the dynamic synaptic efficiency to make the transmission of each input signal controllable. We represent inputs by a simple two-tuple set instead of conventional complex encoding, which achieves end-to-end learning. Experiments on synthetic datasets demonstrate that employing the two-tuple encoding strategy, our method improves the accuracy of the traditional SNN model significantly. In addition, we apply our method to seven real-world datasets and one human motion prediction dataset to investigate its performance. Experimental results show that the proposed functional spike response organization saves the running time of our model compared with the LSTM, GRU and one of the state-of-the-art time series processing methods.
Long-term learning for type-2 neural-fuzzy systems
2019, Fuzzy Sets and Systems
Citation Excerpt :
In the literature, a number of adaptive neural fuzzy systems have been developed to deal with such problems, for example online learning [8], incremental learning [9,10], lifelong learning [11], and knowledge-based learning neural networks [12]. Recent research in online learning concentrates on adaptive learning rates to follow time-varying distributions [5,12,13]. Incremental learning is the process of repeatedly training a model with new data without destroying the old prototype patterns [14], while lifelong learning also termed long-term or “continuous learning” addresses learning through the entire lifespan of a system [15].
The development of a new long-term learning framework for interval-valued neural-fuzzy systems is presented for the first time in this article. The need for such a framework is twofold: to address continuous batch learning of data sets, and to take advantage the extra degree of freedom that type-2 Fuzzy Logic systems offer for better model predictive ability. The presented long-term learning framework uses principles of granular computing (GrC) to capture information/knowledge from raw data in the form of interval-valued sets in order to build a computational mechanism that has the ability to adapt to new information in an additive and long-term learning fashion. The latter, is to accommodate new input–output mappings and new classes of data without significantly disturbing existing input–output mappings, therefore maintaining existing performance while creating and integrating new knowledge (rules). This is achieved via an iterative algorithmic process, which involves a two-step operation: iterative rule-base growth (capturing new knowledge) and iterative rule-base pruning (removing redundant knowledge) for type-2 rules. The two-step operation helps create a growing, but sustainable model structure. The performance of the proposed system is demonstrated using a number of well-known non-linear benchmark functions as well as a highly nonlinear multivariate real industrial case study. Simulation results show that the performance of the original model structure is maintained and it is comparable to the updated model's performance following the incremental learning routine. The study is concluded by evaluating the performance of the proposed framework in frequent and consecutive model updates where the balance between model accuracy and complexity is further assessed.
The implementation of univariable scheme-based air temperature for solar radiation prediction: New development of dynamic evolving neural-fuzzy inference system model
2019, Applied Energy
New development of dynamic evolving neural-fuzzy inference system model was proposed for modeling solar radiation based on univariable air temperature scheme. The proposed predictive model was validated against several robust heuristic regression models including multivariate adaptive regression spline, M5 model tree and least square support vector regression. Historical data of solar radiation and air temperature for two meteorological stations: Adana and Antakya located in Turkey, were investigated. The prediction results were evaluated based on several statistical metrics. The modeling is conducted based on different data division scenarios (training/testing) phases. The attained prediction results evidenced the potential of the dynamic evolving neural-fuzzy inference system over the comparable models and for all the investigated data division scenarios. In quantitative terms, dynamic evolving neural-fuzzy inference system was enhanced the solar radiation prediction capability over the multivariate adaptive regression spline, M5 model tree and least square support vector regression models (Adana-Antakya) by 20–42%, 29–47% and 19–43% based on the root mean square errors metric. The applied predictive models were compared with the field measured average monthly solar radiation values and it was found that the proposed model estimates accurately. However, the comparable models were exhibited a considerable overestimation of the monthly averaged solar radiation values and for both inspected stations specifically in the summer months (June, July and August).
Clustering-based undersampling in class-imbalanced data
2017, Information Sciences
Citation Excerpt :
The threshold method involves setting different threshold values for different classes during the classifier learning stage [37], whereas the one-class learning method entails training the classifier from a training set that contains only one specific class [10,30]. Other types of algorithms, such as evolving clustering in neurofuzzy systems, evolving clustering of dynamic data in spiking neural networks, clustering personalized modeling, and clustering through quantum-inspired evolutionary algorithms, have also been developed to deal with imbalanced data [18–20,28] The cost-sensitive solutions focus on defining different misclassification costs of classifiers for different classes.
Class imbalance is often a problem in various real-world data sets, where one class (i.e. the minority class) contains a small number of data points and the other (i.e. the majority class) contains a large number of data points. It is notably difficult to develop an effective model using current data mining and machine learning algorithms without considering data preprocessing to balance the imbalanced data sets. Random undersampling and oversampling have been used in numerous studies to ensure that the different classes contain the same number of data points. A classifier ensemble (i.e. a structure containing several classifiers) can be trained on several different balanced data sets for later classification purposes. In this paper, we introduce two undersampling strategies in which a clustering technique is used during the data preprocessing step. Specifically, the number of clusters in the majority class is set to be equal to the number of data points in the minority class. The first strategy uses the cluster centers to represent the majority class, whereas the second strategy uses the nearest neighbors of the cluster centers. A further study was conducted to examine the effect on performance of the addition or deletion of 5 to 10 cluster centers in the majority class. The experimental results obtained using 44 small-scale and 2 large-scale data sets revealed that the clustering-based undersampling approach with the second strategy outperformed five state-of-the-art approaches. Specifically, this approach combined with a single multilayer perceptron classifier and C4.5 decision tree classifier ensembles delivered optimal performance over both small- and large-scale data sets.

View all citing articles on Scopus

View full text

Evolving connectionist systems for adaptive learning and knowledge discovery: Trends and directions

Abstract

Section snippets

Hybrid connectionist systems

Fuzzy neurons and fuzzy neural networks. Evolving connectionist systems

Current trends in ECOS: Evolving spiking neural networks (eSNN)

A current trend and a future direction: Evolving Computational Neuro-Genetic Models (eCNGM)

A current trend and a future direction: Quantum inspired eSNN (QeSNN)

A current trend and a future direction: The NeuCube eSNN spatio-temporal data machine

Conclusion

Acknowledgement

Inform. Sci. – Appl.

Neurocomputing

Neural Networks

Neural Networks

Neural Networks

Neural Networks

Neural Networks

Inf. Control

A theory of adaptive pattern classifiers

IEEE Trans. Electron. Comput.

Mathematical foundations of neurocomputing

Proc. IEEEE

Evolving Rule-based Models: A Tool for Design of Flexible Adaptive Systems

Computational Neuro-genetic Modelling

Quantum-inspired evolutionary algorithm: a multi-model EDA

IEEE Trans. Evol. Comput.

Time structure of the activity of neural network models

Phys. Rev.

The Organization of Behavior

A quantitative description of membrane current and its application to conduction and excitation in nerve

J. Physiol.