Elsevier

Neurocomputing

Volume 71, Issues 7–9, March 2008, Pages 1388-1400
Neurocomputing

Time series prediction using evolving radial basis function networks with new encoding scheme

https://doi.org/10.1016/j.neucom.2007.06.004Get rights and content

Abstract

This paper presents a new encoding scheme for training radial basis function (RBF) networks by genetic algorithms (GAs). In general, it is very difficult to select the proper input variables and the exact number of nodes before training an RBF network. In the proposed encoding scheme, both the architecture (numbers and selections of nodes and inputs) and the parameters (centres and widths) of the RBF networks are represented in one chromosome and evolved simultaneously by GAs so that the selection of nodes and inputs can be achieved automatically. The performance and effectiveness of the presented approach are evaluated using two benchmark time series prediction examples and one practical application example, and are then compared with other existing methods. It is shown by the simulation tests that the developed evolving RBF networks are able to predict the time series accurately with the automatically selected nodes and inputs.

Introduction

The artificial neural networks (ANNs) have been effectively applied in many areas [10], such as pattern recognition, function approximation, system identification and control, signal processing, and time series prediction, etc., due to their strong learning capability. Among different kinds of ANNs, the radial basis function (RBF) networks are widely used in time series analysis. The RBF network is a three-layer feed-forward network that generally uses a linear transfer function for the output units and a nonlinear transfer function (normally the Gaussian function) for the hidden units. Its input layer simply consists of the source nodes connected by weighted connections to the hidden layer and the net input to a hidden unit is a distance measure between the input presented at the input layer and the point represented by the hidden unit. The nonlinear transfer function (Gaussian function) is applied to the net input to produce a radial function of the distance. The output units implement a weighted sum of the hidden unit outputs.

Compared with other types of ANNs, such as multilayer perceptron (MLP) neural networks, RBF networks have only one hidden layer, while MLP networks have one or more hidden layers depending on the application task; the hidden and output layers of MLP networks are both nonlinear, while only the hidden layer of RBF networks is nonlinear (the output layer is generally linear); the activation functions in the RBF nodes compute the Euclidean distance between the input examples and the centres, while the activation functions of MLP networks compute inner products from the input examples and the incoming weights, etc. These characteristics give RBF networks many advantages over other ANNs in, e.g., simple architecture and learning scheme, fast training speed (the liner output layer may not be trained), and the possibility of incorporating the qualitative aspects of human experience in the model selection and training. Hence, RBF networks are powerful computation tools and have been extensively used in different areas [17], [26], [29], [30].

In spite of a number of advantages compared with other types of ANNs, such as better approximation capabilities, simple network structures and faster learning algorithms, the development of RBF networks still involves difficulties in selecting the network structure (the number of nodes in the hidden layers, i.e., the number of centres) and calculating the model parameters (e.g., centres, widths and weights), in addition to the problem of selection of inputs to the network which exists in all ANNs. Nowadays, hybridisation in soft computing is becoming a promising research field of computational intelligence focusing on synergistic combinations of multiple soft computing methodologies to develop the next generation of intelligent systems. A fundamental stimulus to the investigations of hybrid approaches is the awareness that combined approaches will be necessary to overcome the limitations of the methods in soft computing [22], [23], [24]. In particular, in order to overcome the existing difficulties in developing an RBF network, the evolving RBF network that combines genetic algorithms (GAs) with other standard learning algorithms for an RBF network has been presented in many papers, such as [2], [27], [32] etc., and a recent review was made in [10]. However, in most of the reported works, an a priori assumption on the selection of model structure is needed or only part of the network parameters are optimised by GAs, such as the number of nodes or the values of centres and widths, etc., combined with the other learning algorithms to obtain the remaining parameters. In addition, no more work has been done on the automatic selection of input variables to RBF networks although some work has been done on fuzzy systems [8]. As in practice, the exact number of nodes and which inputs should be used in training a network cannot be determined a priori; the trial and error method has to be used such that optimal results cannot be guaranteed, and it often consumes time to find acceptable results. Since the number of network nodes and the selection of inputs affect not only the network complexity but also the prediction accuracy of the network, an automatic way of selecting these parameters should be developed to improve both the accuracy and the compactness of RBF networks. This means that the two problems, i.e., the selection of features (such as the input features) from a given, possibly large set of possible features (so-called feature selection) and the optimisation of the model structure with respect to the features selected (so-called model selection), should be addressed simultaneously. Several methods have been proposed for this, such as the separability–correlation measure [7] and discriminative function pruning analysis [19]. And in [1], an evolutionary algorithm is applied to perform the feature and model selection simultaneously for the RBF network. However, regardless of their advantages, these methods are relatively complicated to implement in practice.

To address the above mentioned problems of RBF networks, this paper presents a new encoding scheme such that the network architecture (numbers and selections of nodes and inputs) and the model parameters (centres and widths) can be evolved by GAs simultaneously. In the new encoding scheme, the chromosome consists of three parts, which represent the nodes, the inputs, and the network parameters. For the node and input parts, the position of the binary number ‘1’ indicates which node or input will be selected, and the sum of the binary number ‘1’ indicates the number of nodes or input variables to be used. The centres and widths compose the third part of the chromosome, which will be searched by GAs such that the mean square errors (MSEs) between the true outputs and the network predictions are minimised with respect to the evolved nodes and inputs. This overcomes one drawback of standard RBF networks in selecting centres using a clustering approach, such as k-means clustering algorithm or fuzzy c-means clustering algorithm, which is entirely separated from the actual objective of minimising the prediction error. Only the parameter weights that connect the hidden layer with the output layer are determined by calculating the pseudo-inverse matrix algorithm. To validate the performance and effectiveness of the proposed scheme, the developed evolving RBF networks are used to predict time series for two benchmark examples and one practical application example. By comparing the obtained results with other existing results in the bibliography, we find that the resulting evolving RBF networks can predict the time series satisfactorily with the automatically selected nodes and inputs.

The remainder of this paper is organised as follows. Section 2 introduces the framework of RBF networks and presents the new encoding scheme with the developed algorithm for training the evolving RBF network. In Section 3, the simulation results on time series prediction for two benchmark examples and one practical application example using the presented algorithm are obtained and compared with existing results. Section 4 summarises the findings from this study.

Section snippets

Framework of RBF networks

An RBF network is a three-layer feedforward neural network which consists of an input layer of source nodes, a single layer of nonlinear processing units, and an output layer of linear weights, as depicted in Fig. 1 [11], which includes only one input vector and one output scalar. The input–output relationship of this RBF network can be described byy=k=1Nwkϕ(u,tk)+w0,where N is the number of hidden layer nodes (neurons); the term ϕ(u,tk) is the kth RBF that computes the distance between an

Simulation results

In this section, the presented evolving RBF networks are applied to two time-series prediction problems: Mackey–Glass and Box–Jenkins, which are well-known benchmark examples and are used here for comparison with existing models. The data used in the examples are available from the IEEE Neural Networks Council Standards Committee Working Group on Data Modelling Benchmarks which may be found at: http://neural.cs.nthu.edu.tw/jang/benchmark/. In addition, the presented evolving RBF network will be

Conclusions

In this paper, we present an evolutionary algorithm with a novel encoding scheme for the automatic optimisation of RBF network parameters, especially for the optimisation of the nodes (neurons) in the hidden layer, the inputs to the network, and the values of centres and widths. Two purposes, which are increasing the prediction accuracy and keeping the number of hidden neurons and number of inputs lower, thus being capable of achieving good generalisation performance, are realised

Acknowledgements

The financial support of this work by the University of Technology Sydney, Early Career Research Grant, and the Australian Research Council (DP0560077) is gratefully acknowledged. The authors would like to thank the anonymous reviewers for their constructive comments and suggestions on the improvement of the paper.

Haiping Du received his Ph.D. degree from Shanghai Jiao Tong University, Shanghai, PR China, in 2002. He was awarded the Excellent Ph.D. Thesis Prize by Shanghai Provincial Government in 2004. He is currently a Research Fellow in Mechatronics and Intelligent Systems Group, Faculty of Engineering, University of Technology, Sydney. Previously, he worked as Post-Doctoral Research Associate in the University of Hong Kong and Imperial College London from 2002 to 2003 and 2004 to 2005, respectively.

References (35)

  • N. Roy et al.

    Helicopter rotor blade frequency evolution with damage growth and signal processing

    J. Sound Vib.

    (2005)
  • N. Roy et al.

    Filter design using radial basis function neural network and genetic algorithm for improved operational health monitoring

    Appl. Soft Comput.

    (2006)
  • S.M. Savaresi et al.

    Identification of semi-physical and black-box non-linear models: the case of MR-dampers for vehicles control

    Automatica

    (2005)
  • A.F. Sheta et al.

    Time-series forecasting using GA-tuned radial basis functions

    Inf. Sci.

    (2001)
  • O. Buchtala et al.

    Evolutionary optimization of radial basis function classifiers for data mining applications

    IEEE Trans. Syst. Man Cybernet.

    (2005)
  • C. Chen et al.

    Combined genetic algorithm optimization and regularized orthogonal least squares learning for radial basis function networks

    IEEE Trans. Neural Networks

    (1999)
  • X. Fu et al.

    Data dimensionality reduction with application to simplifying RBF network structure and improving classification performance

    IEEE Trans. Syst. Man Cybernet.

    (2003)
  • Cited by (75)

    View all citing articles on Scopus

    Haiping Du received his Ph.D. degree from Shanghai Jiao Tong University, Shanghai, PR China, in 2002. He was awarded the Excellent Ph.D. Thesis Prize by Shanghai Provincial Government in 2004. He is currently a Research Fellow in Mechatronics and Intelligent Systems Group, Faculty of Engineering, University of Technology, Sydney. Previously, he worked as Post-Doctoral Research Associate in the University of Hong Kong and Imperial College London from 2002 to 2003 and 2004 to 2005, respectively. His research interests include soft computing, robust control theory and engineering applications, dynamic systems modelling, model and controller reductions, smart materials and structures, etc.

    Nong Zhang received his doctorate in 1989 from the University of Tokyo. In the same year, as a Research Assistant Professor, he joined the Faculty of Engineering of the University. In 1992, as a Research Fellow, he joined the Engineering Faculty of the University of Melbourne. In 1995, he joined the Faculty of Engineering of the University of Technology, Sydney. Prof Zhang's research interests include experimental modal analysis, rotor dynamics, vehicle powertrain dynamics, and recently hydraulically interconnected suspension and vehicle dynamics. He is a Member of ASME and a Fellow of the Society of Automotive Engineers, Australasia.

    View full text