Time series prediction using evolving radial basis function networks with new encoding scheme
Introduction
The artificial neural networks (ANNs) have been effectively applied in many areas [10], such as pattern recognition, function approximation, system identification and control, signal processing, and time series prediction, etc., due to their strong learning capability. Among different kinds of ANNs, the radial basis function (RBF) networks are widely used in time series analysis. The RBF network is a three-layer feed-forward network that generally uses a linear transfer function for the output units and a nonlinear transfer function (normally the Gaussian function) for the hidden units. Its input layer simply consists of the source nodes connected by weighted connections to the hidden layer and the net input to a hidden unit is a distance measure between the input presented at the input layer and the point represented by the hidden unit. The nonlinear transfer function (Gaussian function) is applied to the net input to produce a radial function of the distance. The output units implement a weighted sum of the hidden unit outputs.
Compared with other types of ANNs, such as multilayer perceptron (MLP) neural networks, RBF networks have only one hidden layer, while MLP networks have one or more hidden layers depending on the application task; the hidden and output layers of MLP networks are both nonlinear, while only the hidden layer of RBF networks is nonlinear (the output layer is generally linear); the activation functions in the RBF nodes compute the Euclidean distance between the input examples and the centres, while the activation functions of MLP networks compute inner products from the input examples and the incoming weights, etc. These characteristics give RBF networks many advantages over other ANNs in, e.g., simple architecture and learning scheme, fast training speed (the liner output layer may not be trained), and the possibility of incorporating the qualitative aspects of human experience in the model selection and training. Hence, RBF networks are powerful computation tools and have been extensively used in different areas [17], [26], [29], [30].
In spite of a number of advantages compared with other types of ANNs, such as better approximation capabilities, simple network structures and faster learning algorithms, the development of RBF networks still involves difficulties in selecting the network structure (the number of nodes in the hidden layers, i.e., the number of centres) and calculating the model parameters (e.g., centres, widths and weights), in addition to the problem of selection of inputs to the network which exists in all ANNs. Nowadays, hybridisation in soft computing is becoming a promising research field of computational intelligence focusing on synergistic combinations of multiple soft computing methodologies to develop the next generation of intelligent systems. A fundamental stimulus to the investigations of hybrid approaches is the awareness that combined approaches will be necessary to overcome the limitations of the methods in soft computing [22], [23], [24]. In particular, in order to overcome the existing difficulties in developing an RBF network, the evolving RBF network that combines genetic algorithms (GAs) with other standard learning algorithms for an RBF network has been presented in many papers, such as [2], [27], [32] etc., and a recent review was made in [10]. However, in most of the reported works, an a priori assumption on the selection of model structure is needed or only part of the network parameters are optimised by GAs, such as the number of nodes or the values of centres and widths, etc., combined with the other learning algorithms to obtain the remaining parameters. In addition, no more work has been done on the automatic selection of input variables to RBF networks although some work has been done on fuzzy systems [8]. As in practice, the exact number of nodes and which inputs should be used in training a network cannot be determined a priori; the trial and error method has to be used such that optimal results cannot be guaranteed, and it often consumes time to find acceptable results. Since the number of network nodes and the selection of inputs affect not only the network complexity but also the prediction accuracy of the network, an automatic way of selecting these parameters should be developed to improve both the accuracy and the compactness of RBF networks. This means that the two problems, i.e., the selection of features (such as the input features) from a given, possibly large set of possible features (so-called feature selection) and the optimisation of the model structure with respect to the features selected (so-called model selection), should be addressed simultaneously. Several methods have been proposed for this, such as the separability–correlation measure [7] and discriminative function pruning analysis [19]. And in [1], an evolutionary algorithm is applied to perform the feature and model selection simultaneously for the RBF network. However, regardless of their advantages, these methods are relatively complicated to implement in practice.
To address the above mentioned problems of RBF networks, this paper presents a new encoding scheme such that the network architecture (numbers and selections of nodes and inputs) and the model parameters (centres and widths) can be evolved by GAs simultaneously. In the new encoding scheme, the chromosome consists of three parts, which represent the nodes, the inputs, and the network parameters. For the node and input parts, the position of the binary number ‘1’ indicates which node or input will be selected, and the sum of the binary number ‘1’ indicates the number of nodes or input variables to be used. The centres and widths compose the third part of the chromosome, which will be searched by GAs such that the mean square errors (MSEs) between the true outputs and the network predictions are minimised with respect to the evolved nodes and inputs. This overcomes one drawback of standard RBF networks in selecting centres using a clustering approach, such as k-means clustering algorithm or fuzzy c-means clustering algorithm, which is entirely separated from the actual objective of minimising the prediction error. Only the parameter weights that connect the hidden layer with the output layer are determined by calculating the pseudo-inverse matrix algorithm. To validate the performance and effectiveness of the proposed scheme, the developed evolving RBF networks are used to predict time series for two benchmark examples and one practical application example. By comparing the obtained results with other existing results in the bibliography, we find that the resulting evolving RBF networks can predict the time series satisfactorily with the automatically selected nodes and inputs.
The remainder of this paper is organised as follows. Section 2 introduces the framework of RBF networks and presents the new encoding scheme with the developed algorithm for training the evolving RBF network. In Section 3, the simulation results on time series prediction for two benchmark examples and one practical application example using the presented algorithm are obtained and compared with existing results. Section 4 summarises the findings from this study.
Section snippets
Framework of RBF networks
An RBF network is a three-layer feedforward neural network which consists of an input layer of source nodes, a single layer of nonlinear processing units, and an output layer of linear weights, as depicted in Fig. 1 [11], which includes only one input vector and one output scalar. The input–output relationship of this RBF network can be described bywhere N is the number of hidden layer nodes (neurons); the term is the kth RBF that computes the distance between an
Simulation results
In this section, the presented evolving RBF networks are applied to two time-series prediction problems: Mackey–Glass and Box–Jenkins, which are well-known benchmark examples and are used here for comparison with existing models. The data used in the examples are available from the IEEE Neural Networks Council Standards Committee Working Group on Data Modelling Benchmarks which may be found at: http://neural.cs.nthu.edu.tw/jang/benchmark/. In addition, the presented evolving RBF network will be
Conclusions
In this paper, we present an evolutionary algorithm with a novel encoding scheme for the automatic optimisation of RBF network parameters, especially for the optimisation of the nodes (neurons) in the hidden layer, the inputs to the network, and the values of centres and widths. Two purposes, which are increasing the prediction accuracy and keeping the number of hidden neurons and number of inputs lower, thus being capable of achieving good generalisation performance, are realised
Acknowledgements
The financial support of this work by the University of Technology Sydney, Early Career Research Grant, and the Australian Research Council (DP0560077) is gratefully acknowledged. The authors would like to thank the anonymous reviewers for their constructive comments and suggestions on the improvement of the paper.
Haiping Du received his Ph.D. degree from Shanghai Jiao Tong University, Shanghai, PR China, in 2002. He was awarded the Excellent Ph.D. Thesis Prize by Shanghai Provincial Government in 2004. He is currently a Research Fellow in Mechatronics and Intelligent Systems Group, Faculty of Engineering, University of Technology, Sydney. Previously, he worked as Post-Doctoral Research Associate in the University of Hong Kong and Imperial College London from 2002 to 2003 and 2004 to 2005, respectively.
References (35)
- et al.
Time-series prediction using a local linear wavelet neural network
Neurocomputing
(2006) - et al.
Time-series forecasting using flexible neural tree model
Inf. Sci.
(2005) - et al.
Radial basis function based adaptive fuzzy systems and their applications to system identification and prediction
Fuzzy Sets Syst.
(1996) - et al.
Modelling of a magneto-rheological damper by evolving radial basis function networks
Eng. Appl. Artif. Intell.
(2006) - et al.
The effect of different basis functions on a radial basis function network for time series prediction: a comparative study
Neurocomputing
(2006) - et al.
An efficient MDL-based construction of RBF networks
Neural Networks
(1998) - et al.
Evolutionary system for automatically constructing and adapting radial basis function networks
Neurocomputing
(2006) - et al.
Genetic fuzzy system for damage detection in beams and helicopter rotor blades
Comput. Methods Appl. Mech. Eng.
(2003) - et al.
Evolving RBF neural networks for time-series forecasting with EvRBF
Inf. Sci.
(2004) - et al.
Time series analysis using normalized PG-RBF network with regression weights
Neurocomputing
(2002)
Helicopter rotor blade frequency evolution with damage growth and signal processing
J. Sound Vib.
Filter design using radial basis function neural network and genetic algorithm for improved operational health monitoring
Appl. Soft Comput.
Identification of semi-physical and black-box non-linear models: the case of MR-dampers for vehicles control
Automatica
Time-series forecasting using GA-tuned radial basis functions
Inf. Sci.
Evolutionary optimization of radial basis function classifiers for data mining applications
IEEE Trans. Syst. Man Cybernet.
Combined genetic algorithm optimization and regularized orthogonal least squares learning for radial basis function networks
IEEE Trans. Neural Networks
Data dimensionality reduction with application to simplifying RBF network structure and improving classification performance
IEEE Trans. Syst. Man Cybernet.
Cited by (75)
Parameter estimation for a class of radial basis function-based nonlinear time-series models with moving average noises
2021, Journal of the Franklin InstituteAssessing the suitability of soft computing approaches for forest fires prediction
2018, Applied Computing and InformaticsArtificial chemical reaction optimization of neural networks for efficient prediction of stock market indices
2017, Ain Shams Engineering JournalPractical compensation for nonlinear dynamic thrust measurement system
2015, Chinese Journal of Aeronautics
Haiping Du received his Ph.D. degree from Shanghai Jiao Tong University, Shanghai, PR China, in 2002. He was awarded the Excellent Ph.D. Thesis Prize by Shanghai Provincial Government in 2004. He is currently a Research Fellow in Mechatronics and Intelligent Systems Group, Faculty of Engineering, University of Technology, Sydney. Previously, he worked as Post-Doctoral Research Associate in the University of Hong Kong and Imperial College London from 2002 to 2003 and 2004 to 2005, respectively. His research interests include soft computing, robust control theory and engineering applications, dynamic systems modelling, model and controller reductions, smart materials and structures, etc.
Nong Zhang received his doctorate in 1989 from the University of Tokyo. In the same year, as a Research Assistant Professor, he joined the Faculty of Engineering of the University. In 1992, as a Research Fellow, he joined the Engineering Faculty of the University of Melbourne. In 1995, he joined the Faculty of Engineering of the University of Technology, Sydney. Prof Zhang's research interests include experimental modal analysis, rotor dynamics, vehicle powertrain dynamics, and recently hydraulically interconnected suspension and vehicle dynamics. He is a Member of ASME and a Fellow of the Society of Automotive Engineers, Australasia.