A Novel hybrid genetic algorithm for kernel function and parameter optimization in support vector regression

https://doi.org/10.1016/j.eswa.2008.06.046Get rights and content

Abstract

This study developed a novel model, HGA-SVR, for type of kernel function and kernel parameter value optimization in support vector regression (SVR), which is then applied to forecast the maximum electrical daily load. A novel hybrid genetic algorithm (HGA) was adapted to search for the optimal type of kernel function and kernel parameter values of SVR to increase the accuracy of SVR. The proposed model was tested at an electricity load forecasting competition announced on the EUNITE network. The results showed that the new HGA-SVR model outperforms the previous models. Specifically, the new HGA-SVR model can successfully identify the optimal type of kernel function and all the optimal values of the parameters of SVR with the lowest prediction error values in electricity load forecasting.

Introduction

Support vector machines (SVMs) have been successfully applied to a number of applications such as including handwriting recognition, particle identification (e.g., muons), digital images identification (e.g., face identification), text categorization, bioinformatics (e.g., gene expression), function approximation and regression, and database marketing, and so on. Although SVMs have become more widely employed to forecast time-series data (Tay and Cao, 2001, Cao, 2003, Kim, 2003) and to reconstruct dynamically chaotic systems (Müller et al., 1997, Mukherjee et al., 1997, Mattera and Haykin, 1999, Kulkarni et al., 2003), a highly effective model can only be built after the parameters of SVMs are carefully determined (Duan, Keerthi, & Poo, 2003).

Min and Lee (2005) stated that the optimal parameter search on SVM plays a crucial role in building a prediction model with high prediction accuracy and stability. The kernel-parameters are the few tunable parameters in SVMs controlling the complexity of the resulting hypothesis (Cristianini, Campell, & Taylor, 1999). Shawkat and Kate (2007) pointed out that selecting the optimal degree of a polynomial kernel is critical to ensure good generalization of the resulting support vector machine model. They proposed an automatic selection for determining the optimal degree of polynomial kernel in SVM by Bayesian and Laplace approximation method estimation and a rule based meta-learning approach. In addition, to construct an efficient SVM model with RBF kernel, two extra parameters: (a) sigma squared and (b) gamma, have to be carefully predetermined. However, few studies have been devoted to optimizing the parameter values of SVMs. Evolutionary algorithms often have to solve optimization problems in the presence of a wide range of problems (Dastidar et al., 2005, Shin et al., 2005, Yaochu and Branke, 2005, Zhang et al., 2005). In these algorithms, genetic algorithms (GAs) have been widely and successfully applied to various types of optimization problems in recent years (Goldberg, 1989, Fogel, 1994, Cao, 2003, Alba and Dorronsoro, 2005, Alba and Dorronsoro, 2005; Aurnhammer and Tonnies, 2005, Venkatraman and Yen, 2005, Hokey et al., 2006, Cao and Wu, 1999, McCall, 2005). Therefore, this paper proposes a hybrid genetic-based SVR model, HGA-SVR, which can automatically optimize the SVR parameters integrating the real-valued genetic algorithm (RGA) and integer genetic algorithm, for increasing the predictive accuracy and capability of generalization compared with traditional machine learning models.

In addition, a wide range of approaches including time-varying splines (Harvey & Koopman, 1993), multiple regression models (Ramanathan, Engle, Granger, Vahid-Araghi, & Brace, 1997), judgmental forecasts, artificial neural networks (Hippert & Pedreira, 2001) and SVMs (Chen et al., 2004, Tian and Noore, 2004) have been employed to forecast electricity load. One of the most crucial demands for the operation activities of power systems is short-term hourly load forecasting and the extension to several days in the future. Improving the accuracy of short-term load forecasting (STLF) is becoming even more significant than before due to the changing structure of the power utility industry (Tian & Noore, 2004). SVMs have been applied to STLF and performed well. Unfortunately, there is still no consensus as to the perfect approach to electricity demand forecasting (Taylor & Buizza, 2003).

Several studies have proposed optimization methods which used a genetic algorithm for optimizing the SVR parameter values. To overcome the problem of SVR parameters, a GA-SVR has been proposed in a earlier paper (Hsu, Wu, Chen, & Peng, 2006) to take advantage of the GAs optimization technique. However, few studies have focused on concurrently optimizing the type of SVR kernel function and the parameters of SVR kernel function. The present study proposed a novel and specialized hybrid genetic algorithm for optimizing all the SVR parameters simultaneously. Our proposed method was applied to predicting maximum electrical daily load and its performance was analyzed. An actual case of forecasting maximum electrical daily load is illustrated to show the improvement in predictive accuracy and capability of generalization achieved by our proposed HGA-SVR model.

The remainder of this paper is organized as follows. The research gap for obtaining optimal parameters in SVR is reviewed and discussed in Section 2. Section 3 details the proposed HGA-SVR, ideas and procedures. In Section 4 an experimental example for predicting the electricity load is described to demonstrate the proposed method. Discussions are presented in Section 5 and conclusions are drawn in the final Section.

Section snippets

Basic ideas of methods for obtaining optimal parameters in SVR

SVR is a promising technique for data classification and regression (Vapnik, 1998). We briefly introduce the basic idea of SVR in the Section 2.1. To design an effective model, the values of the essential parameters in SVR must be chosen carefully in advance (Duan et al., 2003). Thus, various approaches to determine these values are discussed in Section 2.2. Although many optimization methods have been proposed, GAs is well suited to the concurrent manipulation of models with varying

Design of the hybrid genetic-based SVR (HGA-SVR) model for improving predictive accuracy

In this section, we describe the design of our proposed novel HGA-SVR model. The optimization process of HGA-SVR is introduced in the first section. The basic idea of non-linear SVR model is described in the next section. The design of chromosome representations, fitness function and genetic operators in our novel HGA-SVR are discussed in the final sections.

Experimental example for predicting electricity load

In this section, the effectiveness of the proposed HGA-SVR model was demonstrated by forecasting the daily electricity loading problem as announced on the ’Worldwide Competition within the EUNITE Network1’. The set problem was to predict the maximum daily electricity load for January 1999 using

Conclusions

This study proposed a novel hybrid genetic algorithm for dynamically optimizing all the essential parameters of SVR. Our experimental results demonstrated the successful application of our proposed new model, HGA-SVR, for the complex forecasting problem. It demonstrated that it increased the electricity load forecasting accuracy more than any other model employed in the EUNITE network competition. Specifically, the new HGA-SVR model can successfully identify all the optimal values of the SVR

Acknowledgement

This work was supported by National Science Council of the Republic of China under Grant No. NSC 95-2416-H-147-005.

References (45)

  • Francis E.H. Tay et al.

    Application of support vector machines in financial time series forecasting

    OMEGA The International Journal of Management Science

    (2001)
  • J.W. Taylor et al.

    Using weather ensemble predictions in electricity demand forecasting

    International Journal of Forecasting

    (2003)
  • Adewuya, A.A. (1996) New methods in genetic search with real-valued chromosomes. Master’s thesis, Cambridge:...
  • E. Alba et al.

    The exploration/exploitation tradeoff in dynamic cellular genetic algorithms

    IEEE Transactions on Evolutionary Computation

    (2005)
  • E. Alba et al.

    The exploration/exploitation tradeoff in dynamic cellular genetic algorithms

    IEEE Transactions on Evolutionary Computation

    (2005)
  • M. Aurnhammer et al.

    A genetic algorithm for automated horizon correlation across faults in seismic images

    IEEE Transactions on Evolutionary Computation

    (2005)
  • Y.J. Cao et al.

    Optimization of control parameters in genetic algorithms: A stochastic approach

    International Journal of Systems Science

    (1999)
  • B.J. Chen et al.

    Load forecasting using support vector machines: A study on EUNITE competition 2001

    I EEE Transactions on Power Systems

    (2004)
  • N. Christiani et al.

    An introduction to support vector machines

    (2000)
  • N. Cristianini et al.

    Dynamically adapting kernels in support vector machines

    Advances in Neural Information Processing Systems

    (1999)
  • P.J. Darwen et al.

    Speciation as automatic categorical modularization

    IEEE Transactions on Evolutionary Computation

    (1997)
  • T.R. Dastidar et al.

    A synthesis system for analog circuits based on evolutionary search and topological reuse

    IEEE Transactions on Evolutionary Computation

    (2005)
  • Cited by (255)

    View all citing articles on Scopus
    View full text