A fast differential evolution algorithm using k-Nearest Neighbour predictor

doi:10.1016/j.eswa.2010.09.092

Expert Systems with Applications

Volume 38, Issue 4, April 2011, Pages 4254-4258

https://doi.org/10.1016/j.eswa.2010.09.092 Get rights and content

Abstract

Genetic algorithms (GAs), particle swarm optimisation (PSO) and differential evolution (DE) have proven to be successful in engineering optimisation problems. The limitation of using these tools is their expensive computational requirement. The optimisation process usually needs to run the numerical model and evaluate the objective function thousands of times before converging to an acceptable solution. However, in real world applications, there is simply not enough time and resources to perform such a huge number of model runs. In this study, a computational framework, known as DE-kNN, is presented for solving computationally expensive optimisation problems. The concept of DE-kNN will be demonstrated via one novel approximate model using k-Nearest Neighbour (kNN) predictor. We describe the performance of DE and DE-kNN when applied to the optimisation of a test function. The simulation results suggest that the proposed optimisation framework is able to achieve good solutions as well as provide considerable savings of the function calls compared to DE algorithm.

Research highlights

► A computational framework, known as DE-kNN (Differential Evolution – k-Nearest Neighbour), is presented for solving computationally expensive optimisation problems. ► The proposed method investigates the use of kNN predictor in conjunction with DE in order to accelerate the search process. ► The dynamic learning approach can guarantee the DE to converge towards to global optima with less actual evaluations without compromising the good search capabilities of DE. ► The simulation results suggest that the proposed optimisation framework is able to achieve good solutions as well as provide considerable savings of the function calls compared to DE algorithm.

Introduction

Population-based search strategies such as genetic algorithms (Holland, 1975), particle swarm optimisation (Kennedy & Eberhart, 1995) and differential evolution (Storn & Price, 1995) have found a wide application in various fields of engineering (Liu, 2009). One of the biggest drawbacks of using these population-based optimisation methods for engineering applications is that they require a large number of model evaluations. For example, Madsen (2000) reported that 10,000 model runs was needed in order to calibrate the MIKE 11/NAM rainfall–runoff model with nine parameters. For many real-world engineering applications, the number of calls of running a simulation model is very limited. This is especially true when each evaluation of the quality of solution is very time consuming, generally constrained by the time needed to run simulation models. For example, each simulation of a coarse hydrodynamic 2D model, may take up to 1 min to run on a high performance computer. As the spatial and temporal resolution increases, the simulation time increases. One can expect the simulation time to increase to several minutes or even hours for full hydrodynamic 3D models. Reducing the number of actual evaluations necessary to reach an acceptable solution is thus of major importance. Although the use of parallel computing is a remedy in reducing the computing time required for complex problems, an alternative approach using meta-models has received much attention recently (Jin, 2005, Khu et al., 2004, Liu et al., 2004, Liu and Khu, 2007, Yan and Minsker, 2003). Meta-models can be viewed as approximations of the original models and may be used in place of the original models to reduce the computational time. Since then, there have been numerous developments in the field of meta-models, the most prominent are surrogates of time consuming simulation models. They have been applied to design evaluation and optimisation in many engineering applications.

Jin (2005) presented a complete survey of the research on the use of fitness approximation in evolutionary computation. Jin (2005) has classified meta-models which are used currently in four categories, namely polynomial models, kriging models, neural networks and support vector machine. Main issues like approximation levels, approximate model management schemes and model construction techniques were reviewed. He stated it was difficult to construct a meta-model that was globally correct due to the high dimensionality, ill distribution and limited number of training samples. It has also been suggested that an appropriate procedure must be set up to combine meta-models with original numerical model, a technique known as evolution control or model management. Jin (2005) referred to an individual that was evaluated using the original fitness function as a controlled individual, and a generation in which all its individuals were evaluated using the original fitness function as a controlled generation. Jin (2005) defined two evolution control approaches: in individuals-based evolution control a fraction of each population is evaluated by the true fitness and the remainder of the population is evaluated by the meta-model, while in generation-based evolution control the entire population was either evaluated by the real function or evaluated by the meta-model. In individual-based evolution control, either a random strategy or a best strategy can be used to select the individuals to be controlled (Jin, 2005). In the best strategy, the best individual (based on the ranking evaluated by the meta-model) in the current generation is reevaluated using the original function, while in the random strategy, the individuals to be controlled are selected randomly. It has been shown that the best strategy can reduce the computational cost further and individual-based evolution control can be carried out only in a selected number of generations (Jin, 2005). For both control methods, there is disagreement about how many individuals of a population need to be controlled. The idea of taking less model evaluations in order to reach the optimal solution using fitness estimation with help of radial basis function (RBF) neural network is investigated by Khu et al. (2004) claimed to reduce the exact evaluations about 66.5% for a rainfall–runoff model calibration problem. In the work by Jin, Olhofer, and Sendhoff (2000), the convergence property of an evolution strategy (ES) with multilayer perceptron (MLP) neural network based fitness evaluation was investigated. Yan and Minsker (2003) have proposed a dynamic meta-modelling approach, in which artificial neural networks (ANN) and support vector machines (SVM) were embedded into a genetic algorithm optimisation framework to replace time consuming flow and contaminant transport models.

Therefore, time-efficient approximate model can be particularly beneficial for the expensive function evaluation. The proposed method in this paper investigates the use of kNN predictor in conjunction with DE in order to accelerate the search process. We have named this new technique DE-kNN. The proposed framework uses computationally cheap predictor constructed through dynamic learning to reduce the exact computationally expensive evaluation calls during DE search. The algorithm reported here can serve as a benchmark algorithm aimed at those types of expensive optimisation problems. This approach can substantially reduce the number of function evaluations on computationally expensive problems without compromising the good search capabilities of DE.

The DE and DE-kNN methods in this paper are used to investigate the optimisation of a test function. In Section 2, the differential evolution algorithm is briefly described. In Section 3, the proposed optimisation algorithm using k-Nearest Neighbour predictor for solving the expensive optimisation problem is presented. In Section 4, the test example is presented that illustrates the principles and implications of the proposed optimisation algorithm. Finally, conclusions are given in Section 5.

Section snippets

Differential evolution algorithm

Differential evolution (DE) is a population-based direct-search algorithm for global optimisation (Storn & Price, 1995). The standard DE works as follows: for each vector x_i_,_G, i = 1, 2, … , NP, a trail vector v_i,G₊₁ is generated according to $V_{i, G + 1} = x_{r_{1}, G} + F \cdot (x_{r_{2}, G} - x_{r_{3}, G}),$ with r₁, r₂, r₃ ϵ [0, NP], integer and mutually different, F > 0, and r₁≠ r₂≠ r₃≠ i. F is a real and constant factor which controls the amplification of the differential variation $(x_{r_{2}, G} - x_{r_{3}, G})$ .

In order to increase the diversity of the

k-Nearest Neighbour algorithm

There are many nonlinear predictors such as artificial neural networks (ANNs), support vector machine (SVM) and kNN (Bishop, 1995, Vapnik, 1998). An ANN has many neurons and each neuron accepts input and gives output according its activation function. ANN can learn the mapping relationship between the inputs and the outputs sampled from a training set using a supervised learning algorithm. Then the trained ANN is used to make a prediction for test data. SVM uses a Lagrangian formulation to

Bukin test function

Bukin function is almost fractal (with fine seesaw edges) in the surroundings of their minimal points (see Fig. 2). Due to this property, they are extremely difficult to optimize by any method of global (or local) optimisation (Sudhanshu, 2006). In the search domain x₁ ϵ [−15, −5], x₂ ϵ [−3, 3], the function is defined as follows: $f (x) = 100 \times \sqrt{| x_{2} - 0.01 \times x_{1}^{2} |} + 0.01 \times | x_{1} + 10 |,$ $f_{min} (- 10, 1) = 0 .$

Experimental setup and results

The relevant experiment parameters using the DE and DE-kNN for the test function are listed in Table 1, Table 2. For each

Conclusions

In real-world optimisation problems, the function evaluations usually require a large amount of computation time. In this paper, we presented a hybrid optimisation framework (DE-kNN) that has been developed by combining an approximate technique called kNN predictor which can produce fairly accurate global approximations to actual parameters space to provide the function evaluations efficiently. The two optimisation algorithms have been used for optimisation of the test function. It is clear

Acknowledgement

The author would like to thank ABP Marine Research Ltd., UK for funding the work.

References (15)

Y. Liu
Automatic calibration of a rainfall–runoff model using a fast and Elitist multi-objective particle swarm algorithm
Expert Systems with Applications
(2009)
H. Madsen
Automatic calibration of a conceptual rainfall–runoff model using multiple objectives
Journal of Hydrology
(2000)
C.M. Bishop
Neural networks for pattern recognition
(1995)
H.J. Holland
Adaptation in natural and artificial systems: An introductory analysis with application to biology, control and artificial intelligence
(1975)
Y. Jin
Comprehensive survey of fitness approximation in evolutionary computation
Soft Computing
(2005)
Jin, Y., Olhofer, M., & Sendhoff, B. (2000). On evolutionary optimisation with approximate fitness functions. In...
Kennedy, J., & Eberhart, R. (1995). Particle Swarm Optimisation. In Proceedings of the IEEE international conference on...

There are more references available in the full text version of this article.

Cited by (34)

Optimization of mooring systems in the context of an integrated design methodology
2021, Marine Structures
This work describes an enhanced mooring optimization procedure, oriented towards recent floating production systems (FPS) for oil & gas exploitation in ultra-deep-water scenarios, which may present a large number of risers in an asymmetric layout. Acknowledging that the risers are the key component of an FPS, the optimization procedure is associated to an integrated mooring-riser design methodology; thus, instead of simply minimizing the platform offsets and/or the costs of the mooring system itself, one of the main objectives is to obtain a mooring configuration that ensures the integrity of the risers. Other highlights of the optimization procedure include the following aspects: Enhancements in the modeling of the optimization problem (including the definition of design variables, objective function and constraints that are relevant for such actual applications); The use of the PSO optimization algorithm associated to the ε-constrained method to efficiently handle the constraints; Enhancements in the evaluation of candidate solutions, by full nonlinear time-domain dynamic Finite Element simulations with coupled models; and the implementation in a parallel computing environment to deal with the high associated computational costs. A case study considering an FPS representative of actual applications in deepwater scenarios is presented to illustrate the practical application of the optimization tool.
ACCUGRAM: A novel approach based on classification to frequency band selection for rotating machinery fault diagnosis
2019, ISA Transactions
Frequency band selection (FBS) in rotating machinery fault diagnosis aims to recognize frequency band location including a fault transient out of a full band spectrum, and thus fault diagnosis can suppress noise influence from other frequency components. Impulsiveness and cyclostationarity have been recently recognized as two distinctive signatures of a transient. Thus, many studies have focused on developing quantification metrics of the two signatures and using them as indicators to guide FBS. However, most previous studies almost ignore another aspect of FBS, i.e. health reference, which significantly affect FBS performance. To address this issue, this paper investigates importance of a health reference and recognize it as the third critical aspect in FBS. With help of the health reference, the frequency band where the fault transient exists could be located. A novel approach based on classification is proposed to integrate all three aspects (impulsiveness, cyclostationarity, and health reference) for FBS. Classification accuracy is developed as a novel indicator to select the most sensitive frequency band for rotating machinery fault diagnosis. The proposed method (coined by accugram) has been validated on benchmark and experiment datasets. Comparison results show its effectiveness and robustness over conventional envelope analysis, the kurtogram, and the infogram.
Novel volatility forecasting using deep learning–Long Short Term Memory Recurrent Neural Networks
2019, Expert Systems with Applications
The volatility is related to financial risk and its prediction accuracy is very important in portfolio optimisation. A large body of literature to-date suggests Support Vector Machines (SVM) as the “best of regression” algorithms for financial data regression. Recent work however found that new deep learning––Long Short Term Memory Recurrent Neural Networks (LSTM RNNs) outperformed SVM for classification problems. In the present paper we conduct a new unbiased evaluation of these two modelling techniques for regression problems, and we also compare them with a popular regression model - Generalized Autoregressive Conditional Heteroskedasticity (GARCH) model for financial volatility or risk forecasting. Our experiments using financial data show that the LSTM RNNs performed as good as v-SVR for large interval volatility forecasting and both performed much better than GARCH model for two financial indices (S&P 500 and AAPL). The LSTM RNNS deep learning method can learn from big raw data and can be run with many hidden layers and neurons under GPU to achieve a good prediction for long sequence data compared to the support vector regression. The deep learning technique - LSTM RNNs with big data can be used to improve the volatility prediction instead of v-SVR when the v-SVR does not predict well for some financial stocks of a portfolio. This will help investors to win the competition to maximize their profit.
Hierarchical differential evolution algorithm combined with multi-cross operation
2019, Expert Systems with Applications
In expert systems, complex optimization problems are always characterized by nonlinearity, nonconvexity, multi-modality, discontinuity, and high dimensionality. Although classical optimization algorithms are mature, they readily fall into a local optimum. The differential evolution (DE) algorithm has been successfully applied to solve numerous problems with expert systems. However, balancing the global and local search capabilities of the DE algorithm remains an open issue and has attracted significant research attention. Thus, a hierarchical heterogeneous DE algorithm that incorporates multi-cross operation (MCO) is proposed in this article. In the proposed algorithm, success-history-based adaptive DE (SHADE) is implemented in the bottom layer, while MCO is implemented in the top layer. The MCO search is based on the SHADE results, but its search results do not affect the bottom layer. First-order stability analyses conducted for the presented MCO showed that the individual positions are expected to converge at a fixed point in the search space. The accuracy and convergence speed of the proposed algorithm were also experimentally compared with those of eight other advanced particle swarm optimization techniques and DE variants using benchmark functions from CEC2017. The proposed algorithm yielded better solution accuracy for 30- and 50-dimensional problems than the other variants, and although it did not provide the fastest convergence for all of the functions, it ranked among the top three for the unimodal and simple multimodal functions and achieved fast convergence for the other functions.
Learning enhanced differential evolution for tracking optimal decisions in dynamic power systems
2018, Applied Soft Computing Journal
Citation Excerpt :
Their objective is to enhance the efficiency of NNC, while the objective of our paper is to enhance the efficiency of DE. Both Reference [25] and [26] uses KNN as a fitness predictor to save expensive fitness function calls and thus enhance DE for STATIC optimization problems. In our paper, LEDE is proposed for DYNAMIC OPF problems, and KNN is used to retrieve previous solutions which could be perform well in the current state of the DYNAMIC power systems, which is a new method.
Optimal power flow (OPF) refers to the problem of optimizing the operating decisions such as electric power generation in power systems, which are always subjected to dynamic factors like bus loads. Conventionally, OPF in dynamic environments has been solved by static-oriented optimization methods based on the prediction of the dynamic factors. However, as the dynamics of modern power systems become more and more complex and difficult to predict, research interest of intelligent methods that track the optimal decisions of OPF has been grown recently. Devoted to this objective, a learning enhanced differential evolution (LEDE) is proposed in this paper. LEDE incorporates the idea of nearest-neighbor rule from the field of machine learning, with which decisions of the previous environments are retrieved continually to replace the newly generated individuals of differential evolution. A so-called elitism stochastic ranking strategy is also proposed, used in LEDE to handle constraints of OPF. Experiments are conducted on the dynamic IEEE 30-bus system and IEEE 118-bus system, and the results show the efficiency of LEDE in comparison with other algorithms.
Enhanced differential evolution using local Lipschitz underestimate strategy for computationally expensive optimization problems
2016, Applied Soft Computing Journal
Differential evolution (DE) has been successfully applied in many scientific and engineering fields. However, one of the main problems in using DE is that the optimization process usually needs a large number of function evaluations to find an acceptable solution, which leads to an increase of computational time, particularly in the case of computationally expensive problems. The Lipschitz underestimate method (LUM), a deterministic global optimization technique, can be used to obtain the underestimate of the objective function by building a sequence of piecewise linear supporting hyperplanes. In this paper, an enhanced differential evolution using local Lipschitz underestimate strategy, called LLUDE, is proposed for computationally expensive optimization problems. LLUDE effectively combines the exploration of DE with the underestimation of LUM to obtain promising solutions with less function evaluations. To verify the performance of LLUDE, 18 well-known benchmark functions and one computationally expensive real-world application problem, namely, the protein structure prediction problem, are employed. Results obtained from these benchmark functions show that LLUDE is significantly better than or at least comparable to the state-of-the-art DE variants and non-DE algorithms. Furthermore, the results obtained from the protein structure prediction problem suggest that LLUDE is effective and efficient for computationally expensive problems.

View all citing articles on Scopus

View full text