Global optimization for artificial neural networks: A tabu search application

doi:10.1016/S0377-2217(97)00292-0

European Journal of Operational Research

Volume 106, Issues 2–3, 16 April 1998, Pages 570-584

https://doi.org/10.1016/S0377-2217(97)00292-0 Get rights and content

Abstract

The ability of neural networks to closely approximate unknown functions to any degree of desired accuracy has generated considerable demand for neural network research in business. The attractiveness of neural network research stems from researchers' need to approximate models within the business environment without having a priori knowledge about the true underlying function. Gradient techniques, such as backpropagation, are currently the most widely used methods for neural network optimization. Since these techniques search for local solutions, they are subject to local convergence and thus can perform poorly even on simple problems when forecasting out-of-sample. Consequently, a global search algorithm is warranted. In this paper we examine tabu search (TS) as a possible alternative to the problematic backpropagation approach. A Monte Carlo study was conducted to test the appropriateness of TS as a global search technique for optimizing neural networks. Holding the neural network architecture constant, 530 independent runs were conducted for each of seven test functions, including a production function that exhibits both increasing and diminishing marginal returns and the Mackey-Glass chaotic time series. In the resulting comparison, TS derived solutions that were significantly superior to those of backpropagation solutions for in-sample, interpolation, and extrapolation test data for all seven test functions. It was also shown that fewer function evaluations were needed to find these optimal values.

References (49)

J.A. Bland et al.
Tabu search and design optimisation
Computer Aided Design
(1991)
J.A. Bland
A derivative-free exploratory tool for function minimisation based on tabu search
Advances in Engineering Software
(1994)
K. Funahashi
On the approximate realization of continuous mappings by neural networks
Neural Networks
(1989)
F. Glover
Future paths for integer programming and links to artificial intelligence
Computer Operations Research
(1986)
W.L. Goffe et al.
Global optimization of statistical functions with simulated annealing
Journal of Econometrics
(1994)
K. Hornik et al.
Multilayer feedforward networks are universal approximators
Neural Networks
(1989)
M. Laguna et al.
Tabu search for the multilevel generalized assignment problem
European Journal of Operational Research
(1995)
P.A. Shoemaker et al.
Back propagation learning with trinary quantization of weight updates
Neural Networks
(1991)
J. Sietsma et al.
Creating artificial neural networks that generalize
Neural Networks
(1991)
A. van Ooyen et al.
Improving the convergence of the back propagation algorithm
Neural Networks
(1992)

A.S. Weigend et al.

Back propagation, weight-elimination and time series prediction

N. Archer et al.

Application of the back propagation neural network algorithm with monotonicity constraints for two-group classification problems

Decision Sciences

(1993)

J.A. Bland

Nonlinear optimization of constrained functions using tabu search

International Journal of Mathematical Education in Science and Technology

(1993)

J.R. Chen et al.

Stepsize variation methods for accelerating the back propagation algorithm

T. Crainic et al.

A tabu search procedure for multicommodity location/allocation with balancing requirements

(1991)

D. de Werra et al.

Tabu search techniques: A tutorial and an application to neural networks

Operations Research Spektrum

(1989)

M.A. Franzini

Speech recognition with back propagation

R.A. Gallant et al.

On learning the derivatives of an unknown mapping with multilayer feedforward networks

F. Glover

Tabu search

F. Glover

Tabu search: A tutorial

Interfaces

(1990)

F. Glover et al.

Bandwidth packing: A tabu search approach

Management Science

(1993)

J. Hadar

Mathematical Theory of Economic Behavior

P. Hansen et al.

Algorithms for the maximum satisfiability problem

A. Hertz et al.

The tabu search metaheuristic: How we used it

Annals of Mathematics and Artificial Intelligence

(1990)

Cited by (130)

Machine learning algorithms to predict flow boiling pressure drop in mini/micro-channels based on universal consolidated data
2021, International Journal of Heat and Mass Transfer
Two-phase flow in mini/micro-channels can meet the high heat dissipation requirements of many state-of-the-art cooling solutions. However, there is lack of accurate universal methods for predicting parameters like pressure drop in these configurations. Conventional ways of predicting pressure drop employ either Homogeneous Equilibrium Model (HEM) or semi-empirical correlations. This current study leverages the availability of data collected over the past few decades to build several machine learning models to demonstrate the efficacy and ease of building and deploying such models. A consolidated database of 2787 data points for flow boiling pressure drop in mini/micro-channels is amassed from 21 sources that includes 10 working fluid, reduced pressures of 0.0006 –0.7766, hydraulic diameters of 0.15–5.35 mm, mass velocities of 33.1 < G < 2738 kg/m 2 s, liquid-only Reynolds numbers of 14–27,658, superficial vapor Reynolds number of 75.58–199,453 and flow qualities of 0 and 1. This consolidated database is utilized to develop four machine learning based regression models viz., Artificial Neural Networks (ANN), KNN regression, Extreme Gradient Boosting (XGBoost) and Light GBM. Both input parameters and hyperparameters are optimized for the individual models. The models with dimensionless input parameters: Bd, Bo, Fr_f, Fr_fo, Fr_g, Fr_go, Fr_tp, Pr_f, Pr_g, Pe_g, Pe_f, Re_f, Re_fo, Re_g, Re_go, Re_eq, Su_f, Su_g, We_f, We_fo, We_g, We_go,We_tp predict the test data for ANN model, XGBoost model, KNN model, and LightGBM model with MAEs of 9.58%, 10.38%, 13.52%, and 14.49%, respectively. The optimized machine-learning models performed better than highly reliable generalized pressure drop correlations plus showed good performance across individual datasets, flow regimes, and channel configurations.
Training Feed-Forward Artificial Neural Networks with a modified artificial bee colony algorithm
2020, Neurocomputing
Deep learning is a branch of neural network which has been intensively developed in the last decade. Due to the high-accuracy classification ability, the deep learning algorithms have been widely used in many fields, such as speech recognition, image recognition, and natural speech processing. However, they also show some shortcomings especially on the selection of some parameters in the network, including hyper-parameters, which is still treated as a time consuming task. In this paper, a modified ABC (ABC-ISB) optimization algorithm is proposed to automatically train the parameters of Feed-Forward Artificial Neural Networks, which is a typical a neural network. In the proposed ABC algorithm, we utilize the information of neighbors with better performance to accelerate the convergence of employed and onlooker bees respectively. In addition, a new selection strategy and a gbest-guided strategy are introduced to enhance the global search capability and balance the exploration and exploitation of the algorithm separately. The experimental results show our ABC-ISB is generally leading and competitive.
A hybrid computational intelligence approach to predict spectral acceleration
2019, Measurement: Journal of the International Measurement Confederation
Citation Excerpt :
The main drawback of employing ANNs in prediction problems is the possibility of being trapped in local minima because of inefficient training [23]. Optimization algorithms are proposed as effective tools for training ANNs to overcome this issue [24–30]. Genetic algorithm (GA) as one of the most applicable evolutionary-based algorithms has been tackled for training ANNs several times e.g., da Silva Ferreira [31] for evaluating power coupling efficiency of photonic couplers, Yuce et al. [32] for energy management in the domestic sector, Azadeh et al. [33] for optimizing machinery productivity.
In this study, a new explicit method has been suggested to predict the spectral acceleration characteristic of strong ground-motions based on hybridizing genetic algorithm (GA), multilayer perceptron neural network (MLPNN), and regression analysis (RA), called GA-NN-RA. The predictor variables encompass a period of vibration, magnitude, closest distance co-seismic rupture, shear wave velocity averaged over the top 30 m and flag for reverse faulting earthquakes. To develop the model, a data set of strong ground-motion records gathered by Pacific Earthquake Engineering Research Center has been employed. For confirmation and efficiency of proposed model, an additional set of test that is not involved in the modeling has been applied. The obtained results using GA-NN-RA show good accuracy in comparison with other ground motion models. Also, the proposed model is capable of evaluating the spectral acceleration for any records without restriction in a period of vibration.
New machine learning-based prediction models for fracture energy of asphalt mixtures
2019, Measurement: Journal of the International Measurement Confederation
This paper presents innovative machine learning methods called gene expression programming (GEP) and hybrid artificial neural network/simulated annealing (ANN/SA) to predict the fracture energy of asphalt mixture specimens. The GEP and ANN/SA models are developed using an experimental database including a number of disk-shaped compact tension (DC(T)) test results for fracture energy. The fracture energy is formulated in terms of various predictor variables such as asphalt binder performance grading (PG), asphalt content, aggregate size, aggregate gradation, reclaimed asphalt pavement (RAP) content, reclaimed asphalt shingles (RAS) content, crumb rubber content, and test temperature. A calculation procedure is presented to interpret the models and transform them into practical design equations. A sensitivity analysis is conducted to evaluate the effect of these predictor variables on the fracture energy. Based on the results, the proposed design equations accurately characterize the fracture energy of asphalt mixtures. The GEP model appears to be more practical than the ANN/SA model because of its better generalization and simpler functional structure. The models are recommended for pre-design purposes or as a means to determine asphalt mixture fracture energy when testing is not feasible.
Application of metaheuristic algorithms to the identification of nonlinear magneto-viscoelastic constitutive parameters
2018, Journal of Magnetism and Magnetic Materials
Metaheuristic algorithms offer a robust and convenient method to solve highly nonlinear optimisation problems in engineering applications. In this work we evaluate the effectiveness of a collection of canonical algorithms at performing parameter identification for nonlinear constitutive laws that describe coupled, magnetic-field responsive materials. To achieve this, we define an objective function that captures the influence of many physical measurements recorded during experimental analysis of a coupled material, and incorporates the influence of experimental uncertainty. A benchmark of the algorithms is conducted through the evaluation of a magneto-visco-elastic material by means of numerically-derived parallel-plate rotational rheometry. The effectiveness of each algorithm at matching the fictitious, but representative, experimental data was considered using two different metrics. In addition to the ranking based on a non-parametric statistical test, we consider an ad hoc criterion that accounts for only the top performing candidate solutions. It is determined that the continuous real and discrete bitstring genetic algorithm provide the best overall performance in terms of the accuracy of the predicted parameters, while globally-elitist simulated annealing provides the best compromise between accuracy and computational efficiency. When experimental uncertainties exist (which is always the case for data determined within a laboratory setting), it has been observed that the strong link between constitutive parameters and physical material properties, which is typically assumed, no longer holds.
Next generation prediction model for daily solar radiation on horizontal surface using a hybrid neural network and simulated annealing method
2017, Energy Conversion and Management
Accurate prediction of solar radiation is essential for optimal design of solar systems. This paper presents an innovative artificial intelligence approach for the determination of the daily solar radiation. A new nonlinear model was developed to predict the daily solar radiation on horizontal surface using a hybrid method coupling artificial neural network (ANN) and simulated annealing (SA), called ANN/SA. This method uses SA-based temperature cycling to improve the ANN calibration performance. A calculation procedure was presented to interpret the ANN/SA model and transform it into a practical design equation. The ANN/SA technique formulates the daily solar radiation in terms of several meteorological parameters. Thousands of daily observations during 1995–2014 in a nominal city in Iran were used to develop the solar radiation models. Validity of the model was verified through different phases. Sensitivity analysis was conducted and discussed. The ANN/SA model accurately predicts the daily solar radiation and outperforms the ANN, support vector machines (SVM), and existing regression and machine learning models.

View all citing articles on Scopus

¹: Supported in part by the Mississippi Alabama Sea Grant Consortium through NOAA.

View full text

Global optimization for artificial neural networks: A tabu search application

Abstract

Computer Aided Design

Advances in Engineering Software

Neural Networks

Computer Operations Research

Journal of Econometrics

Neural Networks

European Journal of Operational Research

Neural Networks

Neural Networks

Neural Networks

Application of the back propagation neural network algorithm with monotonicity constraints for two-group classification problems

Decision Sciences

Nonlinear optimization of constrained functions using tabu search

International Journal of Mathematical Education in Science and Technology

Stepsize variation methods for accelerating the back propagation algorithm

A tabu search procedure for multicommodity location/allocation with balancing requirements

Tabu search techniques: A tutorial and an application to neural networks

Operations Research Spektrum

Speech recognition with back propagation

On learning the derivatives of an unknown mapping with multilayer feedforward networks

Tabu search

Tabu search: A tutorial

Interfaces

Bandwidth packing: A tabu search approach

Management Science

Mathematical Theory of Economic Behavior

Algorithms for the maximum satisfiability problem

The tabu search metaheuristic: How we used it

Annals of Mathematics and Artificial Intelligence