Abstract
A general problem in model selection is to obtain the right parameters that make a model fit observed data. If the model selected is a Multilayer Perceptron (MLP) trained with Backpropagation (BP), it is necessary to find appropriate initial weights and learning parameters. This paper proposes a method that combines Simulated Annealing (SimAnn) and BP to train MLPs with a single hidden layer, termed SA-Prop. SimAnn selects the initial weights and the learning rate of the network. SA-Prop combiens the advantages of the stochastic search performed by the Simulated Annealing over the MLP parameter space and the local search of the BP algorithm.
The application of the proposed methodto several real-world benchmark problems shows that MLPs evolved using SA-Prop achieve a higher level of generalization than other perceptron training algorithms, such as QuickPropagation (QP) or RPROP, and other evolutive algorithms, such as G-LVQ.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
E.H.L. Aarts and J. Korst. Simulated Annealing and Boltzmann Machines. John Wiley, Chichester, U.K., 1989.
C. San Martin; C. Gruss; J.M. Carazo. Six molecules of SV40 large tantigen assemble in a propeller-shaped particle around a channel. Journal of Molecular Biology, 268, 15–20, 1997.
S. Fahlman. An empirical study of learning speed in back-propagation networks Technical report, Carnegie Mellon University, 1988.
S.E. Fahlman. Faster-Learning Variations on Back-Propagation: An Empirical Study. Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.
Werner Kinnebrock. A ccelerating the standard backpropagation method using a genetic approach. Neurocomputing, 6, 583–588, 1994.
S. Kirkpatrick Optimization by Simulated Annealing—Quantitative Studies. J. Stat. Phys. 34, 975–986, 1984.
J.J. Merelo; A. Prieto; F. Moran; R. Marabini and J.M Carazo. Automatic Classificati of Biological Particles from Electron-microscopy Images Using Conventional and Genetic-algorithm Optimized Learning Vector Quantization. Neural Proccessing Letters 8: 55–65, 1998, 1998.
Zbigniew Michalewicz. Genetic Algorithms + Data Structures = Evolution Programs, Second, Extended Edition. Springer-Verlag, 1994.
D.J. Montana and L. Davis. Training feedforward neural networks using genetic algorithms. Proc. 11th Internat. Joint Conf. on Artificial Intelligence, 762–767, 1989.
V. Vergara; S. Sinne; C. Moraga. Optimal Identification Using Feed-Forward Neural Networks. Lectures Notes in Computer Science, vol. 930, 1052–1059, 1995.
Lutz Prechelt. PROBEN1—A set of benchmarks and benchmarking rules for neural network training algorithms. Technical Report 21/94, Fakultät für Informatik, Universität Karlsruhe, D-76128 Karlsruhe, Germany, September 1994. Anonymous FTP: /pub/papers/techreports/1994/1994-21.ps.Z on ftp.ira.uka.deurl
P.A. Castillo; J. Gonzalez; J.J. Merelo; V. Rivas; G. Romero; A. Prieto. G-Prop: Global Optimization of Multilayer Perceptrons using GAs. submitted to Neurocomputing, 1998.
M. Riedmiller. Description and Implementation Details. Technical report, University of Karlsruhe, 1994.
M. Riedmiller and H. Braun. A Direct Adaptive Method for Faster Backpropagation Learning; The RPROP Algorithm. In Ruspini, H., (Ed.) Proc. of the ICNN93, San Francisco, pp. 586–591, 1993.
O. L. Mangasarian; R. Setiono and W.H. Wolberg. Pattern recognition via linear programming: Theory and application to medical diagnosis. Large-scale numerical optimization, Thomas F. Coleman and Yuying Li, editors, SIAM Publications, Philadelphia 1990, pp 22–30, 1990.
T. Tambouratzis. Simulated Annealing Artificial Neural Networks For the Satisfiability (SAT) Problem. Artificial Neural Nets and Genetic Algorithms, 340–343, 1995.
N. Metropolis; A.W. Rosenbluth; M.N. Rosenbluth; A.H. Teller; E. Teller. Equations of State Calculations by Fast Computing Machines. J. Chem. Phys. 21, 1087–1092, 1958.
S. Kirkpatrick; C.D. Gerlatt; M.P. Vecchi. Optimization by Simulated Annealing. Science 220, 671–680, 1983.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1999 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Castillo, P.A., Merelo, J.J., González, J., Rivas, V., Romero, G. (1999). SA-prop: Optimization of multilayer perceptron parameters using simulated annealing. In: Mira, J., Sánchez-Andrés, J.V. (eds) Foundations and Tools for Neural Modeling. IWANN 1999. Lecture Notes in Computer Science, vol 1606. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0098224
Download citation
DOI: https://doi.org/10.1007/BFb0098224
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-66069-9
Online ISBN: 978-3-540-48771-5
eBook Packages: Springer Book Archive