Abstract
The formation of a neural network ensemble has attracted much attention in the machine learning literature. A set of trained neural networks are combined using a post-gate to form a single super-network. One main challenge in the literature is to decide on which network to include in, or exclude from the ensemble. Another challenge is how to define an optimum size for the ensemble. Some researchers also claim that for an ensemble to be effective, the networks need to be different. However, there is not a consistent definition of what “different” means. Some take it to mean weakly correlated networks, networks with different bias-variance trade-off, and/or networks which are specialized on different parts of the input space.
In this paper, we present a theoretically sound approach for the formation of neural network ensembles. The approach is based on the dominance concept that determines which network to include/exclude, identifies a suitable size for the ensemble, and provides a mechanism for quantifying differences between networks. The approach was tested on a number of standard dataset and showed competitive results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Abbass, H.A.: A memetic pareto evolutionary approach to artificial neural networks. In: Stumptner, M., Corbett, D., Brooks, M. (eds.) Proceedings of the 14th Australian Joint Conference on Artificial Intelligence (AI 2001), pp. 1–12. Springer, Berlin (2001)
Abbass, H.A.: An evolutionary artificial neural network approach for breast cancer diagnosis. Artificial Intelligence in Medicine 25(3), 265–281 (2002)
Abbass, H.A.: The self-adaptive pareto differential evolution algorithm. In: Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2002), Piscataway, NJ, vol. 1, pp. 831–836. IEEE Press, Los Alamitos (2002)
Abbass, H.A.: Speeding up back-propagation using multiobjective evolutionary algorithms. Neural Computation (2003) (to appear)
Abbass, H.A., Sarker, R.A., Newton, C.S.: PDE: A pareto-frontier differential evolution approach for multi-objective optimization problems. In: Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2001), Piscataway, NJ, vol. 2, pp. 971–978. IEEE Press, Los Alamitos (2001)
Blake, C.L., Merz, C.J.: UCI repository of machine learning databases, University of California, Irvine, Dept. of Information and Computer Sciences (1998) http://www.ics.uci.edu/~mlearn/mlrepository.html
Coello, C.A., Van Veldhuizen, D.A., Lamont, G.B.: Evolutionary Algorithms for Solving Multi-Objective Problems. Kluwer Academic, New York (2002)
Deb, K.: Multi-objective optimization using evolutionary algorithms. John Wiley & Sons, New York (2001)
Haykin, S.: Neural networks - a comprehensive foundation, 2nd edn. Printice Hall, USA (1999)
Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation, and active learning. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems, vol. 7, pp. 231–238. MIT Press, Cambridge (1995)
Liu, Y., Yao, X.: Ensemble learning via negative correlation. Neural Networks 12(10), 1399–1404 (1999)
Liu, Y., Yao, X.: Simultaneous training of negatively correlated neural networks in an ensemble. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 29(6), 716–725 (1999)
Liu, Y., Yao, X., Higuchi, T.: Evolutionary ensembles with negative correlation learning. IEEE Transactions on Evolutionary Computation 4(4), 380–387 (2000)
McKay, R., Abbass, H.A.: Anti-correlation: A diversity promoting mechanisms in ensemble learning. The Australian Journal of Intelligent Information Processing Systems (AJIIPS) 7(3/4), 39–149 (2001)
Michie, D., Spiegelhalter, D.J., Taylor, C.C.: Machine learning, neural and statistical classification. Ellis Horwood (1994)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: McClelland, J.L., Rumelhart, D.E., the PDP Research Group (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition., Foundations, 1, 318, MIT Press, Cambridge (1986)
Sharkey, J.C.: On combining artificial neural nets. Connection Science 8, 299–313 (1996)
Storn, R., Price, K.: Differential evolution: a simple and efficient adaptive scheme for global optimization over continuous spaces. Technical Report TR-95-012, International Computer Science Institute, Berkeley (1995)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Abbass, H.A. (2003). Pareto Neuro-Ensembles. In: Gedeon, T.(.D., Fung, L.C.C. (eds) AI 2003: Advances in Artificial Intelligence. AI 2003. Lecture Notes in Computer Science(), vol 2903. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24581-0_47
Download citation
DOI: https://doi.org/10.1007/978-3-540-24581-0_47
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-20646-0
Online ISBN: 978-3-540-24581-0
eBook Packages: Springer Book Archive