Skip to main content
Springer Nature Link
Log in

Multiobjective Optimization of Ensembles of Multilayer Perceptrons for Pattern Classification

  • Conference paper
Parallel Problem Solving from Nature - PPSN IX (PPSN 2006)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 4193))

Included in the following conference series:

  • 2208 Accesses

Abstract

Pattern classification seeks to minimize error of unknown patterns, however, in many real world applications, type I (false positive) and type II (false negative) errors have to be dealt with separately, which is a complex problem since an attempt to minimize one of them usually makes the other grow. Actually, a type of error can be more important than the other, and a trade-off that minimizes the most important error type must be reached. Despite the importance of type-II errors, most pattern classification methods take into account only the global classification error. In this paper we propose to optimize both error types in classification by means of a multiobjective algorithm in which each error type and the network size is an objective of the fitness function. A modified version of the GProp method (optimization and design of multilayer perceptrons) is used, to simultaneously optimize the network size and the type I and II errors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Abbass, H.A.: A memetic Pareto evolutionary approach to artificial neural networks. In: Stumptner, M., Corbett, D., Brooks, M. (eds.) Proceedings of the 14th Australian Joint Conference on Artificial Intelligence (AI 2001), Berlin, pp. 1–12 (2001)

    Google Scholar 

  2. Abbass, H.A.: Pareto neuro-evolution: constructive ensemble of neural networks using multi-objective optimization. In: IEEE Congress on Evolutionary Computation (CEC 2003), Canberra, Australia, vol. 3, pp. 2074–2080. IEEE Press, Los Alamitos (2003)

    Google Scholar 

  3. Abbass, H.A.: Speeding up back-propagation using multiobjective evlutionary algorithms. In: Neural Computation, vol. 15(11), pp. 2705–2726. MIT Press, Cambridge (2003)

    Google Scholar 

  4. Abbass, H.A., Sarker, R.A., Newton, C.S.: PDE: A Pareto-frontier differential evolution approach for multi-objective optimization problems. In: Proc. of the IEEE Congress on Evolutionary Computation (CEC 2001), vol. 2, pp. 971–978. IEEE Press, Piscataway (2001)

    Google Scholar 

  5. Altman, E.I.: The success of business failure prediction models. an international survey. Journal of Banking, Accounting and Finance 8, 171–198 (1984)

    Google Scholar 

  6. Barton, R.J.: Neyman-Pearson Hypothesis TestingSignal. ECE 6333 - Signal Detection and Estimation (2005), http://www2.egr.uh.edu/~rbarton/Classes/ECE6333/files.pdf/Notes/Notes_9-19-05.pdf

  7. Castillo, P.A., Carpio, J., Merelo, J.J., Rivas, V., Romero, G., Prieto, A.: Evolving Multilayer Perceptrons. Neural Processing Letters 12(2), 115–127 (2000)

    Article  Google Scholar 

  8. Castillo, P.A., González, J., Merelo, J.J., Rivas, V., Romero, G., Prieto, A.: G-Prop-III: Global Optimization of Multilayer Perceptrons using an Evolutionary Algorithm. In: Congress on Evolutionary Computation, In Genetic and Evolutionary Computation Conference, Orlando, USA, vol. I, p. 942 (1999) ISBN:1-55860-611-4

    Google Scholar 

  9. Castillo, P.A., Merelo, J.J., Rivas, V., Romero, G., Prieto, A.: G-Prop: Global Optimization of Multilayer Perceptrons using GAs. Neurocomputing 35/1-4, 149–163 (2000)

    Article  Google Scholar 

  10. Castillo, P.A., Merelo, J.J., Romero, G., Prieto, A., Rojas, I.: Statistical Analysis of the Parameters of a Neuro-Genetic Algorithm. IEEE Transactions on Neural Networks 13(6), 1374–1394 (2002)

    Article  Google Scholar 

  11. Coello Coello, C.A., Van-Veldhuizen, D.A., Lamont, G.B.: Evolutionary algorithms for solving multi-objective problems. Kluwer Academic Publishers, New York (2002)

    Book  Google Scholar 

  12. Coello Coello, C.A., Cortés, N.C.: Solving Multiobjective Optimization Problems using an Artificial Immune System. Genetic Programming and Evolvable Machines 6(2), 163–190 (2005)

    Article  Google Scholar 

  13. de Toro, F., Ortega, J., Fernandez, J., Diaz, A.F.: Parallel genetic algorithm for multiobjective optimization. In: 10th Euromicro Workshop on Parallel, Distributed and Network-based processing, pp. 384–391. IEEE Computer Society, Los Alamitos (2002)

    Chapter  Google Scholar 

  14. Freiman, J.A., Chalmers, T.C., Smith, H.: The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. New England Journal of Medicine 299, 690–694 (1978)

    Article  Google Scholar 

  15. Garcia-Pedrajas, N., Hervas-Martinez, C., Ortiz-Boyer, D.: Cooperative coevolution of artificial neural network ensembles for pattern classification. IEEE Trans. Evolutionary Computation 9(3), 271–302 (2005)

    Article  Google Scholar 

  16. Ishibuchi, H., Yamamoto, T.: Evolutionary multiobjective optimization for generating an ensemble of fuzzy rule-based classifiers. In: Proc. of the Genetic and Evolutionary Computation Conference (GECCO 2003). LNCS, pp. 1077–1088. Springer, Heidelberg (2003)

    MATH  Google Scholar 

  17. Jin, Y., Okabe, T., Sendhoff, B.: Neural network regularization and ensembling using multiobjective evolutionary algorithms. In: Congress on Evolutionary Computation, CEC 2004, vol. 1, pp. 1–8 (2005) ISBN:0-7803-8515-2

    Google Scholar 

  18. Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation and active learning. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems, vol. 7, pp. 231–238. MIT Press, Cambridge (1995)

    Google Scholar 

  19. Liu, Y., Yao, X.: Ensemble leagning via negative correlation. Neural Networks 12(10), 1399–1404 (1999)

    Article  Google Scholar 

  20. Liu, Y., Yao, X., Higuchi, T.: Evolutionary ensembles with negative correlation learning. IEEE Transactions on Evolutionary Computation 4(4), 380–387 (2000)

    Article  Google Scholar 

  21. Mangasarian, O.L., Setiono, R., Wolberg, W.H.: Pattern recognition via linear programming: Theory and application to medical diagnosis. In: Coleman, T.F., Li, Y. (eds.) Large-scale numerical optimization, pp. 22–30. SIAM Publications, Philadelphia (1990)

    Google Scholar 

  22. Olvander, J.: Robustness considerations in multi-objective optimal design. Journal of Engineering Design 16(5), 511–523 (2005)

    Article  Google Scholar 

  23. Perrone, M.P., Cooper, L.N.: When networks disagree: Ensemble methods for hybrid neural networks. In: Mammone, R.J. (ed.) Neural Networks for Speech and Image Processing, pp. 126–142 (1993)

    Google Scholar 

  24. Prechelt, L.: PROBEN1 — A set of benchmarks and benchmarking rules for neural network training algorithms. Technical Report 21/94, Fakultät für Informatik, Universität Karlsruhe, D-76128 Karlsruhe, Germany (September 1994)

    Google Scholar 

  25. Rolf, C., Cooper, T.G., Yeung, C.H., Nieschlag, E.: Antioxidant treatment of patients with asthenozoospermia or moderate oligoasthenozoospermia with high-dose vitamin C and vitamin E: a randomized, placebo-controlled, double blind study. Hum. Reprod. 14, 1028–1033 (1999)

    Article  Google Scholar 

  26. Roman, I., de la Torre, J.M., Castillo, P.A., Merelo, J.J.: Sectorial bankruptcy prediction analysis using artificial neural networks: The spanish companies case. In: 25th Annual Congress European Accounting Association, Copenhagen, April 2002, p. 237 (2002)

    Google Scholar 

  27. Roman, I., de la Torre, J.M., Gomez, M.E., Castillo, P.A., Merelo, J.J.: Bankruptcy prediction adapted to firm characteristics. an empirical study. In: 26th Annual Congress European Accounting Association, Congress Book, April 2003, p. A-108, Sevilla (2003)

    Google Scholar 

  28. Savulescu, J., Chalmers, I., Blunt, J.: Are research ethics committees behaving unethically? some suggestions for improving performance and accountability. Br. Med. J. 313, 1390–1393 (1996)

    Article  Google Scholar 

  29. Sharkey, A.J.C.: On combining artificial neural nets. Connection Science 8, 299–313 (1996)

    Article  Google Scholar 

  30. Smith, S.M.: Statistical scrotal effect. Nature 368, 501–502 (1994)

    Article  Google Scholar 

  31. Sterne, J.A.C.: Teaching hypothesis tests - time for significant change? Statistics in Medicine 21, 985–994 (2002)

    Article  Google Scholar 

  32. Storn, R., Price, K.: Differential evolution: a simple and efficient adaptive scheme for global optimization over continuous spaces. Technical Report TR-95-012, International Computer Science Institute, Berkeley (1995)

    Google Scholar 

  33. Van-Veldhuizen, D.A., Lamont, G.B.: Multiobjective evolutionary algorithms: analyzing the state-of-the-art. Evolutionary Computation 8(2), 125–147 (2000)

    Article  Google Scholar 

  34. Yao, X.: Evolving artificial neural networks. Proceedings of the IEEE 87(9), 1423–1447 (1999)

    Article  Google Scholar 

  35. Zhou, Z.H., Wu, J., Tang, W.: Ensembling neural networks: many could be better than all. Artificial Intelligence 137(1-2), 239–253 (2002)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Architecture and Computer Technology, University of Granada, Spain

    P. A. Castillo, J. J. Merelo & G. Romero

  2. Department of Computer Science, University of Jaén, Spain

    M. G. Arenas & V. M. Rivas

Authors
  1. P. A. Castillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. G. Arenas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. J. Merelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. V. M. Rivas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. G. Romero
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Science Institute, University of Iceland, P.O. Box, Iceland

    Thomas Philip Runarsson

  2. Vorarlberg University of Applied Sciences, Hochschulstr. 1, A-6850, Dornbirn, Austria

    Hans-Georg Beyer

  3. Automated Scheduling, Optimisation and Planning Group, School of Computer Science & IT, University of Nottingham, NG8 1BB, Nottingham, UK

    Edmund Burke

  4. Depto. Arquitectura y Tecnologa de Computadores, ETS Ingeiera Informtica, C/Daniel Saucedo Aranda, s/n, 18071, Granada, Spain

    Juan J. Merelo-Guervós

  5. Colorado State University, 80523, Fort Collins, CO, USA

    L. Darrell Whitley

  6. University of Birmingham, Birmingham, UK

    Xin Yao

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Castillo, P.A., Arenas, M.G., Merelo, J.J., Rivas, V.M., Romero, G. (2006). Multiobjective Optimization of Ensembles of Multilayer Perceptrons for Pattern Classification. In: Runarsson, T.P., Beyer, HG., Burke, E., Merelo-Guervós, J.J., Whitley, L.D., Yao, X. (eds) Parallel Problem Solving from Nature - PPSN IX. PPSN 2006. Lecture Notes in Computer Science, vol 4193. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11844297_46

Download citation

  • DOI: https://doi.org/10.1007/11844297_46

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-38990-3

  • Online ISBN: 978-3-540-38991-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation