Abstract
Evolutionary learning systems (also known as Pittsburgh learningclassifier systems) need to balance accuracy and parsimony for evolving high quality general hypotheses. The learning process used in evolutionary learning systems is based on a set of training instances that sample the target concept to be learned. Thus, the learning process may overfit the learned hypothesis to the given set of training instances. In order to address some of these issues, this paper introduces a multiobjective approach to evolutionary learning systems. Thus, we translate the selection of promising hypotheses into a two-objective problem that looks for: (1) accurate (low error), and (2) compact (low complexity) solutions. Using the proposed multiobjective approach a set of compromise hypotheses are spread along the Pareto front. We also introduce a theory of the impact of noise when sampling the target concept to be learned, as well as the appearance of overfitted hypotheses as the result of perturbations on high quality generalization hypotheses in the Pareto front.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Holland, J.H.: Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control and Artificial Intelligence. MIT Press/Bradford Books (1975)
Goldberg, D.E.: Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley Publishing Company, Inc., Reading (1989)
Wilson, S.W.: Classifier Fitness Based on Accuracy. Evolutionary Computation 3, 149–175 (1995)
Butz, M.V.: Anticipatory learning classifier systems. Genetic Algorithms and Evolutionary Computation. Kluwer Academic Publishers, Boston, MA (2002)
Smith, S.F.: Flexible Learning of Problem Solving Heuristics through Adaptive Search. In: Proceedings of the 8th International Joint Conference on Artificial Intelligence, pp. 422–425 (1983)
De Jong, K.A., Spears, W.M.: Learning Concept Classification Rules Using Genetic Algorithms. In: Proceedings of the International Joint Conference on Artificial Intelligence, Sidney, Australia, pp. 651–656 (1991)
Janikow, C.: Inductive Learning of Decision Rules in Attribute-Based Examples: a Knowledge-Intensive Genetic Algorithm Approach. PhD thesis, University of North Carolina at Chapel Hill (1991)
Michalewicz, Z.: Genetic Algorithms + Data Structures = Evolution Programs. Springer, Heidelberg (1992)
Llorá, X.: Genetic Based Machine Learning using Fine-grained Parallelism for Data Mining. PhD thesis, Enginyeria i Arquitectura La Salle. Ramon Llull University, Barcelona, Catalonia, European Union (February 2002)
Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)
Koza, J.R.: Genetic Programing: On the Programing of Computers by Means of Natural Selection (Complex Adaptive Systems). MIT Press, Cambridge (1992)
Altenberg, L.: Emergent phenomena in genetic programming. In: Proceedings of the Third Annual Conference on Evolutionary Programming, pp. 233–241 (1994)
Blickle, T., Thiele, L.: Genetic programming and redundancy. In: Genetic Algorithms within the Framework of Evolutionary Computation: Proceedings of the KI 1994 Workshop, pp. 33–38 (1994)
Blickle, T.: Evolving compact solutions in genetic programming: A case study. In: Ebeling, W., Rechenberg, I., Voigt, H.-M., Schwefel, H.-P. (eds.) PPSN 1996. LNCS, vol. 1141, pp. 564–573. Springer, Heidelberg (1996)
Angeline, P.J.: Subtree crossover causes bloat. Genetic Programming 98, 745–752 (1998)
Langdon, W.B., Poli, R.: Fitness causes bloat: Mutation. In: Genetic Programming: First European Conference, pp. 37–48 (1998)
Soule, T., Foster, J.A.: Effects of code growth and parsimony pressure on populations in genetic programming. Evolutionary Computation 6, 293–309 (1998)
Langdon, W.B.: Quadratic bloat in genetic programming. In: Proceedings of the Genetic and Evolutionary Computation Conference 2000, pp. 451–458 (2000)
Podgorelec, V., Kokol, P.: Fighting program bloat with the fractal complexity measure. In: Genetic Programming: Third European Conference, pp. 326–337 (2000)
Bleuler, S., Brack, M., Thiele, L., Zitzler, E.: Multiobjective genetic programming: Reducing bloat using SPEA2. In: Proceedings of the 2001 Congress on Evolutionary Computation CEC2001, COEX, World Trade Center, 159 Samseong-dong, Gangnam-gu, Seoul, Korea, pp. 536–543. IEEE Press, Los Alamitos (2001)
Banzhaf, W., Langdon, W.B.: Some Considerations on the Reason for Bloat. Genetic Programming and Evolvable Hardware 3, 81–91 (2002)
Soule, T.: Exons and code growth in genetic programming. In: Foster, J.A., Lutton, E., Miller, J., Ryan, C., Tettamanzi, A.G.B. (eds.) EuroGP 2002. LNCS, vol. 2278, pp. 143–152. Springer, Heidelberg (2002)
Garrell, J.M., Golobardes, E., Bernadó, E., Llorà, X.: Automatic Diagnosis with Genetic Algorithms and Case-Based Reasoning. AIENG 13, 367–372 (1999)
Bassett, J.K., De Jong, K.A.: Evolving Behaviors for Cooperating Agents. In: Ohsuga, S., Raś, Z.W. (eds.) ISMIS 2000. LNCS (LNAI), vol. 1932, pp. 157–165. Springer, Heidelberg (2000)
Bacardit, J., Garrell, J.M.: Métodos de generalizatión para sistemas clasificadores de Pittsburgh. In: Primer Congreso Espaol de Algoritmos Evolutivos y Bioinspirados (AEB 2002), pp. 486–493 (2002)
Pareto, V.: Cours d’Economie Politique, vol. I &II. F. Rouge, Lausanne (1896)
Van Veldhuizen, D.A., Lamont, G.B.: Evolutionary computation and convergence to apareto front. In: Koza, J.R. (ed.) Late Breaking Papers at the Genetic Programming 1998 Conference, pp. 221–228. Omni Press, Madison (1998)
Coello-Coello, C.A.: An updated survey of GA-Based Multiobjective Optimization Techniques. Technical report lania-rd-09-08, Laboratorio Nacional de Informática Avanzada (LA-NIA), Xalapa, Veracruz, México (December 1998)
Van Veldhuizen, D.A., Lamont, G.B.: Multiobjective evolutionary algorithms: Analyzing the state-of-the-art. Evolutionary Computation 8, 125–147 (2000)
Tackett, W.A.: Recombination, selection, and the genetic construction of computer programs. Unpublished doctoral dissertation. University of Southern California (1994)
Zitzler, E.: Evolutionary Algorithms for Multiobjective Optimization: Methods and Applications. PhD thesis, Swiss Federal Institute of Technology (ETH), Zurich (1999)
Zitzler, E., Deb, K., Thiele, L.: Comparison of Multiobjective Evolutionary Algorithms: Empirical Results. Evolutionary Computation 8, 173–195 (2000)
Zitzler, E.: SPEA2: Improving the Strength Pareto Evolutionary Algorithm. Technical report 103, Swiss Federal Institute of Technology (ETH) Zurich, Gloriastrasse 35, CH-8092 Zurich (May 2001)
Nordin, P., Banzhaf, W.: Complexity Compression and Evolution. In: Proceedings of the Sixth International Conference on Genetic Algorithms (1995)
Bernadó, E., Mekaouche, A., Garrell, J.M.: A Study of a Genetic Classifier System Based on the Pittsburgh Approach on a Medical Domain. In: Imam, I., Kodratoff, Y., El-Dessouki, A., Ali, M. (eds.) IEA/AIE 1999. LNCS (LNAI), vol. 1611, pp. 175–184. Springer, Heidelberg (1999)
Gómez-Skarmeta, A.F., Jiménez, F., Ibáez, J.: Pareto-optimality in fuzzy modeling. In: 6th European Congress on Intelligent Techniques and Soft Computing (EUFIT 1998), pp. 694–700 (1998)
Jiménez, F., Gómez-Skarmeta, A.F., Roubos, H., Robert, B.: Accurate, transparent, and compact fuzzy models for function approximation and dynamic modelling through multi-objective evolutionary optimization. In: Zitzler, E., Deb, K., Thiele, L., Coello Coello, C.A., Corne, D.W. (eds.) EMO 2001. LNCS, vol. 1993, pp. 653–667. Springer, Heidelberg (2001)
Breiman, L.: Bagging predictors. Machine Learning 24, 123–140 (1996)
Llorà, X., Garrell, J.M.: Automatic Classification and Artificial Life Models. In: Proceedings of Learning 2000 Workshop, IEEE and Univesidad Carlos III (2000)
Traus, I., Bernadó, E.: Sistema Classificador Pittsburgh basat en Estratégies Evolutives. Technical Report TR-ISRG-2002/0001, Enginyeria i Arquitectura La Salle, Universitat Ramon Llull, Barcelona, European Union (2002)
Llorà, X., Garrell, J.M.: Evolving Partially-Defined Instances with Evolutionary Algorithms. In: Proceedings of the 18th International Conference on Machine Learning (ICML 2001), pp. 337–344. Morgan Kaufmann Publishers, San Francisco (2001)
Llora, X., Garrell, J.M.: Knowledge-Independent Data Mining with Fine-Grained Parallel Evolutionary Algorithms. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2001), pp. 461–468. Morgan Kaufmann Publishers, San Francisco (2001)
Oei, C.K., Goldberg, D.E., Chang, S.J.: Tournament selection, niching, and the preservation of diversity. IlliGAL Report No. 91011, University of Illinois at Urbana-Champaign, Urbana, IL (1991)
Bäck, T.: Generalized convergence models for tournament- and (μ, λ )-selection. In: Proceedings of the Sixth International Conference on Genetic Algorithms, pp. 2–8 (1995)
Miller, B.L., Goldberg, D.E.: Genetic algorithms, tournament selection, and the effects of noise. Complex Systems 9, 193–212 (1995)
Schwefel, H.P.: Kybernetische Evolution als Strategie der experimentellen Forschung in der Strömungstechnik. Technical report, Diplomarbeit, Technische Universität Berlin (1965)
Schwefel, H.P.: Numerische Optimierung von Computer-Modellen mittels der Evolution-sstrategie. In: Interdisciplinary Systems Research, vol. 26, Birkhäuser, Basel (1977)
Bäck, T.: Evolutionary algorithms in theory and practice. Oxford University Press, New York (1996)
Wilson, S.W.: Get real! XCS with continuous-valued intpus. In: Booker, L., Forrest, S., Mitchell, M., Riolo, R.L. (eds.) Festschrift in Honor of John H. Holland, Center for the Study of Complex Systems, pp. 11–121 (1999)
Srinivas, N., Deb, K.: Multiobjective optimization using nondominated sorting in genetic algorithms (1994) (submitted to EC)
Srinivas, N., Deb, K.: Multiobjective optimization using nondominated sorting in genetic algorithms. Evolutionary Computation 2, 221–248 (1995)
Deb, K., Agrawal, S., Pratab, A., Meyarivan, T.: A Fast Elitist Non-Dominated Sorting Genetic Algorithm for Multi-Objective Optimization: NSGA-II. KanGAL report 200001, Indian Institute of Technology (2000)
Goldberg, D.E., Richardson, J.: Genetic algorithms with sharing for multimodal function optimization. In: Proceedings of the Second International Conference on Genetic Algorithms, pp. 41–49 (1987)
Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth International Group, Belmont (1984)
Merz, C.J., Murphy, P.M.: UCI Repository for Machine Learning Data-Bases. University of California, Department of Information and Computer Science, Irvine, CA (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
Bernadó, E., Llorà, X., Garrell, J.M.: XCS and GALE: a Comparative Study of Two Learning Classifier Systems with Six Other Learning Algorithms on Classification Tasks. In: Lanzi, P.L., Stolzmann, W., Wilson, S.W. (eds.) IWLCS 2001. LNCS (LNAI), vol. 2321, p. 115. Springer, Heidelberg (2002) (to appear)
Aha, D., Kibler, D.: Instance-based learning algorithms. Machine Learning 6, 37–66 (1991)
Quinlan, R.: Induction of decision trees. Machine Learning 1, 81–106 (1986)
Quinlan, R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Francisco (1993)
Frank, E., Witten, I.H.: Generating Accurate Rule Sets Without Global Optimization. In: Shavlik, J. (ed.) Machine Learning: Proceedings of the Fifteenth International Conference, pp. 144–151. Morgan Kaufmann, San Francisco (1998)
Witten, I.H., Eibe, F.: Data Mining. Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Francisco (2000)
Llorà, X., Goldberg, D.E.: Minimal Achievable Error in the LED problem. IlliGAL Report No. 2002015, University of Illinois at Urbana-Champaign, Illinois Genetic Algorithms Laboratory, Urbana, IL (2002)
Llorà, X., Goldberg, D.E.: Bounding the effect of noise in Multiobjective Learning Classifier Systems. Evolutionary Computation (2003) (in press)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Llorà, X., Goldberg, D.E., Traus, I., Bernadó, E. (2003). Accuracy, Parsimony, and Generality in Evolutionary Learning Systems via Multiobjective Selection. In: Lanzi, P.L., Stolzmann, W., Wilson, S.W. (eds) Learning Classifier Systems. IWLCS 2002. Lecture Notes in Computer Science(), vol 2661. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-40029-5_8
Download citation
DOI: https://doi.org/10.1007/978-3-540-40029-5_8
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-20544-9
Online ISBN: 978-3-540-40029-5
eBook Packages: Springer Book Archive