Abstract
Inductive logic programming (ILP) algorithms are classification algorithms that construct classifiers represented as logic programs. ILP algorithms have a number of attractive features, notably the ability to make use of declarative background (user-supplied) knowledge. However, ILP algorithms deal poorly with large data sets (>104 examples) and their widespread use of the greedy set-covering algorithm renders them susceptible to local maxima in the space of logic programs.
This paper presents a novel approach to address these problems based on combining the local search properties of an inductive logic programming algorithm with the global search properties of an evolutionary algorithm. The proposed algorithm may be viewed as an evolutionary wrapper around a population of ILP algorithms.
The evolutionary wrapper approach is evaluated on two domains. The chess-endgame (KRK) problem is an artificial domain that is a widely used benchmark in inductive logic programming, and Part-of-Speech Tagging is a real-world problem from the field of Natural Language Processing. In the latter domain, data originates from excerpts of the Wall Street Journal. Results indicate that significant improvements in predictive accuracy can be achieved over a conventional ILP approach when data is plentiful and noisy.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
T.G. Dietterich, “Machine learning research: Four current directions,” AI Magazine, vol. 18, No. 4, pp. 97-136, 1997.
T. Bäck. Evolutionary Algorithms in Theory and Practice. Oxford University Press, 1996.
K.A. DeJong and W.M. Spears, “Using genetic algorithms to solve NP-complete problems,” in International Conference on Genetic Algorithms, 1989, pp. 124-132.
D.E. Goldberg, “Genetic and evolutionary algorithms come of age,” Communications of the ACM, vol. 37, pp. 113-119, 1994.
K.A. DeJong, W.M. Spears, and D.F. Gordon, “Using genetic algorithms for concept learning,” Machine Learning, vol. 13, pp. 161-188, 1993.
A. Giordana and L. Saitta, “Regal: An integrated system for learning relations using genetic algorithms,” in Proceedings of 2nd InternationalWorkshop on Multistrategy Learning, Morgan Kaufmann, pp. 234-249, 1993.
A.C. Schultz and J.J. Grefenstette, “Improving tactical plans with genetic algorithms,” in Proceedings of the 2nd International IEEE Conference on Tools for Artificial Intelligence, number IEEE Cat. No. 90CH2915-7, Herndon,VA, 6-9Nov. 1990. IEEE Computer Society Press: Los Alamitos, CA, pp. 328-334.
M.L. Wong and K.S. Leung, “Inductive logic programming using genetic algorithms,” in Advances in Artificial Intelligence-Theory and Application II, edited by J.W. Brahan and G.E. Lasker, pp. 119-124, 1994.
J.R. Koza, Genetic Programming: On the Programming of Computers by Natural Selection, MIT Press: Cambridge, MA, 1992.
N.J. Radcliffe and P.D. Surry, “Cooperation through hierarchical competition in genetic data mining,” Technical Report TR9409, University of Edinburgh, Parallel Computing Centre, Edinburgh, 1994.
D.J. Montana, “Strongly typed genetic programming,” BBN Technical Report #7866, Bolt Beranek and Newman, Inc., 10 Moulton Street, Cambridge, MA 02138, 1994.
T.D. Haynes, D.A. Schoenefeld, and R.L. Wainwright, “Type inheritance in strongly typed genetic programming,” in Advances in Genetic Programming 2, edited by P.J. Angeline and K.E. Kinnear, Jr., MIT Press: Cambridge, MA, Chap. 18, pp. 359-376, 1996.
M.L. Wong and K.S. Leung, “An induction system that learns programs in different programming languages using genetic programming and logic grammars,” in Seventh International Conference on Tools with Artificial Intelligence, Herndon, Virginia, Nov. 1995, The Chinese University of Hong Kong, IEEE Computer Society Press, pp. 380-387.
M.L. Wong and K.S. Leung, “Combining genetic programming and inductive logic programming using logic grammars,” in Proceedings of the IEEE International Conference on Evolutionary Computation, Perth, Western Australia, Nov. 1995, IEEE Computer Society Press, pp. 733-736.
M.L. Wong and K.S. Leung, “Applying logic grammars to induce sub-functions in genetic programming,” in Proceedings of the IEEE International Conference on Evolutionary Computation, Perth, Western Australia, Nov. 1995, IEEE Computer Society Press, pp. 737-740.
M.L. Wong and K.S. Leung, “Evolving recursive functions for the even-parity problem using genetic programming,” in Advances in Genetic Programming 2, edited by P.J. Angeline and K.E. Kinnear Jr., MIT Press, Cambridge, MA, 1996.
P.A. Whigham, “Grammatically-based genetic programming,” in Proc. Workshop on Genetic Programming: From Theory to Real-World Applications, edited by J. Rosca, Morgan Kaufmann, pp. 33-41, 1995.
P.A. Whigham, “Search bias, language bias, and genetic programming,” in Genetic Programming 1996: Proceedings of the First Conference, Stanford University, CA, 28-31 July 1996, edited by John R. Koza, David E. Goldberg, David B. Fogel, and Rick L. Riolo, MIT Press.
F. Gruau, “On using syntactic constraints with genetic programming,” in Advances in Genetic Programming III, edited by P.J. Angeline and K.E. Kinnear, MIT Press: Cambridge, MA, Chap. 19, pp. 377-394, 1996.
P.S. Ngan, M.L. Wong, W. Lam, K.S. Leung, and J.C.Y. Cheng, “Medical data mining using evolutionary computation,” Artificial Intelligence in Medicine, vol. 16, pp. 73-96, 1999.
P.A. Whigham and P.F. Crapper, “Time series modelling using genetic programming: An application to rainfall-runoff models,” in Advances in Genetic Programming, edited by L. Spector, W.B. Langdon, U.-M. O'Reilly, and P.J. Angeline, MIT Press: Cambridge, MA, Chap. 5, pp. 89-104. 1999.
K.A. DeJong and W.M. Spears, “Learning concept classification rules using genetic algorithms,” in Proceedings of the Twelfth International Joint Conference on Artificial Intelligence, 1991, pp. 651-656.
C.Z. Janikow, “A knowledge-intensive genetic algorithm for supervised learning,” Machine Learning, vol. 13, pp. 189-228, 1993.
J.J. Grefenstette, “A system for learning control strategies with genetic algorithms,” George Mason University, pp. 183-190. Naval Research Laboratory, 1989
J.J. Grefenstette, “Learning decision strategies with genetic algorithms,” in Proceedings of the International Workshop on Analogical and Inductive Inference, Lecture Notes in Artificial Intelligence 642, pp. 35-50, 1992.
J.J. Grefenstette, “The evolution of strategies for multiagent environments,” Adaptive Behaviour, vol. 1, no. 1, pp. 65-90, 1992.
J.J. Grefenstette, “Lamarckian learning in multiagent environments,” in Proceedings of 4th International Conference of Scaling Up Inductive Logic Programming 197 Genetic Algorithms, Morgan Kaufmann, pp. 303-310, 1992.
S. Augier, G. Venturini, and Y. Kodratoff, “Learning first order logic rules with a genetic algorithm,” in Proceedings of the First International Conference on Knowledge Discovery and Data Mining, 1995, pp. 21-26.
A. Giordana, L. Saitta, and F. Zini, “Learning disjunctive concepts by means of genetic algorithms,” in Proceedings of the 11th International Conference on Machine Learning, 1994, pp. 96-104.
F. Neri, “First order logic concept learning by means of a distributed genetic algorithm,” Ph.D. dissertation, Università of Torino, Dipartimento di Informatica, 1997.
L. Davis, Handbook of Genetic Algorithms, Van Nostrand Reinhold: New York, 1991.
W.E. Hart and R.K. Belew, “Optimization with genetic algorithm hybrids that use local search,” in Adaptive Individuals in Evolving Populations: Models and Algorithms, edited by R.K. Belew and M. Mitchell, SFI Studies in the Sciences of Complexity, vol. 26, Chap. 27, pp. 483-496, 1996.
D.E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley: Reading, MA, 1989.
S. Muggleton, “Inductive logic programming,” New Generation Computing, vol. 8, no. 4, pp. 295-318, 1991.
S. Muggleton, “Inductive logic programming: Derivations, successes and shortcomings,” SIGART Bulletin, vol. 5, no. 1, pp. 5-11, 1994.
S. Muggleton, “Inverse entailment and progol,” New Generation Computing, vol. 13, 1995.
U. Pompe, M. Kovačič, and I. Kononenko, “SFOIL: Stochastic approach to inductive logic programming,” in Proceedings of the 2nd Electrotechnical and Computer Science Conference, Portoroz, Slovenija, p. 31, 1993.
M. Kovačič, “Stochastic inductive logic programming,” Ph.D. Thesis, University of Ljubljana, Slovenia, 1994.
L. Breiman, “Bagging predictors,” Technical Report No. 421, University of California, Berkeley, Department of Statistics, 1994.
S.H. Muggleton, M. Bain, J. Hayes-Michie, and D. Michie, “An experimental comparison of human and machine learning formalisms,” in Proc. Sixth International Workshop on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1989. pp. 113-118.
J. Cussens, “Part-of-speech tagging using progol,” in Inductive Logic Programming: Proceedings of the 7th InternationalWorkshop (ILP-97), 1997, volume 1297 of Lecture Notes in Artificial Intelligence, Springer pp. 93-108.
M.P. Marcus, B. Santorini, and M.A. Marcinkiewicz, “Building a large annotated corpus of English: the Penn treebank,” Computational Linguistics, vol. 19, 1993.
J. Cussens, “Part-of-speech disambiguation using ILP,” Technical Report PRG-TR-25-96, Oxford University Computing laboratory, 1996.
J.R. Quinlan, “Discovering rules from large collections of examples: A case study,” in Expert Systems in the Micro-electronic Age, edited by D. Michie, Edinburgh University Press, Edinburgh, pp. 168-201, 1979.
J.W. Lloyd, Foundations of Logic Programming, Springer-Verlag: Berlin, 1984.
T.G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,” Neural Computation, vol. 10, no. 7, pp. 1895-1924, 1998.
D. Fogel and C. deSilva, eds. Proceedings of the IEEE International Conference on Evolutionary Computation, Perth,Western Australia, Nov. 1995, IEEE Computer Society Press.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Reiser, P.G., Riddle, P.J. Scaling Up Inductive Logic Programming: An Evolutionary Wrapper Approach. Applied Intelligence 15, 181–197 (2001). https://doi.org/10.1023/A:1011239013893
Issue Date:
DOI: https://doi.org/10.1023/A:1011239013893