Abstract
In this paper we present 1BC and 1BC2, two systems that perform naive Bayesian classification of structured individuals. The approach of 1BC is to project the individuals along first-order features. These features are built from the individual using structural predicates referring to related objects (e.g., atoms within molecules), and properties applying to the individual or one or several of its related objects (e.g., a bond between two atoms). We describe an individual in terms of elementary features consisting of zero or more structural predicates and one property; these features are treated as conditionally independent in the spirit of the naive Bayes assumption. 1BC2 represents an alternative first-order upgrade to the naive Bayesian classifier by considering probability distributions over structured objects (e.g., a molecule as a set of atoms), and estimating those distributions from the probabilities of its elements (which are assumed to be independent). We present a unifying view on both systems in which 1BC works in language space, and 1BC2 works in individual space. We also present a new, efficient recursive algorithm improving upon the original propositionalisation approach of 1BC. Both systems have been implemented in the context of the first-order descriptive learner Tertius, and we investigate the differences between the two systems both in computational terms and on artificially generated data. Finally, we describe a range of experiments on ILP benchmark data sets demonstrating the viability of our approach.
Article PDF
Similar content being viewed by others
References
Badea, L., & Stanciu, M. (1999). Refinement operators can be (weakly) perfect. In Proceedings of the NinthInternational Workshop on Inductive Logic Programming (pp. 21–32). Springer-Verlag.
Boström, H., & Asker, L. (1999). Combining divide-and-conquer and separate-and-conquer for efficient and effective rule induction. In Proceedings of the Ninth International Workshop on Inductive Logic Programming (pp. 33–43). Springer-Verlag.
Ceci, M., Appice, A., & Malerba, D. (2003). Mr-SBC: A multi-relational naïve Bayes classifier. In Proceedings of the Seventh European Conference on Principles and Practice of Knowledge Discovery in Databases (pp. 95–106). Springer-Verlag.
Cussens, J. (2001). Parameter estimation in stochastic logic programs. Machine Learning, 43, 245–271.
Date, C. (1995). An introduction to database systems. Addison Wesley.
Dehaspe, L. (1997). Maximum entropy modeling with clausal constraints. In Proceedings of the Seventh International Workshop on Inductive Logic Programming (pp. 109–124). Springer-Verlag.
Dietterich, T., Lathrop, R., & Lozano-Perez, T. (1997). Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence, 89, 31–71.
Dol?sak, B., & Muggleton, S. (1992). The application of inductive logic programming to finite-element mesh design. In S. Muggleton (Ed.), Inductive logic programming. Academic Press.
Domingos, P., & Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 29, 103–130.
D?zeroski, S., Schulze-Kremer, S., Heidtke, K., Siems, K., Wettschereck, D., & Blockeel, H. (1998). Diterpene structure elucidation from 13C NMR spectra with inductive logic programming. Applied Artificial Intelligence,12, 363–383.
D?zeroski, S., & Lavrač, N. (Eds.). (2001). Relational data mining. Springer-Verlag.
Fawcett, T. (2003). ROC graphs: Notes and practical considerations for data mining researchers. Technical report HPL-2003-4. HP Laboratories, Palo Alto, CA, USA. Available at http://www.hpl.hp.com/techreports/ 2003/HPL-2003-4.pdf.
Flach, P., Giraud-Carrier, C., & Lloyd, J. (1998). Strongly typed inductive concept learning. Proceedings of the Eighth International Conference on Inductive Logic Programming (pp. 185–194). Springer-Verlag.
Flach, P., & Lachiche, N. (1999). 1BC: A first-order Bayesian classifier. In Proceedings of the Ninth International Workshop on Inductive Logic Programming (pp. 92–103). Springer-Verlag.
Flach, P. (1999). Knowledge representation for inductive learning. Symbolic and Quantitative Approaches toReasoning and Uncertainty (pp. 160–167). Springer-Verlag.
Flach, P., Gyftodimos, E., & Lachiche, N. (to appear). Probabilistic reasoning with terms. Linkoping Electronic Articles in Computer and Information Science. Available at http://www.ida.liu.se/ext/epa/cis/2002/011/tcover.html.
Flach, P., & Lachiche, N. (2001). Confirmation-guided discovery of first-order rules with tertius. Machine Learning, 42, 61–95.
Gärtner, T., Lloyd, J., & Flach, P. (2004). Kernels and distances for structured data. Machine Learning, 57:3, 205–232.
Getoor, L., Friedman, N., Koller, D., & Pfeffer, A. (2001). Learning probabilistic relational models. In S. D?zeroski and N. Lavrač (Eds.), Relational data mining. Springer-Verlag.
Gyftodimos, E., & Flach, P. (2003). Hierarchical Bayesian networks: an approach to classification and learning for structured data. In Proceedings of the Work-in-Progress Track at the Thirteenth International Conference on Inductive Logic Programming (pp. 12–21). Department of Informatics, University of Szeged.
Hand, D., & Till, R. (2001). A simple generalisation of the Area Under the ROC Curve for multiple class classification problems. Machine Learning, 45, 171–186.
John, G., & Langley, P. (1995). Estimating continuous distributions in Bayesian classifiers. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence (pp. 338–345). Morgan Kaufmann.
Kersting, K., & De Raedt, L. (2000). Bayesian logic programs. In Proceedings of theWork-in-Progress Track at the Tenth International Conference on Inductive Logic Programming (pp. 138–155). CEURWorkshop Proceedings Series, Vol. 35.
Kersting, K., & De Raedt, L. (2001). Towards combining inductive logic programming with Bayesian networks. In Proceedings of the Eleventh International Conference on Inductive Logic Programming (pp. 118–131). Springer-Verlag.
Knobbe, A., de Haas, M., & Siebes, A. (2001). Propositionalisation and aggregates. In Proceedings of the Fifth European Conference on Principles of Data Mining and Knowledge Discovery (pp. 277–288). Springer-Verlag.
Kramer, S., Lavrač, N., & Flach, P. (2001). Propositionalization approaches to relational data mining. In S. D?zeroski and N. Lavrač (Eds.), Relational data mining. Springer-Verlag.
Krogel, M.-A., & Wrobel, S. (2001). Transformation-based learning using multirelational aggregation. In Proceedings of the Eleventh International Conference on Inductive Logic Programming (pp. 142–155). Springer-Verlag.
Lachiche, N., & Flach, P. (2002). 1BC2: a true first-order Bayesian classifier. In Proceedings of the Twelfth International Conference on Inductive Logic Programming (pp. 133–148). Springer-Verlag.
Lachiche, N., & Flach, P. (2003). Improving accuracy and cost of two-class and multi-class probabilistic classifiers using ROC curves. In Proceedings of the Twentieth International Conference on Machine Learning (pp. 416–423). AAAI Press.
Lavrač, N., & Flach, P. (2001). An extended transformation approach to inductive logic programming. ACM Transactions on Computational Logic, 2, 458–494.
Lavrač, N., Železný, F., & Flach, P. (2002). RSD: Relational subgroup discovery through first-order feature construction. In Proceedings of the Twelfth International Conference on Inductive Logic Programming (pp. 149–165). Springer-Verlag.
Lloyd, J. (1999). Programming in an integrated functional and logic language. Journal of Functional and Logic Programming, 1999.
Lloyd, J. (2003). Logic for learning: learning comprehensible theories from structured data. Springer-Verlag.
Lu, Q., & Getoor, L. (2003). Link-based classification. In Proceedings of the Twentieth International Conference on Machine Learning (pp. 496–503). AAAI Press.
Muggleton, S. (Ed.). (1992). Inductive logic programming. Academic Press.
Muggleton, S. (1995). Inverse entailment and Progol. New Generation Computing,13, 245–286.
Muggleton, S. (1996). Stochastic logic programs. In L. De Raedt (Ed.), Advances in inductive logic programming. IOS Press.
Muggleton, S., & De Raedt, L. (1994). Inductive logic programming: Theory and methods. Journal of Logic Programming, 19/20, 629–679.
Muggleton, S., Srinivasan, A., King, R., & Sternberg, M. (1998). Biochemical knowledge discovery using Inductive Logic Programming. In Proceedings of the first International Conference on Discovery Science (pp. 326–341). Springer-Verlag.
Pompe, U., & Kononenko, I. (1995). Naive Bayesian classifier within ILP-R. In Proceedings of the Fifth International Workshop on Inductive Logic Programming (pp. 417–436). Department of Computer Science, Katholieke Universiteit Leuven.
Rouveirol, C. (1994). Flattening and saturation: Two representation changes for generalization. Machine Learning, 14, 219–232.
Sato, T., & Kameya, Y. (1997). Prism: A symbolic-statistical modeling language. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (pp. 1330–1335). Morgan Kaufmann.
Sato, T., & Kameya, Y. (2001). Parameter learning of logic programs for symbolic-statistical modeling. Journal of Artificial Intelligence Research, 15, 391–454.
Slattery, S., & Craven, M. (1998). Combining statistical and relational methods for learning in hypertext domains. In Proceedings of the Eighth International Conference on Inductive Logic Programming (pp. 38–52). Springer-Verlag.
Srinivasan, A., Muggleton, S., King, R., & Sternberg, M. (1994). Mutagenesis: ILP experiments in a nondeterminate biological domain. In Proceedings of the Fourth International Workshop on Inductive Logic Programming (pp. 217–232). Gesellschaft für Mathematik und Datenverarbeitung MBH.
Taskar, B., Segal, E., & Koller, D. (2001). Probabilistic clustering in relational data. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (pp. 870–87). Morgan Kaufmann.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Flach, P.A., Lachiche, N. Naive Bayesian Classification of Structured Data. Machine Learning 57, 233–269 (2004). https://doi.org/10.1023/B:MACH.0000039778.69032.ab
Issue Date:
DOI: https://doi.org/10.1023/B:MACH.0000039778.69032.ab