Abstract
In many real-world applications, the i.i.d. assumption does not hold and thus capturing the interactions between instances is essential for the task at hand. Recently, a clear connection between predictive modelling such as decision trees and probabilistic circuits, a form of deep probabilistic model, has been established although it is limited to propositional data. We introduce the first connection between relational rule models and probabilistic circuits, obtaining tractable inference from discriminative rule models while operating on the relational domain. Specifically, given a relational rule model, we make use of Mixed Sum-Product Networks (MSPNs)—a deep probabilistic architecture for hybrid domains—to equip them with a full joint distribution over the class and how (often) the rules fire. Our empirical evaluation shows that we can answer a wide range of probabilistic queries on relational data while being robust to missing, out-of-domain data and partial counts. We show that our method generalizes to different distributions outperforming strong baselines. Moreover, due to the clear probabilistic semantics of MSPNs we have informative model interpretations.
F. Ventola and D. S. Dhami—Equal contribution.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
We are not strict on “density” vs. “distribution”.
- 2.
A set of random variables is finitely exchangeable with respect to a joint distribution P, if all permutations of the variables result in the same joint probabilities. Note that finite exchangeable does not require independence; the random variables can have strong dependencies.
- 3.
In the figure and in the following text \(\boldsymbol{1}()\) represents an indicator function.
- 4.
- 5.
- 6.
Both Scikit-learn implementation with default hyperparameters.
References
Altman, N.S.: An introduction to kernel and nearest-neighbor nonparametric regression. The Am. Stat. (1992)
Austin, P.C., White, I.R., Lee, D.S., van Buuren, S.: Missing data in clinical research: a tutorial on multiple imputation. Can. J. Cardiol. (2020)
Blockeel, H., de Raedt, L.: Top-down induction of first-order logical decision trees. AI (1998)
van Buuren, S., Groothuis-Oudshoorn, K.: mice: Multivariate imputation by chained equations in r. J. Stat. Softw. (2011)
Cawley, G.C., Talbot, N.L.: On over-fitting in model selection and subsequent selection bias in performance evaluation. JMLR (2010)
Chattopadhyay, P., Balaji, Y., Hoffman, J.: Learning to balance specificity and invariance for in and out of domain generalization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 301–318. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_18
Cheng, W., Kok, S., Pham, H.V., Chieu, H.L., Chai, K.M.A.: Language modeling with sum-product networks. In: INTERSPEECH (2014)
Choi, Y., Vergari, A., Van den Broeck, G.: Probabilistic circuits: a unifying framework for tractable probabilistic models (2020)
Correia, A.H.C., Peharz, R., de Campos, C.P.: Joints in random forests. In: NeurIPS (2020)
Darwiche, A.: A differential approach to inference in Bayesian networks. JACM (2003)
De Raedt, L., Kersting, K., Natarajan, S., Poole, D.: Statistical relational artificial intelligence: logic, probability, and computation (2016)
Dhami, D.S., Kunapuli, G., Das, M., Page, D., Natarajan, S.: Drug-drug interaction discovery: kernel learning from heterogeneous similarities. Smart Health (2018)
Dhami, D.S., Yen, S., Kunapuli, G., Natarajan, S.: Non-parametric learning of Gaifman models. arXiv preprint arXiv:2001.00528 (2020)
Flach, P.A., Lachiche, N.: Naive Bayesian classification of structured data. Mach. Learn. (2004)
Gangal, V., Arora, A., Einolghozati, A., Gupta, S.: Likelihood ratios and generative classifiers for unsupervised out-of-domain detection in task oriented dialog. In: AAAI (2020)
Gens, R., Domingos, P.: Learning the structure of sum-product networks. In: ICML (2013)
Getoor, L., Taskar, B.: Statistical relational learning (2007)
Khosravi, P., Vergari, A., Choi, Y., Liang, Y., den Broeck, G.V.: Handling missing data in decision trees: a probabilistic approach. arXiv preprint arXiv:2006.16341 (2020)
Khot, T., Natarajan, S., Kersting, K., Shavlik, J.: Learning Markov logic networks via functional gradient boosting. In: ICDM (2011)
Kok, S., et al.: The alchemy system for statistical relational AI (2005)
Landwehr, N., Kersting, K., De Raedt, L.: nfoil: Integrating Naıve Bayes and foil. In: AAAI (2005)
Landwehr, N., Kersting, K., De Raedt, L.: Integrating Naive Bayes and foil. JMLR (2007)
Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In: NeurIPS (2018)
Mitchell, T., et al.: Never-ending learning. Commun. ACM (2018)
Molina, A., Natarajan, S., Kersting, K.: Poisson sum-product networks: a deep architecture for tractable multivariate Poisson distributions. In: AAAI (2017)
Molina, A., Vergari, A., Mauro, N.D., Natarajan, S., Esposito, F., Kersting, K.: Mixed sum-product networks: a deep architecture for hybrid domains. In: AAAI (2018)
Natarajan, S., Khot, T., Kersting, K., Gutmann, B., Shavlik, J.: Gradient-based boosting for statistical relational learning: the relational dependency network case. Mach. Learn. (2012)
Nath, A., Domingos, P.: Learning relational sum-product networks. In: AAAI (2015)
Neville, J., Jensen, D.: Relational dependency networks. JMLR (2007)
Peharz, R., Gens, R., Pernkopf, F., Domingos, P.M.: On the latent variable interpretation in sum-product networks. TPAMI (2017)
Peharz, R., Kapeller, G., Mowlaee, P., Pernkopf, F.: Modeling speech with sum-product networks: application to bandwidth extension. In: ICASSP (2014)
Peharz, R., et al.: Random sum-product networks: a simple and effective approach to probabilistic deep learning. In: UAI (2019)
Poon, H., Domingos, P.M.: Sum-product networks: a new deep architecture. In: UAI (2011)
Richardson, M., Domingos, P.: Markov logic networks. Mach. Learn. (2006)
Tan, M., et al.: Out-of-domain detection for low-resource text classification tasks. In: EMNLP-IJCNLP (2019)
Tran, D., Vafa, K., Agrawal, K.K., Dinh, L., Poole, B.: Discrete flows: invertible generative models of discrete data. In: NeurIPS 2019 (2019)
Vapnik, V., Vashist, A.: A new learning paradigm: learning using privileged information. Neural Netw. (2009)
Vergari, A., Di Mauro, N., Esposito, F.: Visualizing and understanding sum-product networks. Mach. Learn. (2019)
Yang, E., Ravikumar, P.K., Allen, G.I., Liu, Z.: On Poisson graphical models. In: NIPS (2013)
Yu, Z., Ventola, F., Kersting, K.: Whittle networks: a deep likelihood model for time series. In: ICML (2021)
Yuan, Z., Wang, H., Wang, L., Lu, T., Palaiahnakote, S., Tan, C.L.: Modeling spatial layout for scene image understanding via a novel multiscale sum-product network. Expert Syst. Appl. (2016)
Acknowledgments
This work was supported by the ICT-48 Network of AI Research Excellence Center “TAILOR” (EU Horizon 2020, GA No 952215), the Federal Ministry of Education and Research (BMBF; Competence Center for AI and Labour; “kompAKI”, FKZ 02L19C150), the German Science Foundation (DFG, German Research Foundation; GRK 1994/1 “AIPHES”), the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK; projects “The Third Wave of AI” and “The Adaptive Mind”), the Hessian research priority programme LOEWE within the project “WhiteBox”, and the Collaboration Lab “AI in Construction” (AICO).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Ventola, F., Dhami, D.S., Kersting, K. (2022). Generative Clausal Networks: Relational Decision Trees as Probabilistic Circuits. In: Katzouris, N., Artikis, A. (eds) Inductive Logic Programming. ILP 2021. Lecture Notes in Computer Science(), vol 13191. Springer, Cham. https://doi.org/10.1007/978-3-030-97454-1_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-97454-1_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-97453-4
Online ISBN: 978-3-030-97454-1
eBook Packages: Computer ScienceComputer Science (R0)