Abstract
Neuro-Symbolic AI (NeSy) holds promise to ensure the safe deployment of AI systems, as interpretable symbolic techniques provide formal behaviour guarantees. The challenge is how to effectively integrate neural and symbolic computation, to enable learning and reasoning from raw data. Existing pipelines that train the neural and symbolic components sequentially require extensive labelling, whereas end-to-end approaches are limited in terms of scalability, due to the combinatorial explosion in the symbol grounding problem. In this paper, we leverage the implicit knowledge within foundation models to enhance the performance in NeSy tasks, whilst reducing the amount of data labelling and manual engineering. We introduce a new architecture, called NeSyGPT, which fine-tunes a vision-language foundation model to extract symbolic features from raw data, before learning a highly expressive answer set program to solve a downstream task. Our comprehensive evaluation demonstrates that NeSyGPT has superior accuracy over various baselines, and can scale to complex NeSy tasks. Finally, we highlight the effective use of a large language model to generate the programmatic interface between the neural and symbolic components, significantly reducing the amount of manual engineering required. The Appendix is presented in the longer version of this paper, which contains additional results and analysis [8].
Daniel was previously employed at IBM Research, when this work was performed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
A hypothesis is said to cover an example, if the target label is deduced given the input features.
- 2.
where h is learned using predictions from f, once f is learned.
References
Aspis, Y., Broda, K., Lobo, J., Russo, A.: Embed2sym-scalable neuro-symbolic reasoning via clustered embeddings. In: Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, vol. 19, pp. 421–431 (2022)
Badreddine, S., d’Avila Garcez, A., Serafini, L., Spranger, M.: Logic tensor networks. Artif. Intell. 303, 103649 (2022). https://doi.org/10.1016/j.artint.2021.103649
Besold, T.R., et al.: Neural-symbolic learning and reasoning: a survey and interpretation. In: Neuro-Symbolic Artificial Intelligence: The State of the Art. IOS Press (2022)
Byerly, A., Kalganova, T., Dear, I.: No routing needed between capsules. Neurocomputing 463, 545–553 (2021)
Charalambous, T., Aspis, Y., Russo, A.: NeuralFastLAS: fast logic-based learning from raw data. arXiv preprint arXiv:2310.05145 (2023)
Cunnington, D., Law, M., Lobo, J., Russo, A.: FFNSL: feed-forward neural-symbolic learner. Mach. Learn. 112(2), 515–569 (2023)
Cunnington, D., Law, M., Lobo, J., Russo, A.: Neuro-symbolic learning of answer set programs from raw data. In: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 3586–3596 (2023)
Cunnington, D., Law, M., Lobo, J., Russo, A.: The role of foundation models in neuro-symbolic learning and reasoning. arXiv preprint arXiv:2402.01889 (2024)
Dai, W.Z., Muggleton, S.: Abductive knowledge induction from raw data. In: Zhou, Z.H. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1845–1851. International Joint Conferences on Artificial Intelligence Organization (2021). https://doi.org/10.24963/ijcai.2021/254
Dai, W.Z., Xu, Q., Yu, Y., Zhou, Z.H.: Bridging machine learning and logical reasoning by abductive learning. Adv. Neural Inf. Process. Syst. 32 (2019)
Daniele, A., Campari, T., Malhotra, S., Serafini, L.: Deep symbolic learning: discovering symbols and rules from perceptions. In: Elkind, E. (ed.) Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 3597–3605. International Joint Conferences on Artificial Intelligence Organization (2023). https://doi.org/10.24963/ijcai.2023/400
Defresne, M., Barbe, S., Schiex, T.: Scalable coupling of deep learning with logical reasoning. In: Elkind, E. (ed.) Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 3615–3623. International Joint Conferences on Artificial Intelligence Organization (2023). https://doi.org/10.24963/ijcai.2023/402
Eiter, T., Higuera, N., Oetsch, J., Pritz, M.: A neuro-symbolic ASP pipeline for visual question answering. Theory Pract. Logic Program. 22(5), 739–754 (2022)
Evans, R., et al.: Making sense of raw input. Artif. Intell. 299, 103521 (2021). https://doi.org/10.1016/j.artint.2021.103521
d’Avila Garcez, A., Gori, M., Lamb, L.C., Serafini, L., Spranger, M., Tran, S.N.: Neural-symbolic computing: an effective methodology for principled integration of machine learning and reasoning. FLAP 6(4), 611–632 (2019)
Gelfond, M., Kahl, Y.: Knowledge Representation, Reasoning, and the Design of Intelligent Agents: The Answer-set Programming Approach. Cambridge University Press, Cambridge, UK (2014)
Harnad, S.: The symbol grounding problem. Phys. D 42(1–3), 335–346 (1990)
Hutchins, D., Schlag, I., Wu, Y., Dyer, E., Neyshabur, B.: Block-recurrent transformers. Adv. Neural. Inf. Process. Syst. 35, 33248–33261 (2022)
Ishay, A., Yang, Z., Lee, J.: Leveraging large language models to generate answer set programs. In: Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning, pp. 374–383 (2023). https://doi.org/10.24963/kr.2023/37
Karp, R.M.: Reducibility among combinatorial problems. In: Miller, R.E., Thatcher, J.W., Bohlinger, J.D. (eds.) Complexity of Computer Computations, pp. 85–103. Springer, Boston, MA, US (1972). https://doi.org/10.1007/978-1-4684-2001-2_9
Kassner, N., Schütze, H.: Negated and misprimed probes for pretrained language models: birds can talk, but cannot fly. arXiv preprint arXiv:1911.03343 (2019)
Law, M., Russo, A., Bertino, E., Broda, K., Lobo, J.: FastLAS: scalable inductive logic programming incorporating domain-specific optimisation criteria. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 2877–2885 (2020)
Law, M., Russo, A., Broda, K.: Logic-based learning of answer set programs. In: Reasoning Web, Explainable Artificial Intelligence - 15th International Summer School 2019, Bolzano, Italy, September 20-24, 2019, Tutorial Lectures, pp. 196–231 (2019)
Law, M., Russo, A., Broda, K.: The ilasp system for inductive learning of answer set programs. arXiv preprint arXiv:2005.00904 (2020)
Li, D., Li, J., Le, H., Wang, G., Savarese, S., Hoi, S.C.: Lavis: a library for language-vision intelligence. arXiv preprint arXiv:2209.09019 (2022)
Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning, pp. 12888–12900. PMLR (2022)
Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T., De Raedt, L.: Deepproblog: neural probabilistic logic programming. Adv. Neural Inf. Process. Syst. 31 (2018)
Marconato, E., Teso, S., Vergari, A., Passerini, A.: Not all neuro-symbolic concepts are created equal: analysis and mitigation of reasoning shortcuts. Adv. Neural Inf. Process. Syst. 36 (2024)
Nye, M., Tessler, M., Tenenbaum, J., Lake, B.M.: Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Adv. Neural. Inf. Process. Syst. 34, 25192–25204 (2021)
OpenAI, et al.: GPT-4 technical report (2023)
Pryor, C., Dickens, C., Augustine, E., Albalak, A., Wang, W.Y., Getoor, L.: NeuPSL: neural probabilistic soft logic. In: Elkind, E. (ed.) Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 4145–4153. International Joint Conferences on Artificial Intelligence Organization (2023). https://doi.org/10.24963/ijcai.2023/461
Riegel, R., et al.: Logical neural networks. arXiv preprint arXiv:2006.13155 (2020)
Shindo, H., Pfanschilling, V., Dhami, D.S., Kersting, K.: \(\alpha \) ILP: thinking visual scenes as differentiable logic programs. Mach. Learn. 112(5), 1465–1497 (2023)
Singh, D., Jain, N., Jain, P., Kayal, P., Kumawat, S., Batra, N.: PlantDoc: a dataset for visual plant disease detection. In: Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pp. 249–253 (2020)
Skryagin, A., Ochs, D., Dhami, D.S., Kersting, K.: Scalable neural-probabilistic answer set programming. arXiv preprint arXiv:2306.08397 (2023)
Stammer, W., Schramowski, P., Kersting, K.: Right for the right concept: revising neuro-symbolic concepts by interacting with their explanations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3619–3629 (2021)
Surís, D., Menon, S., Vondrick, C.: ViperGPT: visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128 (2023)
Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., Goodman, N.D.: Hypothesis search: inductive reasoning with language models. arXiv preprint arXiv:2309.05660 (2023)
Yang, Z., Ishay, A., Lee, J.: NeurASP: embracing neural networks into answer set programming. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 1755–1762. International Joint Conferences on Artificial Intelligence Organization (2020). https://doi.org/10.24963/ijcai.2020/243
Yang, Z., Ishay, A., Lee, J.: Coupling large language models with logic programming for robust and general reasoning from text. arXiv preprint arXiv:2307.07696 (2023)
Yujian, L., Bo, L.: A normalized Levenshtein distance metric. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 1091–1095 (2007)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cunnington, D., Law, M., Lobo, J., Russo, A. (2024). The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning. In: Besold, T.R., d’Avila Garcez, A., Jimenez-Ruiz, E., Confalonieri, R., Madhyastha, P., Wagner, B. (eds) Neural-Symbolic Learning and Reasoning. NeSy 2024. Lecture Notes in Computer Science(), vol 14979. Springer, Cham. https://doi.org/10.1007/978-3-031-71167-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-71167-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-71166-4
Online ISBN: 978-3-031-71167-1
eBook Packages: Computer ScienceComputer Science (R0)