Skip to main content
SpringerLink
Log in

Commonsense Reasoning and Explainable Artificial Intelligence Using Large Language Models

  • Conference paper
  • First Online:
Artificial Intelligence. ECAI 2023 International Workshops (ECAI 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1947))

Included in the following conference series:

  • 374 Accesses

Abstract

Commonsense reasoning is a difficult task for a computer, but a critical skill for an artificial intelligence (AI). It can enhance the explainability of AI models by enabling them to provide intuitive and human-like explanations for their decisions. This is necessary in many areas but especially in the field of question answering (QA), which is one of the most important tasks of natural language processing (NLP). Over time, a multitude of methods have emerged for solving commonsense reasoning problems such as knowledge-based approaches using formal logic or linguistic analysis.

In this paper, we investigate the effectiveness of large language models (LLMs) on different QA tasks with focus on their abilities on reasoning and producing explanations. For this, we study the recent and very prominent LLM ChatGPT and evaluate the results by means of a questionnaire. We demonstrate ChatGPT’s ability to reason with common sense, and although ChatGPT’s accuracy ranges from 56% to 93% on various QA benchmarks, it outperforms human accuracy. Furthermore we can appraise that, in the sense of explainable artificial intelligence (XAI), ChatGPT gives good explanations for its decisions. In our questionnaire we found that 68% of the participants quantify ChatGPT’s explanations as “good” or “excellent”. Taken together, these findings enrich our understanding of current LLMs and pave the way for future investigations of reasoning and explainability.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://chat.openai.com/.

  2. 2.

    https://www.soscisurvey.de/.

References

  1. Álvez, J., Lucio, P., Rigau, G.: Adimen-SUMO: Reengineering an ontology for first-order reasoning. Int. J. Semant. Web Int. Syst. (IJSWIS) 8(4), 80–116 (2012), https://ideas.repec.org/a/igg/jswis0/v8y2012i4p80-116.html

  2. Barredo Arrieta, A., et al.: Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020). https://doi.org/10.1016/j.inffus.2019.12.012

    Article  Google Scholar 

  3. Brown, T., et al.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pp. 1877–1901 (2020). https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

  4. Burkart, N., Huber, M.F.: A survey on the explainability of supervised machine learning. J. Artif. Intell. Res. 70, 245–317 (2021). https://doi.org/10.1613/jair.1.12228

    Article  MathSciNet  Google Scholar 

  5. Chen, M., D’arcy, M., Liu, A., Fernandez, J., Downey, D.: CODAH: An adversarially-authored question answering dataset for common sense. In: Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pp. 63–69 (2019). https://www.jaredfern.com/publication/codah/

  6. Clark, P., et al.: Think you have solved question answering? Try ARC, the AI2 reasoning challenge. CoRR - Computing Research Repository abs/1803.05457, Cornell University Library (2018), https://arxiv.org/abs/1803.05457

  7. Davis, E.: Logical formalizations of commonsense reasoning: a survey. J. Artif. Intell. Res. 59, 651–723 (2017). https://doi.org/10.1613/jair.5339

    Article  MathSciNet  Google Scholar 

  8. Davis, E., Marcus, G.: Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM 58(9), 92–103 (2015). https://doi.org/10.1145/2701413

    Article  Google Scholar 

  9. Deng, J., Lin, Y.: The benefits and challenges of ChatGPT: An overview. Front. Comput. Intell. Syst. 2(2), 81–83 (2023). https://doi.org/10.54097/fcis.v2i2.4465

  10. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. https://arxiv.org/pdf/1810.04805

  11. Du, L., Ding, X., Xiong, K., Liu, T., Qin, B.: e-CARE: a new dataset for exploring explainable causal reasoning. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 432–446. Association for Computational Linguistics (2022). https://aclanthology.org/2022.acl-long.33/

  12. Forbus, K.D.: Qualitative process theory. Artif. Intell. 24(1–3), 85–168 (1984). https://doi.org/10.1016/0004-3702(84)90038-9

    Article  Google Scholar 

  13. Furbach, U., Hölldobler, S., Ragni, M., Schon, C., Stolzenburg, F.: Cognitive reasoning: A personal view. KI 33(3), 209–217 (2019). https://link.springer.com/article/10.1007/s13218-019-00603-3

  14. d’Avila Garcez, A.S., Broda, K., Gabbay, D.M.: Symbolic knowledge extraction from trained neural networks: A sound approach. Artificial Intelligence 125(1–2), 155–207 (2001). https://doi.org/10.1016/S0004-3702(00)00077-1

  15. d’Avila Garcez, A., Lamb, L., Gabbay, D.: Neural-Symbolic Cognitive Reasoning. Springer, Berlin, Heidelberg (2009). https://doi.org/10.1007/978-3-540-73246-4

    Book  Google Scholar 

  16. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  17. Huang, L., Le Bras, R., Bhagavatula, C., Choi, Y.: Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2391–2401. Association for Computational Linguistics (2019). https://aclanthology.org/D19-1243/

  18. Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Müller, K.R.: Unmasking clever hans predictors and assessing what machines really learn. Nature Communications 10(1), 1096 (2019). https://www.nature.com/articles/s41467-019-08987-4

  19. Lenat, D.B.: CYC: a large-scale investment in knowledge infrastructure. Commun. ACM 38(11), 33–38 (1995). https://doi.org/10.1145/219717.219745

    Article  Google Scholar 

  20. Lewis, M., et al.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension (2019). arXiv:1910.13461

  21. Liu, Hugo, Singh, Push: Commonsense Reasoning in and Over Natural Language. In: Negoita, Mircea Gh., Howlett, Robert J.., Jain, Lakhmi C.. (eds.) KES 2004. LNCS (LNAI), vol. 3215, pp. 293–306. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30134-9_40

    Chapter  Google Scholar 

  22. Liu, Y., et al: RoBERTa: A robustly optimized BERT pretraining approach. https://arxiv.org/pdf/1907.11692

  23. McCarthy, J.: Programs with common sense (1959). https://www.cs.rit.edu/~rlaz/is2014/files/McCarthyProgramsWithCommonSense.pdf

  24. McCarthy, J.: Artificial intelligence, logic and formalizing common sense. In: Thomason, R.H. (ed.) Philosophical Logic and Artificial Intelligence, pp. 161–190. Springer, Netherlands, Dordrecht (1989). https://doi.org/10.1007/978-94-009-2448-2_6

  25. Mostafazadeh, N., Roth, M., Louis, A., Chambers, N., Allen, J.: LSDSem 2017 shared task: The Story Cloze Test. In: Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 46–51 (2017). https://aclanthology.org/W17-0906.pdf

  26. Onoe, Y., Zhang, M.J.Q., Choi, E., Durrett, G.: CREAK: A dataset for commonsense reasoning over entity knowledge. https://arxiv.org/pdf/2109.01653

  27. Pal, A., Umapathi, L.K., Sankarasubbu, M.: MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. ACM Conference on Health (2022). https://arxiv.org/pdf/2203.14371

  28. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. Tech. rep., OpenAI (2019). https://paperswithcode.com/paper/language-models-are-unsupervised-multitask

  29. Roemmele, M., Bejan, C.A., Gordon, A.S.: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In: AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, pp. 90–95 (2011), https://aaai.org/papers/02418-choice-of-plausible-alternatives-an-evaluation-of-commonsense-causal-reasoning/

  30. Sap, M., et al.: ATOMIC: an atlas of machine commonsense for if-then reasoning. Proc. AAAI Conf. Artif. Intell. 33(01), 3027–3035 (2019). https://doi.org/10.1609/aaai.v33i01.33013027

    Article  Google Scholar 

  31. Sap, M., Rashkin, H., Chen, D., LeBras, R., Choi, Y.: Social IQa: Commonsense reasoning about social interactions. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463–4473. Association for Computational Linguistics (2019). https://aclanthology.org/D19-1454/

  32. Siebert, S., Schon, C., Stolzenburg, F.: Commonsense reasoning using theorem proving and machine learning. In: Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) Machine Learning and Knowledge Extraction - 3rd IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2019, pp. 395–413. LNCS 11713, Springer Nature Switzerland, Canterbury, UK (2019). https://doi.org/10.1007/978-3-030-29726-8_25

  33. Singh, S., et al.: COM2SENSE: A commonsense reasoning benchmark with complementary sentences. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 883–898. Association for Computational Linguistics (2021). https://aclanthology.org/2021.findings-acl.78

  34. Speer, R., Chin, J., Havasi, C.: ConceptNet 5.5: An open multilingual graph of general knowledge. Proceedings of the AAAI Conference on Artificial Intelligence 31(1) (2017). https://doi.org/10.1609/aaai.v31i1.11164

  35. Talmor, A., Herzig, J., Lourie, N., Berant, J.: CommonsenseQA: A question answering challenge targeting commonsense knowledge. https://arxiv.org/pdf/1811.00937

  36. Taylor, J.E.T., Taylor, G.W.: Artificial cognition: how experimental psychology can help generate explainable artificial intelligence. Psychon. Bull. Rev. 28(2), 454–475 (2021). https://doi.org/10.3758/s13423-020-01825-5

    Article  MathSciNet  Google Scholar 

  37. Xu, Feiyu, Uszkoreit, Hans, Du, Yangzhou, Fan, Wei, Zhao, Dongyan, Zhu, Jun: Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges. In: Tang, Jie, Kan, Min-Yen., Zhao, Dongyan, Li, Sujian, Zan, Hongying (eds.) NLPCC 2019. LNCS (LNAI), vol. 11839, pp. 563–574. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32236-6_51

    Chapter  Google Scholar 

  38. Zellers, R., Bisk, Y., Schwartz, R., Choi, Y.: SWAG: A large-scale adversarial dataset for grounded commonsense inference. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 93–104. Association for Computational Linguistics (2018). https://aclanthology.org/D18-1009/

  39. Zimmer, M., et al.: Differentiable logic machines. CoRR - Computing Research Repository abs/2102.11529, Cornell University Library (2021). https://arxiv.org/abs/2102.11529, latest revision 2023

Download references

Acknowledgments

We would like to thank Oliver Obst for sharing our questionnaire on LinkedIn and Osama Siddiqui for his help testing ChatGPT on different datasets. We would also like to thank the anonymous reviewers for their thoughtful reading and comments.

Author information

Authors and Affiliations

  1. Automation and Computer Sciences Department, Harz University of Applied Sciences, Friedrichstr. 57–59, 38855, Wernigerode, Germany

    Stefanie Krause & Frieder Stolzenburg

Authors
  1. Stefanie Krause
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frieder Stolzenburg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefanie Krause .

Editor information

Editors and Affiliations

  1. Halmstad University, Halmstad, Sweden

    Sławomir Nowaczyk

  2. Warsaw University of Technology, Warsaw, Poland

    Przemysław Biecek

  3. Warsaw University, Warsaw, Poland

    Neo Christopher Chung

  4. University of Huddersfield, Huddersfield, UK

    Mauro Vallati

  5. AGH University of Science and Technology, Kraków, Poland

    Paweł Skruch

  6. AGH University of Science and Technology, Kraków, Poland

    Joanna Jaworek-Korjakowska

  7. University of Huddersfield, Huddersfield, UK

    Simon Parkinson

  8. University of Huddersfield, Huddersfield, UK

    Alexandros Nikitas

  9. Universität Osnabrück, Osnabrück, Germany

    Martin Atzmüller

  10. University of Economics Prague, Prague, Czech Republic

    Tomáš Kliegr

  11. University of Bamberg, Bamberg, Germany

    Ute Schmid

  12. Jagiellonian University, Kraków, Poland

    Szymon Bobek

  13. Jožef Stefan Institute, Ljubljana, Slovenia

    Nada Lavrac

  14. HU University of Applied Sciences Utrecht, Utrecht, The Netherlands

    Marieke Peeters

  15. Rotterdam University of Applied Sciences, Rotterdam, The Netherlands

    Roland van Dierendonck

  16. Amsterdam University of Applied Sciences, Amsterdam, The Netherlands

    Saskia Robben

  17. University of Reims Champagne-Ardenne, Reims, France

    Eunika Mercier-Laurent

  18. Istanbul Technical University, Istanbul, Türkiye

    Gülgün Kayakutlu

  19. Wroclaw University of Economics and Business, Wrocław, Poland

    Mieczyslaw Lech Owoc

  20. University of Galway, Galway, Ireland

    Karl Mason

  21. University of Galway, Galway, Ireland

    Abdul Wahid

  22. University of Calabria, Rende, Italy

    Pierangela Bruno

  23. University of Calabria, Rende, Italy

    Francesco Calimeri

  24. Marche Polytechnic University, Ancona, Italy

    Francesco Cauteruccio

  25. University of Calabria, Rende, Italy

    Giorgio Terracina

  26. University of Bamberg, Bamberg, Germany

    Diedrich Wolter

  27. Coburg University of Applied Sciences, Coburg, Germany

    Jochen L. Leidner

  28. FAU Erlangen-Nürnberg, Erlangen, Germany

    Michael Kohlhase

  29. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Krause, S., Stolzenburg, F. (2024). Commonsense Reasoning and Explainable Artificial Intelligence Using Large Language Models. In: Nowaczyk, S., et al. Artificial Intelligence. ECAI 2023 International Workshops. ECAI 2023. Communications in Computer and Information Science, vol 1947. Springer, Cham. https://doi.org/10.1007/978-3-031-50396-2_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-50396-2_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-50395-5

  • Online ISBN: 978-3-031-50396-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation