Skip to main content
Springer Nature Link
Log in

Extracting Norms from Contracts Via ChatGPT

Opportunities and Challenges

  • Conference paper
  • First Online:
Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XVII (COINE 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 15398))

  • 71 Accesses

Abstract

We investigate the effectiveness of ChatGPT in extracting norms from contracts. Norms provide a natural way to engineer multiagent systems by capturing how to govern the interactions between two or more autonomous parties. We extract norms of commitment, prohibition, authorization, and power, along with associated norm elements (the parties involved, antecedents, and consequents) from contracts. Our investigation reveals ChatGPT’s effectiveness and limitations in norm extraction from contracts. ChatGPT demonstrates promising performance in norm extraction without requiring training or fine-tuning, thus obviating the need for annotated data, which is not generally available in this domain. However, we found some limitations of ChatGPT in extracting these norms that lead to incorrect norm extractions. The limitations include oversight of crucial details, hallucination, incorrect parsing of conjunctions, and inferring incorrect norm types and elements. Enhanced norm extraction from contracts can foster the development of more transparent and trustworthy formal agent interaction specifications, thereby contributing to improving multiagent systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aires, J.P., Meneguzzi, F.: Norm conflict identification using a convolutional neural network. In: International Workshops on Coordination, Organizations, Institutions, Norms, and Ethics (COINE) for Governance of Multi-Agent Systems XIII, LNCS, vol. 12298, pp. 3–19, Springer (2020). https://doi.org/10.1007/978-3-030-72376-7_1

  2. AlAfnan, M.A., Dishari, S., Jovic, M., Lomidze, K.: ChatGPT as an educational tool: opportunities, challenges, and recommendations for communication, business writing, and composition courses. J. Artif. Intell. Technol. 3(2), 60–68 (2023). https://doi.org/10.37965/jait.2023.0184

  3. Ali, S.R., Dobbs, T.D., Hutchings, H.A., Whitaker, I.S.: Using ChatGPT to write patient clinic letters. Lancet Digital Health 5(4), e179–e181 (2023). https://doi.org/10.1016/S2589-7500(23)00048-1

  4. Baidoo-Anu, D., Owusu Ansah, L.: Education in the era of generative artificial intelligence (AI): understanding the potential benefits of ChatGPT in promoting teaching and learning. J. AI 7(1), 52–62 (2023). https://doi.org/10.61969/jai.1337500

  5. Bubeck, S., et al.: Sparks of artificial general intelligence: early experiments with GPT-4. arXiv preprint arXiv:2303.12712 (2023)

  6. Chalkidis, I., Androutsopoulos, I., Michos, A.: Obligation and prohibition extraction using hierarchical RNNs. In: Gurevych, Iryna and Miyao, Yusuke (ed.) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 254–259, Association for Computational Linguistics, Melbourne, Australia (2018). https://doi.org/10.18653/v1/P18-2041

  7. Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Androutsopoulos, I.: Neural contract element extraction revisited. In: Workshop on Document Intelligence at NeurIPS 2019 (2019). https://openreview.net/forum?id=B1x6fa95UH

  8. Chopra, A.K., Singh, M.P.: Custard: computing norm states over information stores. In: Proceedings of the 15th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pp. 1096–1105, IFAAMAS, Singapore (2016). https://doi.org/10.5555/2936924.2937085

  9. Chopra, A.K., Singh, M.P.: Accountability as a foundation for requirements in sociotechnical systems. IEEE Internet Comput. (IC) 25(6), 33–41 (2021). https://doi.org/10.1109/MIC.2021.3106835

    Article  MATH  Google Scholar 

  10. Christie V, S.H., Chopra, A.K., Singh, M.P.: Hercule: representing and reasoning about norms as a foundation for declarative contracts over blockchain. IEEE Internet Comput. (IC) 25(4), 67–75 (2021). https://doi.org/10.1109/MIC.2021.3080982

  11. Cox, C., Tzoc, E.: ChatGPT: implications for academic libraries. Coll. Res. Libr. News 84(3), 99 (2023). https://doi.org/10.5860/crln.84.3.99

    Article  Google Scholar 

  12. Curtotti, M., McCreath, E.: Corpus based classification of text in Australian contracts. In: Indurkhya, N., Zwarts, S. (eds.) Proceedings of the Australasian Language Technology Association Workshop 2010, pp. 18–26, Melbourne, Australia (2010). URL https://aclanthology.org/U10-1005

  13. D’Agostini Bueno, T.C., Von Wangenheim, C.G., Da Silva Mattos, E., Hoeschl, H.C., Barcia, R.M.: Jurisconsulto: retrieval in jurisprudencial text bases using juridical terminology. In: Proceedings of the 7th International Conference on Artificial Intelligence and Law, pp. 147–155. ICAIL, Association for Computing Machinery, New York, NY, USA (1999). https://doi.org/10.1145/323706.323789

  14. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 17th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 4171–4186, Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1423

  15. Gao, X., Singh, M.P.: Extracting normative relationships from business contracts. In: International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS, Paris, France, May 5-9, 2014, pp. 101–108, IFAAMAS/ACM (2014). http://dl.acm.org/citation.cfm?id=2615751

  16. Gao, X., Singh, M.P.: Mining contracts for business events and temporal constraints in service engagements. IEEE Trans. Serv. Comput. 7(3), 427–439 (2014). https://doi.org/10.1109/TSC.2013.21

    Article  MATH  Google Scholar 

  17. George, A.S., George, A.S.H.: A review of ChatGPT AI’s impact on several business sectors. Partners Univers. Int. Innovation J. 1(1), 9–23 (2023). https://doi.org/10.5281/zenodo.7644359

    Article  MATH  Google Scholar 

  18. Hashmi, M.: A methodology for extracting legal norms from regulatory documents. In: 2015 IEEE 19th International Enterprise Distributed Object Computing Workshop, pp. 41–50 (2015). https://doi.org/10.1109/EDOCW.2015.29

  19. Hendrycks, D., Burns, C., Chen, A., Ball, S.: CUAD: an expert-annotated NLP dataset for legal contract review. In: Vanschoren, J., Yeung, S. (eds.) Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks, December 2021 (2021). https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/6ea9ab1baa0efb9e19094440c317e21b-Abstract-round1.html

  20. Howard, A., Hope, W., Gerada, A.: ChatGPT and antimicrobial advice: the end of the consulting infection doctor? Lancet. Infect. Dis. 23(4), 405–406 (2023)

    Article  MATH  Google Scholar 

  21. Jalil, S., Rafi, S., LaToza, T.D., Moran, K., Lam, W.: ChatGPT and software testing education: promises & perils. In: IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 4130–4137 (2023). https://doi.org/10.1109/ICSTW58534.2023.00078

  22. Kafalı, Ö., Ajmeri, N., Singh, M.P.: Desen: specification of sociotechnical systems via patterns of regulation and control. ACM Trans. Softw. Eng. Methodol. (TOSEM) 29(1), 7:1–7:50 (2020). https://doi.org/10.1145/3365664

  23. Kampik, T., et al.: Governance of autonomous agents on the web: challenges and opportunities. ACM Trans. Internet Technol. (TOIT) 22(4), 104:1–104:31 (2022). https://doi.org/10.1145/3507910

  24. Kashefi, A., Mukerji, T.: ChatGPT for programming numerical methods. J. Mach. Learn. Model. Comput. 4(2), 1–74 (2023)

    Google Scholar 

  25. Kasneci, E., et al.: ChatGPT for good? On opportunities and challenges of large language models for education. Learn. Individ. Differ. 103, 102274 (2023). https://doi.org/10.1016/j.lindif.2023.102274

  26. Khan, R.A., Jawaid, M., Khan, A.R., Sajjad, M.: ChatGPT - reshaping medical education and clinical management. Pak. J. Med. Sci. 39(2), 605–607 (2023). https://doi.org/10.12669/pjms.39.2.7653

  27. Kiyavitskaya, N., et al.: Automating the extraction of rights and obligations for regulatory compliance. In: Li, Q., Spaccapietra, S., Yu, E., Olivé, A. (eds.) ER 2008. LNCS, vol. 5231, pp. 154–168. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87877-3_13

    Chapter  MATH  Google Scholar 

  28. Koreeda, Y., Manning, C.: ContractNLI: a dataset for document-level natural language inference for contracts. In: Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1907–1919. Association for Computational Linguistics, Punta Cana, Dominican Republic (2021). https://doi.org/10.18653/v1/2021.findings-emnlp.164

  29. Lee, J., Yi, J.S., Son, J.: Development of automatic-extraction model of poisonous clauses in international construction contracts using rule-based NLP. J. Comput. Civ. Eng. 33(3), 04019003 (2019). https://doi.org/10.1061/(ASCE)CP.1943-5487.0000807

    Article  MATH  Google Scholar 

  30. Leivaditi, S., Rossi, J., Kanoulas, E.: A benchmark for lease contract review. arXiv preprint arXiv:2010.10386 (2020)

  31. Lippi, M., et al.: Automated detection of unfair clauses in online consumer contracts. In: Legal Knowledge and Information Systems - JURIX 2017: The 13th Annual Conference, Luxembourg, 13-15, Frontiers in Artificial Intelligence and Applications, vol. 302, pp. 145–154. IOS Press (2017). https://doi.org/10.3233/978-1-61499-838-9-145

  32. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  33. Mahala, G., Kafalı, Ö., Dam, H.K., Ghose, A., Singh, M.P.: A normative approach for resilient multiagent systems. JAAMAS 37(2), 46:1–46:40 (2023). https://doi.org/10.1007/s10458-023-09627-4

  34. Mamakas, D., Tsotsi, P., Androutsopoulos, I., Chalkidis, I.: Processing long legal documents with pre-trained transformers: modding LegalBERT and longformer. In: Proceedings of the Natural Legal Language Processing Workshop 2022, pp. 130–142. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid) (2022). https://doi.org/10.18653/v1/2022.nllp-1.11

  35. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training. OpenAI (2018). https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf

  36. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners, Technical report, Open AI (2019). https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf

  37. Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: unanswerable questions for SQuAD. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789. Association for Computational Linguistics, Melbourne, Australia (2018). https://doi.org/10.18653/v1/P18-2124

  38. Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392. Association for Computational Linguistics, Austin, Texas (2016). https://doi.org/10.18653/v1/D16-1264

  39. Sainani, A., Anish, P.R., Joshi, V., Ghaisas, S.: Extracting and classifying requirements from software engineering contracts. In: IEEE 28th International Requirements Engineering Conference (RE), pp. 147–157 (2020). https://doi.org/10.1109/RE48521.2020.00026

  40. Sallam, M.: ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare 11(6) (2023). https://doi.org/10.3390/healthcare11060887

  41. Sharifi, S., Parvizimosaed, A., Amyot, D., Logrippo, L., Mylopoulos, J.: Symboleo: towards a specification language for legal contracts. In: 2020 IEEE 28th International Requirements Engineering Conference (RE), pp. 364–369 (2020). https://doi.org/10.1109/RE48521.2020.00049

  42. Singh, M.P.: Norms as a basis for governing sociotechnical systems. ACM Trans. Intell. Syst. Technol. (TIST) 5(1), 21:1–21:23 (2013). https://doi.org/10.1145/2542182.2542203

  43. Singh, M.P., Chopra, A.K.: Computational governance and violable contracts for blockchain applications. IEEE Comput. 53(1), 53–62 (2020). https://doi.org/10.1109/MC.2019.2947372

    Article  MATH  Google Scholar 

  44. Singh, M.P., Gao, X.: A gray box conceptual model for accountability and ethics in business contracts. IEEE Internet Comput. (IC) 25(4), 13–19 (2021). https://doi.org/10.1109/MIC.2021.3083295

    Article  MATH  Google Scholar 

  45. Sleimi, A., Sannier, N., Sabetzadeh, M., Briand, L.C., Ceci, M., Dann, J.: An automated framework for the extraction of semantic legal metadata from legal texts. Empir. Softw. Eng. 26(3), 43 (2021). https://doi.org/10.1007/S10664-020-09933-5

    Article  Google Scholar 

  46. Taecharungroj, V.: What can ChatGPT do? Analyzing early reactions to the innovative AI chatbot on twitter. Big Data Cogn. Comput. 7(1) (2023). https://doi.org/10.3390/bdcc7010035

  47. Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., Long Beach, CA (2017). https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

  48. Von Wright, G.H.: Norm and action: a logical enquiry. In: International Library of Philosophy and Scientific Method. Humanities Press, New York (1963)

    MATH  Google Scholar 

  49. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.: GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353–355. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/W18-5446

  50. Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sentence understanding through inference. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112–1122. Association for Computational Linguistics, New Orleans, Louisiana (2018). https://doi.org/10.18653/v1/N18-1101

  51. Xu, W., Deng, Y., Lei, W., Zhao, W., Chua, T.S., Lam, W.: ConReader: exploring implicit relations in contracts for contract clause extraction. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 2581–2594. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (2022). https://doi.org/10.18653/v1/2022.emnlp-main.166

  52. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., Vancouver (2019). https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf

Download references

Acknowledgments

Thanks to the US NSF (grant IIS-1908374) for support for this research.

Author information

Authors and Affiliations

  1. North Carolina State University, Raleigh, USA

    Amanul Haque & Munindar P. Singh

Authors
  1. Amanul Haque
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Munindar P. Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amanul Haque .

Editor information

Editors and Affiliations

  1. University of Otago, Dunedin, New Zealand

    Stephen Cranefield

  2. IMT Mines Saint-Etienne, Saint‐Étienne, France

    Luis Gustavo Nardin

  3. Ontario Tech University, Ontario, ON, Canada

    Nathan Lloyd

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Haque, A., Singh, M.P. (2025). Extracting Norms from Contracts Via ChatGPT. In: Cranefield, S., Nardin, L.G., Lloyd, N. (eds) Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XVII. COINE 2024. Lecture Notes in Computer Science(), vol 15398. Springer, Cham. https://doi.org/10.1007/978-3-031-82039-7_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-82039-7_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-82038-0

  • Online ISBN: 978-3-031-82039-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics