skip to main content
10.1145/3594536.3595170acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicailConference Proceedingsconference-collections
research-article

Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction

Published: 07 September 2023 Publication History

Abstract

Legal syllogism is a form of deductive reasoning commonly used by legal professionals to analyze cases. In this paper, we propose legal syllogism prompting (LoT), a simple prompting method to teach large language models (LLMs) for legal judgment prediction. LoT teaches only that in the legal syllogism the major premise is law, the minor premise is the fact, and the conclusion is judgment. Then the models can produce a syllogism reasoning of the case and give the judgment without any learning, fine-tuning, or examples. On CAIL2018, a Chinese criminal case dataset, we performed zero-shot judgment prediction experiments with GPT-3 models. Our results show that LLMs with LoT achieve better performance than the baseline and chain of thought prompting, the state-of-art prompting method on diverse reasoning tasks. LoT enables the model to concentrate on the key information relevant to the judgment and to correctly understand the legal meaning of acts, as compared to other methods. Our method enables LLMs to predict judgment along with law articles and justification, which significantly enhances the explainability of models.

References

[1]
Zhenwei An, Quzhe Huang, Cong Jiang, Yansong Feng, and Dongyan Zhao. 2022. Do Charge Prediction Models Learn Legal Theory? arXiv preprint arXiv:2210.17108 (2022).
[2]
Kevin D Ashley. 2019. A brief history of the changing roles of case prediction in AI and law. Law Context: A Socio-Legal J. 36 (2019), 93.
[3]
Katie Atkinson, Trevor Bench-Capon, and Danushka Bollegala. 2020. Explanation in AI and law: Past, present and future. Artificial Intelligence 289 (2020), 103387.
[4]
Trevor Bench-Capon. 2020. Explaining legal decisions using IRAC. In CEUR Workshop Proceedings, Vol. 2669. 74--83.
[5]
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. arXiv preprint arXiv:2010.02559 (2020).
[6]
Yi Feng, Chuanyi Li, and Vincent Ng. 2022. Legal Judgment Prediction: A Survey of the State of the Art. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. 5461--5469.
[7]
Łukasz Górski and Shashishekar Ramakrishna. 2021. Explainable artificial intelligence, lawyer's perspective. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law. 60--68.
[8]
Guido Governatori, Trevor Bench-Capon, Bart Verheij, Michał Araszkiewicz, Enrico Francesconi, and Matthias Grabmair. 2022. Thirty years of Artificial Intelligence and Law: the first decade. Artificial Intelligence and Law 30, 4 (2022), 481--519.
[9]
Wilson Huhn. 2001. The Use and Limits of Syllogistic Reasoning in Briefing Cases. Santa Clara L. Rev. 42 (2001), 813.
[10]
Daniel Martin Katz, Michael J Bommarito, and Josh Blackman. 2017. A general approach for predicting the behavior of the Supreme Court of the United States. PloS one 12, 4 (2017), e0174698.
[11]
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 (2022).
[12]
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1--35.
[13]
Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to predict charges for criminal cases with legal basis. arXiv preprint arXiv:1707.09168 (2017).
[14]
Masha Medvedeva, Michel Vols, and Martijn Wieling. 2018. Judicial decisions of the European Court of Human Rights: Looking into the crystal ball. In Proceedings of the conference on empirical legal studies. 24.
[15]
Richard A Posner. 1990. The problems of jurisprudence. Harvard University Press.
[16]
Swarnadeep Saha, Peter Hase, Nazneen Rajani, and Mohit Bansal. 2022. Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations. arXiv preprint arXiv:2211.07517 (2022).
[17]
TYS Santosh, Shanshan Xu, Oana Ichim, and Matthias Grabmair. 2022. Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts. arXiv preprint arXiv:2210.13836 (2022).
[18]
Giovanni Sartor, Michał Araszkiewicz, Katie Atkinson, Floris Bex, Tom van Engers, Enrico Francesconi, Henry Prakken, Giovanni Sileno, Frank Schilder, Adam Wyner, et al. 2022. Thirty years of Artificial Intelligence and Law: the second decade. Artificial Intelligence and Law (2022), 1--37.
[19]
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.00093 (2023).
[20]
Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. arXiv preprint arXiv:2203.13224 (2022).
[21]
Dietrich Trautmann, Alina Petrova, and Frank Schilder. 2022. Legal Prompt Engineering for Multilingual Legal Judgement Prediction. arXiv preprint arXiv:2212.02199 (2022).
[22]
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022).
[23]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022).
[24]
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv:1807.02478 (2018).
[25]
Fangyi Yu, Lee Quartey, and Frank Schilder. 2022. Legal Prompting: Teaching a Language Model to Think Like a Lawyer. arXiv preprint arXiv:2212.01326 (2022).
[26]
Haoxi Zhong, Chaojun Xiao, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. Overview of CAIL2018: Legal judgment prediction competition. arXiv preprint arXiv:1810.05851 (2018).

Cited By

View all
  • (2025)CoLE: A collaborative legal expert prompting framework for large language models in lawKnowledge-Based Systems10.1016/j.knosys.2025.113052311(113052)Online publication date: Feb-2025
  • (2025)Simulating judicial trial logic: Dual residual cross-attention learning for predicting legal judgment in long documentsExpert Systems with Applications10.1016/j.eswa.2024.125462261(125462)Online publication date: Feb-2025
  • (2024)Implementación de inteligencia artificial en el derecho bolivianoImplementation of artificial intelligence in Bolivian lawImplementação da inteligência artificial na legislação bolivianaYUYAY: Estrategias, Metodologías & Didácticas Educativas10.59343/yuyay.v3i2.683:2(120-138)Online publication date: 30-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICAIL '23: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law
June 2023
499 pages
ISBN:9798400701979
DOI:10.1145/3594536
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

  • IAAIL: Intl Asso for Artifical Intel & Law

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 September 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. chain of thought
  2. large language models
  3. legal judgment prediction
  4. legal syllogism

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICAIL 2023
Sponsor:
  • IAAIL

Acceptance Rates

Overall Acceptance Rate 69 of 169 submissions, 41%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)274
  • Downloads (Last 6 weeks)31
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)CoLE: A collaborative legal expert prompting framework for large language models in lawKnowledge-Based Systems10.1016/j.knosys.2025.113052311(113052)Online publication date: Feb-2025
  • (2025)Simulating judicial trial logic: Dual residual cross-attention learning for predicting legal judgment in long documentsExpert Systems with Applications10.1016/j.eswa.2024.125462261(125462)Online publication date: Feb-2025
  • (2024)Implementación de inteligencia artificial en el derecho bolivianoImplementation of artificial intelligence in Bolivian lawImplementação da inteligência artificial na legislação bolivianaYUYAY: Estrategias, Metodologías & Didácticas Educativas10.59343/yuyay.v3i2.683:2(120-138)Online publication date: 30-Jul-2024
  • (2024)Can Large Language Models Assess Serendipity in Recommender Systems?Journal of Advanced Computational Intelligence and Intelligent Informatics10.20965/jaciii.2024.p126328:6(1263-1272)Online publication date: 20-Nov-2024
  • (2024)Legal Judgment Prediction with LLM and Graph Contrastive Learning NetworksProceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence10.1145/3709026.3709068(424-432)Online publication date: 6-Dec-2024
  • (2024)LAPIS: Language Model-Augmented Police Investigation SystemProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680044(4637-4644)Online publication date: 21-Oct-2024
  • (2024)LawLLM: Law Large Language Model for the US Legal SystemProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680020(4882-4889)Online publication date: 21-Oct-2024
  • (2024)Learning Defendant-aware Label Representation for Multi-Defendant Charge Prediction2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650211(1-8)Online publication date: 30-Jun-2024
  • (2024)It cannot be right if it was written by AI: on lawyers’ preferences of documents perceived as authored by an LLM vs a humanArtificial Intelligence and Law10.1007/s10506-024-09422-wOnline publication date: 3-Dec-2024
  • (2024)AI, Law and beyond. A transdisciplinary ecosystem for the future of AI & LawArtificial Intelligence and Law10.1007/s10506-024-09404-yOnline publication date: 16-May-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media