Skip to main content
SpringerLink
Log in

Improving abstractive summarization of legal rulings through textual entailment

  • Original Research
  • Published:
Artificial Intelligence and Law Aims and scope Submit manuscript

Abstract

The standard approach for abstractive text summarization is to use an encoder-decoder architecture. The encoder is responsible for capturing the general meaning from the source text, and the decoder is in charge of generating the final text summary. While this approach can compose summaries that resemble human writing, some may contain unrelated or unfaithful information. This problem is called “hallucination” and it represents a serious issue in legal texts as legal practitioners rely on these summaries when looking for precedents, used to support legal arguments. Another concern is that legal documents tend to be very long and may not be fed entirely to the encoder. We propose our method called LegalSumm for addressing these issues by creating different “views” over the source text, training summarization models to generate independent versions of summaries, and applying entailment module to judge how faithful these candidate summaries are with respect to the source text. We show that the proposed approach can select candidate summaries that improve ROUGE scores in all metrics evaluated.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Beltagy I, Peters ME, Cohan A (2020) Longformer: The long-document transformer arXiv preprint arXiv:200405150

  • Cao Z, Wei F, Li W, Li S (2018) Faithful to the original: fact aware neural abstractive summarization. In: thirty-second AAAI conference on artificial intelligence

  • Carbonell J, Goldstein J (1998) The use of mmr, diversity-based reranking for reordering documents and producing summaries. In: proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval, pp 335–336

  • Chandrasekaran MK, Yasunaga M, Radev D, Freitag D, Kan MY (2019) Overview and results: Cl-scisumm shared task 2019. In: in proceedings of joint workshop on bibliometric-enhanced information retrieval and NLP for digital libraries (BIRNDL 2019)

  • Child R, Gray S, Radford A, Sutskever I (2019) Generating long sequences with sparse transformers. arXiv preprint arXiv:190410509

  • Compton P, Jansen R (1990) Knowledge in context: A strategy for expert system maintenance. In: proceedings of the second Australian joint conference on artificial intelligence, Springer-Verlag, Berlin, Heidelberg, AI ’88, p 292–306

  • Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805

  • Falke T, Ribeiro LF, Utama PA, Dagan I, Gurevych I (2019) Ranking generated summaries by correctness: an interesting but challenging application for natural language inference. In: proceedings of the 57th annual meeting of the association for computational linguistics, pp 2214–2220

  • Fan A, Grangier D, Auli M (2018) Controllable abstractive summarization. In: proceedings of the 2nd workshop on neural machine translation and generation, 45–54

  • Feijo D, Moreira V (2018) Rulingbr: A summarization dataset for legal texts. In: computational processing of the portuguese language (PROPOR 2018), Springer International Publishing, pp 255–264

  • Feijo D, Moreira V (2019) Summarizing legal rulings: comparative experiments. in: proceedings of the international conference on recent advances in natural language processing (RANLP 2019), pp 313–322

  • Galgani F, Compton P, Hoffmann A (2012). Combining different summarization techniques for legal text. In: proceedings of the workshop on innovative hybrid approaches to the processing of textual data, Association for Computational Linguistics, pp 115–123

  • Gelbart D, Smith J (1991) Beyond boolean search: Flexicon, a legal text-based intelligent system. In: proceedings of the 3rd international conference on Artificial intelligence and law, pp 225–234

  • Goldhan D, Eckart T, Quasthoff U (2012) Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In: proceedings of the 8th international language resources and evaluation (LREC’12)

  • Goodrich B, Rao V, Liu PJ, Saleh M (2019) Assessing the factual accuracy of generated text. In: proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp 166–175

  • Grover C, Hachey B, Hughson I, Korycinski C (2003a) Automatic summarisation of legal documents. In: proceedings of the 9th international conference on artificial intelligence and law, association for computing machinery, ICAIL ’03, p 243–251

  • Grover C, Hachey B, Korycinski C (2003b). Summarising legal texts: Sentential tense and argumentative roles. In: proceedings of the HLT-NAACL 03 on text summarization workshop-volume 5, association for computational linguistics, pp 33–40

  • Guimarães JAC (2011) Elaboração de ementas jurisprudenciais: elementos teórico-metodológicos. Série Monografias do CEJ 9

  • Kitaev N, Kaiser Ł, Levskaya A (2020) Reformer: the efficient transformer arXiv preprint arXiv:200104451

  • Klein G, Kim Y, Deng Y, Senellart J, Rush A (2017). OpenNMT: Open-source toolkit for neural machine translation. In: proceedings of ACL 2017, system demonstrations, association for computational linguistics, Vancouver, Canada, pp 67–72

  • Kryściński W, McCann B, Xiong C, Socher R (2020) Evaluating the factual consistency of abstractive text summarization. In: proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 9332–9346

  • Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L (2020) BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: proceedings of the 58th annual meeting of the association for computational linguistics, association for computational linguistics, Online, pp 7871–7880, https://doi.org/10.18653/v1/2020.acl-main.703, https://aclanthology.org/2020.acl-main.703

  • Lin CY (2004) Rouge: a package for automatic evaluation of summaries. In: text summarization branches out, pp 74–81

  • Liu Y (2019) Fine-tune BERT for extractive summarization. arXiv preprint arXiv:190310318

  • Liu Y, Lapata M (2019) Fine-tune BERT for extractive summarization. arXiv preprint arXiv:190310318

  • Luijtgaarden N (2019) Automatic summarization of legal text. Utrecht University Master’s thesis

  • Matsumaru K, Takase S, Okazaki N (2020) Text summarization with pretrained encoders. In: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 3721–3731

  • Maynez J, Narayan S, Bohnet B, McDonald R (2020) On faithfulness and factuality in abstractive summarization. In: proceedings of the 58th annual meeting of the association for computational linguistics, pp 1906–1919

  • Moens MF, Uyttendaele C (1997) Automatic text structuring and categorization as a first step in summarizing legal cases. Inf Proces Manag 33(6):727–737

    Article  Google Scholar 

  • Mudrakarta PK, Taly A, Sundararajan M, Dhamdhere K (2018) Did the model understand the question? In: proceedings of the 56th annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp 1896–1906

  • Pandya V (2019) Automatic text summarization of legal cases: A hybrid approach. 5th international conference on advances in computer science and information technology (ACSTY-2019)

  • Paulus R, Xiong C, Socher R (2017) A deep reinforced model for abstractive summarization. arXiv preprint arXiv:170504304

  • Pires T, Schlinger E, Garrette D (2019) How multilingual is multilingual BERT? In: proceedings of the 57th annual meeting of the association for computational linguistics, pp 4996–5001

  • Roy A, Saffar M, Vaswani A, Grangier D (2021) Efficient content-based sparse attention with routing transformers. Transac Assoc Comput Linguistics 9:53–68

    Article  Google Scholar 

  • See A, Liu PJ, Manning CD (2017) Get to the point: Summarization with pointer-generator networks. In: proceedings of the 55th annual meeting of the association for computational linguistics 1:1073–1083

  • Tay Y, Bahri D, Yang L, Metzler D, Juan DC (2020) Sparse sinkhorn attention. In: international conference on machine learning, PMLR, pp 9438–9447

  • Turtle H (1995) Text retrieval in the legal world. Artif Intell and Law 3(1):5–54

    Article  Google Scholar 

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: proceedings of the 31st international conference on neural information processing systems, pp 6000–6010

  • Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, et al. (2019) Huggingface’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:191003771

  • Yousfi-Monod M, Farzindar A, Lapalme G (2010) Supervised machine learning for summarizing legal documents. In: Canadian conference on artificial intelligence, Springer, pp 51–62

  • Zhang J, Zhao Y, Saleh M, Liu P (2020) Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In: international conference on machine learning, PMLR, pp 11328–11339

  • Zhang X, Wei F, Zhou M (2019) HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In: proceedings of the 57th annual meeting of the association for computational linguistics, pp 5059–5069

  • Zhao Z, Cohen SB, Webber B (2020) Reducing quantity hallucinations in abstractive summarization. arXiv preprint arXiv:200913312

  • Zhong L, Zhong Z, Zhao Z, Wang S, Ashley KD, Grabmair M (2019) Automatic summarization of legal decisions using iterative masking of predictive sentences. In: proceedings of the seventeenth international conference on artificial intelligence and law, ICAIL ’19, p 163–172

Download references

Acknowledgements

This work was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001. The authors thank the two anonymous reviewers whose suggestions helped improve and clarify this manuscript.

Author information

Authors and Affiliations

  1. Institute of informatics, Universidade Federal do Rio Grande do Sul, Porto Alegre, RS, Brazil

    Diego de Vargas Feijo & Viviane P. Moreira

Authors
  1. Diego de Vargas Feijo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Viviane P. Moreira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diego de Vargas Feijo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feijo, D.d., Moreira, V.P. Improving abstractive summarization of legal rulings through textual entailment. Artif Intell Law 31, 91–113 (2023). https://doi.org/10.1007/s10506-021-09305-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10506-021-09305-4

Keywords

Search

Navigation