research-article

Automatic Summarization of Legal Decisions using Iterative Masking of Predictive Sentences

Authors:

Kevin D. Ashley,

Matthias GrabmairAuthors Info & Claims

ICAIL '19: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law

Pages 163 - 172

https://doi.org/10.1145/3322640.3326728

Published: 17 June 2019 Publication History

Abstract

We report on a pilot experiment in automatic, extractive summarization of legal cases concerning Post-traumatic Stress Disorder from the US Board of Veterans' Appeals. We hypothesize that length-constrained extractive summaries benefit from choosing among sentences that are predictive for the case outcome. We develop a novel train-attribute-mask pipeline using a CNN classifier to iteratively select predictive sentences from the case, which measurably improves prediction accuracy on partially masked decisions. We then select a subset for the summary through type classification, maximum marginal relevance, and a summarization template. We use ROUGE metrics and a qualitative survey to evaluate generated summaries along with expert-extracted and expert-drafted summaries. We show that sentence predictiveness does not reliably cover all decision-relevant aspects of a case, illustrate that lexical overlap metrics are not well suited for evaluating legal summaries, and suggest that future work should focus on case-aspect coverage.

References

[1]

L. K. Branting, A. Yeh, B. Weiss, E. Merkhofer, and B. Brown. 2015. Inducing predictive models for decision support in administrative adjudication. In AI Approaches to the Complexity of Legal Systems. 465--477.

[2]

J. Carbonell and J. Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. Proc. SIGIR 1998, 335--336.

Digital Library

[3]

D. Cer, Y. Yang, S.-Y.i Kong, N. Hua, N. Limtiaco, R. St. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, B. Strope, and R. Kurzweil. 2018. Universal Sentence Encoder for English. In Proc. EMNLP 2018. ACL, 169--174.

[4]

G. Erkan and D. R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. J. of Artificial Intelligence Research 22 (2004), 457--479.

Digital Library

[5]

A. Farzindar and G. Lapalme. 2004. Legal text summarization by exploration of the thematic structures and argumentative roles. Text Summarization Branches Out, Conference held in conjunction with ACL 2004 (2004), 27--38.

[6]

K. Ganesan, C. Zhai, and J. Han. 2010. Opinosis: a graph-based approach to abstractive summarization of highly redundant opinions. Proc. COLING 2010, 340--348.

Digital Library

[7]

Y. Gong and X. Liu. 2001. Generic text summarization using relevance measure and latent semantic analysis. Proc. SIGIR 2001, 19--25.

Digital Library

[8]

C. Grover, B. Hachey, I. Hughson, and C. Korycinski. 2003. Automatic summarisation of legal documents. Proc. ICAIL 2003, 243--251.

Digital Library

[9]

C. Grover, B. Hachey, and C. Korycinski. 2003. Summarising legal texts: Sentential tense and argumentative roles. Proc. of the HLT-NAACL 2003 Workshop on Text summarization, 33--40.

Digital Library

[10]

B. Hachey and C. Grover. 2006. Extractive summarisation of legal texts. Artificial Intelligence and Law 14, 4 (2006), 305--345.

Digital Library

[11]

M.-Y. Kim, Y. Xu, and R. Goebel. 2012. Summarization of legal texts with high cohesion and automatic compression rate. JSAI International Symposium on Artificial Intelligence, 190--204.

Digital Library

[12]

Y. Kim. 2014. Convolutional Neural Networks for Sentence Classification. Proc. EMNLP 2014, 1746--1751.

[13]

R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, and S. Fidler. 2015. Skip-thought vectors. Advances in neural information processing systems, 3294--3302.

Digital Library

[14]

C.-Y. Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, Conference held in conjunction with ACL 2004 (2004), 74--81.

[15]

A. M. Rush, S. Chopra, and J. Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. Proc. EMNLP 2015, 379--389.

[16]

J. Savelka, V. R. Walker, M. Grabmair, and K. D. Ashley. 2017. Sentence boundary detection in adjudicatory decisions in the united states. Traitement automatique des langues 58, 2 (2017), 21--45.

[17]

F. Schilder and H. Molina-Salgado. 2006. Evaluating a summarizer for legal text with a large text collection. 3rd Midwestern Computational Linguistics Colloquium (MCLC).

[18]

C. Stab and I. Gurevych. 2014. Identifying Argumentative Discourse Structures in Persuasive Essays. Proc. EMNLP 2014, 46--56.

[19]

M. Sundararajan, A. Taly, and Q. Yan. 2017. Axiomatic Attribution for Deep Networks. Proc. ICML 2017, 3319--3328.

Digital Library

[20]

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 5998--6008.

Digital Library

[21]

T. Vodolazova, E. Lloret, R. Muñoz, and M. Palomar. 2013. The role of statistical and semantic features in single-document extractive summarization. Artificial Intelligence Research 2, 3 (2013), 35.

[22]

V. R. Walker, J. H. Han, X. Ni, and K. Yoseda. 2017. Semantic types for computational legal reasoning: propositional connectives and sentence roles in the veterans' claims dataset. Proc. ICAIL 2017, 217--226.

Digital Library

[23]

Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy. 2016. Hierarchical attention networks for document classification. Proc. NAACL 2016, 1480--1489.

[24]

M. Yousfi-Monod, A. Farzindar, and G. Lapalme. 2010. Supervised machine learning for summarizing legal documents. Canadian Conference on Artificial Intelligence, 51--62.

Digital Library

Cited By

Benedetto ILa Quatra MCagliero L(2025)LegItBART: a summarization model for Italian legal documentsArtificial Intelligence and Law10.1007/s10506-025-09436-yOnline publication date: 22-Feb-2025
https://doi.org/10.1007/s10506-025-09436-y
Benedetto ICagliero LFerro MTarasconi FBernini CGiacalone G(2025)Leveraging large language models for abstractive summarization of Italian legal newsArtificial Intelligence and Law10.1007/s10506-025-09431-3Online publication date: 3-Feb-2025
https://doi.org/10.1007/s10506-025-09431-3
McMeel D(2024)Digital systems supporting improvements to material information disseminationSustainability and Toxicity of Building Materials10.1016/B978-0-323-98336-5.00028-5(627-646)Online publication date: 2024
https://doi.org/10.1016/B978-0-323-98336-5.00028-5
Show More Cited By

Index Terms

Automatic Summarization of Legal Decisions using Iterative Masking of Predictive Sentences
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Information extraction

Recommendations

Keyword-based Augmentation Method to Enhance Abstractive Summarization for Legal Documents
ICAIL '23: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law

Since state-of-the-art machine learning models like Transformers can not handle long text well, the quality of the summarization of legal documents is still not desirable. In order to improve the ability of machine learning models to understand the ...
Extraction of unexpected sentences: A sentiment classification assessed approach

Sentiment classification in text documents is an active data mining research topic in opinion retrieval and analysis. Different from previous studies concentrating on the development of effective classifiers, in this paper, we focus on the extraction ...
EPICURE - Aspect-based Multimodal Review Summarization
WebSci '18: Proceedings of the 10th ACM Conference on Web Science

Restaurant reviews are popular and a valuable source of information. Often, large number of reviews are written for restaurants which warrants the need for automated summarization systems. In this paper we present epicure, a novel text and image ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICAIL '19: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law

June 2019

312 pages

ISBN:9781450367547

DOI:10.1145/3322640

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

In-Cooperation

Univ. of Montreal: University of Montreal
AAAI
IAAIL: Intl Asso for Artifical Intel & Law

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICAIL '19

Sponsor:

SIGAI

ICAIL '19: Seventeenth International Conference on Artificial Intelligence and Law

June 17 - 21, 2019

QC, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 69 of 169 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
637
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)7

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Benedetto ILa Quatra MCagliero L(2025)LegItBART: a summarization model for Italian legal documentsArtificial Intelligence and Law10.1007/s10506-025-09436-yOnline publication date: 22-Feb-2025
https://doi.org/10.1007/s10506-025-09436-y
Benedetto ICagliero LFerro MTarasconi FBernini CGiacalone G(2025)Leveraging large language models for abstractive summarization of Italian legal newsArtificial Intelligence and Law10.1007/s10506-025-09431-3Online publication date: 3-Feb-2025
https://doi.org/10.1007/s10506-025-09431-3
McMeel D(2024)Digital systems supporting improvements to material information disseminationSustainability and Toxicity of Building Materials10.1016/B978-0-323-98336-5.00028-5(627-646)Online publication date: 2024
https://doi.org/10.1016/B978-0-323-98336-5.00028-5
Lüders KStohlmann B(2024)Classifying proportionality - identification of a legal argumentArtificial Intelligence and Law10.1007/s10506-024-09415-9Online publication date: 3-Aug-2024
https://doi.org/10.1007/s10506-024-09415-9
Ragazzi LMoro GGuidi SFrisoni G(2024)LAWSUIT: a LArge expert-Written SUmmarization dataset of ITalian constitutional court verdictsArtificial Intelligence and Law10.1007/s10506-024-09414-wOnline publication date: 9-Sep-2024
https://doi.org/10.1007/s10506-024-09414-w
Deroy AGhosh KGhosh S(2024)Applicability of large language models and generative models for legal case judgement summarizationArtificial Intelligence and Law10.1007/s10506-024-09411-zOnline publication date: 15-Jul-2024
https://doi.org/10.1007/s10506-024-09411-z
Dadas SKozłowski MPoświata RPerełkiewicz MBiałas MGrębowiec M(2024)A support system for the detection of abusive clauses in B2C contractsArtificial Intelligence and Law10.1007/s10506-024-09408-8Online publication date: 26-Jun-2024
https://doi.org/10.1007/s10506-024-09408-8
Benedetto IKoudounas AVaiani LPastor ECagliero LTarasconi FBaralis E(2024)Boosting court judgment prediction and explanation using legal entitiesArtificial Intelligence and Law10.1007/s10506-024-09397-8Online publication date: 18-Mar-2024
https://doi.org/10.1007/s10506-024-09397-8
Goswami SSaini NShukla S(2024)Incorporating Domain Knowledge in Multi-objective Optimization Framework for Automating Indian Legal Case SummarizationPattern Recognition10.1007/978-3-031-78495-8_17(265-280)Online publication date: 4-Dec-2024
https://doi.org/10.1007/978-3-031-78495-8_17
Freire Dde Almeida ADias MRivolli APereira Fde Godoi Gde Carvalho A(2024)LegalSum: Towards Tool for Evaluation for Extractive Summarization of Brazilian LawsuitsInformation Technology and Systems10.1007/978-3-031-54256-5_24(258-267)Online publication date: 27-Feb-2024
https://doi.org/10.1007/978-3-031-54256-5_24
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten