skip to main content
10.1145/3209978.3210161acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
short-paper

A Test Collection for Evaluating Legal Case Law Search

Published: 27 June 2018 Publication History

Abstract

Test collection based evaluation represents the standard of evalua- tion for information retrieval systems. Legal IR, more speci cally case law retrieval, has no such standard test collection for evalua- tion. In this paper, we present a test collection for use in evaluating case law search, being the retrieval of judicial decisions relevant to a particular legal question. The collection is made available at ielab.io/caselaw.

References

[1]
Omar Alonso, Catherine C Marshall, and Marc A Najork . 2013. A human-centered framework for ensuring reliability on crowdsourced labeling tasks First AAAI Conference on Human Computation and Crowdsourcing.
[2]
Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas . 2016. UQV100: A test collection with query variability. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 725--728.
[3]
Jody J Daniels and Edwina L Rissland . 1997. What you saw is what you want: Using cases to seed information retrieval International Conference on Case-Based Reasoning. Springer, 325--336.
[4]
Phong-Khac Do, Huy-Tien Nguyen, Chien-Xuan Tran, Minh-Tien Nguyen, and Minh-Le Nguyen . 2017. Legal Question Answering using Ranking SVM and Deep Convolutional Neural Network. arXiv preprint arXiv:1703.05320 (2017).
[5]
James H Fowler, Timothy R Johnson, James F Spriggs, Sangick Jeon, and Paul J Wahlbeck . 2007. Network Analysis and the Law: Measuring the Legal Importance of Precedents at the US Supreme Court. Political Analysis (2007), 324--346.
[6]
Dephne Gelbart and JC Smith . 1991. Beyond boolean search: FLEXICON, a legal tex-based intelligent system Proceedings of the 3rd international conference on Artificial intelligence and law. ACM, 225--234.
[7]
Kevin Gerson . 1999. Evaluating Legal Information Retrieval Systems: How do the Ranked-retrieval Methods of WESTLAW and LEXIS Measure Up? Legal Reference Services Quarterly Vol. 17, 4 (1999), 53--67.
[8]
Marios Koniaris, Ioannis Anagnostopoulos, and Yannis Vassiliou . 2016. Multi-dimension Diversification in Legal Information Retrieval International Conference on Web Information Systems Engineering (WISE'16). Springer, 174--189.
[9]
Marios Koniaris, Ioannis Anagnostopoulos, and Yannis Vassiliou . 2017. Evaluation of Diversification Techniques for Legal Information Retrieval. Algorithms Vol. 10, 1 (2017), 22.
[10]
Daniel Locke and Guido Zuccon . 2017. Automatic cited decision retrieval: Working notes of Ielab for FIRE Legal Track Precedence Retrieval Task. In Working notes of FIRE 2017-Forum for Information Retrieval Evaluation (CEUR Workshop Proceedings).
[11]
Daniel Locke, Guido Zuccon, and Harrisen Scells . 2017. Automatic Query Generation from Legal Texts for Case Law Retrieval. Springer International Publishing, 181--193.
[12]
E. Maddalena, M. Basaldella, D. De Nart, D. Degl'Innocenti, S. Mizzaro, and G. Demartini . 2016. Crowdsourcing relevance assessments: The unexpected benefits of limiting the time to judge. In HCOMP'16.
[13]
Arpan Mandal, Kripabandhu Ghosh, Arnab Bhattacharya, Arindam Pal, and Saptarshi Ghosh . 2017. Overview of the FIRE 2017 track: Information Retrieval from Legal Documents (IRLeD) Working notes of FIRE 2017-Forum for Information Retrieval Evaluation (CEUR Workshop Proceedings).
[14]
Bhaskar Mitra and Nick Craswell . 2017. Neural Models for Information Retrieval. arXiv preprint arXiv:1705.01509 (2017).
[15]
Anselmo Pe nas, Pamela Forner, Richard Sutcliffe, Álvaro Rodrigo, Corina Foruascu, I naki Alegria, Danilo Giampiccolo, Nicolas Moreau, and Petya Osenova . 2010. Overview of ResPubliQA 2009: Question Answering Evaluation Over European Legislation. Multilingual information access evaluation I. text retrieval experiments (2010), 174--196.
[16]
Harrisen Scells, Guido Zuccon, Bevan Koopman, Anthony Deacon, Leif Azzopardi, and Shlomo Geva . 2017. A test collection for evaluating retrieval of studies for inclusion in systematic reviews. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1237--1240.
[17]
Colin Tapper . 1974. Legal Information Retrieval by Computer: Applications and Implications. McGill Law Journal Vol. 20 (1974), 26.
[18]
Howard Turtle . 1994. Natural Language vs. Boolean Query Evaluation: A Comparison of Retrieval Performance. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 212--220.
[19]
Robert A Wilson . 1962. Computer retrieval of case law. Sw. LJ Vol. 16 (1962), 409.

Cited By

View all
  • (2024)Searching in Professional Instant Messaging Applications: User Behaviour, Intent, and Pain-pointsProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698417(82-91)Online publication date: 8-Dec-2024
  • (2024)Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language ModelsProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680025(4554-4562)Online publication date: 21-Oct-2024
  • (2024)I beg to differ: how disagreement is handled in the annotation of legal machine learning data setsArtificial Intelligence and Law10.1007/s10506-023-09369-432:3(839-862)Online publication date: 1-Sep-2024
  • Show More Cited By

Index Terms

  1. A Test Collection for Evaluating Legal Case Law Search

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
    June 2018
    1509 pages
    ISBN:9781450356572
    DOI:10.1145/3209978
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 June 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. case law search
    2. legal
    3. test collection

    Qualifiers

    • Short-paper

    Funding Sources

    • Google
    • Australian Research Council

    Conference

    SIGIR '18
    Sponsor:

    Acceptance Rates

    SIGIR '18 Paper Acceptance Rate 86 of 409 submissions, 21%;
    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)27
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 15 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Searching in Professional Instant Messaging Applications: User Behaviour, Intent, and Pain-pointsProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698417(82-91)Online publication date: 8-Dec-2024
    • (2024)Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language ModelsProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680025(4554-4562)Online publication date: 21-Oct-2024
    • (2024)I beg to differ: how disagreement is handled in the annotation of legal machine learning data setsArtificial Intelligence and Law10.1007/s10506-023-09369-432:3(839-862)Online publication date: 1-Sep-2024
    • (2023)CLosER: Conversational Legal Longformer with Expertise-Aware Passage Response Ranker for Long ContextsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614812(25-35)Online publication date: 21-Oct-2023
    • (2023)Techniques to Estimate the Status of Legal Proceedings Considering Sequential Text Data2023 International Conference on Emerging Smart Computing and Informatics (ESCI)10.1109/ESCI56872.2023.10099995(1-6)Online publication date: 1-Mar-2023
    • (2023)Semantic matching based legal information retrieval system for COVID-19 pandemicArtificial Intelligence and Law10.1007/s10506-023-09354-x32:2(397-426)Online publication date: 14-Mar-2023
    • (2023)ECIR 2023 Workshop: Legal Information RetrievalAdvances in Information Retrieval10.1007/978-3-031-28241-6_46(412-419)Online publication date: 16-Mar-2023
    • (2022)PARM: A Paragraph Aggregation Retrieval Model for Dense Document-to-Document RetrievalAdvances in Information Retrieval10.1007/978-3-030-99736-6_2(19-34)Online publication date: 10-Apr-2022
    • (2021)VerbCL: A Dataset of Verbatim Quotes for Highlight Extraction in Case LawProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482021(4554-4563)Online publication date: 26-Oct-2021
    • (2021)LeCaRD: A Legal Case Retrieval Dataset for Chinese Law SystemProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3463250(2342-2348)Online publication date: 11-Jul-2021
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media