short-paper

Conversational vs Traditional: Comparing Search Behavior and Outcome in Legal Case Retrieval

Authors:

Shaoping MaAuthors Info & Claims

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1622 - 1626

https://doi.org/10.1145/3404835.3463064

Published: 11 July 2021 Publication History

Abstract

In recent years, legal case retrieval has attracted much attention in the IR research community. It aims to retrieve supporting cases for a given query case and contributes to better legal systems. While using a legal case retrieval system, users always feel difficult to construct accurate queries to express their information need, especially when they lack sufficient domain knowledge. Since conversational search has been widely recognized to fulfill users' complex and exploratory information need, we investigate whether conversational search paradigm can be adopted to improve users' legal case retrieval experience. We design a laboratory-based study to collect users' interaction behaviors and explicit feedback signals while using traditional and agent-mediated conversational legal case retrieval systems. Based on the collected data, we compare search behavior and outcome of these two different kinds of interaction paradigms. Compared with the traditional one, experimental results show that users can achieve better retrieval performance with the conversational case retrieval system. Moreover, conversational system can also save users' efforts in formulating queries and examining results.

References

[1]

Mikhail Ageev, Qi Guo, Dmitry Lagun, and Eugene Agichtein. 2011. Find it if you can: a game for modeling different types of web search success using interaction data. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. 345--354.

Digital Library

[2]

Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd international acm sigir conference on research and development in information retrieval. 475--484.

Digital Library

[3]

Krisztian Balog, Lucie Flekova, Matthias Hagen, Rosie Jones, Martin Potthast, Filip Radlinski, Mark Sanderson, Svitlana Vakulenko, and Hamed Zamani. 2020. Common Conversational Community Prototype: Scholarly Conversational Assistant. arXiv preprint arXiv:2001.06910 (2020).

[4]

Nicholas J Belkin, Colleen Cool, Adelheit Stein, and Ulrich Thiel. 1995. Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems. Expert systems with applications 9, 3 (1995), 379--395.

[5]

Ben Carterette, Evangelos Kanoulas, Mark M. Hall, and Paul D. Clough. 2014. Overview of the TREC 2014 Session Track. In Proceedings of The Twenty-Third Text REtrieval Conference, TREC 2014 (NIST Special Publication, Vol. 500--308).

[6]

John Doyle. 1992. WESTLAW and the American digest classification scheme. Law. Libr. J. 84 (1992), 229.

[7]

Ángel Sancho Ferrer, Carlos Fernández Hernández, and Pierre Boulat. [n.d.]. LEGAL SEARCH: foundations, evolution and next challenges. The Wolters Kluwer experience LA BUSQUEDA DE INFORMACIÓN LEGAL: fundamentos, evoluciony proximos desafios. La Experiencia de Wolters Kluwer. ([n. d.]).

[8]

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378.

[9]

Hanjo Hamann. 2019. The German Federal Courts Dataset 1950--2019: From Paper Archives to Linked Open Data. Journal of Empirical Legal Studies 16, 3 (2019), 671--688.

[10]

Diane Kelly. 2009. Methods for evaluating interactive information retrieval systems with users. Now Publishers Inc.

Digital Library

[11]

Jiaxin Mao, Yiqun Liu, Noriko Kando, Min Zhang, and Shaoping Ma. 2018. How Does Domain Expertise Affect Users' Search Interaction and Outcome in Exploratory Search? ACM Transactions on Information Systems (TOIS) 36, 4 (2018), 1--30.

Digital Library

[12]

John O McGinnis and Russell G Pearce. 2019. The great disruption: How machine intelligence will transform the role of lawyers in the delivery of legal services. Actual Probs. Econ. & L. (2019), 1230.

[13]

John O McGinnis and StevenWasick. 2014. Law's algorithm. Fla. L. Rev. 66 (2014), 991.

[14]

Daan Odijk, RyenWWhite, Ahmed Hassan Awadallah, and Susan T Dumais. 2015. Struggling and success in web search. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. 1551--1560.

Digital Library

[15]

Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 conference on conference human information interaction and retrieval. 117--126.

Digital Library

[16]

Paul Solomon. 1997. Conversation in information-seeking contexts: A test of an analytical framework. Library & Information Science Research 19, 3 (1997), 217--248.

[17]

Howard Turtle. 1995. Text retrieval in the legal world. Artificial Intelligence and Law 3, 1 (1995), 5--54.

Digital Library

[18]

Marc Van Opijnen and Cristiana Santos. 2017. On the concept of relevance in legal information retrieval. Artificial Intelligence and Law 25, 1 (2017), 65--87.

Digital Library

[19]

Ryen W White, Susan T Dumais, and Jaime Teevan. 2009. Characterizing the influence of domain expertise on web search behavior. In Proceedings of the second ACM international conference on web search and data mining. 132--141.

Digital Library

[20]

Hamed Zamani and Nick Craswell. 2020. Macaw: An extensible conversational information seeking platform. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2193--2196.

Digital Library

[21]

Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W Bruce Croft. 2018. Towards conversational search and recommendation: System ask, user respond. In Proceedings of the 27th acm international conference on information and knowledge management. 177--186.

Digital Library

Cited By

Guan JYu ZLiao YTang RDuan MHan G(2024)Predicting Critical Path of Labor Dispute Resolution in Legal Domain by Machine Learning Models Based on SHapley Additive exPlanations and Soft Voting StrategyMathematics10.3390/math1202027212:2(272)Online publication date: 14-Jan-2024
https://doi.org/10.3390/math12020272
Liu SHu YTian ZJin ZRuan SMao JSakai TIshita EOhshima HHasibi FMao JJose J(2024)Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search TasksProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698406(103-113)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698406
Liu BZhu ZAi QLiu YWu YSerra ESpezzano F(2024)LeDQA: A Chinese Legal Case Document-based Question Answering DatasetProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679154(5385-5389)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679154
Show More Cited By

Index Terms

Conversational vs Traditional: Comparing Search Behavior and Outcome in Legal Case Retrieval
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Collaborative search

Recommendations

Investigating User Behavior in Legal Case Retrieval
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Legal case retrieval is a specialized IR task aiming to retrieve supporting cases given a query case. While recent research efforts are committed to improving the automatic retrieval models' performances, little attention has been paid to the practical ...
LeCaRD: A Legal Case Retrieval Dataset for Chinese Law System
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Legal case retrieval is of vital importance for ensuring justice in different kinds of law systems and has recently received increasing attention in information retrieval (IR) research. However, the relevance judgment criteria of previous retrieval ...
Understanding Relevance Judgments in Legal Case Retrieval
Legal case retrieval, which aims to retrieve relevant cases given a query case, has drawn increasing research attention in recent years. While much research has worked on developing automatic retrieval models, how to characterize relevance in this ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2021

2998 pages

ISBN:9781450380379

DOI:10.1145/3404835

General Chairs:
Fernando Diaz
(Google)
,
Chirag Shah
University of Washington
,
Torsten Suel
New York University
,
Program Chairs:
Pablo Castells
Universidad Autónoma de Madrid, Amazon
,
Rosie Jones
Spotify
,
Tetsuya Sakai
Waseda University

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

National Key Research and Development Program of China
Natural Science Foundation of China

Conference

SIGIR '21

Sponsor:

SIGIR

SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2021

Virtual Event, Canada

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
400
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)6

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Guan JYu ZLiao YTang RDuan MHan G(2024)Predicting Critical Path of Labor Dispute Resolution in Legal Domain by Machine Learning Models Based on SHapley Additive exPlanations and Soft Voting StrategyMathematics10.3390/math1202027212:2(272)Online publication date: 14-Jan-2024
https://doi.org/10.3390/math12020272
Liu SHu YTian ZJin ZRuan SMao JSakai TIshita EOhshima HHasibi FMao JJose J(2024)Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search TasksProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698406(103-113)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698406
Liu BZhu ZAi QLiu YWu YSerra ESpezzano F(2024)LeDQA: A Chinese Legal Case Document-based Question Answering DatasetProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679154(5385-5389)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679154
Ye FLi SChua TNgo CKa-Wei Lee RKumar RLauw H(2024)MileCut: A Multi-view Truncation Framework for Legal Case RetrievalProceedings of the ACM Web Conference 202410.1145/3589334.3645349(1341-1349)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645349
Yue LLiu QJin BWu HAn Y(2024)A Circumstance-Aware Neural Framework for Explainable Legal Judgment PredictionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.338758036:11(5453-5467)Online publication date: Nov-2024
https://doi.org/10.1109/TKDE.2024.3387580
Vitório DSouza EMartins Lda Silva Nde Carvalho AOliveira Ade Andrade F(2024)Building a relevance feedback corpus for legal information retrieval in the real-case scenario of the Brazilian Chamber of DeputiesLanguage Resources and Evaluation10.1007/s10579-024-09767-3Online publication date: 18-Aug-2024
https://doi.org/10.1007/s10579-024-09767-3
Shao YLi HWu YLiu YAi QMao JMa YMa S(2023)An Intent Taxonomy of Legal Case RetrievalACM Transactions on Information Systems10.1145/362609342:2(1-27)Online publication date: 11-Dec-2023
https://dl.acm.org/doi/10.1145/3626093
Li QHu YYao FXiao CLiu ZSun MShen WFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)MUSER: A Multi-View Similar Case Retrieval DatasetProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615125(5336-5340)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615125
Liu BHu YAi QLiu YWu YLi CShen WFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Leveraging Event Schema to Ask Clarifying Questions for Conversational Legal Case RetrievalProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614953(1513-1522)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614953
Shao YWu YLiu YMao JMa S(2023)Understanding Relevance Judgments in Legal Case RetrievalACM Transactions on Information Systems10.1145/356992941:3(1-32)Online publication date: 7-Feb-2023
https://dl.acm.org/doi/10.1145/3569929
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten