extended-abstract

The role of evaluation in AI and law: an examination of its different forms in the AI and law journal

Authors:

Jack G. Conrad,

John ZeleznikowAuthors Info & Claims

ICAIL '15: Proceedings of the 15th International Conference on Artificial Intelligence and Law

Pages 181 - 186

https://doi.org/10.1145/2746090.2746116

Published: 08 June 2015 Publication History

Get Access

Abstract

This paper explores the presence and forms of evaluation in articles published in the journal Artificial Intelligence and Law for the ten-year period from 2005 through 2014. It represents a meta-level study of some the most significant works produced by the AI and Law community, in this case nearly 140 research articles published in the AI and Law journal. It also compares its findings to previous work conducted on evaluation appearing in the Proceedings of the International Conference on Artificial Intelligence and Law (ICAIL). In addition, the paper highlights works harnessing performance evaluation as one of their chief scientific tools and the means by which they use it. It extends the argument for why evaluation is essential in formal Artificial Intelligence and Law reports such as those in the journal. As in the case of two earlier works on the topic, it pursues answers to the questions: how good is the system, algorithm or proposal?, how reliable is the approach or technique?, and, ultimately, does the method work? The paper investigates the role of performance evaluation in scientific research reports, underscoring the argument that a performance-based 'ethic' signifies a level of maturity and scientific rigor within a community. In addition, the work examines recent publications that address the same critical issue within the broader field of Artificial Intelligence.

References

[1]

F. J. Bex, H. Prakken, and B. Verheij:. Formalising argumentative story-based analysis of evidence. In Proceedings of the 12th International Conference on Artificial Intelligence and Law (ICAIL 2007) (Stanford, CA), pages 1--10. IAAIL, ACM Press, 2007.

Digital Library

Google Scholar

[2]

M. Caminada and L. Amgoud. On the evaluation of argumentation formalisms. Artificial Intelligence, 17(5): 286--310, 2007.

Digital Library

Google Scholar

[3]

P. R. Cohen and A. E. Howe. How evaluation guides AI research. AI Magazine, 9(4): 35--43, Winter 1988.

Digital Library

Google Scholar

[4]

J. G. Conrad, K. Al-Kofahi, Y. Zhao, and G. Karypis. Effective document clustering for large heterogeneous law firm collections. In Proceedings of the 10th International Conference on Artificial Intelligence and Law (ICAIL 2005) (Bologna, Italy), pages 177--187. IAAIL, ACM Press, June 2005.

Digital Library

Google Scholar

[5]

J. G. Conrad and J. Zeleznikow. Acknowledging insufficiency in evaluation of legal knowledge-based systems: Strategies towards a broad-based evaluation model. In Proceedings of the 14th International Conference on Artificial Intelligence and Law (ICAL 2013) (Rome, Italy), pages 186--191. IAAIL, ACM Press, June 2013.

Google Scholar

[6]

M. J. J. Hall and J. Zeleznikow. Acknowledging insufficiency in evaluation of legal knowledge-based systems: Strategies towards a broad-based evaluation model. In Proceedings of the 8th International Conference on Artificial Intelligence and Law (ICAIL 2001) (St. Louis, Missouri), pages 146--156. IAAIL, ACM Press, May 2001.

Digital Library

Google Scholar

[7]

P. E. Jackson. Personal communication, 2009.

Google Scholar

[8]

S. Joseph and H. Prakken. Coherence-driven argumentation to norm consensus. In Proceedings of the 13th International Conference on Artificial Intelligence and Law (ICAIL 2009) (Barcelona, Spain), pages 58--67. IAAIL, ACM Press, 2009.

Digital Library

Google Scholar

[9]

Q. Lu, J. G. Conrad, K. Al-Kofahi, and W. Keenan. Legal document clustering with built-in topic segmentation. In Proceedings of the 20th International Conference on Information and Knowledge Management (CIKM11), pages 383--392. ACM Press, 2011.

Digital Library

Google Scholar

[10]

H. Prakken. Modelling reasoning about evidence in legal procedure. In Proceedings of the 9th International Conference on Artificial Intelligence and Law (ICAIL 2001) (St. Louis, Missouri), pages 119--128. IAAIL, ACM Press, 2001.

Digital Library

Google Scholar

[11]

A. Stranieri and J. Zeleznikow. Evaluating legal expert systems. In Proceedings of the 7th International Conference on Artificial Intelligence and Law (ICAIL 1999) (Oslo, Norway), pages 18--24. IAAIL, ACM Press, 1999.

Digital Library

Google Scholar

Cited By

View all

Wei BYu YGan LWu F(2025)An LLMs-based neuro-symbolic legal judgment prediction framework for civil casesArtificial Intelligence and Law10.1007/s10506-025-09433-1Online publication date: 8-Feb-2025
https://doi.org/10.1007/s10506-025-09433-1
Bex F(2024)AI, Law and beyond. A transdisciplinary ecosystem for the future of AI & LawArtificial Intelligence and Law10.1007/s10506-024-09404-y33:1(253-270)Online publication date: 16-May-2024
https://doi.org/10.1007/s10506-024-09404-y
Conrad JRay Chaudhuri SPaul SGhosh S(2023)AI & Law: Formative Developments, State-of-the-Art Approaches, Challenges & OpportunitiesProceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD)10.1145/3570991.3571050(320-323)Online publication date: 4-Jan-2023
https://dl.acm.org/doi/10.1145/3570991.3571050
Show More Cited By

Index Terms

Recommendations

The significance of evaluation in AI and law: a case study re-examining ICAIL proceedings
ICAIL '13: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Law

This paper examines the presence of performance evaluation in works published at ICAIL conferences since 2000. As such, it is a self-reflexive, meta-level study that investigates the proportion of works that include some form of performance assessment ...
Rediscovering artificial intelligence and law: an inadequate jurisprudence?

Headlines suggesting that Google scientists had developed the first computer programme capable of learning a wide variety of tasks independently, in what has been hailed as a significant step towards ‘true artificial intelligence’¹ may or may not ...
Theory and Practice in AI and Law: A Response to Branting
Proceedings of the 2005 conference on Legal Knowledge and Information Systems: JURIX 2005: The Eighteenth Annual Conference

In this paper we use our previous work which has examined the different levels involved in reasoning about legal cases to examine some challenges to the relevance of current theoretical work in AI and Law made by Branting. In our model we view the ...

Comments

Information & Contributors

Information

Published In

ICAIL '15: Proceedings of the 15th International Conference on Artificial Intelligence and Law

June 2015

246 pages

ISBN:9781450335225

DOI:10.1145/2746090

Conference Chair:
Ted Sichelman
University of San Diego School of Law, San Diego, CA
,
Program Chair:
Katie Atkinson
University of Liverpool, Liverpool, UK

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

In-Cooperation

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 June 2015

Check for updates

Author Tags

Qualifiers

Extended-abstract

Conference

ICAIL '15

Sponsor:

Center for IP Law & Markets
TrademarkNow
Davis Polk
Legal Robot
Thomson Reuters

ICAIL '15: 15th International Conference on Artificial Intelligence and Law

June 8 - 12, 2015

California, San Diego

Acceptance Rates

ICAIL '15 Paper Acceptance Rate 30 of 58 submissions, 52%;

Overall Acceptance Rate 69 of 169 submissions, 41%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
479
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)4

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wei BYu YGan LWu F(2025)An LLMs-based neuro-symbolic legal judgment prediction framework for civil casesArtificial Intelligence and Law10.1007/s10506-025-09433-1Online publication date: 8-Feb-2025
https://doi.org/10.1007/s10506-025-09433-1
Bex F(2024)AI, Law and beyond. A transdisciplinary ecosystem for the future of AI & LawArtificial Intelligence and Law10.1007/s10506-024-09404-y33:1(253-270)Online publication date: 16-May-2024
https://doi.org/10.1007/s10506-024-09404-y
Conrad JRay Chaudhuri SPaul SGhosh S(2023)AI & Law: Formative Developments, State-of-the-Art Approaches, Challenges & OpportunitiesProceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD)10.1145/3570991.3571050(320-323)Online publication date: 4-Jan-2023
https://dl.acm.org/doi/10.1145/3570991.3571050
Ganguly DConrad JGhosh KGhosh SGoyal PBhattacharya PNigam SPaul S(2023)Legal IR and NLP: The History, Challenges, and State-of-the-ArtAdvances in Information Retrieval10.1007/978-3-031-28241-6_34(331-340)Online publication date: 16-Mar-2023
https://doi.org/10.1007/978-3-031-28241-6_34
Francesconi E(2022)The winter, the summer and the summer dream of artificial intelligence in lawArtificial Intelligence and Law10.1007/s10506-022-09309-830:2(147-161)Online publication date: 3-Feb-2022
https://doi.org/10.1007/s10506-022-09309-8
Verheij B(2022)The Study of Artificial Intelligence as LawLaw and Artificial Intelligence10.1007/978-94-6265-523-2_24(477-502)Online publication date: 6-Jul-2022
https://doi.org/10.1007/978-94-6265-523-2_24
Novotná TLibal T(2022)An Evaluation of Methodologies for Legal FormalizationExplainable and Transparent AI and Multi-Agent Systems10.1007/978-3-031-15565-9_12(189-203)Online publication date: 23-Sep-2022
https://doi.org/10.1007/978-3-031-15565-9_12
Verheij B(2020)Artificial intelligence as lawArtificial Intelligence and Law10.1007/s10506-020-09266-0Online publication date: 14-May-2020
https://doi.org/10.1007/s10506-020-09266-0
Lauritsen MSteenhuis Q(2019)Substantive Legal Software QualityProceedings of the Seventeenth International Conference on Artificial Intelligence and Law10.1145/3322640.3326706(52-62)Online publication date: 17-Jun-2019
https://dl.acm.org/doi/10.1145/3322640.3326706
Bench-Capon T(2017)HYPO’S legacy: introduction to the virtual special issueArtificial Intelligence and Law10.1007/s10506-017-9201-125:2(205-250)Online publication date: 2-Jun-2017
https://doi.org/10.1007/s10506-017-9201-1
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

The significance of evaluation in AI and law: a case study re-examining ICAIL proceedings

Rediscovering artificial intelligence and law: an inadequate jurisprudence?

Theory and Practice in AI and Law: A Response to Branting

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations