research-article

The significance of evaluation in AI and law: a case study re-examining ICAIL proceedings

Authors:

Jack G. Conrad,

John ZeleznikowAuthors Info & Claims

ICAIL '13: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Law

Pages 186 - 191

https://doi.org/10.1145/2514601.2514624

Published: 10 June 2013 Publication History

Abstract

This paper examines the presence of performance evaluation in works published at ICAIL conferences since 2000. As such, it is a self-reflexive, meta-level study that investigates the proportion of works that include some form of performance assessment in their contribution. It also reports on the categories of evaluation present as well as their degree. In addition, the paper compares current trends in performance measurement with those of earlier ICAILs, as reported in the Hall and Zeleznikow work on the same topic (ICAIL 2001). The paper also develops an argument for why evaluation in formal Artificial Intelligence and Law reports such as ICAIL proceedings is imperative. It underscores the importance of answering the question: how good is the system?, how reliable is the approach?, or, more succinctly, does it work? The paper argues that the presence of a performance-based ethic within a scientific research community is a sign of maturity and essential scientific rigor. Finally the work references an evaluation checklist and presents a set of recommended best practices for the inclusion of evaluation methods going forward.

References

[1]

E. Chelimsky. The coming transformations in evaluation. Evaluation for the 21st Century: A Handbook, pages 1--26, 1997.

[2]

P. R. Cohen and A. E. Howe. How evaluation guides AI research. AI Magazine, 9(4):35--43, Winter 1988.

Digital Library

[3]

J. G. Conrad, K. Al-Kofahi, Y. Zhao, and G. Karypis. Effective document clustering for large heterogeneous law firm collections. In Proceedings of the 10th International Conference on Artificial Intelligence and Law (ICAIL 2005) (Bologna, Italy), pages 177--187. IAAIL, ACM Press, June 2005.

Digital Library

[4]

M. J. J. Hall. An Adaptive, Subsumptive Evaluation Framework for Intelligent Legal Systems. PhD thesis, La Trobe University, Faculty of Science Technology and Engineering, Bundoora, VIC, 3086, Australia, Jan. 2005.

[5]

M. J. J. Hall, R. Hall, and J. Zeleznikow. A process for evaluating legal knowledge-based systems based upon the context criteria contingency-guidelines framework. In Proceedings of the 9th International Conference on Artificial Intelligence and Law (ICAIL 2003) (Edinburgh, Scotland), pages 274--283. IAAIL, ACM Press, June 2003.

Digital Library

[6]

M. J. J. Hall and J. Zeleznikow. Acknowledging insufficiency in evaluation of legal knowledge-based systems: Strategies towards a broad-based evaluation model. In Proceedings of the 8th International Conference on Artificial Intelligence and Law (ICAL 2001) (St. Louis, Missouri), pages 146--156. IAAIL, ACM Press, May 2001.

Digital Library

[7]

P. H. Hendriks and D. J. Vriens. Knowledge-based systems and knowledge management: friends or foes? Information & Management, 35(2):113--125, 1999.

Digital Library

[8]

P. E. Jackson. Personal communication, 2009.

[9]

A. S. Jadhav and R. M. Sonar. Evaluating and selecting software packages: A review. Information and Software Technology, 51(3):555--563, 2009.

Digital Library

[10]

S. Joseph and H. Prakken. Coherence-driven argumentation to norm consensus. In Proceedings of the 13th International Conference on Artificial Intelligence and Law (ICAIL 2009) (Barcelona, Spain), pages 58--67. IAAIL, ACM Press, 2009.

Digital Library

[11]

J. Karlsson, C. Wohlin, and B. Regnell. An evaluation of methods for prioritizing software requirements. Information and Software Technology, 39(14):939--947, 1998.

[12]

A. W. Koers. Criteria for the classification of legal knowledge systems advisory systems on legal questions. Legal Knowledge Based Systems, pages 23--35, 1990.

[13]

Q. Lu, J. G. Conrad, K. Al-Kofahi, and W. Keenan. Legal document clustering with built-in topic segmentation. In Proceedings of the 20th International Conference on Information and Knowledge Management (CIKM11), pages 383--392. ACM Press, 2011.

Digital Library

[14]

H. Prakken. Modelling reasoning about evidence in legal procedure. In Proceedings of the 9th International Conference on Artificial Intelligence and Law (ICAIL 2001) (St. Louis, Missouri), pages 119--128. IAAIL, ACM Press, 2001.

Digital Library

[15]

H. Prakken. A study of accrual of arguments, with applications to evidential reasoning. In Proceedings of the 11th International Conference on Artificial Intelligence and Law (ICAIL 2005) (Bologna, Italy), pages 85--94. IAAIL, ACM Press, 2005.

Digital Library

[16]

K. E. Sanders. Chiron: Planning in an Open-textured Domain. PhD thesis, Brown Univ., Providence, RI, 1995.

Digital Library

[17]

T. V. E. Silvie Spreeuwenberg and R. Gerrits. The role of verification in improving the quality of legal decision-making. In Fourteenth Annual International Conference on Legal Knowledge and Information Systems. Frontiers in Artificial Intelligence and Applications (JURIX 2001), pages 1--15, Amsterdam, Netherlands, 2001. IOS Press.

[18]

R. E. Susskind. Expert systems in law. In Proceedings of the 1st International Conference on Artificial Intelligence and Law (ICAIL 1987) (Boston, MA), pages 1--8. IAAIL, ACM Press, May 1987.

Digital Library

Cited By

Bex F(2024)AI, Law and beyond. A transdisciplinary ecosystem for the future of AI & LawArtificial Intelligence and Law10.1007/s10506-024-09404-y33:1(253-270)Online publication date: 16-May-2024
https://doi.org/10.1007/s10506-024-09404-y
Conrad JRay Chaudhuri SPaul SGhosh S(2023)AI & Law: Formative Developments, State-of-the-Art Approaches, Challenges & OpportunitiesProceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD)10.1145/3570991.3571050(320-323)Online publication date: 4-Jan-2023
https://dl.acm.org/doi/10.1145/3570991.3571050
Ganguly DConrad JGhosh KGhosh SGoyal PBhattacharya PNigam SPaul S(2023)Legal IR and NLP: The History, Challenges, and State-of-the-ArtAdvances in Information Retrieval10.1007/978-3-031-28241-6_34(331-340)Online publication date: 16-Mar-2023
https://doi.org/10.1007/978-3-031-28241-6_34
Show More Cited By

Index Terms

Recommendations

The role of evaluation in AI and law: an examination of its different forms in the AI and law journal
ICAIL '15: Proceedings of the 15th International Conference on Artificial Intelligence and Law

This paper explores the presence and forms of evaluation in articles published in the journal Artificial Intelligence and Law for the ten-year period from 2005 through 2014. It represents a meta-level study of some the most significant works produced by ...
Rediscovering artificial intelligence and law: an inadequate jurisprudence?

Headlines suggesting that Google scientists had developed the first computer programme capable of learning a wide variety of tasks independently, in what has been hailed as a significant step towards ‘true artificial intelligence’¹ may or may not ...
Acknowledging insufficiency in the evaluation of legal knowledge-based systems: strategies towards a broadbased evaluation model
ICAIL '01: Proceedings of the 8th international conference on Artificial intelligence and law

This paper considers the need for evaluation of knowledge-based systems in general and legal knowledge-based systems in particular. Some special features of legal knowledge-based systems pertinent to their evaluation are presented. The expected benefits ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICAIL '13: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Law

June 2013

277 pages

ISBN:9781450320801

DOI:10.1145/2514601

Conference Chair:
Enrico Francesconi
Institute of Legal Information Theory and Techniques - ITTIG-CNR Florence, Italy
,
Program Chair:
Bart Verheij
Institute of Artificial Intelligence and Cognitive Engineering, University of Groningen, Groningen, The Netherlands

Copyright © 2013 Authors.

Sponsors

ITTIG-CNR: Istituto di Teoria e Tecniche dell'Informazione Giuridica - Consiglio Nazionale delle Ricerche
IAAIL: Intl Asso for Artifical Intel & Law

In-Cooperation

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 June 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Thomson Reuters Global Resources

Conference

ICAIL '13

Sponsor:

ITTIG-CNR
IAAIL

ICAIL '13: International Conference on Artificial Intelligence and Law

June 10 - 14, 2013

Rome, Italy

Acceptance Rates

ICAIL '13 Paper Acceptance Rate 17 of 53 submissions, 32%;

Overall Acceptance Rate 69 of 169 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
224
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)2

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bex F(2024)AI, Law and beyond. A transdisciplinary ecosystem for the future of AI & LawArtificial Intelligence and Law10.1007/s10506-024-09404-y33:1(253-270)Online publication date: 16-May-2024
https://doi.org/10.1007/s10506-024-09404-y
Conrad JRay Chaudhuri SPaul SGhosh S(2023)AI & Law: Formative Developments, State-of-the-Art Approaches, Challenges & OpportunitiesProceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD)10.1145/3570991.3571050(320-323)Online publication date: 4-Jan-2023
https://dl.acm.org/doi/10.1145/3570991.3571050
Ganguly DConrad JGhosh KGhosh SGoyal PBhattacharya PNigam SPaul S(2023)Legal IR and NLP: The History, Challenges, and State-of-the-ArtAdvances in Information Retrieval10.1007/978-3-031-28241-6_34(331-340)Online publication date: 16-Mar-2023
https://doi.org/10.1007/978-3-031-28241-6_34
Novotná TLibal T(2022)An Evaluation of Methodologies for Legal FormalizationExplainable and Transparent AI and Multi-Agent Systems10.1007/978-3-031-15565-9_12(189-203)Online publication date: 23-Sep-2022
https://doi.org/10.1007/978-3-031-15565-9_12
Sovrano FPalmirani MDistefano BSapienza SVitali FMaranhão JWyner A(2021)A dataset for evaluating legal question answering on private international lawProceedings of the Eighteenth International Conference on Artificial Intelligence and Law10.1145/3462757.3466094(230-234)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3462757.3466094
Bartolini CLenzini GSantos C(2019)An Agile Approach to Validate a Formal Representation of the GDPRNew Frontiers in Artificial Intelligence10.1007/978-3-030-31605-1_13(160-176)Online publication date: 11-Oct-2019
https://doi.org/10.1007/978-3-030-31605-1_13
Hernández-Orallo J(2017)Evaluation in artificial intelligenceArtificial Intelligence Review10.1007/s10462-016-9505-748:3(397-447)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1007/s10462-016-9505-7

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten