Skip to main content
SpringerLink
Log in
Book cover

JSAI International Symposium on Artificial Intelligence

JSAI-isAI 2018: New Frontiers in Artificial Intelligence pp 177–192Cite as

COLIEE-2018: Evaluation of the Competition on Legal Information Extraction and Entailment

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11717))

Abstract

We summarize the evaluation of the 5th Competition on Legal Information Extraction/Entailment 2018 (COLIEE-2018). The COLIEE-2018 tasks include two tasks in each of statute law and case law. The case law component includes an information retrieval (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component includes information retrieval (Task 3) and entailment/question answering (Task 4). Participation was open to any group based on any approach. 13 teams participated in the case law competition, and we received results from 7 teams where 6 submissions to Task 1 (12 runs), and 4 submissions to Task 2 (8 runs). Regarding the statute law, there were submissions of 17 runs from 8 teams (including 2 organizers’ runs) for Task 3 and 7 runs from 3 teams for Task 4. We describe each team’s approaches, our official evaluation, and analysis on our data and submission results. We also discuss possibilities for future competition tasks.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Kim, M.Y., Goebel, R., Satoh, K.: COLIEE-2015: evaluation of legal question answering. In: Ninth International Workshop on Juris-Informatics (JURISIN 2015) (2015)

    Google Scholar 

  2. Kim, M.Y., Goebel, R., Kano, Y., Satoh, K.: COLIEE-2016: evaluation of the competition on legal information extraction/entailment. In: Tenth International Workshop on Juris-Informatics (JURISIN 2016) (2016)

    Google Scholar 

  3. Kano, Y., Kim, M.Y., Goebel, R., Satoh, K.: Overview of COLIEE 2017. Epic Ser. Comput. 47, 1–8 (2017)

    Google Scholar 

  4. Yoshioka, M.: Analysis of COLIEE information retrieval task data. In: Arai, S., Kojima, K., Mineshima, K., Bekki, D., Satoh, K., Ohta, Y. (eds.) New Frontiers in Artificial Intelligence, pp. 5–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93794-6_1

    Chapter  Google Scholar 

  5. Draijer, W., Verberne, S.: Case law retrieval with doc2vec and elastic search. In: Twelfth International Workshop on Juris-Informatics (JURISIN 2018), 2018

    Google Scholar 

  6. Chen, Y., Zhou, Y., Lu, Z., Sun, H., Yang, W.: Legal information retrieval by association rules. In: JURISIN 2018 (2018)

    Google Scholar 

  7. Tran, V., Truong, S.N., Le Nguyen, M.: JNLP group: legal information retrieval with summary and logical structure analysis. In: International Workshop on Juris-Informatics (JURISIN 2018) (2018)

    Google Scholar 

  8. Yoshioka, M., Song, Z.: HUKB at COLIEE2018 information retrieval task. In: International Workshop on Juris-Informatics (JURISIN 2018) (2018)

    Google Scholar 

  9. Rabelo, J., Kim, M.Y., Babiker, H., Goebel, R., Farruque, N.: Legal information extraction and entailment for statute law and case law. In: JURISIN 2018 (2018)

    Google Scholar 

  10. Lefoane, M., Koboyatshwene, T., Narasimhan, L.: KNN clustering approach to legal precedence retrieval. In: Twelfth International Workshop on Juris-Informatics (JURISIN 2018) (2018)

    Google Scholar 

  11. Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: a language-model based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis, pp. 2–6 (2005)

    Google Scholar 

  12. Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 133–142 (2002)

    Google Scholar 

  13. Kim, M.Y., Goebel, R.: Two-step cascaded textual entailment for legal bar exam question answering. In: 16th International Conference on Artificial Intelligence and Law (ICAIL 2017), 2017

    Google Scholar 

  14. Hoshino, R., Taniguchi, R., Kiyota, N., Kano, Y.: Question answering system for legal bar examination using predicate argument structure. In: Twelfth International Workshop on Juris-Informatics (JURISIN 2018) (2018)

    Google Scholar 

  15. Taniguchi, R., Hoshino, R., Kano, Y.: Legal question answering system using framenet. In: Twelfth International Workshop on Juris-Informatics (JURISIN 2018) (2018)

    Google Scholar 

Download references

Acknowledgements

This research was supported by Alberta Machine Intelligence Institute (AMII), National Institute of Informatics, Shizuoka University and Hokkaido University. Thanks to Colin Lachance from vLex for his constant support in the development of the case law data set, and to support from Ross Intelligence and Intellicon. This work was partially supported by JSPS KAKENHI Grant Number 16H01756, 18H0333808, 17H06103, and JST CREST.

Author information

Authors and Affiliations

  1. Department of Science, Augustana Faculty, University of Alberta, Camrose, AB, Canada

    Mi-Young Kim

  2. Alberta Machine Intelligence Institute, University of Alberta, Edmonton, AB, Canada

    Mi-Young Kim, Yao Lu, Juliano Rabelo & Randy Goebel

  3. Department of Computing Science, University of Alberta, Edmonton, AB, Canada

    Randy Goebel

  4. Graduate School of Information Science and Technology, Hokkaido University, N14 W9, Kita-ku, Sapporo-shi, Hokkaido, Japan

    Masaharu Yoshioka

  5. Global Station for Big Date and Cybersecurity, Global Institution for Collaborative, Research and Education, Hokkaido University, Kita-ku, Sapporo-shi, Hokkaido, Japan

    Masaharu Yoshioka

  6. Faculty of Informatics, Shizuoka University, 3-5-1 Johoku, Naka-ku, Hamamatsu-shi, Shizuoka, Japan

    Yoshinobu Kano & Naoki Kiyota

  7. National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, Japan

    Ken Satoh

Authors
  1. Yoshinobu Kano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mi-Young Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Masaharu Yoshioka
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yao Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Juliano Rabelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Naoki Kiyota
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Randy Goebel
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ken Satoh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yoshinobu Kano .

Editor information

Editors and Affiliations

  1. National Institute of Advanced Industrial Science and Technology, Ibaraki, Japan

    Kazuhiro Kojima

  2. University of Electro-Communications, Tokyo, Japan

    Maki Sakamoto

  3. Ochanomizu University, Tokyo, Japan

    Koji Mineshima

  4. National Institute of Informatics, Tokyo, Japan

    Ken Satoh

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kano, Y. et al. (2019). COLIEE-2018: Evaluation of the Competition on Legal Information Extraction and Entailment. In: Kojima, K., Sakamoto, M., Mineshima, K., Satoh, K. (eds) New Frontiers in Artificial Intelligence. JSAI-isAI 2018. Lecture Notes in Computer Science(), vol 11717. Springer, Cham. https://doi.org/10.1007/978-3-030-31605-1_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-31605-1_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-31604-4

  • Online ISBN: 978-3-030-31605-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation