Hostname: page-component-669899f699-swprf Total loading time: 0 Render date: 2025-04-30T20:30:10.282Z Has data issue: false hasContentIssue false
  • English
  • Français

TempCourt: evaluation of temporal taggers on a new corpus of court decisions

Published online by Cambridge University Press:  17 December 2019

María Navas-Loro Open the ORCID record for María Navas-Loro [Opens in a new window] ,
Erwin Filtz Open the ORCID record for Erwin Filtz [Opens in a new window] ,
Víctor Rodríguez-Doncel Open the ORCID record for Víctor Rodríguez-Doncel [Opens in a new window] ,
Axel Polleres Open the ORCID record for Axel Polleres [Opens in a new window]  and
Sabrina Kirrane Open the ORCID record for Sabrina Kirrane [Opens in a new window]
María Navas-Loro
Affiliation:
D3206 – Ontology Engineering Group, Universidad Politécnica de Madrid, Montegancedo Campus, Madrid, Spain e-mails: mnavas@fi.upm.es, vrodriguez@fi.upm.es
Erwin Filtz
Affiliation:
Institute for Information Business, Vienna University of Economics and Business, Vienna, Austria e-mails: Erwin.Filtz@wu.ac.at, Axel.Polleres@wu.ac.at, Sabrina.Kirrane@wu.ac.at
Víctor Rodríguez-Doncel
Affiliation:
D3206 – Ontology Engineering Group, Universidad Politécnica de Madrid, Montegancedo Campus, Madrid, Spain e-mails: mnavas@fi.upm.es, vrodriguez@fi.upm.es
Axel Polleres
Affiliation:
Institute for Information Business, Vienna University of Economics and Business, Vienna, Austria e-mails: Erwin.Filtz@wu.ac.at, Axel.Polleres@wu.ac.at, Sabrina.Kirrane@wu.ac.at
Sabrina Kirrane
Affiliation:
Institute for Information Business, Vienna University of Economics and Business, Vienna, Austria e-mails: Erwin.Filtz@wu.ac.at, Axel.Polleres@wu.ac.at, Sabrina.Kirrane@wu.ac.at
Get access Rights & Permissions [Opens in a new window]

Abstract

The extraction and processing of temporal expressions (TEs) in textual documents have been extensively studied in several domains; however, for the legal domain it remains an open challenge. This is possibly due to the scarcity of corpora in the domain and the particularities found in legal documents that are highlighted in this paper. Considering the pivotal role played by temporal information when it comes to analyzing legal cases, this paper presents TempCourt, a corpus of 30 legal documents from the European Court of Human Rights, the European Court of Justice, and the United States Supreme Court with manually annotated TEs. The corpus contains two different temporal annotation sets that adhere to the TimeML standard, the first one capturing all TEs and the second dedicated to TEs that are relevant for the case under judgment (thus excluding dates of previous court decisions). The proposed gold standards are subsequently used to compare ten state-of-the-art cross-domain temporal taggers, and to identify not only the limitations of cross-domain temporal taggers but also limitations of the TimeML standard when applied to legal documents. Finally, the paper identifies the need for dedicated resources and the adaptation of existing tools, and specific annotation guidelines that can be adapted to different types of legal documents.

Type
Research Article
Information
The Knowledge Engineering Review , Volume 34 , 2019 , e24
Copyright
© Cambridge University Press 2019

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

The two first authors equally contributed to this work.

References

Bethard, S. 2013. ClearTK-TimeML: a minimalist approach to TempEval 2013. In Proceedings of the Workshop SemEval 2013, 10–14. ACL.Google Scholar
Chambers, N. 2013. NavyTime: event and time ordering from raw text. In Proceedings of the Workshop SemEval 2013, 2, 73–77.Google Scholar
Chambers, N., Cassidy, T., McDowell, B. & Bethard, S. 2014. Dense event ordering with a multi-pass architecture. Transactions of the ACL, 2, 273–284.Google Scholar
Chang, A. X. & Manning, C. D. 2012. SUTime: a library for recognizing and normalizing time expressions. In Proceedings of LREC 2012.Google Scholar
Chang, A. X., et al. 2014. TokensRegex: Defining Cascaded Regular Expressions Over Tokens. Technical report CSTR 2014-02, Department of Computer Science, Stanford University.Google Scholar
Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37–46.CrossRefGoogle Scholar
Cunningham, H., Tablan, V., Roberts, A. & Bontcheva, K. 2013. Getting more out of biomedical documents with GATE’s full lifecycle open source text analytics. PLOS Computational Biology 9(2), 1–16.CrossRefGoogle Scholar
Dell’Orletta, F., Marchi, S., Montemagni, S., Plank, B. & Venturi, G. 2012. The splet–2012 shared task on dependency parsing of legal texts. In Semantic Processing of Legal Texts (SPLeT-2012) Workshop Programme, 42.Google Scholar
Derczynski, L. & Gaizauskas, R. 2010. USFD2: annotating temporal expressions and TLINKs for TempEval-2. In Proceedings of the Workshop SemEval, 337–340. ACL.Google Scholar
Ferro, L., Gerber, L., Mani, I., Sundheim, B. & Wilson, G. 2005. Tides 2005 Standard for the Annotation of Temporal Expressions. Technical report, MITRE.Google Scholar
Filannino, M. & Nenadic, G. 2015. Temporal expression extraction with extensive feature type selection and a posteriori label adjustment. Data & Knowledge Engineering 100, 19–33. ISSN 0169-023X. 10.1016/j.datak.2015.09.002. http://www.sciencedirect.com/science/article/pii/S0169023X15000725.CrossRefGoogle Scholar
Freund, Y., Schapire, R. & Abe, N. 1999. A short introduction to boosting. Journal-Japanese Society For Artificial Intelligence 14(771–780), 1612.Google Scholar
Gildea, D. & Jurafsky, D. 2002. Automatic labeling of semantic roles. Computational Linguistics 28(3), 245–288. 10.1162/089120102760275983. https://doi.org/10.1162/089120102760275983.CrossRefGoogle Scholar
Han, B., Gates, D. & Levin, L. 2006. From language to time: a temporal expression anchorer. In Thirteenth International Symposium on Temporal Representation and Reasoning, 2006, TIME 2006, 196–203. IEEE.CrossRefGoogle Scholar
Hellmann, S.Lehmann, J. & Auer, S. 2012. NIF: an ontology-based and linked-data-aware NLP Interchange Format. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.1215&rep=rep1&type=pdfGoogle Scholar
Isemann, D., Ahmad, K., Fernando, T. & Vogel, C. 2013. Temporal Dependence in Legal Documents, 497–504. Springer. ISBN 978-3-642-41278-3.CrossRefGoogle Scholar
Ji, H., Cassidy, T., Li, Q. & Tamang, S. 2014. Tackling representation, annotation and classification challenges for temporal knowledge base population. Knowledge and Information Systems 41(3), 611–646. ISSN 0219-3116. 10.1007/s10115-013-0675-1. https://doi.org/10.1007/s10115-013-0675-1.CrossRefGoogle Scholar
Kajiwara, T. & Komachi, M. 2016. Building a monolingual parallel corpus for text simplification using sentence similarity based on alignment between word embeddings. In Proceedings of COLING 2016: Technical Papers, 1147–1158.Google Scholar
Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A. & Bandyopadhyay, S. 2013. JU_CSE: a CRF based approach to annotation of temporal expression, event and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), 2, 64–72.Google Scholar
Lafferty, J. D., McCallum, A. & Pereira, F. C. N. 2001. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001),Williams College, Williamstown, MA, USA, June 28–July 1, 2001, Brodley, C. E. & Danyluk, A. P. (eds). Morgan Kaufmann, 282–289. ISBN 1-55860-778-1.Google Scholar
Lee, K., Artzi, Y., Dodge, J., & Zettlemoyer, L. 2014. Context-dependent semantic parsing for time expressions. In Proceedings of the 52nd Annual Meeting of the ACL,1, 1437–1447.Google Scholar
Llorens, H., Saquete, E. & Navarro, B. 2010. TIPSem (english and spanish): evaluating CRFs and semantic roles inTempEval-2. In Proceedings of the Workshop SemEval, 284–291. ACL.Google Scholar
Llorens, H., Derczynski, L., Gaizauskas, R. J. & Saquete, E. 2012. TIMEN: an open temporal expression normalisation resource. In Proceedings of LREC 2012, 3044–3051.Google Scholar
Loper, E. & Bird, S. 2002. NLTK: The Natural Language Toolkit. CoRR, cs.CL/0205028. http://arxiv.org/abs/cs.CL/0205028.CrossRefGoogle Scholar
Mani, I. & Wilson, G. 2000. Robust temporal processing of news. In Proceedings of the 38th Annual Meeting on ACL, 69–76. ACL.CrossRefGoogle Scholar
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. & McClosky, D. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of the 52nd Annual Meeting of the ACL 2014, System Demonstrations, 55–60.Google Scholar
Mazur, P. & Dale, R. 2009. The dante temporal expression tagger. In Human Language Technology. Challenges of the Information Society, Vetulani, Z. & Uszkoreit, H. (eds). 245–257. Springer. ISBN 978-3-642-04235-5.Google Scholar
Mazur, P. & Dale, R. 2010. Wikiwars: a new corpus for research on temporal expressions. In Proceedings of EMNLP 2010, 913–922. ACL.Google Scholar
Minard, A.-L., Speranza, M., Urizar, R., Altuna, B., Van Erp, M., Schoen, A. & Van Son, C. 2016. Meantime, the newsreader multilingual event and time corpus. In Proceedings of LREC 2016, European Language Resources Association.Google Scholar
Naik, V., Vanitha, G. & Inturi, S. 2011. Reasoning in legal text documents with extracted event information. International Journal of Computer Applications 28, 8–-13.CrossRefGoogle Scholar
Nivre, J., et al. 2007. The CoNLL 2007 shared task on dependency parsing. In Proceedings of EMNLP-CoNLL 2007.Google Scholar
Northwood, C. 2010. TERNIP: Temporal Expression Recognition and Normalisation in Python. Master’s thesis, University of Sheffield.Google Scholar
Pellow, D. & Eskenazi, M. 2014. An open corpus of everyday documents for simplification tasks. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), 84–93.Google Scholar
Pustejovsky, J., Hanks, P., Sauri, R., See, A., Gaizauskas, R., Setzer, A., Radev, D., Sundheim, B., Day, D., Ferro, L. & Lazo, M. 2003a. The TimeBank corpus. In Corpus Linguistics, Lancaster University (UK), 2003, 28–31.Google Scholar
Pustejovsky, J., Castano, J. M., Ingria, R., Sauri, R., Gaizauskas, R. J., Setzer, A.Katz, G. & Radev, D. R. 2003bb. TimeML: robust specification of event and temporal expressions intext. In New Directions in Question Answering, Papers from 2003 AAAI Spring Symposium, Maybury, M. T. (ed). AAAI Press, 28–34. ISBN 1-57735-184-3.Google Scholar
Quinlan, J. R. 1993. C4.5: Programs for Machine Learning. Morgan Kaufmann. ISBN 1-55860-238-0.Google Scholar
Rijsbergen, C. J. V. 1979. Information Retrieval, 2nd edition. Butterworth-Heinemann. ISBN 0408709294.Google Scholar
, R., Littman, J., Knippen, B., Gaizauskas, R., Setzer, A. &Pustejovsky, J. TimeML Annotation Guidelines. Version 1.2.1, 2006.Google Scholar
Schilder, F. 2005. Event extraction and temporal reasoning in legal documents. In Annotating, Extracting and Reasoning about Time and Events,International Seminar, Dagstuhl Castle. Revised Papers, 59–71.Google Scholar
Scott, W. A. 1955. Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly, 19(3), 321–325.CrossRefGoogle Scholar
Steedman, M.Jason, B. 2011. Combinatory categorial grammar, In Non-Transformational Syntax: Formal and Explicit Models of Grammar, Borsley, R. & Borjars, K. (eds). Wiley-Blackwell, 181–224.Google Scholar
Strötgen, J. & Gertz, M. 2016. Domain-sensitive temporal tagging. Synthesis Lectures on Human Language Technologies 9(3), 1–151.CrossRefGoogle Scholar
Strötgen, J. & Gertz, M. 2012. Temporal tagging on different domains: challenges, strategies, and gold standards. In Proceedings of LREC 2012, 12, 3746–3753.Google Scholar
Styler IV, W. F., Bethard, S., Finan, S., Palmer, M., Pradhan, S., de Groen, P. C., Erickson, B., Miller, T., Lin, C., Savova, G., et al. 2014. Temporal annotation in the clinical domain. Transactions of the Association of Computational Linguistics 2(1), 143–154.CrossRefGoogle Scholar
Tabassum, J., Ritter, A. & Xu, W. TweeTime: a minimally supervised method for recognizing and normalizing time expressions in Twitter. arXiv preprint arXiv:1608.02904, 2016.Google Scholar
UzZaman, N. & Allen, J. F. 2010. Event and temporal expression extraction from raw text: first step towards a temporally aware system. International Journal of Semantic Computing 4(04), 487–508.CrossRefGoogle Scholar
UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M. & Pustejovsky, J. 2013. SemEval-2013 Task 1: TempEval-3: evaluating time expressions, events, and temporal relations. In Proceedings of the Workshop SemEval 2013, 1–9.Google Scholar
Verhagen, M., Mani, I., Sauri, R., Littman, J., Knippen, R., Jang, S.B, Rumshisky, A., Phillips, J., & Pustejovsky, J. 2005. Automating temporal annotation with TARSQI. In Proceedings of the ACL 2005 on Interactive Poster and Demonstration Sessions, ACLdemo ’05, 81–84. ACL.CrossRefGoogle Scholar
Vlek, C. S., Prakken, H., Renooij, S. & Verheij, B. 2013. Representing and evaluating legal narratives with subscenarios in a Bayesian network. In OASIcs-OpenAccess Series in Informatics, 32. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.Google Scholar
Zhong, X., Sun, A. & Cambria, E. 2017 Time expression analysis and recognition using syntactic token types and general heuristic rules. In Proceedings of the 55th Annual Meeting of the ACL, 1, 420–429.Google Scholar