Scalable and explainable legal prediction

Abstract

Legal decision-support systems have the potential to improve access to justice, administrative efficiency, and judicial consistency, but broad adoption of such systems is contingent on development of technologies with low knowledge-engineering, validation, and maintenance costs. This paper describes two approaches to an important form of legal decision support—explainable outcome prediction—that obviate both annotation of an entire decision corpus and manual processing of new cases. The first approach, which uses an attention network for prediction and attention weights to highlight salient case text, was shown to be capable of predicting decisions, but attention-weight-based text highlighting did not demonstrably improve human decision speed or accuracy in an evaluation with 61 human subjects. The second approach, termed semi-supervised case annotation for legal explanations, exploits structural and semantic regularities in case corpora to identify textual patterns that have both predictable relationships to case decisions and explanatory value.

Appendix: Annotation

A key goal of SCALE is a methodology that permits development of explainable legal prediction systems by agencies that lack the resources to engineer domain-specific feature sets, a process that requires both extensive expertise in the particular legal domain and experience in feature engineering. Instead, SCALE requires only the linguistic skills necessary to annotate the decision portion of representative subset of cases, a much more limited process.

Our annotation schema for WIPO decisions consists of three layers: Argument Elements, Issues, and Factors (sub-issues).

Tags are applied to clauses and sentences, as opposed to shorter units such as noun phrases, in order to identify the complete linguistic proposition corresponding to the annotation label. The MITRE Annotation Toolkit (MAT)¹³ was used to perform the annotation.

1.1 Argument elements

Although our approach to predictive-text identification is to leverage the Factual Findings and Legal Findings, the annotation schema is designed to capture the full range of argument elements present in cases. These argument elements are as follows:

• 1. Policy

• 2. Contention

• 3. Factual Finding

• 4. Legal Finding

• 5. Case Rule

• 6. Decision

We have found that with these six argument elements, the majority of sentences within the “Findings” and “Decision” sections of WIPO cases can be assigned an argument element label. These argument elements are not specific to WIPO decisions and should be applicable in other domains.

1.2 Issues

Each Argument Element tag is assigned an Issue. The Issue tags include the three required elements that the complainant must establish in order to prevail in a WIPO case. These issues, which are documented in the Uniform Domain Name Dispute Resolution Policy, paragraph 4,¹⁴ form the backbone of every decision:

• (1) ICS: Domain name is Identical or Confusingly Similar to a trademark or service mark in which the complainant has rights

• (2) NRLI: Respondent has No Rights or Legitimate Interests in respect of the domain name

• (3) Bad Faith: Domain name has been registered and is being used in Bad Faith

For element (2), NRLI, although the dispute is typically approached from the point of view of the complainant demonstrating that the respondent has NRLI, it is very often the case that the panel considers the rights or legitimate interests of the complainant and/or the respondent. In that case, RLI is available as an Issue tag. In addition, the domain name resolution procedure allows for situations in which the complainant abuses the process by filing the complaint in bad faith (CIBF).¹⁵

The schema thus consists of five Issue tags, plus an Other category:

• ICS

• NRLI

• RLI

• BadFaith

• CIBF

• OTHER.

1.3 Factors

In our annotation scheme factors are the elements which we hypothesize will prove most useful for explainable legal prediction. The factors and corresponding tags are specific to the WIPO issues. For ICS, the ICANN policy does not explicitly identify specific factors that will be considered by the panel, so our tag set for ICS is derived from factual findings commonly observed in the data, such as CownsTM (Complainant owns Trademark) and TMentire (Trademark is contained in its entirety within the Domain Name). For NRLI/RLI, the policy establishes three factors, and for Bad Faith, four factors. Each of these has a corresponding tag. For example, under NRLI there is PriorBizUse from 4(c)(i) of the policy (“Bona fide business use of Domain Name or demonstrable preparations to do so, prior to notice of the dispute”) and under BadFaith there is Confusion4CommGain from 4(b)(iv) of the policy “For commercial gain from confusion with complainant’s mark”). The tag set also includes labels for other common factors observed in the data, such as PrimaFacieEst (Prima Facie Case Established). For CIBF, two factor tags are available: RDNH (Reverse Domain Name Hijacking) and Harass (complaint brought primarily to harass DN holder).

Each level of annotation also has an “Other” option to be used when none of the predefined tags is appropriate, and there is a free-form Comment field which the annotator can use to capture ad hoc labels and enter notes.

Fig. 5

Four text spans annotated with factual and legal-findings features

1.4 Attributes

A Citation attribute is used to capture the paragraph citation of Policy and Case Rule argument elements. A polarity attribute is used to capture positive/negative values for issues and factors. Figure 5 shows four typical annotations.