Evaluating machine-assisted annotation in under-resourced settings

Abstract

Machine assistance is vital to managing the cost of corpus annotation projects. Identifying effective forms of machine assistance through principled evaluation is particularly important and challenging in under-resourced domains and highly heterogeneous corpora, as the quality of machine assistance varies. We perform a fine-grained evaluation of two machine-assistance techniques in the context of an under-resourced corpus annotation project. This evaluation requires a carefully controlled user study crafted to test a number of specific hypotheses. We show that human annotators performing morphological analysis of text in a Semitic language perform their task significantly more accurately and quickly when even mediocre pre-annotations are provided. When pre-annotations are at least 70 % accurate, annotator speed and accuracy show statistically significant relative improvements of 25–35 and 5–7 %, respectively. However, controlled user studies are too costly to be suitable for under-resourced corpus annotation projects. Thus, we also present an alternative analysis methodology that models the data as a combination of latent variables in a Bayesian framework. We show that modeling the effects of interesting confounding factors can generate useful insights. In particular, correction propagation appears to be most effective for our task when implemented with minimal user involvement. More importantly, by explicitly accounting for confounding variables, this approach has the potential to yield fine-grained evaluations using data collected in a natural environment outside of costly controlled user studies.

Appendices

Appendix 1: RStan time model

Appendix 2: RStan accuracy model

Appendix 3: Derivation of complete conditionals

This appendix walks through a mathematical derivation of the complete conditional distributions necessary to use Gibbs sampling to obtain samples from the posterior probability distributions over the unobserved parameters in the latent variable model of annotation time first presented in Sect. 5.3.1.

Note that the following derivation is not required by those simply wishing to formulate and run a model such as that described in Sect. 5.3 using existing statistical computing libraries (see "Appendix 2" section). The following derivation is for those readers whose background may not be in statistics, but wish to acquaint themselves with the details required to implement own Gibbs sampler.

1.1 Introduction

A complete conditional distribution for a variable represents the probability of that variable given the priors, the data, and values for every other parameter in the model. These distributions are obtained by first calculating ln(g), the unnormalized joint posterior distribution. After it is defined, ln(g) is used as a basis for finding the complete conditional distributions over each parameter.

We first take a moment to summarize the model. We model the number of seconds taken to annotate a word y _hatbro as a combination of the following variables:

1.2 Variables

• σ ² Variance common to all words

• θ _h Annotators

• α _a Current condition

• τ _t Grammatical Category

• β _b Bucketed word position (0,1,2,3+)

• ρ _r Hyperlinks clicked

• ω _o Hyperlinks shown

• κ Offset common to all words

1.3 Priors

Prior justifications may be found in Sect. 5.3.1. Gamma distributions are parameterized by shape and scale.

• σ² ∼ Gamma(50,50)

• $\theta_h \sim N(0,\frac{40}{3})$

• $\alpha_a \sim N(0,\frac{40}{3})$

• $\tau_t \sim N(0,\frac{40}{3})$

• $\beta_b \sim N(0,\frac{40}{3})$

• $\rho_r \sim N(0,\frac{40}{3})$

• $\omega_o \sim N(0,\frac{40}{3})$

• $\kappa \sim N(90,\frac{50}{3})$

1.4 Likelihood

The density of a single data point is distributed as y _hatbro |θ _h , α _a , τ _t , β _b , ρ _r , ω _o , κ ∼ N(θ _h  + α _a  + τ _t  + β _b  + ρ _r  + ω _o  + κ, σ²) Assuming that the probability of each data point is independent, the likelihood, or probability of the data set, may be written as the product of the probability of each data point.

$$ \begin{aligned} L(\underline{y}|\underline{\varTheta}) &= L(\underline{y}|\sigma^2,\underline{\theta},\underline{\alpha},\underline{\tau}, \underline{\beta},\underline{\rho},\underline{\omega},\kappa)\\ &= \prod_{y_{hatbro} \in \underline{y}} p(y_{hatbro}|\sigma^2,\theta_h,\alpha_a,\tau_t,\beta_b,\rho_r,\omega_o,\kappa)\\ &= \prod_{y_{hatbro} \in \underline{y}} (2\pi)^{-\frac{1}{2}} (\sigma^2)^{-\frac{1}{2}} e^{-\frac{1}{2\sigma^2} (y_{hatbro}-(\theta_h+\alpha_a+\tau_t+\beta_b+\rho_r+\omega_o+\kappa))^2}\\ &= \prod_{y_{hatbro} \in \underline{y}} (2\pi)^{-\frac{1}{2}} (\sigma^2)^{-\frac{1}{2}} e^{-\frac{1}{2\sigma^2} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2}\\ &= (2\pi)^{-\frac{N}{2}} (\sigma^2)^{-\frac{N}{2}} e^{-\frac{1}{2\sigma^2} \sum_{y_{hatbro} \in \underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2} \end{aligned} $$

1.5 Joint posterior

Our end goal is to estimate the joint posterior distribution over all of our parameters given the evidence provided by the data, $p(\underline{\varTheta}|\underline{y})$, where $\underline{\varTheta}$ represents all of our parameters. Using Bayes’ rule, we write the posterior as a combination of the likelihood of the data and our prior probability distributions. $p(\underline{\varTheta}|\underline{y}) = \frac{L(\underline{y}|\underline{\varTheta})p(\underline{\varTheta})}{p(\underline{y})} = \frac{L(\underline{y}|\underline{\varTheta})p(\underline{\varTheta})}{\int\cdots\int L(\underline{y}|\underline{\varTheta})p(\underline{\varTheta}) d\underline{\varTheta}}$. Because the normalizing constant in the denominator of this quantity involves integrating over all $\underline{\varTheta}$’s it is intractable to compute. Fortunately, using Gibbs sampling to get samples from the joint posterior does not require being able to compute the normalizing constant. We drop the constant and calculate the numerator of our joint posterior, which we will call g. Recall that because the denominator of the posterior is a constant, g is proportional to the posterior distribution. We will similarly drop any other constants we find as we derive g. Finally, because of machine precision issues when working with small probabilities, it is most useful to work directly with the logarithm of g.

$$ ln(g) = ln(L(\underline{y}|\underline{\varTheta})p(\underline{\varTheta})) $$

Now insert our own parameter names.

$$= ln(L(\underline{y}|\sigma^2,\underline{\theta},\underline{\alpha}, \underline{\tau},\underline{\beta},\underline{\rho},\underline{\omega},\kappa) p(\sigma^2,\underline{\theta},\underline{\alpha}, \underline{\tau},\underline{\beta},\underline{\rho},\underline{\omega},\kappa)) $$

Because our parameters are all independent of one another, we can write their joint prior probabilities as a product of individual prior probabilities.

$$ \begin{aligned} &=ln(L(\underline{y}|\sigma^2,\underline{\theta},\underline{\alpha},\underline{\tau}, \underline{\beta},\underline{\rho},\underline{\omega},\kappa) p(\sigma^2) \prod_{h=1}^H p(\theta_h) \prod_{a=1}^A p(\alpha_a)\\ &\quad\prod_{t=1}^T p(\tau_t) \prod_{b=1}^B p(\beta_b) \prod_{r=1}^R p(\rho_r) \prod_{o=1}^O p(\omega_o) p(\kappa) ) \end{aligned} $$

Distribute the logarithm.

$$ \begin{aligned} &= ln(L(\underline{y}|\sigma^2,\underline{\theta},\underline{\alpha}, \underline{\tau},\underline{\beta},\underline{\rho},\underline{\omega},\kappa))\\ &+ln(p(\sigma^2)) +\sum_{h=1}^H ln(p(\theta_h))\\ &+\sum_{a=1}^A ln(p(\alpha_a))\\ & +\sum_{t=1}^T ln(p(\tau_t))\\ &+\sum_{b=1}^B ln(p(\beta_b))\\ & +\sum_{r=1}^R ln(p(\rho_r))\\ & +\sum_{o=1}^O ln(p(\omega_o))\\ &+ln(p(\kappa)) \end{aligned} $$

Now substitute the numerical form of the likelihood and priors.

$$ \begin{aligned} &=ln((2\pi)^{-\frac{N}{2}} (\sigma^2)^{-\frac{N}{2}} e^{-\frac{1}{2\sigma^2} \sum_{y_{hatbro} \in \underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2})\\ &\quad+ln(\frac{1}{\Upgamma(50)50^{50}} \sigma^{2(50-1)} e^{-\frac{\sigma^2}{50}})\\ &\quad+\sum_{h=1}^H ln\left((2\pi*(\frac{40}{3})^2)^{-\frac{1}{2}} e^{-\frac{(\theta_h-0)^2}{2*(\frac{40}{3})^2} }\right)\\ &\quad+\sum_{a=1}^A ln\left( (2\pi*(\frac{40}{3})^2)^{-\frac{1}{2}} e^{-\frac{(\alpha_a-0)^2}{2*(\frac{40}{3})^2} }\right)\\ &\quad+\sum_{t=1}^T ln\left( (2\pi*(\frac{40}{3})^2)^{-\frac{1}{2}} e^{-\frac{(\tau_t-0)^2}{2*(\frac{40}{3})^2} }\right)\\ &\quad+\sum_{b=1}^B ln\left( (2\pi*(\frac{40}{3})^2)^{-\frac{1}{2}} e^{-\frac{(\beta_b-0)^2}{2*(\frac{40}{3})^2} }\right)\\ &\quad+\sum_{r=1}^R ln\left( (2\pi*(\frac{40}{3})^2)^{-\frac{1}{2}} e^{-\frac{(\rho_r-0)^2}{2*(\frac{40}{3})^2} }\right)\\ &\quad+\sum_{o=1}^O ln\left( (2\pi*(\frac{40}{3})^2)^{-\frac{1}{2}} e^{-\frac{(\omega_o-0)^2}{2*(\frac{40}{3})^2} }\right)\\ & \quad + ln\left((2\pi*(\frac{50}{3})^2)^{-\frac{1}{2}} e^{-\frac{(\kappa-90)^2}{2*(\frac{50}{3})^2} }\right) \end{aligned} $$

Distribute logarithms deeper into terms and drop additive constants.

$$ \begin{aligned} &\propto - \frac{N}{2}ln(\sigma^2) - \frac{1}{2\sigma^2} \sum_{y_{hatbro} \in \underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2\\ &\quad+49ln(\sigma^2) - \frac{\sigma^2}{50}\\ &- \sum_{h=1}^H \frac{\theta_h^2}{2(\frac{40}{3})^2}\\ &\quad- \sum_{a=1}^A \frac{\alpha_a^2}{2(\frac{40}{3})^2}\\ &\quad- \sum_{t=1}^T \frac{\tau_t^2}{2(\frac{40}{3})^2}\\ &\quad- \sum_{b=1}^B\frac{\beta_b^2}{2(\frac{40}{3})^2}\\ &\quad- \sum_{r=1}^R \frac{\rho_r^2}{2(\frac{40}{3})^2}\\ &\quad- \sum_{o=1}^O \frac{\omega_o^2}{2(\frac{40}{3})^2}\\ &\quad-\frac{(\kappa-90)^2}{2(\frac{50}{3})^2} \end{aligned} $$

1.6 Complete conditionals

Now it remains to derive the complete conditional distributions of each parameter. A complete conditional distribution over parameter $\varTheta$ represents the probability of that parameter given the data and the value of every other parameter in the graph, and is necessary for Gibbs sampling to function correctly. It turns out that we can use g, which we have already calculated, to derive the complete conditional of each parameter simply by treating all variables except the parameter of interest as constants, and dropping as many of these constants as possible. Because g is not normalized, and complete conditionals are defined as proper probability distributions, we symbolically add to each complete conditional the constant c that would correctly normalize it. However, this is only for form’s sake, since Gibbs sampling does not require normalized complete conditionals. Again, because of machine precision issues, we express our conditionals in log space. Because these distribution are not in the form of a known distribution that we can easily sample from, it is necessary to do Metropolis-Hastings within Gibbs sampling. For details on the Metropolis-Hastings algorithm, we refer the reader to Gelman (2004).

$$ \begin{aligned} &\left[ \sigma^2 \right] = (49-\frac{N}{2})ln(\sigma^2) - \frac{\sigma^2}{50} - \frac{1}{2\sigma^2} \sum_{\underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2+ c \\ &\left[ \alpha_a \right] = - \frac{\alpha_a^2}{2(\frac{40}{3})^2} - \frac{1}{2\sigma^2} \sum_{\underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2 + c \\ &\left[ \theta_h \right] = - \frac{\theta_h^2}{2(\frac{40}{3})^2} - \frac{1}{2\sigma^2} \sum_{\underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2 + c \\\ &\left[ \beta_b \right] = - \frac{\beta_b^2}{2(\frac{40}{3})^2} - \frac{1}{2\sigma^2} \sum_{\underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2 + c \\ &\left[ \tau_t \right] = - \frac{\tau_t^2}{2(\frac{40}{3})^2} - \frac{1}{2\sigma^2} \sum_{\underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2 + c \\ &\left[ \rho_r \right] = - \frac{\rho_r^2}{2(\frac{40}{3})^2} - \frac{1}{2\sigma^2} \sum_{\underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2 + c \\ &\left[ \omega_o \right] = - \frac{\omega_o^2}{2(\frac{40}{3})^2} - \frac{1}{2\sigma^2} \sum_{\underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2 + c \\ &\left[ \kappa \right] = -\frac{(\kappa-90)^2}{2(\frac{50}{3})^2} - \frac{1}{2\sigma^2} \sum_{\underline{y}} (y_{hatbro}-\theta_h-\alpha_a-\tau_t-\beta_b-\rho_r-\omega_o-\kappa)^2 + c\\ \end{aligned} $$