Context-aware MIML instance annotation: exploiting label correlations with classifier chains

Abstract

In multi-instance multi-label (MIML) instance annotation, the goal is to learn an instance classifier while training on a MIML dataset, which consists of bags of instances paired with label sets; instance labels are not provided in the training data. The MIML formulation can be applied in many domains. For example, in an image domain, bags are images, instances are feature vectors representing segments in the images, and the label sets are lists of objects or categories present in each image. Although many MIML algorithms have been developed for predicting the label set of a new bag, only a few have been specifically designed to predict instance labels. We propose MIML-ECC (ensemble of classifier chains), which exploits bag-level context through label correlations to improve instance-level prediction accuracy. The proposed method is scalable in all dimensions of a problem (bags, instances, classes, and feature dimension) and has no parameters that require tuning (which is a problem for prior methods). In experiments on two image datasets, a bioacoustics dataset, and two artificial datasets, MIML-ECC achieves higher or comparable accuracy in comparison with several recent methods and baselines.

Appendix

The correlation coefficient $\rho (X,Y)$ between two random variables X and Y is given by

$$\begin{aligned} \rho (X,Y) = \frac{\hbox {Cov}(X,Y)}{\sqrt{\hbox {Var}(X)}\sqrt{\hbox {Var}(Y)}} \end{aligned}$$

(14)

Let $X$ be a Bernoulli RV with $P(X=1)=\frac{1}{2}$. Similarly, let $Y$ conditioned on $X$ be a Bernoulli RV with $P(Y=1|X)=\frac{1}{2} (1+\rho ) X + \frac{1}{2} (1-\rho ) (1-X)$, as in Sect. 5.5, Eq. (12). The correlation coefficient $\rho (X,Y) = \rho$.

Proof we begin by noting the property that the expected value of an arbitrary Bernoulli RV $T$ with $P(T=1)=p$ satisfies $E[T] = P(T=1)=p$. Moreover, since $T \in \{0,1\}$, $T^k=T$ for $k=1,2,\ldots$ and consequently $E[T^k]=p$. The variance of $T$ is given by $\hbox {Var}(T)=E[T^2]-E[T]^2 = p - p^2 =p(1-p)$. To compute the correlation coefficient $\rho (X,Y)$, we first compute $E[X]$, $E[Y]$, $\hbox {Var}(X)$, $\hbox {Var}(Y)$, and $\hbox {Cov}(X,Y)$. Since $P(X=1)=\frac{1}{2}$, we have $E[X]=P(X=1)=\frac{1}{2}$. The expectation of $Y$ is computed as follows

$$\begin{aligned} E[Y]&= E_X [ E_Y [Y |X]] \nonumber \\&= E_X [P(Y=1|X)] \nonumber \\&= E_X \left[ \frac{1}{2} (1+\rho ) X + \frac{1}{2} (1-\rho ) (1-X)\right] \nonumber \\&= \frac{1}{2} (1+\rho ) E_X [X] + \frac{1}{2} (1 -\rho ) (1 -E[X]) \nonumber \\&= \frac{1}{2} (1+\rho ) \frac{1}{2} + \frac{1}{2} (1-\rho ) \frac{1}{2} \nonumber \\&= \frac{1}{2}. \end{aligned}$$

(15)

Since $X$ and $Y$ are Bernoulli RVs with $P(X=1)=P(Y=1)=E[X]=E[Y]=\frac{1}{2}$, we also have $\hbox {Var}(X)=\hbox {Var}(Y)=\frac{1}{2}(1-\frac{1}{2}) =\frac{1}{4}$. Next, we compute

$$\begin{aligned} E[X Y]&= E_X [ E_Y [X Y |X]] \nonumber \\&= E_X [ X E_Y [ Y |X]] \nonumber \\&= E_X [X P(Y=1|X)]\nonumber \\&= E_X\left[ X (\frac{1}{2} (1+\rho ) X + \frac{1}{2} (1-\rho ) (1-X))\right] \nonumber \\&= \frac{1}{2} (1+\rho ) E_X [X^2] + \frac{1}{2} (1 -\rho ) E[X(1-X)] \nonumber \\&= \frac{1}{2} (1+\rho ) \frac{1}{2} \nonumber \\&= \frac{1}{4} (1+\rho ). \end{aligned}$$

(16)

The covariance is therefore $\hbox {Cov}(X,Y)=E[X Y] -E[X]E[Y] \!=\! \frac{1}{4} (1+\rho ) \!-\! \frac{1}{2}^2 = \frac{1}{4} \rho .$ Finally, substituting $\hbox {Var}(X)\!=\!\hbox {Var}(Y)\!=\!\frac{1}{4}$ and $\hbox {Cov}(X,Y)\!=\!\frac{1}{4} \rho$ into (14), yields $\rho (X,Y)~=~\rho$.