Inference Methods for CRFs with Co-occurrence Statistics

Abstract

The Markov and Conditional random fields (CRFs) used in computer vision typically model only local interactions between variables, as this is generally thought to be the only case that is computationally tractable. In this paper we consider a class of global potentials defined over all variables in the CRF. We show how they can be readily optimised using standard graph cut algorithms at little extra expense compared to a standard pairwise field. This result can be directly used for the problem of class based image segmentation which has seen increasing recent interest within computer vision. Here the aim is to assign a label to each pixel of a given image from a set of possible object classes. Typically these methods use random fields to model local interactions between pixels or super-pixels. One of the cues that helps recognition is global object co-occurrence statistics, a measure of which classes (such as chair or motorbike) are likely to occur in the same image together. There have been several approaches proposed to exploit this property, but all of them suffer from different limitations and typically carry a high computational cost, preventing their application on large images. We find that the new model we propose produces a significant improvement in the labelling compared to just using a pairwise model and that this improvement increases as the number of labels increases.

Appendix

Proof of Lemma 1

First we show that:

$$\begin{aligned} E_\alpha (\mathbf{t})&= \min _{z_\alpha } [(C_{\alpha \beta } - C_{\beta }) (1-z_\alpha ) \nonumber \\&+ \sum _{i \in \mathcal{V}_{\alpha \beta }} (C_{\alpha \beta } - C_{\beta }) (1-t_i) z_\alpha ] \nonumber \\&= \left\{ \begin{array}{ll} 0&\text{ if} \forall i \in \mathcal{V}_{\alpha \beta }: t_i = 1,\\ C_{\alpha \beta } - C_\beta&\text{ otherwise}. \end{array} \right. \end{aligned}$$

(61)

If $\forall i \in \mathcal{V}_{\alpha \beta } : t_i = 1$ then $\sum _{i \in \mathcal{V}_{\alpha \beta }} (C_{\alpha \beta } - C_{\beta }) (1-t_i) z_\alpha = 0$ and the minimum cost cost $0$ occurs when $z_\alpha =1.$ If $\exists i \in \mathcal{V}_{\alpha \beta } , t_i = 0$ the minimum cost labelling occurs when $z_\alpha =0$ and the minimum cost is $C_{\alpha \beta }-C_\beta .$ Similarly:

$$\begin{aligned} E_\beta (\mathbf{t})&= \min _{z_\beta } [(C_{\alpha \beta } - C_{\alpha }) z_\beta \nonumber \\&+ \sum _{i \in \mathcal{V}_{\alpha \beta }} (C_{\alpha , \beta } - C_{\alpha }) t_i (1 - z_\beta )] \nonumber \\&= \left\{ \begin{array}{l@{\quad }l} 0&\text{ if} \forall i \in \mathcal{V}_{\alpha \beta }: t_i = 0,\\ C_{\alpha \beta } - C_\alpha&\text{ otherwise}. \end{array} \right. \end{aligned}$$

(62)

By inspection, if $\forall i \in \mathcal{V}_{\alpha \beta } : t_i = 0$ then $\sum _{i \in \mathcal{V}_{\alpha \beta }}(C_{\alpha , \beta } - C_{\alpha }) t_i (1 - z_\beta ) = 0$ and the minimum cost cost 0 occurs when $z_\beta =0.$ If $\exists i \in \mathcal{V}_{\alpha \beta } , t_i = 1$ the minimum cost labelling occurs when $z_\beta =1$ and the minimum cost is $C_{\alpha \beta } - C_\alpha .$

For all three cases (all pixels take label $\alpha ,$ all pixels take label $\beta$ and mixed labelling) $E(\mathbf{t}) = E_\alpha (\mathbf{t}) + E_\beta (\mathbf{t}) + C_{\alpha } + C_{\beta } - C_{\alpha \beta }.$ The construction of the $\alpha \beta {\text{-swap} }$ move is similar to the Robust $P^N$ model (Kohli et al. 2008).$\square$

See Figs. 2 and  3 for graph construction.

Proof of Lemma 2

Similarly to the $\alpha \beta {\text{-swap} }$ proof we can show:

$$\begin{aligned} E_\alpha (\mathbf{t})&= \min _{z_\alpha } \bigg [k^{\prime }_\alpha (1-z_\alpha ) + \sum _{i \in \mathcal{V}} k^{\prime }_\alpha (1-t_i) z_\alpha \bigg ]\nonumber \\&= \left\{ \begin{array}{l@{\quad }l} k^{\prime }_\alpha&\text{ if} \exists i \in \mathcal{V} \text{ s.t.} t_i = 0,\\ 0&\text{ otherwise} . \end{array} \right. \end{aligned}$$

(63)

If $\exists i \in \mathcal{V} s.t. t_i = 0,$ then $\sum _{i \in \mathcal{V}} k^{\prime }_\alpha (1-t_i) \ge k^{\prime }_\alpha ,$ the minimum is reached when $z_\alpha = 0$ and the cost is $k^{\prime }_\alpha .$

If $\forall i \in \mathcal{V} : t_i = 1$ then $k^{\prime }_\alpha (1-t_i) z_\alpha = 0,$ the minimum is reached when $z_\alpha = 1$ and the cost becomes $0.$

For all other $l \in A$:

$$\begin{aligned} E_b(\mathbf{t})&= \min _{z_l} \bigg [k^{\prime \prime }_l z_l + \sum _{i \in \mathcal{V}_l} k^{\prime \prime }_l t_i (1 - z_l) \bigg ]\nonumber \\&= \left\{ \begin{array}{l@{\quad }l} k^{\prime \prime }_l&\text{ if} \exists i \in \mathcal{V}_l \text{ s.t.} t_i = 1,\\ 0&\text{ otherwise} . \end{array} \right. \end{aligned}$$

(64)

If $\exists i \in \mathcal{V}_l$ s.t. $t_i = 1,$ then $\sum _{i \in \mathcal{V}_l} k^{\prime \prime }_l t_i \ge k^{\prime \prime }_l,$ the minimum is reached when $z_l = 1$ and the cost is $k^{\prime \prime }_l.$

If $\forall i \in \mathcal{V}_l : t_i = 0$ then $\sum _{i \in \mathcal{V}_l} k^{\prime \prime }_l t_i (1 - z_l) = 0,$ the minimum is reached when $z_l = 1$ and the cost becomes $0.$

By summing up the cost $E_\alpha (\mathbf{t})$ and $|A|$ costs $E_l(\mathbf{t})$ we get $E^{\prime }(\mathbf{t}) = E_\alpha (\mathbf{t}) + \sum _{l \in A} E_l(\mathbf{t}).$ If $\alpha$ is already present in the image $k^{\prime }_\alpha = 0$ and edges with this weight and variable $z_\alpha$ can be ignored. $\square$

See Figs. 2 and  3 for graph construction.