Approximate computations for binary Markov random fields and their use in Bayesian models

Abstract

Discrete Markov random fields form a natural class of models to represent images and spatial datasets. The use of such models is, however, hampered by a computationally intractable normalising constant. This makes parameter estimation and a fully Bayesian treatment of discrete Markov random fields difficult. We apply approximation theory for pseudo-Boolean functions to binary Markov random fields and construct approximations and upper and lower bounds for the associated computationally intractable normalising constant. As a by-product of this process we also get a partially ordered Markov model approximation of the binary Markov random field. We present numerical examples with both the pairwise interaction Ising model and with higher-order interaction models, showing the quality of our approximations and bounds. We also present simulation examples and one real data example demonstrating how the approximations and bounds can be applied for parameter estimation and to handle a fully Bayesian model computationally.

Appendices

Appendix: Proof of Theorem 3

Expanding $\hbox {SSE}(f,\tilde{\tilde{f}}) = \sum _{x \in {\varOmega }} \left\{ f(x)-\tilde{\tilde{f}}(x) \right\} ^2$ we get,

$$\begin{aligned}&\sum _{x \in {\varOmega }} \left\{ f(x)-\tilde{\tilde{f}}(x) \right\} ^2 = \sum _{x \in {\varOmega }} \left\{ f(x)-\tilde{f}(x) + \tilde{f}(x)-\tilde{\tilde{f}}(x) \right\} ^2\\&= \sum _{x \in {\varOmega }} \left\{ f(x)-\tilde{f}(x) \right\} ^2 + \sum _{x \in {\varOmega }} \left\{ \tilde{f}(x)-\tilde{\tilde{f}}(x) \right\} ^2\\&\qquad +\sum _{x \in {\varOmega }} \left\{ f(x)-\tilde{f}(x)\right\} \left\{ \tilde{f}(x)-\tilde{\tilde{f}}(x)\right\} \\&= \hbox {SSE}(f,\tilde{f}) + \hbox {SSE}(\tilde{f},\tilde{\tilde{f}})\\&\qquad + \sum _{x \in {\varOmega }} \{f(x)-\tilde{f}(x)\}\tilde{f}(x)-\sum _{x \in {\varOmega }} \{f(x)-\tilde{f}(x)\}\tilde{\tilde{f}}(x). \end{aligned}$$

To prove the theorem it is thereby sufficient to show that,

$$\begin{aligned} \sum _{x \in {\varOmega }}\left\{ f(x)-\tilde{f}(x) \right\} \tilde{f}(x)-\sum _{x \in {\varOmega }}\left\{ f(x)-\tilde{f}(x)\right\} \tilde{\tilde{f}}(x) = 0. \end{aligned}$$

First recall that we from (9) know that,

$$\begin{aligned} \sum _{x \in {\varOmega }_\lambda }\left\{ f(x) - \tilde{f}(x) \right\} = 0, \quad \text { } \forall \text { } \lambda \in \tilde{S}. \end{aligned}$$

(42)

Also, since $\tilde{\tilde{S}} \subseteq \tilde{S}$,

$$\begin{aligned} \sum _{x \in {\varOmega }_\lambda }\left\{ f(x) - \tilde{f}(x) \right\} = 0, \quad \text { } \forall \text { } \lambda \in \tilde{\tilde{S}}. \end{aligned}$$

(43)

We study the first term, $\sum _{x \in {\varOmega }}\{f(x)-\tilde{f}(x)\}\tilde{f}(x)$, expand the expression for $\tilde{f}(x)$ outside the parenthesis and change the order of summation,

$$\begin{aligned}&\sum _{x \in {\varOmega }}\{f(x)-\tilde{f}(x)\}\tilde{f}(x)\\&\quad = \sum _{x \in {\varOmega }}\left[ \{f(x)-\tilde{f}(x)\} \sum _{{\varLambda }\in \tilde{S}} \tilde{\beta }^{{\varLambda }} \prod _{k \in {\varLambda }}x_k \right] \\&\quad = \sum _{{\varLambda }\in \tilde{S}} \left( \tilde{\beta }^{{\varLambda }} \sum _{x \in {\varOmega }} \left[ \prod _{k \in {\varLambda }}x_k\{f(x)-\tilde{f}(x)\} \right] \right) \\&\quad = \sum _{{\varLambda }\in \tilde{S}} \left[ \tilde{\beta }^{{\varLambda }} \sum _{x \in {\varOmega }_{\varLambda }} \left\{ f(x)-\tilde{f}(x) \right\} \right] = 0, \end{aligned}$$

where the last transition follows from (42). Using (43) we can correspondingly show that $\sum _{x \in {\varOmega }} \{f(x)-\tilde{f}(x)\}\tilde{\tilde{f}}(x) = 0$.

Proof of Theorem 4

We study the error sum of squares,

$$\begin{aligned}&\sum _{x \in {\varOmega }} \left\{ f(x) - \tilde{f}(x) \right\} ^2\\&\quad = \sum _{x \in {\varOmega }}\left[ \{f(x)-\tilde{f}(x)\}f(x) \right] \\&\qquad -\sum _{x \in {\varOmega }}\left[ \{f(x)-\tilde{f}(x)\}\tilde{f}(x) \right] \\&\quad = \sum _{x \in {\varOmega }} \left[ \sum _{{\varLambda }\in S} \beta ^{{\varLambda }} \{f(x)-\tilde{f}(x)\}\prod _{k \in {\varLambda }}x_k \right] \\&\quad \quad - \sum _{x \in {\varOmega }} \left[ \sum _{{\varLambda }\in \tilde{S}} \tilde{\beta }^{{\varLambda }} \{f(x)-\tilde{f}(x)\}\prod _{k \in {\varLambda }}x_k \right] \\&\quad = \sum _{{\varLambda }\in S} \beta ^{{\varLambda }} \left[ \sum _{x \in {\varOmega }_{{\varLambda }}} \{f(x)-\tilde{f}(x)\} \right] \\&\quad \quad - \sum _{{\varLambda }\in \tilde{S}} \tilde{\beta }^{{\varLambda }} \left[ \sum _{x \in {\varOmega }_{{\varLambda }}} \{f(x)-\tilde{f}(x)\} \right] , \end{aligned}$$

where the second sum is always zero by (43). Since $\tilde{S} \subseteq S$, the first sum can be further split into two parts,

$$\begin{aligned}&\sum _{{\varLambda }\in S} \beta ^{{\varLambda }} \left[ \sum _{x \in {\varOmega }_{{\varLambda }}} \{f(x)-\tilde{f}(x)\} \right] \\&\quad = \sum _{{\varLambda }\in \tilde{S}} \beta ^{{\varLambda }} \left[ \sum _{x \in {\varOmega }_{{\varLambda }}} \{f(x)-\tilde{f}(x)\} \right] \\&\quad \quad + \sum _{{\varLambda }\in S\setminus \tilde{S}} \beta ^{{\varLambda }} \left[ \sum _{x \in {\varOmega }_{{\varLambda }}} \{f(x)-\tilde{f}(x)\} \right] , \end{aligned}$$

where once again the first sum is zero.

Proof of Theorem 5

From Theorem 1 it follows that it is sufficient to consider a function f(x) with non-zero interactions $\beta ^{\varLambda }$ only for ${\varLambda }\in S_{\{ i,j\}}$, since we only need to focus on the interactions we want to remove. Thus we have

$$\begin{aligned} f(x) = \sum _{{\varLambda }\in S_{\{ i,j\}}} \beta ^{\varLambda }\prod _{k\in {\varLambda }}x_k \hbox {~~~and~~~} \widetilde{f}(x) = \sum _{{\varLambda }\in \widetilde{S}} \widetilde{\beta }^{\varLambda }\prod _{k\in {\varLambda }} x_k, \end{aligned}$$

and we need to show that then

$$\begin{aligned} \tilde{\beta }^{\varLambda }= \left\{ \begin{array}{@{}ll} - \frac{1}{4}\beta ^{{\varLambda }\cup \{ i,j\}} &{} \hbox {~~if }{\varLambda }\cup \{i,j\} \in S, \\ \frac{1}{2}\beta ^{{\varLambda }\cup \{ i\}} &{} \hbox {~~if }{\varLambda }\cup \{ i\}\in S \hbox { and }{\varLambda }\cup \{ j\}\not \in S, \\ \frac{1}{2}\beta ^{{\varLambda }\cup \{ j\}} &{} \hbox {~~if }{\varLambda }\cup \{ i\}\not \in S \hbox { and }{\varLambda }\cup \{ j\}\in S, \\ 0 &{} \hbox {~~otherwise.} \end{array}\right. \end{aligned}$$

(44)

We start by defining the sets

$$\begin{aligned} R_{\varLambda }= \{ {\varLambda }\setminus \{ i\},{\varLambda }\setminus \{ j\}, {\varLambda }\setminus \{ i,j\}\} \hbox {~~for }{\varLambda }\in S_{\{i,j\}}, \end{aligned}$$

and note that these sets are disjoint, and, since we have assumed S to be dense, $R_{\varLambda }\subseteq \widetilde{S}$. Defining also the residue set

$$\begin{aligned} T = \widetilde{S} \setminus \left( \bigcup _{{\varLambda }\in S_{\{ i,j\}}} R_{\varLambda }\right) \end{aligned}$$

we may write the approximation error $f(x)-\widetilde{f}(x)$ in the following form,

$$\begin{aligned}&f(x)-\widetilde{f}(x) \\&\quad =\sum _{{\varLambda }\in S_{\{ i,j\}}} \left\{ \left( \beta ^{\varLambda }x_ix_j - \sum _{\lambda \in R_{\varLambda }} \widetilde{\beta }^\lambda \prod _{k\in {\varLambda }\setminus \lambda } x_k \right) \prod _{k\in {\varLambda }\setminus \{ i,j\}} x_k\right\} \\&\qquad - \sum _{{\varLambda }\in T} \widetilde{\beta }^{\varLambda }\prod _{k\in {\varLambda }} x_k. \end{aligned}$$

Defining

$$\begin{aligned} \Delta f^{\varLambda }(x_i,x_j)= & {} \beta ^{\varLambda }x_i x_j - \sum _{\lambda \in R_{\varLambda }} \widetilde{\beta }^\lambda \prod _{k\in {\varLambda }\setminus \lambda }x_k\\= & {} \beta ^{\varLambda }x_ix_j - \left( \widetilde{\beta }^{{\varLambda }\setminus \{ i,j\}} + \widetilde{\beta }^{{\varLambda }\setminus \{ j\}}x_i + \widetilde{\beta }^{{\varLambda }\setminus \{ i\}}x_j\right) \end{aligned}$$

we have

$$\begin{aligned} f(x)-\widetilde{f}(x)= & {} \sum _{{\varLambda }\in S_{\{ i,j\}}} \Delta f^{\varLambda }(x_i,x_j) \prod _{k\in {\varLambda }\setminus \{ i,j\}} x_k\nonumber \\&-\sum _{{\varLambda }\in T} \widetilde{\beta }^{\varLambda }\prod _{k\in {\varLambda }}x_k. \end{aligned}$$

(45)

Inserting this into (9) and switching the order of summation we get

$$\begin{aligned}&\sum _{x\in {\varOmega }_\lambda }\left\{ f(x)-\widetilde{f}(x)\right\} \nonumber \\&\quad =\sum _{x\in {\varOmega }_\lambda }\left( \sum _{{\varLambda }\in S_{\{ i,j\}}} \Delta f^{\varLambda }(x_i,x_j) \prod _{k\in {\varLambda }\setminus \{ i,j\}} x_k\right. \nonumber \\&\left. \qquad - \sum _{{\varLambda }\in T} \widetilde{\beta }^{\varLambda }\prod _{k\in {\varLambda }}x_k\right) \nonumber \\&\quad = \sum _{{\varLambda }\in S_{\{ i,j\}}} \left( \sum _{x\in {\varOmega }_\lambda } \Delta f^{\varLambda }(x_i,x_j) \prod _{k\in {\varLambda }\setminus \{ i,j\}} x_k\right) \nonumber \\&\qquad - \sum _{{\varLambda }\in T}\left( \sum _{x\in {\varOmega }_\lambda } \widetilde{\beta }^{\varLambda }\prod _{k\in {\varLambda }} x_k\right) \nonumber \\&\quad =\sum _{{\varLambda }\in S_{\{ i,j\}}} \left( \sum _{x\in {\varOmega }_{\lambda \cup ({\varLambda }\setminus \{ i,j\}}} \Delta f^{\varLambda }(x_i,x_j)\right) \nonumber \\&\quad \quad - \sum _{{\varLambda }\in T}\left( \sum _{x\in {\varOmega }_{\lambda \cup T}} \widetilde{\beta }^{\varLambda }\right) = 0 \end{aligned}$$

(46)

for all $\lambda \in \widetilde{S}$. We now proceed to show that this system of equations has a solution where $\widetilde{\beta }^{\varLambda }= 0$ for ${\varLambda }\in T$ and $\sum _{x\in {\varOmega }_{\lambda \cup ({\varLambda }\setminus \{ i,j\}}} \Delta f^{\varLambda }(x_i,x_j) = 0$ for each ${\varLambda }\in S_{\{ i,j\}}$. Obviously for each ${\varLambda }$ the function $\Delta f^{\varLambda }(x_i,x_j)$ has only our possible values, namely $\Delta f^{\varLambda }(0,0)$, $\Delta f^{\varLambda }(1,0)$, $\Delta ^{\varLambda }(0,1)$ and $\Delta f^{\varLambda }(1,1)$. Thus the sum $\sum _{x\in {\varOmega }_{\lambda \cup ({\varLambda }\setminus \{ i,j\}}} \Delta f^{\varLambda }(x_i,x_j)$ is simply given as a sum over these four values multiplied by the number of times they occur. Consider first the case where $\lambda$, and thereby also $\lambda \cup ({\varLambda }\setminus \{ i,j\})$ does not contain i or j. Then the four values $\Delta f^{\varLambda }(0,0)$, $\Delta f^{\varLambda }(1,0)$, $\Delta ^{\varLambda }(0,1)$ and $\Delta f^{\varLambda }(1,1)$ will occur the same number of times, so

$$\begin{aligned}&\sum _{x\in {\varOmega }_{\lambda \cup ({\varLambda }\setminus \{ i,j\})}} \Delta f^{\varLambda }(x_i,x_j) \nonumber \\&\quad =\frac{|{\varOmega }_{\lambda \cup ({\varLambda }\setminus \{ i,j\}}|}{4} \left( \Delta f^{\varLambda }(0,0)+\Delta f^{\varLambda }(1,0)\right. \nonumber \\&\qquad \left. +\,\Delta f^{\varLambda }(0,1) + \Delta f^{\varLambda }(1,1)\right) . \end{aligned}$$

Next consider the case when $\lambda$, and thereby also $\lambda \cup ({\varLambda }\setminus \{ i,j\})$, contains i, but not j. Then $x_i=1$ in all terms in the sum, so the values $\Delta f^{\varLambda }(0,0)$ and $\Delta f^{\varLambda }(0,1)$ will not occur, whereas the values $\Delta f^{\varLambda }(1,0)$ and $\Delta f^{\varLambda }(1,1)$ will occur the same number of times. Thus,

$$\begin{aligned} \sum _{x\in {\varOmega }_{\lambda \cup ({\varLambda }\setminus \{ i,j\})}} \Delta f^{\varLambda }(x_i,x_j)= & {} \frac{|{\varOmega }_{\lambda \cup ({\varLambda }\setminus \{ i,j\}}|}{2} \left\{ \Delta f^{\varLambda }(1,0)\right. \\&+\left. \Delta f^{\varLambda }(1,1)\right\} . \end{aligned}$$

When $\lambda$ contains j, but not i we correspondingly get

$$\begin{aligned} \sum _{x\in {\varOmega }_{\lambda \cup ({\varLambda }\setminus \{ i,j\})}} \Delta f^{\varLambda }(x_i,x_j)= & {} \frac{|{\varOmega }_{\lambda \cup ({\varLambda }\setminus \{ i,j\}}|}{2} \left\{ \Delta f^{\varLambda }(0,1)\right. \\&+ \left. \Delta f^{\varLambda }(1,1)\right\} . \end{aligned}$$

The final case, that $\lambda$ contains both i and j, will never occur since $\lambda \in \widetilde{S}$ and all interaction involving both i and j have been removed from $\widetilde{S}$. We can now reach the conclusion that if we can find a solution for

$$\begin{aligned}&\Delta f^{\varLambda }(0,0)+\Delta f^{\varLambda }(1,0)+\Delta f^{\varLambda }(0,1)+\Delta (1,1) = 0,\\&\Delta f^{\varLambda }(1,0)+\Delta f^{\varLambda }(1,1) = 0, \\&\Delta f^{\varLambda }(0,1)+\Delta f^{\varLambda }(1,1) = 0, \end{aligned}$$

for all ${\varLambda }\in S_{\{ i,j\}}$ we also have a solution for (46) as discussed above. Using our expression for $\Delta f^{\varLambda }(x_i,x_j)$, the above three equations become

$$\begin{aligned}&\beta ^{\varLambda }- \left( 4\widetilde{\beta }^{{\varLambda }\setminus \{ i,j\}} + 2\widetilde{\beta }^{{\varLambda }\setminus \{ i\}} + 2\widetilde{\beta }^{{\varLambda }\setminus \{ j\}}\right) =0,\\&\beta ^{\varLambda }- \left( 2\widetilde{\beta }^{{\varLambda }\setminus \{ i,j\}} + \widetilde{\beta }^{{\varLambda }\setminus \{ i\}} + 2\widetilde{\beta }^{{\varLambda }\setminus \{ j\}}\right) =0,\\&\beta ^{\varLambda }- \left( 2\widetilde{\beta }^{{\varLambda }\setminus \{ i,j\}} + 2\widetilde{\beta }^{{\varLambda }\setminus \{ i\}} + \widetilde{\beta }^{{\varLambda }\setminus \{ j\}}\right) =0. \end{aligned}$$

Since the sets $R_{\varLambda }$ are disjoint, the three equations above can be solved separately for each ${\varLambda }$, and the solution is $\widetilde{\beta }^{{\varLambda }\setminus \{ i,j\}} = -\frac{1}{4}\beta ^{\varLambda }$ and $\widetilde{\beta }^{{\varLambda }\setminus \{ i\}}=\widetilde{\beta }^{{\varLambda }\setminus \{ j\}}= \frac{1}{2}\beta ^{\varLambda }$. Together with $\widetilde{\beta }^{\varLambda }=0$ for ${\varLambda }\in T$ this is equivalent to (13) in the theorem. Inserting the values we have found for $\widetilde{\beta }^{\varLambda }$ in (45) we get

$$\begin{aligned} \Delta f^{\varLambda }(x_i,x_j) = \left( x_ix_j + \frac{1}{4} - \frac{1}{2}x_i - \frac{1}{2}x_j\right) \beta ^{\varLambda }. \end{aligned}$$

Inserting this into the above expression for $f(x)-\widetilde{f}(x)$, and using that we know $\widetilde{\beta }^{\varLambda }=0$ for ${\varLambda }\in T$ we get (14) given in the theorem.