Efficient designs for Bayesian networks with sub-tree bounds

Abstract

We present upper and lower bounds for information measures, and use these to find the optimal design of experiments for Bayesian networks. The bounds are inspired by properties of the junction tree algorithm, which is commonly used for calculating conditional probabilities in graphical models like Bayesian networks. We demonstrate methods for iteratively improving the upper and lower bounds until they are sufficiently tight. We illustrate properties of the algorithm by tutorial examples in the case where we want to ensure optimality and for the case where the goal is an approximate solution with a guarantee. We further use the bounds to accelerate established algorithms for constructing useful designs. An example with petroleum fields in the North Sea is studied, where the design problem is related to exploration drilling campaigns. All of our examples consider binary random variables, but the theory can also be applied to other discrete or continuous distributions.

Appendices

Appendices

1.1 Appendix 1: The junction tree algorithm

A junction tree (JT) is an undirected tree (N, L) whose nodes $N_i \in N$ are associated with a variable set $\phi (N_i)$ each. For a standard JT, each edge $(N_i, N_j) \in L$ is correspondingly associated with the variable set $S_{i,j}=\phi (N_i) \cap \phi (N_j)$, which we refer to as the separator between the two nodes. A JT is required to have the running intersection property; for any pair of JT nodes $N_i$ and $N_j$ having a non-empty intersection $S=\phi (N_i) \cap \phi (N_j)$, this intersection S is contained in the corresponding node set $\phi (N_k)$ for each node $N_k$ on the unique path from node $N_i$ to node $N_j$. In addition to the running intersection property, the key requirements for the construction of a JT for a given BN with (V, E), is that $\cup _{N_i \in N}N_i=V$ and that $\ \exists N_i \in N \ :\ \{v\} \cup \text {Pa}(v)\subseteq N_i \quad \forall v\in V$.

For each BN with (V, E), there are in general several possible JTs, for example the trivial $\mathcal {T}=(\{V\}, \emptyset )$. For efficient calculations, the choice of JT matters. When calculating queries for a BN, we choose to work on a JT where the node set N is exactly the set of maximal cliques of a triangulated moralized version of the BN. Within this restriction, there are still several possible JTs in general. However, finding the optimal one is an NP-complete problem, so in applications it is common to use heuristics to choose a JT which is “good enough” for practical purposes.

The different concepts discussed in this appendix are illustrated for the BN in Fig. 11. In the following, when discussing this network, we do not distinguish between a JT node and its corresponding set of nodes in the underlying BN, which we both refer to as the clique $C_i$.

Fig. 11

Example BN. There are three (latent) parent nodes $P_1$, $P_2$, $P_3$, and a total number of $M_1$, $M_2$, $M_3$ (observable) nodes connected to each parent node, respectively

Fig. 12

Example JTs for the BN in Fig. 11. They can be described as a straight chain JT (JT on the left) and consisting of two star-like parts (JT on the right), respectively. There are also other several versions which could be described as something in between those two, and also several rearrangements of the one on the left. Each node of the JT represents cliques derived from the BN, and is visualized as a labelled circle. Each edge has its corresponding separator visualized as a labelled box

The JTA instructs a sequence of message passings on a JT for computing queries $\mathbb {P}\left( X_{A_i}|X_{B_i}\right)$ from the underlying BN. In practice, each message passing scheme computes a representation of $\mathbb {P}\left( X_{B_i^C}|X_{B_i}\right)$ from which it is efficient to compute $\mathbb {P}\left( X_{A_i}|X_{B_i}\right)$ by marginalization, so the JTA is especially efficient when several of our queries have the same evidence assignment $X_{B_i}=x_{B_i}$. The message passing scheme visits each node in the JT two times, which makes the procedure linear in the number of nodes in the JT. The time complexity of visiting a JT node $C_j$ is equal to the dimension of its probability table $\varPsi _j$, i.e. $2^{|C_j|}$ in the case of binary random variables (see Sect. 1). It is also an assumption that $A_i$ is represented in a single JT-node or a few neighboring ones, such that computing $\mathbb {P}\left( X_{A_i}|X_{B_i}\right)$ from the representation of (${X_{B_i^C}|X_{B_i}}$) has constant time complexity.

If the evidence assignments are different for all queries, a modified (local) JTA which skips unnecessary messages could be more efficient. We discuss how we choose to build the JT in the following subsection, and after that we describe the calculation of a query when the JT is already fixed.

1.1.1 Appendix 1.1: Almond trees

Cowell et al. (2007) show that we can find a standard JT with (N, L) for a BN by the following steps:

• 1. Moralize the DAG: marry all unlinked parents by adding an edge between any pair of non-neighboring nodes that has a common child.

• 2. Remove the direction of all edges.

• 3. Triangulate the undirected graph by repeatedly linking a pair of non-neighboring nodes in any cycle of length $\ge 3$ that does not (already) have a chord.

• 4. Let N be the set of all the maximal cliques in the triangulated graph.

• 5. Construct a complete graph $(N,\hat{L})$. Define the weight of each edge $(N_i,N_j) \in \hat{L}$ to be $w\left( (N_i,N_j)\right) =|N_i \cap N_j|$, and find a maximum weight spanning tree (N, L).

Fig. 13

The AT for the BN in Fig. 11. Each clique-node of the AT is visualized as a labelled circle, and each almond/separator-node is visualized as a labelled box

Performing the 1–5 steps procedure for the BN in Fig. 11, the final result is something like one of the JTs in Fig. 12. Note that also many of the separators in Fig. 12 represent the same node set. In (standard) JTA, we pass computed messages over separators from a node to another. Actually, every pair of equal separators (representing the same node set) end up having the exact same message computed in the second part of the message passing (and many also in the first), and that means the JTA is doing unnecessary work.

An almond tree (AT) is a version of the standard junction trees (JT). Doing computations in an AT instead of a standard JT makes computations more efficient (time and memory), and it requires no significant changes to the algorithm itself. For instance, we simplify message passing by working on the corresponding AT with equal separators merged. This is visualized in Fig. 13. Recall that each of the edges $(C_i,C_j)$ in the JT is associated with what we call a separator $S_{i,j}$ which represents the node set $S_{i,j}=C_i \cap C_j$. For an AT these separators are viewed as (a special kind of) nodes, and we call these almonds. In a sense, each one-edge long path $C_i, (C_i,C_j), C_j$ between two neighboring cliques $C_i,C_j$ in the original JT is expanded to $C_i, (C_i,S_{i,j}),S_{i,j},(S_{i,j},C_j), C_j$ in the AT. Also, equal separators in the original JT are merged into one single separator in the AT, having multiplicity equal to the number of separators merged to one almond. Suppose a standard JT has M separators $S_{i,j}$ which all represent the same node set. We can assume their corresponding edges $(C_i,C_j)$ in the JT together with their incident cliques $C_i$ and $C_j$ constitute a connected component in the JT. (This will always correspond to a maximum weight spanning tree.) Now, when the equal separators $S_{i,j}$ are merged to one single separator S, all edges $(C_i,S_{i,j})$ are translated to $(C_i,S)$ and duplicate edges are deleted.

In building a standard JT, steps (3) and (4) are equivalent to using variable elimination rules to add necessary edges sequentially as we at each step k remove one node $j_k$ from the undirected graph. For each such node $j_k$, denote its neighboring set $D_k=\text {Ne}(j_k)$ in the current (time k) graph as a clique, and let D be the set of all the maximal sets (cliques) in $\{D_k\}_{k=1}^{|V|}$. As in Almond and Kong (1991), we construct an AT directly by further changing step (5): Moving backwards in the elimination order, $k=|V|, \ldots , 2,1$, we use the collection $\{(j_k,D_k)\}_{k=1}^{|V|}$ to sequentially build the AT with $\mathcal {T}=\mathcal {T}^{|V|}=\left( \mathcal {C}^{|V|} \cup \mathcal {A}^{|V|},L^{|V|}\right)$, where $\mathcal {C}^{|V|}$ is the collection of clique nodes and $\mathcal {A}^{|V|}$ is the collection of almond nodes. Initially, $\mathcal {T}^1=\left( \mathcal {C}^1 \cup \mathcal {A}^1,L^1\right) \equiv \left( \{j_{|V|}\} \cup \emptyset ,\emptyset \right)$, and at each step $t=2,3, \ldots , |V|$, a tree $\mathcal {T}^t=\left( \mathcal {C}^t \cup \mathcal {A}^t,L^t\right)$ is constructed from $\mathcal {T}^{t-1}$ and the pair $(j_{|V|-(t-1)},D_{|V|-(t-1)})$ according to the following rules: Iteratively at step t, define $D^\star = D_{|V|-(t-1)}$ and $C^\star = D^\star \cup \{j_{|V|-(t-1)}\}$, and then:

• If $D^\star \in \mathcal {A}^{t-1}$:

  • $\mathcal {A}^t=\mathcal {A}^{t-1}$

  • $\mathcal {C}^t=\mathcal {C}^{t-1} \cup \{C^\star \}$

  • $L_t=L_{t-1} \cup \{D^\star ,C^\star \}$

• Else-If $D^\star \subset A \in \mathcal {A}^{t-1}$ and A is the smallest such almond node:

  • $\mathcal {A}^t=\mathcal {A}^{t-1} \cup \{D^\star \}$

  • $\mathcal {C}^t=\mathcal {C}^{t-1} \cup \{C^\star \}$

  • $L_t=L_{t-1} \cup \{\{D^\star ,C^\star \},\{D^\star ,A\}\}$

• Else-If $D^\star \in \mathcal {C}^{t-1}$:

  • $\mathcal {A}^t=\mathcal {A}^{t-1}$

  • $\mathcal {C}^t=(\mathcal {C}^{t-1} \cup \{C^\star \}) \setminus \{D^\star \}$

  • $L_t=\{e \in L_{t-1}\ :\ D^\star \not \in e\} \cup \{ \{A,C^\star \}\ :\ \{A,D^\star \}\in E_{t-1}\}$

• Else: Find the clique $C \in \mathcal {C}^{t-1}$ with $D^\star \subset C$, and

  • $\mathcal {A}^t=\mathcal {A}^{t-1} \cup \{D^\star \}$

  • $\mathcal {C}^t=\mathcal {C}^{t-1} \cup \{C^\star \}$

  • $L_t=L_{t-1} \cup \{\{D^\star ,C^\star \},\{D^\star ,C\}\}$

For the BN in Fig. 11, the constructed AT would look like Fig. 13.

1.1.2 Appendix 1.2: Prerequisites to message passing in the JTA

Note that the following subsection is written in a standard JT notation: Clique nodes are denoted $C_j$, and separators are denoted $(C_j,C_k)$. The JTA for ATs is equivalent: The almond nodes have multiplicity equal to one less than the number of clique node neighbors, which equals the number of times this separator would be present as an edge in a standard JT.

• Our JT is a tree whose nodes $C_i$ represent different subsets of the nodes in the BN, and we can assume the JT nodes are numbered $C_1, \ldots \,C_m$ according to an topological ordering for the directed graph induced by a choice of root $C_1$. The choice of root could be arbitrary, but it determines the ordering of the message passing. The numbering of the nodes further implies:

  • $\forall j \le m$ the (unique) path $(C_{k_1}, C_{k_2}, \ldots , C_{k_s})$ from $C_j$ to $C_1$ has $j=k_1>k_2>\ldots >k_s=1$.

  • We can define $r_j<j \quad \forall j$ such that $C_{r_j}$ is the unique lower-numbered neighbor of $C_j$. Also, let $C_{r_1}=\emptyset$ be a fictious neighbor of the root $C_1$, catching the result of the first message passing but not replying.

  • In the first pass, messages start from leaves, propagate and evolve towards the root; from each j to $r_j$.

  • In the second pass, each node replies to the ones it received a message from in the first pass.

• Each JT node $C_j$ is associated with a function $\varPsi _{C_j}(X_{C_j})$ of the random vector $X_{C_j}$. The function $\varPsi _{C_j}(X_{C_j})$ is represented as a table, where each entry is the function value of a given assignment to $X_{C_j}$. If all variables $X_k,\ k \in C_j,$ are binary, the full table $\varPsi _{C_j}(X_{C_j})$ has size $2^{|C_j|}$.

• Initially in each message passing, we assume $\mathbb {P}\left( X_V\right) =\frac{\prod _{j=1}^m \varPsi _{C_j}(X_{C_j})}{\prod _{(C_j,C_k) \in L} \varPsi _{C_j,C_k}}$ is a probability distribution; that is, it sums to 1. In fact, before the first message passing, the JT is constructed such that $\mathbb {P}\left( X_V\right) =\prod _{j=1}^m \varPsi _j(X_{C_j})=\prod _{i=1}^{|V|} P(X_i|X_{\text {Pa}(i)})$ with all $\varPsi _{C_j,C_k}\equiv 1$. Since $\exists C_j\ :\ \{v\} \cup \text {Pa}(v) \subset C_j \quad \forall v \in V$, this can be done in several ways. Iteratively for each $v \in V$, choose a JT node $C_j$ such that $\{v\} \cup \text {Pa}(v) \subset C_j$, and redefine $\varPsi _{C_j}$ as its former table multiplied by $P(X_v|X_{\text {Pa}(v)})$.

Fig. 14

Workflow for the AT in Fig. 13

1.1.3 Appendix 1.3: Work flow for calculating a query $\mathbb {P}\left( X_{A_i}|X_{B_i}\right)$ in a given JT representation

• Insert evidence $B_i$ For each variable $X_k,\ k \in B_i$, find a clique $C_j$ containing k. Set $\varPsi _j(X_{C_j})=0$ for all assignments to $X_{C_j}$ not consistent with the value assigned to $X_k$ in the evidence. Each $\varPsi _{C_j}(X_{C_j})$ is now effectively a function of the random vector $X_{C_j \setminus B_i}$. The joint conditional distribution for the non-evidenced variables is given by $\frac{1}{W}\prod _j \varPsi _{C_j}(X_{C_j})$, effectively as a function of $X_{B_i^C}$. The unknown normalization constant W has the same value as the probability $\mathbb {P}\left( X_{B_i}\right)$ of the evidence.

• Message passing towards root From each node $C_j$ in order $j=m,\ldots , 1$: Collect

  $$\begin{aligned} \phi _j=\varPsi _{C_j} (X_{C_j}) \cdot \prod _{C_k \in \text {Ne}(C_j) \setminus \{C_{r_j}\}} \frac{M^\rightarrow _k (X_{C_k \cap C_j}) }{\varPsi _{C_j,C_k} (X_{C_k \cap C_j}) }. \end{aligned}$$

  Pass message $M^\rightarrow _j(X_{C_j \cap C_{r_j}})=\sum _{X_{C_j \setminus C_{r_j}}} \phi _j(X_{C_j})$ to $C_{r_j}$, and let remaining potential in clique $C_j$ be

  $$\begin{aligned} \frac{\phi _j(X_{C_j})}{M^\rightarrow _j (X_{C_j \cap C_{r_j}}) }. \end{aligned}$$

• Intermediate result The equality

  $$\begin{aligned} \prod _j \varPsi _{C_j}(X_{C_j})=M^\rightarrow _1 \prod _{j=1}^m \frac{\phi _j(X_{C_j})}{M^\rightarrow _j (X_{C_j \cap C_{r_j}}) } \end{aligned}$$

  implies factorization of conditional joint as

  $$\begin{aligned} \prod _{j=1}^{m}\mathbb {P}\left( X_{C_j \setminus C_{r_j}}|X_{C_j \cap C_{r_j}}\right) , \end{aligned}$$

  i.e. $\mathbb {P}\left( X_{B_i}\right) =M^\rightarrow _1$   and

  $$\begin{aligned} \mathbb {P}\left( X_{B_i^C}|X_{B_i}\right) =\prod _{j=1}^m \frac{\phi _j(X_{C_j})}{M^\rightarrow _j (X_{C_j \cap C_{r_j}}) }. \end{aligned}$$

• Message replying From each JT node $C_j$, in order $j=1,\ldots , m$, reply to each JT node $C_{k}$ with $r_k=j$. Redefine $\varPsi _1=\frac{\phi _1 (X_{C_1})}{M^\rightarrow _1}$, or collect and redefine

  $$\begin{aligned} \varPsi _j= M^{j\leftarrow }_{r_j} (X_{C_j \cap C_{r_j}}) \cdot \frac{\phi _j(X_{C_j})}{M^\rightarrow _j (X_{C_j \cap C_{r_j}})} \quad \text {for }j>1. \end{aligned}$$

  For each k such that $r_k=j$, send copies of

  $$\begin{aligned} M^{k\leftarrow }_j (X_{C_j \cap C_k})=\sum _{X_{C_j \setminus C_k}}\varPsi _j(X_{C_j}) \end{aligned}$$

  over $S_{j,k}$ to $C_k$, so that $S_{j,k}$ also keeps a copy as $\varPsi _{j,k}(X_{C_j \cap C_k})$.

• Result For any sub-tree $\mathcal {T}_J=(C_J,L_J)$ with separators associated with the edges $L_J=L\cap (C_J \times C_J)$, we have

  $$\begin{aligned} \mathbb {P}\left( C_J{\setminus }B_i|B_i\right) =\frac{\prod _{j \in J} \varPsi _j(X_{C_j})}{\prod _{(C_j,C_k) \in L_J} \varPsi _{j,k}(X_{C_j \cap C_k})}. \end{aligned}$$

  The simplest examples are single cliques $J=\{j\}$, or all variables $\mathcal {T}_J=\mathcal {T}$. We will later use that this formula, and hence updates, is correct both globally and locally.

Assume we are given an AT like in Fig. 13, and that we would like to answer queries like $\mathbb {P}\left( c_{j}|\cup _{i=1}^3 c_{I_i}\right)$, where $I_1 \subseteq \{1, \ldots M_1\}$, $I_2 \subseteq \{M_1+1, \ldots M_2\}$, $I_3 \subseteq \{M_2+1, \ldots M_3\}$ for $M_1<M_2<M_3$. In this example, we let the variables $c_1, \ldots , c_{M_3}$ be our observable variables in the BN, while the variables $P_1,P_2,P_3$ are latent variables that ties the observable ones together in the probabilistic model (Fig. 11). Assume for this example that all the AT nodes (clique nodes and almond nodes) are labelled with increasing numbers with the distance from $C_1=\{P_3\}$. Define for each almond $P_\ell$,

$$\begin{aligned} EF^\ell (P^\ell )=\prod _{i \in I_\ell }\mathbb {P}\left( c_{i}| P^\ell \right) . \end{aligned}$$

As a simple local example, we have $\mathbb {P}\left( c_j|c_I\right) \propto \sum _{P^\ell } \mathbb {P}\left( c_j,P^\ell \right) \cdot EF(P^\ell )$ if $j \cup I \subseteq I_\ell$ for some $\ell \in \{1,2,3\}$. However, we might also want to answer queries where $j \cup I \not \subseteq I_\ell$ for any $\ell \in \{1,2,3\}$, for example when $I=\{M_1, M_2, M_3\}$. In this more general case, we need to do message passings between the almonds as well. The calculations for the full JTA is explained in Fig. 14.

Similar to the simple local part of this example we now have

$$\begin{aligned} \mathbb {P}\left( c_j|c_I\right) = \sum _{P^\ell } \mathbb {P}\left( c_j,P^\ell \right) \cdot \tilde{EF}^\ell (P^\ell ) \end{aligned}$$

where $\ell \in \{1,2,3\}$ is chosen such that $j \subseteq I_\ell$. Recall that we do not have the requirement $I \subseteq I_\ell$ for the evidence to be local to the same almond as the query variable $c_j$.