An Efficient Bayesian Network Structure Learning Strategy

Abstract

This paper addresses the problem of efficiently finding an optimal Bayesian network structure for maximizing the posterior probability. In particular, we focus on the B& B strategy to save the computational effort associated with finding the largest score. To make the search more efficient, we need a tighter upper bound so that the current score can exceed it more easily. We find two upper bounds and prove that they are tighter than the existing one (Campos and Ji, J Mach Learn Res 12(3):663–689, 2011). Finally, we demonstrate that the proposed two bounds render the search to be much more efficient using the Alarm and Insurance data sets. For example, the search is twice to three times faster for $n=100$ and almost twice faster for $n=500$. We also experimentally verify that the overhead due to replacing the existing pruning rule by the proposed one is negligible.

Appendix

We assume that z takes on a value in $\{1,\ldots ,\gamma \}$ with $\gamma \ge 2$.

Proof of Lemma 1

In general,

$$\begin{aligned} c(z)+a-j\le \sum _{z'=z}^{\gamma }c(z')+a-j \end{aligned}$$

for $j=1,\ldots ,c(z)$ and $z=1,\ldots ,\gamma$. By multiplying both sides over all $j=1,\ldots ,c(z)$, we have

$$\begin{aligned} \frac{\Gamma (c(z)+a)}{\Gamma (a)}\le \frac{\Gamma (\sum _{z'=z}^{\gamma }c(z')+a)}{\Gamma (\sum _{z'=z+1}^{\gamma }c(z')+a)} \end{aligned}$$

for $z=1,\ldots ,\gamma$. By further multiplying both sides over all $z=1,\ldots ,\gamma$, we obtain the lemma.

Proof of Lemma 2

Note that in general, for $0<p<q$, the function $f_{p,q}(u):=\frac{u+p}{u+q}<1$ is monotonously increasing with $u\ge 0$. Thus,

$$\begin{aligned} \frac{c(z)+a-j}{c(z)+b-j}\le \frac{\sum _{z'=z}^\gamma c(z')+a-j}{\sum _{z'=z}^\gamma c(z')+b-j} \end{aligned}$$

for $j=1,\ldots ,c(z)$ and $z=1,\ldots ,\gamma$. By multiplying both sides over all $j=1,\ldots ,c(z)$, we have

$$\begin{aligned}&\frac{\Gamma (c(z)+a)}{\Gamma (a)}\cdot \frac{\Gamma (b)}{\Gamma (c(z)+b)}\\&\quad \le \frac{\Gamma (\sum _{z'=z}^\gamma c(z')+a)}{\Gamma (\sum _{z'=z+1}^\gamma c(z')+a)}\cdot \frac{\Gamma (\sum _{z'=z+1}^\gamma c(z')+b)}{\Gamma (\sum _{z'=z}^\gamma c(z')+b)} \\ \end{aligned}$$

for $z=1,\ldots ,\gamma$. By further multiplying both sides over all $z=1,\ldots ,\gamma$, we obtain the lemma.

Proof of Lemma 3

Similar to the proof of Lemma 2, we have

$$\begin{aligned} \frac{c(z)+a(z)-j}{c(z)+\sum _{z'=1}^\gamma a(z')-j}\le & {} \frac{\sum _{z'=z}^\gamma c(z')+a(z)-j}{\sum _{z'=z}^\gamma c(z')+\sum _{z'=1}^\gamma a(z')-j}\\\le & {} \frac{\sum _{z'=z}^\gamma c(z')+\max _{z'} a(z')-j}{\sum _{z'=z}^\gamma c(z')+\sum _{z'=1}^\gamma a(z')-j} \end{aligned}$$

for $j=1,\ldots ,c(z)$ and $z=1,\ldots ,\gamma$. By multiplying both sides over all $j=1,\ldots ,c(z)$, we have

$$\begin{aligned}&\frac{\Gamma (c(z)+a(z))}{\Gamma (a(z))}\cdot \frac{\Gamma (\sum _{z'=1}^\gamma a(z'))}{\Gamma (c(z)+\sum _{z'=1}^\gamma a(z'))}\\&\quad \le \frac{\sum _{z'=z}^\gamma c(z')+\max _{z'} a(z')-1}{\sum _{z'=z}^\gamma c(z')+\sum _{z'} a(z')-1}\cdots \frac{\sum _{z'=z+1}^\gamma c(z')+\max _{z'} a(z')}{\sum _{z'=z+1}^\gamma c(z')+\sum _{z'=1}^\gamma a(z')}\\&\quad = \frac{\Gamma (\sum _{z'=z}^\gamma c(z')+\max _{z'}a(z'))}{\Gamma (\sum _{z'=z+1}^{\gamma } c(z')+\max _{z'} a(z'))} \frac{\Gamma (\sum _{z'=z+1}^{\gamma } c(z')+\sum _{z'=1}^\gamma a(z'))}{\Gamma (\sum _{z'=z}^\gamma c(z')+\sum _{z'=1}^\gamma a(z'))} \end{aligned}$$

for $z=1,\ldots ,\gamma$. By further multiplying both sides over all $z=1,\ldots ,\gamma$, we obtain the lemma.

Proof of Theorem 5

We prove

$$\begin{aligned}&\frac{\Gamma (\beta /2)}{\Gamma (n+\beta /2)}\prod _y \frac{\Gamma (c(y)+1/2)}{\Gamma (1/2)}\cdot \prod _y \left\{ \frac{\Gamma (\alpha /2)}{\Gamma (c(y)+\alpha /2)} \prod _x \frac{\Gamma (c(x,y)+1/2)}{\Gamma (1/2)}\right\} \\&\quad \ge \frac{\Gamma (\alpha \beta /2)}{\Gamma (n+\alpha \beta /2)}\prod _x\prod _y \frac{\Gamma (c(x,y)+1/2)}{\Gamma (1/2)}, \end{aligned}$$

which is equivalent to

$$\begin{aligned} \frac{\prod _y \Gamma (c(y)+1/2)}{\Gamma (n+\beta /2)}\cdot \frac{\Gamma (\beta /2)}{\Gamma (1/2)^\beta } \ge \frac{\prod _y \Gamma (c(y)+\alpha /2)}{\Gamma (n+\alpha \beta /2)}\cdot \frac{\Gamma (\alpha \beta /2)}{\Gamma (1/2)^{\alpha \beta }} \end{aligned}$$

(27)

We regard (27) as a function of $\alpha \ge 1$. We find that the both sides are equal when $\alpha =1$, that $\Gamma (1/2)^{\alpha \beta }/\Gamma (\alpha \beta )$ decreases with $\alpha \ge 1$, and that $B(x+r_1,\ldots ,x+r_m)$ with constants $r_1,\ldots ,r_m>0$ decreases with $x>0$, where $B(r_1,\ldots ,r_n)$ is the Beta function defined by $\frac{\Gamma (\sum _{i=1}^m r_i)}{\prod _{i=1}^m \Gamma (r_i)}$. Those three facts imply the theorem.