Bayesian probabilistic extensions of a deterministic classification model

Summary

This paper extends deterministic models for Boolean regression within a Bayesian framework. For a given binary criterion variable Y and a set of k binary predictor variables X₁,…, X_k, a Boolean regression model is a conjunctive (or disjunctive) logical combination consisting of a subset S of the X variables, which predicts Y. Formally, Boolean regression models include a specification of a k-dimensional binary indicator vector (θ₁,…,θ_k) with θ_j = 1 iff X_j ∈ S. In a probabilistic extension, a parameter π is added which represents the probability of the predicted value ${\hat y_i}$ and the observed value y_i differing (for any observation i). Within a Bayesian framework, a posterior distribution of the parameters (θ₁,…, θ_k, π) is looked for. The advantages of such a Bayesian approach include a proper account of the uncertainty in the model estimates and various possibilities for model checking (using posterior predictive checks). We illustrate this method with an example using real data.

Appendix: Deriving posterior distributions

We first compute the prior predictive distribution p(y):

$$\begin{aligned} p(y) &=\sum_{\vartheta \in \Theta}\left[\int_{0}^{1} p(y | \vartheta, \pi) p(\vartheta) p(\pi) d \pi\right] \\ &=\sum_{\vartheta \in \Theta}\left[\int_{0}^{1} \pi^{D_{\vartheta}}(1-\pi)^{n-D_{\vartheta}} \frac{1}{2^{k}} 1\; d \pi\right] \\ &=\sum_{\vartheta \in \Theta}\left[\frac{1}{2^{k}} \frac{\Gamma\left(D_{\vartheta}+1\right) \Gamma\left(n-D_{\vartheta}+1\right)}{\Gamma(n+2)} \times \right.\\&\qquad\qquad\qquad\qquad\left.\int_{0}^{1} \frac{\Gamma(n+2)}{\Gamma\left(D_{\vartheta}+1\right) \Gamma\left(n-D_{\vartheta}+1\right)} \pi^{D_{\vartheta}}(1-\pi)^{n-D_{\vartheta}} d \pi\right]\\ &=\sum_{\vartheta \in \Theta}\left[\frac{1}{2^{k}} \frac{D_{\vartheta} !,\left(n-D_{\vartheta}\right) !}{(n+1) !}\right] \\ &=\frac{1}{2^{k}(n+1)} \sum_{\vartheta \in \Theta} \frac{1}{\left(D_{\vartheta}\right)}, \end{aligned}$$

the integral in the third step being equal to 1 as it is the area under a Beta density.

For the posterior distribution of (θ,π), we start from Eq. (5):

$$\begin{array}{l}p\left( {\theta ,\pi |y} \right) = {{p\left( {y|\theta ,\pi } \right)p\left( {\theta ,\pi } \right)} \over {p\left( y \right)}}\\{\qquad\qquad=\frac{\pi^{D_{\theta}}(1-\pi)^{n-D_{\theta}} \frac{1}{2^{\mathrm{k}}}}{\frac{1}{2^{k}(n+1)} \sum_{\vartheta \in \Theta} \frac{1}{\left(\begin{array}{c}{n} \\ {D_{s}}\end{array}\right)}}} \\\qquad\qquad {=\frac{(n+1) \pi^{D_{\theta}}(1-\pi)^{n-D_{\theta}}}{\sum_{\vartheta \in \Theta} \frac{1}{\left(D_{\vartheta}^{n}\right)}}}\end{array}$$

To derive the marginal posterior distribution of θ, π is integrated out in the joint posterior distribution for θ and π in the formula above.

$$\begin{array}{*{20}c}{p\left( {\theta |y} \right)\;= \;\int_0^1 {p\left( {\theta ,\,\pi |y} \right)} } d\pi \\ {\quad \quad \quad\qquad = \int_0^1 {{{(n + 1){\pi ^{{D_\theta }}}{{(1 - \pi )}^{n - {D_\theta }}}} \over {\sum\limits_{\vartheta \in \Theta } {{1 \over {\left( {_{{D_\vartheta }}^{\;\;n}} \right)}}} }}} }d\pi \\ {\qquad \qquad \qquad \qquad\qquad\qquad\qquad\qquad\quad = {{{1 \over {\left( {_{{D_\theta }}^{\;\;n}} \right)}}} \over {\sum\limits_{\vartheta \in \Theta } {{1 \over {\left( {_{{D_\vartheta }}^{\;\;n}} \right)}}} }}\int_0^1 {\left( {\begin{array}{*{20}c}n \\ {{D_\theta }} \\ \end{array} } \right)(n + 1){\pi ^{{D_\theta }}}{{(1 - \pi )}^{n - {D_\theta }}}d\pi ,} } \\ \end{array}$$

the latter integral being 1 as it is again the area under a Beta density.