Discrete Stochastic Search and Its Application to Feature-Selection for Deep Relational Machines

Abstract

We use a model for discrete stochastic search in which one or more objects (“targets”) are to be found by a search over n locations (“boxes”), where n is infinitely large. Each box has some probability that it contains a target, resulting in a distribution H over boxes. We model the search for the targets as a stochastic procedure that draws boxes using some distribution S. We derive first a general expression on the expected number of misses $\text {E}[Z]$ made by the search procedure in terms of H and S. We then obtain an expression for an optimal distribution $S^{*}$ to minimise $\text {E}[Z]$. This results in a relation between: the entropy of H and the KL-divergence between H and $S^{*}$. This result induces a 2-partitions over the boxes consisting of those boxes with H probability greater than $\frac{1}{n}$ and the rest. We use this result to devise a stochastic search procedure for the practical situation when H is unknown. We present results from simulations that agree with theoretical predictions; and demonstrate that the expected misses by the optimal seeker decreases as the entropy of H decreases, with a maximum obtained for uniform H. Finally, we demonstrate applications of this stochastic search procedure with a coarse assumption about H. The theoretical results and the procedure are applicable to stochastic search over any aspect of machine learning that involves a discrete search-space: for example, choice over features, structures or discretized parameter-selection. In this work, the procedure is used to select features for Deep Relational Machines (DRMs) which are Deep Neural Networks (DNNs) defined in terms of domain-specific knowledge and built with features selected from large, potentially infinite-attribute space. Empirical results obtained across over 70 real-world datasets show that using the stochastic search procedure results in significantly better performances than the state-of-the-art.

Appendix: Proofs

Proof of Lemma 1

Proof

The ideal case is $\mathrm {E}[Z]=0$. That is, on average, the search opens the correct box k on its first attempt. Now $\mathrm {P}(Z=0| \text{ the } \text{ ball } \text{ is } \text{ in } \text{ box } k) = h_k s_k = h_k (1-s_k)^0 s_k$. Since the ball can be in any of the n boxes, $\mathrm {P}(Z=0) = \sum _{k=1}^{n} h_k (1-s_k)^0 s_k$. More generally, for $Z=j$, the search opens wrong boxes j times, and $\mathrm {P}(Z=j) = \sum _{k=1}^{n} h_k (1-s_k)^j s_k$. The expected number of misses can now be computed:

$$\begin{aligned} \begin{aligned} \mathrm {E}[Z]|_{H,S}&= \sum _{j=0}^{\infty }{j\times \mathrm {P}(Z=j)}\\&= \sum _{j=0}^{\infty }{j\sum _{k=1}^{n} h_k (1-s_k)^j s_k} \\ \end{aligned} \end{aligned}$$

Swapping the summations over j and k, we get

$$\begin{aligned} \begin{aligned} \mathrm {E}[Z]|_{H,S}&= \sum _{k=1}^{n} h_k s_k \sum _{j=0}^{\infty }{j (1-s_k)^j} \\&= \sum _{k=1}^{n}{h_k s_k \frac{1-s_k}{s_k ^ 2}} \end{aligned} \end{aligned}$$

This simplifies to:

$$\begin{aligned} \mathrm {E}[Z]|_{H,S} = \sum _{k=1}^{n}{\frac{h_k}{s_k}} - 1 \end{aligned}$$

Proof of Lemma 2

Proof

This extends the Lemma 1 (Expected Cost of Misses by the Seeker) to a general case of multiple (K) stationary hiders. The number of ways the K hiders can choose to hide in n boxes is $^nP_K$ and let $\mathsf {P}(n,K)$ denote a set of all such permutations. For example, $\mathsf {P}(3,2) = \{(1,2), (1,3), (2,1), (2,3), (3,1), (3,2)\}$.

All the K hiders can hide in any one of these choices with probability $\left( h_{\sigma (i)}^{(1)} h_{\sigma (i)}^{(2)} \dots h_{\sigma (i)}^{(K)}\right)$, where $h_{\sigma (i)}^{(k)}$ denotes the probability of the hider in kth place in the selected choice of $\sigma (i)$. Analogously, the seeker can find any one of these hiders with probability $\left( s_{\sigma (i)}^{(1)} + s_{\sigma (i)}^{(2)} + \dots + s_{\sigma (i)}^{(K)}\right)$, and would not find the hider once is $1 - \left( s_{\sigma (i)}^{(1)} + s_{\sigma (i)}^{(2)} + \dots + s_{\sigma (i)}^{(K)}\right)$. If the hider makes j such misses, then it is $\left\{ 1-\left( s_{\sigma (i)}^{(1)} + s_{\sigma (i)}^{(2)} + \dots s_{\sigma (i)}^{(K)}\right) \right\} ^j$. Now, the expected misses for this multiple hider formulation is given as

$$\begin{aligned} \text {E}[Z] = \sum _{i, \sigma (i) \in \mathsf {P}(n,K)} \prod _{k=1}^{K} h_{\sigma (i)}^{(k)} \sum _{j = 0}^{\infty }{j \left( 1-\sum _{k=1}^{K}s_{\sigma (i)}^{(k)}\right) ^j \sum _{k=1}^{K}s_{\sigma (i)}^{(k)}} \end{aligned}$$

This further simplifies to

$$\begin{aligned} \text {E}[Z]|_{H,S} = \sum _{i, \sigma (i) \in \mathsf {P}(n,K)} \prod _{k=1}^{K} h_{\sigma (i)}^{(k)} \left( \frac{1}{\sum _{k=1}^{K}s_{\sigma (i)}^{(k)}} - 1 \right) \end{aligned}$$

Proof of Theorem 1

Proof

The problem can be posed as a constrained optimisation problem in which the objective function that is to be minimized is

$$ f = \sum _{i=1}^{n}{\frac{h_i}{s_i} - 1} $$

Our objective is to minimize the function f given any hider distribution H. Let us represent $\mathbf {\nabla } f = \left( \dfrac{\partial f}{\partial s_1}, \dfrac{\partial f}{\partial s_2}, \ldots , \dfrac{\partial f}{\partial s_n}\right)$. In this problem, $\mathbf {\nabla } f = \left( -\frac{h_1}{s_1^2}, -\frac{h_2}{s_2^2}, \ldots , -\frac{h_n}{s_n^2} \right)$. Now, computing the double derivative $\mathbf {\nabla }^2 f$, we get

$$ \mathbf {\nabla }^2 f = \mathbf {\nabla } (\mathbf {\nabla } f) = 2\left( \frac{h_1}{s_1^3}, \frac{h_2}{s_2^3}, \dots , \frac{h_n}{s_n^3}\right) $$

Since, $\forall i, h_i \ge 0, s_i \ge 0$, we can claim that $\mathbf {\nabla }^2 f$ has all non-negative second derivative components. And, therefore f is convex.

Proof of Theorem 2

Proof

We will write $\text {E}[Z]|_{H,S}$ as a function of S i.e. f(S). Our objective is to minimise $f(S) = \sum _{i=1}^{n}{\frac{h_i}{s_i}}$ subject to constraint $\sum _{i=1}^{n}s_i = 1$. The corresponding dual form (unconstrained) of this minimisation problem can be written as

$$ g(S,\lambda ) = \sum _{i=1}^{n}{\frac{h_i}{s_i}} + \lambda \left( 1 - \sum _{i=1}^{n}{s_i} \right) $$

To obtain the optimal values of S and $\lambda$, we set $\frac{\partial g}{\partial s_i} = 0$ for $i = 1,\ldots ,n$, and $\frac{\partial g}{\partial \lambda } = 0$. This gives: $-\frac{h_i}{s_i^2} - \lambda = 0~\text {and}~ \sum _{i=1}^{n}s_i = 1$. From this: $s_i = -\frac{\sqrt{h_i}}{\sqrt{\lambda }},~\forall i$. Applying this quantity for $s_i$ in $\sum _{i=1}^{n}s_i = 1$ and the value of the parameter $\lambda = -\frac{h_i}{s_i^2}$, we get: $-\frac{\sum _{i=1}^{n}\sqrt{h_i}}{-\frac{\sqrt{h_i}}{s_i}} = 1$. Simplifying the above, we obtain the desired optimal seeker distribution $S^*$: $s^*_i = \frac{\sqrt{h_i}}{\sum _{j=1}^{n}{\sqrt{h_j}}},~\forall i \in \{1,\ldots ,n\}$.

Proof of Corollary 1

Proof

If S is non-uniform with $\forall s_i>0$, we have $\text {E}[Z]|{H,S} = \frac{1}{n}\sum _{i=1}^n{\frac{1}{s_i}} - 1 \ge \frac{n}{\sum _{i=1}^n{s_i}} - 1$ and the denominator is 1 because S is a distribution. So, $S^*$ must be a uniform distribution and in this case, the quantity $\text {E}[Z]|_{H,S} = \sum _{i=1}^{n}{\frac{1/n}{1/n}} - 1 = \sum _{i=1}^{n}1 - 1 = n - 1.$

Proof of Corollary 2

Proof

The proof is as follows:

$$\begin{aligned} \sum _{i=1}^{n}\frac{h_i}{s^*_i} = \sum _{i=1}^{n}\frac{h_i}{\left( \frac{\sqrt{h_i}}{\sum _{j=1}^{j}{\sqrt{h_j}}}\right) } = \sum _{i=1}^{n}\sqrt{h_i}\sum _{j=1}^{j}{\sqrt{h_j}} = \sum _{i=1}^{n}\left( {\sqrt{h_i}}\right) ^2 \end{aligned}$$

Hence, the result follows.

Proof of Theorem 3

Proof

The KL-divergence between the two distribution H and $S^*$ is defined as:

$$\begin{aligned} \text {KLD}(H\Vert S^*)&= \sum _{i=1}^n{h_i\log _2{\frac{h_i}{s^*_i}}} \\&= \sum _{i=1}^n{h_i\log _2{h_i}} - \sum _{i=1}^n{h_i\log _2{\frac{\sqrt{h_i}}{\sum _{j=1}^n {\sqrt{h_j}}}}} ~~\text {(using Theorem}~2)\\&= \frac{1}{2}\sum _{i=1}^n{h_i\log _2{h_i}} + \log _2\left( \sum _{j=1}^n{\sqrt{h_j}}\right) \left( \sum _{i=1}^n h_i\right) \\&= -\frac{1}{2}Entropy(H) + \log _2\left( \sum _{j=1}^n{\sqrt{h_j}}\right) \\&= -\frac{1}{2}Entropy(H) + \log _2\left( \text {E}[Z]|_{H,S^*} + 1\right) ^{\frac{1}{2}} ~~\text {(using Corollary}~2)\\&= \frac{1}{2}\left[ -Entropy(H) + \log _2\left( \text {E}[Z]|_{H,S^*} + 1\right) \right] \end{aligned}$$

Simplifying, we get:

$$ \text {E}[Z]|_{H,S^*} = 2^{2\text {KLD}(H||S^*)+Entropy(H)} - 1 $$

Proof of Lemma 3

Proof

The probability that a randomly drawn box is not in the U partition is $(1-p)$. The probability that in a sample of s boxes, none are from the U partition is $(1-p)^s$, and therefore the probability that there is at least 1 box amongst the s from the U partition is $1 - (1-p)^s$. We want this probability to be at least $\alpha$. That is:

$$ 1-(1-p)^s \ge \alpha $$

With some simple arithmetic, it follows that

$$ s \ge \frac{\mathrm {log}(1-\alpha )}{\mathrm {log}(1-p)} $$