EP-Based Infinite Inverted Dirichlet Mixture Learning: Application to Image Spam Detection

Abstract

We propose in this paper a new fully unsupervised model based on a Dirichlet process prior and the inverted Dirichlet distribution that allows the automatic inferring of clusters from data. The main idea is to let the number of mixture components increases as new vectors arrive. This allows answering the model selection problem in a elegant way since the resulting model can be viewed as an infinite inverted Dirichlet mixture. An expectation propagation (EP) inference methodology is developed to learn this model by obtaining a full posterior distribution on its parameters. We validate the model on a challenging application namely image spam filtering to show the merits of the framework.

A The calculation of \(Z_i\) in Eq. (21)

The normalized constant $Z_i$ in Eq. (21) can be calculated as

$$\begin{aligned} Z_i = \int f_i(\varTheta )q^{\setminus i}(\varTheta )d\varTheta =\sum _{j=1}^J \bar{\lambda }_j\prod _{s=1}^{j-1}(1-\bar{\lambda }_s) \int \mathcal {ID}(\varvec{X}_i|\varvec{\alpha }_j) N(\varvec{\alpha }_j|\varvec{\mu }_j^{\setminus i},A_j^{\setminus i})\mathrm {d}\varvec{\alpha }_j \end{aligned}$$

(30)

where $\bar{\lambda }_j$ is the expected value of $\lambda _j$. Since the integration involved in Eq. (30) is analytically intractable, we tackle this problem by adopting the Laplace approximation to approximate the integrand with a Gaussian distribution [19]. First, we define $h(\varvec{\alpha }_j)$ as the integrand in Eq. (30):

$$\begin{aligned} h(\varvec{\alpha }_j) =\mathcal {ID}(\varvec{X}_i|\varvec{\alpha }_j)\mathcal {N}(\varvec{\alpha }_j|\varvec{\mu }_j^{\setminus i},A^{\setminus i}_{j}) \end{aligned}$$

(31)

Then, the normalized distribution for this integrand which is indeed a product of a Dirichlet distribution and a Gaussian distribution is given by

$$\begin{aligned} \mathcal {H}(\varvec{\alpha }_j) =\frac{h(\varvec{\alpha }_j)}{\int h(\varvec{\alpha }_j)d\varvec{\alpha }_j} \end{aligned}$$

(32)

Our goal for the Laplace method is to find a Gaussian approximation which is centered on the mode of the distribution $\mathcal {H}(\varvec{\alpha }_j)$. We may obtain the mode $\varvec{\alpha }_j^*$ numerically by setting the first derivative of $\ln h(\varvec{\alpha }_j)$ to 0, where

$$\begin{aligned}&\ln h(\varvec{\alpha }_j) = \ln \frac{\varGamma (\sum _{l=1}^{D+1}\alpha _{jl})}{\prod _{l=1}^{D+1}\varGamma (\alpha _{jl})} + \sum _{l=1}^D(\alpha _{jl}-1)\ln X_{il} - \sum _{l=1}^{D+1}\alpha _{jl} \nonumber \\&\ln (1+\sum _{l=1}^DX_{il}) - \frac{1}{2}(\varvec{\alpha }_j - \varvec{\mu }^{\setminus i}_{j})^T A^{\setminus i}_j (\varvec{\alpha }_j - \varvec{\mu }^{\setminus i}_{j})+\text{ const. } \end{aligned}$$

(33)

We can calculate the first and second derivatives with respect to $\varvec{\alpha }_j$ as

$$\begin{aligned} \frac{\partial \ln h(\varvec{\alpha }_j)}{\partial \varvec{\alpha }_j} =\begin{bmatrix} \varPsi (\mathop {\sum }\nolimits _{l=1}^{D+1}\alpha _{jl}) - \varPsi (\alpha _{j1}) + \ln X_{i1}-\ln (1+\mathop {\sum }\nolimits _{l=1}^DX_{il})\\ \vdots \\ \varPsi (\sum _{l=1}^{D+1}\alpha _{jl}) - \varPsi (\alpha _{jD}) + \ln X_{iD}-\ln (1+\mathop {\sum }\nolimits _{l=1}^DX_{il}) \end{bmatrix}-A_j^{\setminus i}(\varvec{\alpha }_j- \varvec{\mu }^{\setminus i}_j) \end{aligned}$$

(34)

$$\begin{aligned} \frac{\partial ^2\ln h(\varvec{\alpha }_j)}{\partial \varvec{\alpha }_j^2} = \begin{bmatrix} \varPsi '(\mathop {\sum }\nolimits _{l=1}^D\alpha _{jl}) - \varPsi '(\alpha _{j1})&\cdots&\varPsi '(\mathop {\sum }\nolimits _{l=1}^D\alpha _{jl})\\ \vdots&\ddots&\vdots \\ \varPsi '(\mathop {\sum }\nolimits _{l=1}^D\alpha _{jl})&\cdots&\varPsi '(\mathop {\sum }\nolimits _{l=1}^D\alpha _{jl}) - \varPsi '(\alpha _{jD}) \end{bmatrix}-A^{\setminus i}_{j} \end{aligned}$$

(35)

where $\varPsi (\cdot )$ is the digamma function. Then, we can approximate $h(\varvec{\alpha }_j)$

$$\begin{aligned} h(\varvec{\alpha }_j)\simeq h(\varvec{\alpha }_j^*)\exp \bigg (-\frac{1}{2}(\varvec{\alpha }_j-\varvec{\alpha }_j^*)\widehat{A}_{j}(\varvec{\alpha }_j-\varvec{\alpha }_j^*)\bigg ) \end{aligned}$$

(36)

where the precision matrix $\widehat{A}_{j}$ is given by

$$\begin{aligned} \widehat{A}_{j} = - \left. \frac{\partial ^2\ln h(\varvec{\alpha }_j)}{\partial \varvec{\alpha }_j^2} \right| _{\varvec{\alpha }_j =\varvec{\alpha }_j^*} \end{aligned}$$

(37)

Therefore, the integration of $h(\varvec{\alpha }_j)$ can be approximated by using Eq. (36) as

$$\begin{aligned} \int h(\varvec{\alpha }_j)d\varvec{\alpha }_j \simeq h(\varvec{\alpha }_j^*)\int \exp (-\frac{1}{2}(\varvec{\alpha }_j-\varvec{\alpha }_j^*)\widehat{A}_{j}(\varvec{\alpha }_j-\varvec{\alpha }_j^*))d\varvec{\alpha }_j= h(\varvec{\alpha }_j^*) \frac{(2\pi )^{(D+1)/2}}{|\widehat{A}_j|^{1/2}} \end{aligned}$$

(38)

Finally, we can rewrite Eq. (30) as following:

$$\begin{aligned} Z_i=\sum _{j=1}^J \bar{\lambda }_j\prod _{s=1}^{j-1}(1-\bar{\lambda }_s)h(\varvec{\alpha }_j^*)\frac{(2\pi )^{{(D+1)}/2}}{|\widehat{A}_j|^{1/2}} \end{aligned}$$

(39)