Online variational learning of finite Dirichlet mixture models

Abstract

In this paper, we present an online variational inference algorithm for finite Dirichlet mixture models learning. Online algorithms allow data points to be processed one at a time, which is important for real-time applications, and also where large scale data sets are involved so that batch processing of all data points at once becomes infeasible. By adopting the variational Bayes framework in an online manner, all the involved parameters and the model complexity (i.e. the number of components) of the Dirichlet mixture model can be estimated simultaneously in a closed form. The proposed algorithm is validated through both synthetic data sets and a challenging real-world application namely video background subtraction.

Appendices

Appendix 1: Proof of Equation (15)

The variational parameter r _ij is calculated by by setting the derivative of $\mathcal{L}(Q)$ with respect to r _ij to 0. Here we must take account of the constraint that ∑ ^M _j=1 r _ij  = 1. This can be achieved by adding a Lagarange multiplier to $\mathcal{L}(Q).$ Taking the derivative w.r.t. r _ij and setting the result to zero we get

$$ \begin{aligned} \frac{\partial {\mathcal{L}}(Q)}{\partial r_{ij}} =\,& 0\\ =\,& \frac{\partial}{\partial r_{ij}} \left\{ \sum_{i=1}^{N}\sum_{j=1}^Mr_{ij}\left[\ln\pi_{j} + {\mathcal{R}}_{j}+ \sum_{l=1}^{D}(\bar{\alpha}_{jl}-1)\ln X_{il}\right.\right.\\ &\qquad\quad\left. \left.-\ln r_{ij}\vphantom{\sum_{l=1}^{D}}\right] + \lambda\left(\sum_{j=1}^Mr_{ij}-1\right)\right\}\\ =\,& {\mathcal{R}}_{j}+ \sum_{l=1}^{D}(\bar{\alpha}_{jl}-1)\ln X_{il} + \ln\pi_{j} -(\ln r_{ij}+1) + \lambda\\ \end{aligned} $$

(39)

where

$$ {\mathcal{R}}_j = \left\langle\ln\frac{\Upgamma(\sum_{l=1}^D \alpha_{jl})}{\prod_{l=1}^D\Upgamma(\alpha_{jl})}\right\rangle \;, \qquad \bar{\alpha}_{jl} = \left\langle\alpha_{jl}\right\rangle=\frac{u_{jl}}{v_{jl}} $$

(40)

Unfortunately, a closed-form expression cannot be found for $\mathcal{R}_{j},$ so the variational inference can not be applied directly. Here, we approximate the function $\mathcal{R}_j$ using a second-order Taylor expansion about the expected values of the parameters $\varvec{\alpha}_j.$ This approximation function is denoted as $\mathcal{\widetilde{R}}_j$ and is defined in (16). By substituting $\mathcal{\widetilde{R}}_j$ into (39) for $\mathcal{R}_j$ and applying some algebra, we then have

$$ \lambda = 1- \ln \sum\limits_{j=1}^{M} \exp\left(\widetilde{R}_{j}+ \sum\limits_{l=1}^{D}(\bar{\alpha}_{jl}-1)\ln X_{il}+\ln\pi_{j}\right) $$

(41)

By substituting (41) back into (39), we then obtain

$$ r_{ij} = \frac{\exp \left[\ln\pi_j + {\mathcal{\widetilde{R}}}_j+ \sum_{l=1}^{D}(\bar{\alpha}_{jl}-1)\ln X_{il}\right]}{\sum_{j=1}^{M}\exp \left[\ln\pi_{j} + {\mathcal{\widetilde{R}}}_j+ \sum_{l=1}^{D}(\bar{\alpha}_{jl}-1)\ln X_{il}\right]} $$

(42)

Appendix 2: proof of equation (17)

The value of the mixing coefficients $\varvec{\pi}$ is calculated by maximizing the lower bound with respect to $\varvec{\pi}.$ Adding a Lagarange term to the lower bound is required due to the constraint of ∑ ^M _j=1 π _j  = 1. By taking the derivative w.r.t. π _j and setting the result to zero, we then have

$$ \begin{aligned} \frac{\partial {\mathcal{L}}(Q)}{\partial \pi_{j}} = &\frac{\partial}{\partial \pi_{j}} \sum\limits_{i=1}^{N}\sum\limits_{j=1}^{M}r_{ij}\,\hbox{ln}\,\pi_{j} + \lambda\left(\sum_{j=1}^{M}\pi_{j} -1\right)\\ = & \sum_{i=1}^{N}r_{ij}(1/\pi_{j}) + \lambda = 0 \\ \end{aligned} $$

(43)

$$ \Longrightarrow\quad \sum_{i=1}^{N}r_{ij} = -\lambda\pi_{j} $$

(44)

Summing both sides of (44) over j, we can obtain that λ =  − N. By substituting the value of λ back into (43), we then acquire

$$ \pi_j = \frac{1}{N}\sum\limits_{i=1}^{N}r_{ij} $$

(45)

Appendix 3: proof of equations (18) and (19)

For the variational factor Q(α _jl ), instead of using the gradient method, it is more straightforward to use (11) for computing the variational solution. Therefore, the logarithm of Q(α _jl ) is given by

$$ \begin{aligned} \ln Q(\alpha_{jl}) =\,& \left\langle {\mathbb{L}}({\mathcal{X}},\Uptheta)\right\rangle_{\Uptheta \neq \alpha_{jl}}\\ =& \sum\limits_{i=1}^{N} \left\langle Z_{ij}\right\rangle \left\langle\ln\frac{\Upgamma(\sum_{s=1}^{D} \alpha_{js})}{\prod_{s=1}^{D}\Upgamma(\alpha_{js})}\right\rangle_{\Uptheta \neq \alpha_{jl}}\\ & + \alpha_{jl}\sum_{i=1}^{N} \left\langle Z_{ij}\right\rangle\ln X_{il} + (u_{jl}-1)\hbox{ln}\alpha_{jl}\\ & -v_{jl}\alpha_{jl}+ \hbox{const.}\\ =& \sum_{i=1}^{N} r_{ij} {\mathcal{J}}(\alpha_{jl})+ \alpha_{jl} \sum_{i=1}^{N}r_{ij}\ln X_{il}\\ & + (u_{jl}-1)\ln\alpha_{jl} -v_{jl}\alpha_{jl} + \hbox{const.}\\ \end{aligned} $$

(46)

where

$$ {\mathcal{J}}(\alpha_{jl}) = \left\langle\ln\frac{\Upgamma(\alpha_l + \sum_{s\neq l}^{D} \alpha_{js})}{\Upgamma(\alpha_l)\prod_{s \neq l}^{D}\Upgamma(\alpha_{js})}\right\rangle_{\Uptheta \neq \alpha_{jl}} $$

(47)

Notice that, $\mathcal{J}(\alpha_{jl})$ is a function of α _jl and is unfortunately analytically intractable. We can obtain a approximate lower bound by applying first-order Taylor expansion about $\bar{\alpha}_{jl}$ (the expected value of α _jl ) as

$$ \begin{aligned} {\mathcal{J}}(\alpha_{jl})\geq& \bar{\alpha}_{jl} \hbox{ln} \alpha_{jl} \biggl[\Uppsi\biggl(\sum_{s=1}^{D}\bar{\alpha}_{js}\biggr) - \Uppsi(\bar{\alpha}_{jl}) \\ & + \sum_{s \neq l}^{D} \bar{\alpha}_{js}\Uppsi^{\prime}\left(\sum_{s=1}^D\bar{\alpha}_{js}\right)(\left\langle \hbox{ln} \alpha_{js}\right\rangle- \ln\bar{\alpha}_{js}) \biggr] + \hbox{const.}\\ \end{aligned} $$

(48)

Substituting (48) back into (46), this gives

$$ \begin{aligned} \hbox{ln}\, Q(\alpha_{jl})=&\sum_{i=1}^{N}r_{ij}\bar{\alpha}_{jl}\ln \alpha_{jl}\biggl[\Uppsi\biggl(\sum_{s=1}^{D}\bar{\alpha}_{js}\biggr) -\Uppsi(\bar{\alpha}_{jl})\\ & + \sum_{s \neq l}^{D} \Uppsi^{\prime}\biggl(\sum_{s=1}^{D}\bar{\alpha}_{js}\biggr) \bar{\alpha}_{js}(\left\langle \hbox{ln}\, \alpha_{js}\right\rangle- \ln\bar{\alpha}_{js}) \biggr]\\ & + \alpha_{jl}\sum_{i=1}^{N}r_{ij}\ln\, X_{il} + (u_{jl}-1)\ln\alpha_{jl} -v_{jl}\alpha_{jl} \\ & + \hbox{const.}\\ =\,& \hbox{ln}\, \alpha_{jl}(u_{jl}+\varphi_{jl}-1) - \alpha_{jl}(v_{jl} - \vartheta_{jl} ) + \hbox{const.}\\ \end{aligned} $$

(49)

where

$$ \begin{aligned} \varphi_{jl} =& \sum\limits_{i=1}^{N}r_{ij}\bar{\alpha}_{jl} \biggl[\Uppsi\biggl(\sum\limits_{s=1}^{D}\bar{\alpha}_{js}\biggr)-\Uppsi(\bar{\alpha}_{jl})\\ & +\sum\limits_{s \neq l}^{D} \Uppsi^{\prime}\biggl(\sum\limits_{s=1}^{D}\bar{\alpha}_{js}\biggr) \bar{\alpha}_{js}(\left\langle\ln \alpha_{js}\right\rangle- \ln\bar{\alpha}_{js})\biggr] \end{aligned} $$

(50)

$$ \vartheta_{jl} = \sum\limits_{i=1}^{N}r_{ij}\,\hbox{ln}\,X_{il} $$

(51)

It is obvious that (49) has the logarithmic form of a Gamma distribution. Taking the exponential of its both sides, it gives

$$ Q(\alpha_{jl}) \propto \alpha_{jl}^{u_{jl}+\varphi_{jl}-1} e^{-(v_{jl}-\vartheta_{jl})\alpha_{jl}} $$

(52)

Thus, the optimal solutions to the hyper-parameters $\varvec{u}$ and $\varvec{v}$ can be obtained as

$$ u_{jl}^{\ast} = u_{jl} + \varphi_{jl} \;, \qquad v_{jl}^\ast = v_{jl} - \vartheta_{jl}. $$

(53)