Bayesian nonnegative matrix factorization in an incremental manner for data representation

Abstract

Nonnegative matrix factorization (NMF) is a novel paradigm for feature representation and dimensionality reduction. However, the performance of the NMF model is affected by two critical and challenging problems. One is that the original NMF does not consider the distribution information of data and parameters, resulting in inaccurate representations. The other is the high computational complexity in online processing. Bayesian approaches are proposed to address the former problem of NMF. However, most existing Bayesian-based NMF models utilize an exponential prior, which only guarantees the nonnegativity of parameters without fully considering the prior information of the parameters. Thus, a new Bayesian-based NMF model is constructed based on the Gaussian likelihood and a truncated Gaussian prior, called the truncated Gaussian-based NMF (TG-NMF) model, in which a truncated Gaussian prior can prevent overfitting while ensuring nonnegativity. Furthermore, Bayesian inference-based incremental learning is introduced to reduce the high computational complexity of TG-NMF; this model is called TG-INMF. We adopt variational Bayesian to estimate all parameters of TG-NMF and TG-INMF. Experiments on genetic data-based tumor recognition demonstrate that our models are competitive with other existing methods for classification problems.

Appendix

Firstly, the optimization problem of the TG-NMF model is solved by the variational Bayesian inference algorithm. The variational posterior distribution can be derived using the (15), and then the update rules are obtained according to those distributions. The derivations for W_ir, H_rj, τ are shown below. For a random variable x and the function f of x, we adopt $\widetilde {f(x)}$ as a shorthand for E_q[f(x)].

Following (15), the variational posterior distribution of W_ir is truncated Gaussian distribution, i.e., ${W_{ir}} \sim TG\left ({\left .{{W_{ir}}} \right |\mu _{ir}^{W},(\tau _{ir}^{W})^{-1},0,+\infty }\right )$.

$$ \begin{array}{@{}rcl@{}} {q^{*}}\left( {{W_{ir}}} \right) &\propto &\exp \left\{ {{E_{q({{\theta_{-\!{W_{ir}}}}})}}\left[ {\log p\left( {{{V_{ij}}}|{W_{ir}},{H_{rj}}} \right) + \log p\left( {{{W_{ir}}}|\mu_{ir},(\tau_{ir})^{-1},0, + \infty } \right)} \right]} \right\} \times I\left( x \right) \\ &\propto &\exp \left\{ {{E_{q({{\theta_{-\!{W_{ir}}}}})}}\left[ { - \frac{\tau }{2}\sum\limits_{j} {{{\left( {{V_{ij}} - {{\left( {WH} \right)}_{ij}}} \right)}^{2}} - \frac{{{\tau_{ir}}}}{2}{{\left( {{W_{ir}} - \mu_{ir}^{}} \right)}^{2}}} } \right]} \right\} \times I\left( x \right) \\ &\propto &\exp \left\{ {{E_{q({{\theta_{-\!{W_{ir}}}}})}}\left[ {\left( {-\frac{\tau }{2}\sum\limits_{j} {H_{rj}^{2}-\frac{{\tau_{ir}^{}}}{2}}}\right)W_{ir}^{2}+A(W_{ir},H_{rj},\tau){W_{ir}}} \right]} \right\} \times I\left( x \right) \\ &\propto &\exp \left\{{-\frac{{\tau_{ir}^{W}}}{2}{{\left( {{W_{ir}}-\mu_{ir}^{W}}\right)}^{2}}}\right\} \\ &\propto &TG\left( {\left. {{W_{ir}}} \right|\mu_{ir}^{W},(\tau_{ir}^{W})^{-1},0,\infty } \right), \end{array} $$

(24)

In the above formula, A(W_ir,H_rj,τ) is shown in (25) respectively.

$$ A(W_{ir},H_{rj},\tau)={\tau \sum\limits_{j} {\left( {{V_{ij}}-\sum\limits_{r^{\prime}\ne r} {{W_{ir^{\prime}}}{H_{r^{\prime}j}}} } \right)}{H_{rj}} + {\tau_{ir}}\mu_{ir}}. $$

(25)

And the parameters $\tau _{ir}^{W}$ and $\mu _{ir}^{W}$ corresponding to variational posterior distribution of W_ir are shown in (26) and (26).

$$ \tau_{ir}^{W} = \tilde \tau \sum\limits_{j} {\tilde H_{rj}^{2}}+{\tilde \tau_{ir}}, $$

(26)

$$ \mu_{ir}^{W} = \frac{{\tilde \tau \sum\limits_{j} {\left( {{V_{ij}} - \sum\limits_{r^{\prime} \ne r} {{{\tilde W}_{ir^{\prime}}}{{\tilde H}_{r^{\prime}j}}} } \right){{\tilde H}_{rj}} + {{\tilde \tau }_{ir}}{{\tilde \mu }_{ir}}} }}{{\tilde \tau \sum\limits_{j} {\tilde H_{rj}^{2}} { +}{{\tilde \tau }_{ir}}}}. $$

(27)

In the same way, the variational posterior distribution of H_rj is also truncated Gaussian distribution, that is, ${H_{rj}} \sim TG\left ({\left .{{H_{rj}}} \right |\mu _{rj}^{H},(\tau _{rj}^{H})^{-1},0,+\infty }\right )$.

$$ \begin{array}{@{}rcl@{}} {q^{*}}\left( {{H_{rj}}} \right) &\propto& \exp \left\{ {{E_{q({{\theta_{-\!{H_{rj}}}}})}}\left[ {\log p\left( {\left. {{V_{ij}}} \right|{W_{ir}},{H_{rj}}} \right) + \log p\left( {\left. {{H_{rj}}} \right|{\mu_{rj}},\tau_{rj}^{- 1},0, + \infty } \right)} \right]} \right\} \times I\left( x \right) \\ &\propto& \exp \left\{ {{E_{q({{\theta_{-\!{H_{rj}}}}})}}\left[ { - \frac{\tau }{2}\sum\limits_{i} {{{\left( {{V_{ij}} - {{\left( {WH} \right)}_{ij}}} \right)}^{2}} - \frac{{{\tau_{rj}}}}{2}{{\left( {{H_{rj}} - \mu_{rj}^{}} \right)}^{2}}} } \right]} \right\} \times I\left( x \right) \\ &\propto& \exp \left\{ {{E_{q({{\theta_{-\!{H_{rj}}}}})}}\left[ {\left( {-\frac{\tau}{2}\sum\limits_{i} {W_{ir}^{2}-\frac{{\tau_{rj}}}2}} \right)H_{rj}^{2}+B(W_{ir},H_{rj},\tau){H_{rj}}} \right]} \right\} \times I\left( x \right) \\ &\propto& \exp \left\{ { - \frac{{\tau_{rj}^{H}}}{2}{{\left( {{H_{rj}} - \mu_{rj}^{H}} \right)}^{2}}} \right\} \\ &\propto& TG\left( {\left. {{H_{rj}}} \right|\mu_{rj}^{H},(\tau_{rj}^{H})^{-1},0, + \infty } \right), \end{array} $$

(28)

The B(W_ir,H_rj,τ) contained in above (28) is shown in (29).

$$ B(W_{ir},H_{rj},\tau)={\tau \sum\limits_{i} {\left( {{V_{ij}} - \sum\limits_{r^{\prime}\ne r} {{W_{ir^{\prime}}}{H_{r^{\prime}j}}} } \right)} {W_{ir}} + {\tau_{rj}}\mu_{rj}}.\\ $$

(29)

The parameters $\tau _{rj}^{H}$ and $\mu _{rj}^{H}$ in the variational posterior distribution of H_rj are written as in (30) and (31).

$$ \tau_{rj}^{H} = \tilde \tau \sum\limits_{i} {\tilde W_{ir}^{2}} {\text{ + }}{\tilde \tau_{rj}}, $$

(30)

$$ \mu_{ir}^{H} = \frac{{\tilde \tau \sum\limits_{i} {\left( {{V_{ij}} - \sum\limits_{r^{\prime} \ne r} {{{\tilde W}_{ir^{\prime}}}{{\tilde H}_{r^{\prime}j}}} } \right){{\tilde W}_{ir}} + {{\tilde \tau }_{rj}}{{\tilde \mu }_{rj}}} }}{{\tilde \tau \sum\limits_{i} {\tilde W_{ir}^{2}} {\text{ + }}{{\tilde \tau }_{rj}}}}. $$

(31)

For the parameter τ, the variational posterior distribution takes the same form as the prior distribution, i.e., $\tau \sim Gamma(\alpha _{\tau }^{*},\beta _{\tau }^{*})$,

$$ \begin{array}{@{}rcl@{}} {q^{*}}\left( \tau \right) &\propto& \exp \left\{ {{E_{q\left( {{\theta_{-\!\tau }}} \right)}}\left[ {\log p\left( {\left. {{V_{ij}}}\right|{W_{ir}},{H_{rj}},{\tau^{-1}}}\right)+ \log p\left( {\left. \tau \right|{\alpha_{\tau} },{\beta_{\tau} }} \right)} \right]} \right\} \\ &\propto& \exp \left\{ {{E_{q\left( {{\theta_{-\!\tau }}}\right)}}\left[ {\frac{{nm}}{2}\log \tau - \frac{\tau }{2}\sum\limits_{i,j} {{{\left( {{V_{ij}} - {{\left( {WH} \right)}_{ij}}} \right)}^{2}} + \left( {{\alpha_{\tau} } - 1} \right)\log \tau - {\beta_{\tau} }\tau } } \right]} \right\} \\ &\propto& \exp \left\{ {\left( {{\alpha_{\tau} } - 1 +\!\frac{{nm}}{2}} \right)\log \tau - \left[ {{\beta_{\tau} } + \frac{1}{2}\sum\limits_{i,j} {{E_{q}}\left[ {{{\left( {{V_{i,j}} - {{\left( {WH} \right)}_{i,j}}} \right)}^{2}}} \right]} } \right]\tau } \right\} \\ &\propto& Gamma\left( \tau|{\alpha_{\tau}^{*},\beta_{\tau}^{*}} \right), \end{array} $$

(32)

where the parameters $\alpha _{\tau }^{*},\beta _{\tau }^{*}$ corresponding to the variational posterior distribution of τ are shown as follows.

$$ \begin{array}{@{}rcl@{}} &&\alpha_{\tau}^{*} = {\alpha_{\tau} } + \frac{{nm}}{2}, \end{array} $$

(33)

$$ \begin{array}{@{}rcl@{}} &&\beta_{\tau}^{*} = {\beta_{\tau} } + \frac{1}{2}\sum\limits_{i,j} {{E_{q}}\left[ {{{\left( {{V_{ij}} - {{\left( {WH} \right)}_{ij}}} \right)}^{2}}} \right]}. \end{array} $$

(34)

The above is the detailed optimization process of TG-NMF model. In fact, the prior distribution of current samples in the TG-INMF model equals to the posterior distribution of previous samples, which can be obtained by the TG-NMF model. Therefore, based on the optimal estimation of TG-NMF model, we then give the optimization process of the TG-INMF model.

The parameters to be optimized in the TG-INMF model can be summarized as $\theta ^{\prime }=\{W_{ir}^{k+1},h_{ir}^{k+1},\tau ^{k+1}\}$. According to (15), the variational posterior distribution of $W_{ir}^{k+1}$ is first derived and $W_{ir}^{k+1}$ obeys the truncated Gaussian distribution with parameters $\tau _{ir}^{W^{k+1}},\mu _{ir}^{W^{k+1}}$.

$$ \begin{array}{@{}rcl@{}} {q^{*}}\!\left( {W_{ir}^{k + 1}}\right)\!&\propto&\!\exp\!\left\{{{E_{q({{\theta_{ - W_{ir}^{k + 1}}}})}}\!\left[{\log \left( {{p_{k + 1}}\left( {\left. {{v^{k + 1}}} \right|\theta^{\prime}} \right)} \right) + \log \left( {p\left. {\left( {W_{ir}^{k + 1}} \right)} \right|\mu\!_{ir}^{W}\!,(\tau_{ir}^{W})^{ - 1}\!,0,\!+\infty } \right)} \right]}\!\right\}\!\times\!I\left( x \right) \\ &\propto&\!\exp\!\left\{{{E_{q({{\theta_{ - W_{ir}^{k + 1}}}})}}\left[ {\!-\frac{{{\tau^{k+1}}}}{2}{{\left( {v_{i}^{k + 1} - {{\left( {{W^{k + 1}}{h^{k + 1}}} \right)}_{i}}} \right)}^{2}} - \frac{{\tau_{ir}^{W}}}{2}{{\left( {W_{ir}^{k + 1} - \mu\!_{ir}^{W}}\right)}^{2}}} \right]} \right\}\!\times\!I\left( x \right) \\ &\propto&\!\exp\!\left\{ { - \frac{{\tau_{ir}^{{W^{k + 1}}}}}{2}\left( {W_{ir}^{k + 1} - \mu_{ir}^{{W^{k+1}}}}\right)}\right\}\!\times\!I\left( x \right) \\ &\propto&\!TG\left( {\left. {W_{ir}^{k + 1}} \right|\mu_{ir}^{{W^{k+1}}}\!,(\tau_{ir}^{{W^{k+1}}})^{ - 1}\!,0, + \infty } \right), \end{array} $$

(35)

And (36) and (37) are the parameters $\tau _{ir}^{W^{k+1}}$, $\mu _{ir}^{W^{k+1}}$ of variational posterior distribution of $W_{ir}^{k+1}$.

$$ \tau_{ir}^{{W^{k+1}}} = \tilde \tau {\left( {\tilde h_{r}^{k + 1}} \right)^{2}} + \tilde \tau_{ir}^{W}, $$

(36)

$$ \mu_{ir}^{{W^{k+1}}} = \frac{{\tilde \tau \left( \! {v_{i}^{k + 1}\! -\! \sum\limits_{r^{\prime} \ne r} {\tilde W_{ir^{\prime}}^{k + 1}\tilde h_{r^{\prime}}^{k + 1}} } \right)\tilde h_{r}^{k + 1} + \tilde \tau_{ir}^{W}\tilde \mu_{ir}^{W}}}{{\tilde \tau {{\left( {\tilde h_{r}^{k + 1}} \right)}^{2}} + \tilde \tau_{ir}^{W}}}. $$

(37)

Secondly, (38) shows the optimization process of the variational posterior distribution corresponding to $h_{r}^{k+1}$,

$$ \begin{array}{@{}rcl@{}} {q^{*}}\left( {h_{r}^{k + 1}} \right) &\propto&\!\exp\!\left\{ {{E_{q({{\theta_{ - h_{r}^{k + 1}}}})}}\left[{\log p\left( {\left. {v_{i}^{k + 1}} \right|\theta^{\prime}} \right) + \log p\left( {\left. {h_{r}^{k + 1}} \right|\mu_{rj}^{H}\!,(\tau_{rj}^{H})^{ - 1}\!,0, + \infty } \right)} \right]} \right\}\!\times\!I\left( x \right) \\ &\propto&\!\exp\!\left\{ {{E_{q({{\theta_{ - h_{r}^{k + 1}}}})}}\left[ {\sum\limits_{i = 1}^{n} { - \frac{{{\tau^{k + 1}}}}{2}{{\left( {v_{i}^{k + 1} - {{\left( {{W^{k + 1}}{h^{k + 1}}} \right)}_{i}}} \right)}^{2}} - \frac{{\tau_{rj}^{H}}}{2}{{\left( {h_{r}^{k + 1} - \mu_{rj}^{H}} \right)}^{2}}} } \right]} \right\}\!\times\!I\left( x \right) \\ &\propto& \exp \left\{ { - \frac{{\tau_{r}^{{h_{k+1}}}}}{2}{{\left( {h_{r}^{k + 1} - \mu_{r}^{{h_{k + 1}}}} \right)}^{2}}} \right\} \\ &\propto&\!TG\!\left( {\left. {h_{r}^{k + 1}} \right|\mu_{r}^{{h^{k + 1}}}\!,(\tau_{r}^{{h^{k + 1}}})^{ - 1}\!,0, + \infty } \right), \end{array} $$

(38)

where the parameters $\tau _{r}^{h^{k+1}}$, $\mu _{r}^{h^{k+1}}$ in the variational posterior distribution of $h_{r}^{k+1}$ are shown below.

$$ \tau_{r}^{{h_{k + 1}}} = \tilde \tau \sum\limits_{i = 1}^{n} {\left( {{{\tilde W}_{k + 1}}} \right)_{ir}^{2} + \tilde \tau_{rj}^{H}}, $$

(39)

$$ \mu_{r}^{{h_{k + 1}}} = \frac{{\sum\limits_{i = 1}^{n} {\tilde \tau \left( {v_{i}^{k + 1} - \sum\limits_{r^{\prime} \ne r} {\tilde W_{ir^{\prime}}^{k + 1}\tilde h_{r^{\prime}}^{k + 1}} } \right)\tilde W_{ir}^{k + 1} + \tilde \tau_{rj}^{H}\tilde \mu_{rj}^{H}} }}{{\tilde \tau \sum\limits_{i = 1}^{n} {{{\left( {\tilde W_{ir}^{k + 1}} \right)}^{2}} + {{\tilde \tau }_{r}}} }}. $$

(40)

Finally, the variational posterior distribution of τ^k+ 1 is illustrated in (41).

$$ \begin{array}{@{}rcl@{}} {q^{*}}\!\left( {{\tau^{k + 1}}} \right) &\propto& \exp \left\{ {{E_{q({{\theta\!_{ - {\tau^{k + 1}}}}})}}\left[ {\log p\left( \left. {v_{i}^{k + 1}} \right|\theta^{\prime}\right) + \log p\left( {\left. {{\tau^{k + 1}}} \right|{\alpha_{\tau}^{*}},{\beta_{\tau}^{*}}}\right)}\right]}\right\} \\ &\propto&\!\exp\!\left\{ {{E_{q({{\theta_{ - {\tau^{k + 1}}}}})}}\left\{{\left( {{\alpha_{\tau}^{*}} - 1 + \frac{n}{2}} \right)\log {\tau^{k + 1}} - \left[ {{\beta_{\tau}^{*}} + \frac{1}{2}\sum\limits_{i = 1}^{n} {{{\left( {v_{i}^{k + 1} - {{\left( {{W^{k + 1}}{h^{k + 1}}} \right)}_{i}}} \right)}^{2}}} } \right]{\tau^{k + 1}}} \right\}} \right\} \\ &\propto& Gamma(\tau^{k+1}|\alpha^{k+1},\beta^{k+1}), \end{array} $$

(41)

where the parameters α^k+ 1, β^k+ 1 of new variational posterior distribution are written as (42) and (43).

$$ \begin{array}{@{}rcl@{}} &&{\alpha^{k+1}}= {\alpha_{\tau}^{*}} + \frac{n}{2}, \end{array} $$

(42)

$$ \begin{array}{@{}rcl@{}} &&{\beta^{k+1}}= {\beta_{\tau}^{*}}+\frac{1}{2}\sum\limits_{i = 1}^{n} {{{\left( {v_{i}^{k + 1} - {{\left( {{W^{k + 1}}{h^{k + 1}}} \right)}_{i}}} \right)}^{2}}}. \end{array} $$

(43)

After obtaining the variational posterior distributions of the parameters, we can derive the optimal updates of ${W^{k+1}}_{ir}$, ${h^{k+1}}_{r}$, τ^k+ 1.

$$ { {W^{k+1}_{ir}}} \!\leftarrow\! \mu_{ir}^{{W^{k +1}}} + \frac{1}{{\sqrt {\tau_{ir}^{{W^{k+1}}}} }}\lambda \left( {\! - \!\mu_{ir}^{{W^{k\! +\! 1}}}\sqrt {\tau_{ir}^{{W^{k + 1}}}} } \right), $$

(44)

$$ {h^{k+1}_{r}}\! \leftarrow\! \mu_{r}^{{h^{k +1}}}\! +\! \frac{1}{{\sqrt {\tau_{r}^{{h^{k+1}}}} }}\lambda \left( {\! -\! \mu_{r}^{{h^{k+1}}}\sqrt {\tau_{r}^{{h^{k + 1}}}} } \right), $$

(45)

$$ \tau^{k+1} \leftarrow \frac{{\alpha_{\tau}^{*}}}{{\beta_{\tau}^{*}}}. $$

(46)