Bayesian Inference via Variational Approximation for Collaborative Filtering

Abstract

Variational approximation method finds wide applicability in approximating difficult-to-compute probability distributions, a problem that is especially important in Bayesian inference to estimate posterior distributions. Latent factor model is a classical model-based collaborative filtering approach that explains the user-item association by characterizing both items and users on latent factors inferred from rating patterns. Due to the sparsity of the rating matrix, the latent factor model usually encounters the overfitting problem in practice. In order to avoid overfitting, it is necessary to use additional techniques such as regularizing the model parameters or adding Bayesian priors on parameters. In this paper, two generative processes of ratings are formulated by probabilistic graphical models with corresponding latent factors, respectively. The full Bayesian frameworks of such graphical models are proposed as well as the variational inference approaches for the parameter estimation. The experimental results show the superior performance of the proposed Bayesian approaches compared with the classical regularized matrix factorization methods.

Appendix

Proof of Lemma 1

Noting that the ELBO can be written as:

$$\begin{aligned} ELBO\left( Q\right)&=E_{Q\left( A\right) ,Q\left( B\right) ,w}[\log P\left( A,B,w,R\right) ]-E_{Q\left( A\right) ,Q\left( B\right) ,w}[\log Q\left( A,B,w\right) ]\\&=E_{Q\left( A,B,w\right) }\left[ -\frac{1}{2}\sum _{i=1}^M\sum _{k=1}^K \left( \log \left( 2\pi {\sigma _k}^2\right) +\frac{a_{ik}^2}{{\sigma _k}^2}\right) -\frac{1}{2}\sum _{j=1}^N\sum _{k=1}^K \left( \log \left( 2\pi {\rho _k}^2\right) +\frac{b_{jk}^2}{{\rho _k}^2}\right) \right. \\&\quad -\left. \frac{1}{2}\sum _{l=1}^{L_w}\left( \log 2\pi +w_l^2\right) -\frac{1}{2}\sum _{\left( i,j\right) \in \varOmega }\left( \log \left( 2\pi \tau ^2\right) + \frac{\left( r_{ij}-\hat{r}_{ij}\right) ^2}{\tau ^2}\right) \right]&\\&\quad -E_{Q\left( A\right) }\left( \log Q\left( A\right) \right) -E_{Q\left( B\right) }\left( \log Q\left( B\right) \right) -E_{Q\left( w\right) }\left( \log Q\left( w\right) \right) \\&=-\frac{M}{2}\sum _{k=1}^K\log \left( 2\pi {\sigma _k}^2\right) -\frac{N}{2}\sum _{k=1}^K\log \left( 2\pi {\rho _k}^2\right) -\frac{{L_w}\log \left( 2\pi \right) }{2} -\frac{|\varOmega |}{2}\log \left( 2\pi \tau ^2\right) \\&\quad -\frac{1}{2}\sum _{k=1}^K\left( \frac{\sum _{i=1}^ME_{Q\left( A\right) }\left( a_{ik}^2\right) }{{\sigma _k}^2} +\frac{\sum _{j=1}^NE_{Q\left( B\right) }\left( b_{jk}^2\right) }{{\rho _k}^2}\right) -\frac{1}{2}\sum _l^{L_w}E_{Q\left( w\right) }\left( w_l^2\right) \\&\quad -\frac{1}{2}\sum _{\left( i,j\right) \in \varOmega }\frac{E_{Q\left( A\right) Q\left( B\right) Q\left( w\right) }\left( r_{ij}-\hat{r}_{ij}\right) ^2}{\tau ^2}\\&\quad -E_{Q\left( A\right) }\left( \log Q\left( A\right) \right) -E_{Q\left( B\right) }\left( \log Q\left( B\right) \right) -E_{Q\left( w\right) }\left( \log Q\left( w\right) \right) ) \end{aligned}$$

To achieve the optimal Q(A), we can maximize ELBO by fixing $Q(B),Q(\alpha )$ and $Q(\beta )$. This gives,

$$\begin{aligned} logQ\left( A\right)&=E_{Q\left( B\right) Q\left( w\right) }[\log p\left( R,A,B,w\right) ] \propto E_{Q\left( B\right) Q\left( w\right) }[\log p\left( R|A,B,w\right) +logp\left( A\right) ]\\&=-\frac{1}{2}\sum _{k=1}^K\sum _{i=1}^M\frac{a_{ik}^2}{{\sigma _k}^2} - \frac{1}{2}\sum _{\left( i,j\right) \in \varOmega } \frac{E_{Q\left( B\right) Q\left( w\right) }\left( r_{ij}-a_i^Tb_j-l\left( w\right) \right) ^2}{\tau ^2}\\&\quad \propto -\frac{1}{2}\sum _{i=1}^M a_i^T\varLambda _1 a_i+ \sum _{j\in N\left( i\right) } \frac{-2a_i^TE_{Q\left( B\right) }\left( b_j\right) E_{Q\left( w\right) } \left( r_{ij}-l\left( w\right) \right) +a_i^TE\left( b_jb_j^T\right) a_i}{\tau ^2}\\&\quad \propto -\frac{1}{2}\sum _{i=1}^Ma_i^T\varLambda _1 a_i+\sum _{j\in N\left( i\right) }a_i^T\left( \varPsi _j+{\bar{b}}_j {\bar{b}}_j^T\right) a_i\\&\qquad -2a_i^T\sum _{j\in N\left( i\right) } \frac{{\bar{b}}_j(r_{ij}-l\left( \bar{w}\right) }{\tau ^2} =-\frac{1}{2}\sum _{i=1}^M \left( a_i-{\bar{a}}_i\right) ^T\varPhi _i^{-1}\left( a_i-{\bar{a}}_i\right) \end{aligned}$$

Thus, Q(A) is given :

$$\begin{aligned}&Q\left( A\right) \propto \prod _{j=1}^M \exp \left( -\frac{1}{2}\left( a_i-{\bar{a}}_i\right) ^T\varPhi _i^{-1}\left( a_i-{\bar{a}}_i \right) \right) \\&\varLambda _1=\begin{pmatrix} \frac{1}{\sigma _1^2}&{} &{}0\\ &{} \ddots &{}\\ 0&{}&{}\frac{1}{\sigma _K^2} \end{pmatrix}, \varPhi _i=\left( \varLambda _1+\sum _{j\in N\left( i\right) } \frac{\varPsi _j+{\bar{b}}_j{\bar{b}}_j^T}{\tau ^2}\right) ^{-1}, \\&{\bar{a}}_i =\varPhi _i\sum _{j\in N\left( i\right) }\frac{{\bar{b}}_j(r_{ij}-l\left( \bar{w}\right) )}{\tau ^2}, \end{aligned}$$

where N(i) is the set of j’s such that $r_{ij}$ is observed. $\varPhi _i$ and $\bar{a_i}$ are the covariance and the mean of $a_i$ respectively. $\varPsi _j$ and $\bar{b_j}$ are the covariance and the mean of $b_j$respectively. $\bar{w}$ is the mean of w. Similarly, the optimal Q(B) is gained by the same method.

$$\begin{aligned} logQ(B)&=E_{Q(A)Q(w)}[\log p(R,A,B,w)] \\&\quad \propto -\frac{1}{2}\sum _{k=1}^K\sum _{j=1}^N\frac{b_{jk}^2}{{\rho _k}^2} - \frac{1}{2}\sum _{(i,j)\in \varOmega } \frac{E_{Q(A)Q(w)}(r_{ij}-a_i^Tb_j-l(w))^2}{\tau ^2}\\&\quad \propto -\frac{1}{2}\sum _{j=1}^Nb_j^T\varLambda _2 b_j+\sum _{i\in N(j)}b_j^T(\varPhi _i+{\bar{a}}_i {\bar{a}}_i^T)b_j-2b_j^T\sum _{i\in N(j)} \frac{{\bar{a}}_i(r_{ij}-l(\bar{w}) }{\tau ^2} \\&=-\frac{1}{2}\sum _{j=1}^N (b_j-{\bar{b}}_j)^T\varPsi _j^{-1}(b_j-{\bar{b}}_j) \end{aligned}$$

Thus, Q(B) is gained:

$$\begin{aligned}&Q\left( B\right) \propto \prod _{j=1}^N\exp \left( -\frac{1}{2}\left( b_j-{\bar{b}}_j\right) ^T\varPsi ^{-1}\left( b_j-{\bar{b}}_j\right) \right) \\&\varLambda _2=\begin{pmatrix} \frac{1}{\rho _1^2}&{} &{}0\\ &{} \ddots &{}\\ 0&{}&{}\frac{1}{\rho _K^2} \end{pmatrix}, \varPsi _j=\left( \varLambda _2+\sum _{i\in N\left( j\right) } \frac{\varPhi _i+{\bar{a}}_i{\bar{a}}_i^T}{\tau ^2}\right) ^{-1},\\&{\bar{b}}_j =\varPsi _j\sum _{i\in N\left( j\right) }\frac{{\bar{a}}_j (r_{ij}-l\left( w\right) }{\tau ^2} \end{aligned}$$

Assume the linear function $l(w)=x^Tw$, where x represents the known sample.

$$\begin{aligned} \log Q\left( w\right)&=E_{Q\left( A\right) Q\left( B\right) }[\log p\left( R,A,B,w\right) ] \propto E_{Q\left( A\right) Q\left( B\right) }[\log p\left( R|A,B,w\right) +\log p\left( w\right) ]\\&=E_{Q\left( A\right) Q\left( B\right) }[-\frac{1}{2}\sum _{\left( i,j\right) \in \varOmega }\frac{\left( r_{ij}-a_i^Tb_j-x^Tw\right) ^2}{\tau ^2}-\frac{1}{2}\sum _{l=1}^{L_w}w_l^2]\\&\quad \propto -\frac{1}{2}\sum _{\left( i,j\right) \in \varOmega }E_{Q\left( A\right) ,Q\left( B\right) }\frac{2x^Tw\left( r_{ij}-a_ib_j\right) +w^Txx^Tw}{\tau ^2}-\frac{1}{2}w^Tw\\&=-\frac{1}{2}w^T\varDelta w-\frac{1}{2}\sum _{\left( i,j\right) \in \varOmega }\frac{2x^Tw\left( r_{ij}-{\bar{a}}_i{\bar{b}}_j\right) +w^Txx^Tw}{\tau ^2}\\&=-\frac{1}{2}\sum _{l=1}^{L_w} \left( w_l-\bar{w}_l\right) ^T\varDelta ^{-1}\left( w_l-\bar{w}_l\right) Q\left( w\right) \propto \exp -\left( -\frac{1}{2} \left( w -\bar{w}\right) ^T\varDelta ^{-1}\left( w-\bar{w}\right) \right) \\&\varDelta =\left( I+\sum _{\left( i,j\right) \in \varOmega }\frac{xx^T}{\tau ^2}\right) ^{-1}, \bar{w}=\varDelta \sum _{\left( i,j\right) \in \varOmega }\frac{x^T\left( r_{ij}-{\bar{a}}_i^T{\bar{b}}_j\right) }{\tau ^2} \end{aligned}$$

Therefore, the local optimal $Q(A,B,w)=Q(A)Q(B)Q(w)$ is given. $\square$