BMF: Matrix Factorization of Large Scale Data Using Block Based Approach

Abstract

Matrix Factorization on large scale matrices is a memory intensive task. Alternative convergence techniques are needed when the size of the input matrix and the latent feature matrices are higher than the available memory, both on a Central Processing Unit (CPU) as well as a Graphical Processing Unit (GPU). While alternating least squares (ALS) convergence on a CPU could last forever, loading all the required matrices on to a GPU memory may not be possible when the dimensions are significantly high. In this paper, we introduce a novel technique based on dividing the entire data into block matrices and make use of the Stochastic Gradient Descent (SGD) based factorization at the block level.

A Derivation of SGD Update Equations for BMF

Our objective is to find $\underline{U_i}, \underline{V_j}$ for $\underline{X_{ij}}$ such that $\underline{X_{ij}} \approx \underline{U_i}.{\underline{V_j}}^T$. Let $\underline{E_{ij}}$ represent the deviation of estimate $\underline{X_{ij}}^\prime$ from actual ($\underline{X_{ij}}$). Hence, sum of squared deviations ($\mathcal {E}$) can be represented as:

$$\begin{aligned} \displaystyle {\mathcal {E}} = ||\underline{X_{ij}} -\underline{X_{ij}}^\prime ||^2 = ||\underline{X_{ij}} - \underline{U_{i}} {\underline{V_j}}^T||^2 \end{aligned}$$

(4)

The goal is to find $\underline{U_i}$ and $\underline{V_j}$ such that the sum of squared deviations is minimal. The optimum value for $g^{th}$ latent feature of $\underline{U_i}$ and $\underline{V_j}$ blocks can be obtained by minimizing Eq. (4) with respect to $\underline{U_i}$ and $\underline{V_j}$ respectively as:

$$\begin{aligned}&\displaystyle {\frac{\partial }{\partial \underline{U_i}}_{*g}\mathcal {E}} = -2 ( \underline{X_{ij}} - \underline{X_{ij}}^\prime ).\underline{V_j}_{*g} \end{aligned}$$

(5)

$$\begin{aligned}&\displaystyle {\frac{\partial }{\partial \underline{V_j}}_{*g}\mathcal {E}} = -2{( \underline{X_{ij}} - \underline{X_{ij}}^\prime )^T}\underline{U_i}_{*g} \end{aligned}$$

(6)

Using the block level gradients for a latent feature, the update equations for $g^{th}$ feature of $\underline{U_i}$ and $\underline{V_j}$ can be arrived as:

$$\begin{aligned}&\displaystyle {\underline{U_i}_{*g}^\prime } = \underline{U_i}_{*g} + \alpha \frac{\partial }{\partial \underline{U_i}_{*g}} \mathcal {E} = \underline{U_i}_{*g} + 2\alpha \underline{E}_{ij}\underline{V_j}_{*g} \end{aligned}$$

(7)

$$\begin{aligned}&\displaystyle {\underline{V_j}_{*g}^\prime } = \underline{V_j}_{*g} + \alpha \frac{\partial }{\partial \underline{V_j}_{*g}} \mathcal {E} = \underline{V_j}_{*g} + 2\alpha \underline{E_{ij}}^{T}\underline{U_i}_{*g} \end{aligned}$$

(8)

Factoring in regularization term into the Eqs. (7) and (8) gives us:

$$\begin{aligned}&\displaystyle {\underline{U_i}_{*g}^\prime } = \underline{U_i}_{*g} + \alpha (2\underline{E_{ij}}\underline{V_j}_{*g} - \beta \underline{U_i}_{*g}) \end{aligned}$$

(9)

$$\begin{aligned}&\displaystyle {\underline{V_i}_{*g}^\prime } = \underline{V_j}_{*g} + \alpha (2\underline{E_{ij}}^{T}\underline{U_i}_{*g} - \beta \underline{V_j}_{*g}) \end{aligned}$$

(10)