Iterative rank-one matrix completion via singular value decomposition and nuclear norm regularization

Abstract

Matrix completion is widely used in many fields. In existing matrix completion methods, such as rank minimization and matrix factorization, the hyperparameters must be learned. However, hyperparameter tuning is a time-consuming and tedious process. In this paper, we propose a novel matrix completion method called IMC, i.e., iterative rank-one matrix completion via singular value decomposition (SVD) and nuclear norm regularization. First, we construct a rank-one matrix completion model using nuclear norm regularization. Then, the variables to be optimized in the model are divided into several blocks. Finally, the blocks are iteratively optimized one by one until convergence. For the optimization of each block, we propose an efficient solution scheme based on SVD, in which only the maximum singular value and leading singular vectors of a sparse matrix must be calculated. Further, a nonparametric singular value penalty function is designed to ensure a low-rank completion matrix. In addition, the optimization of each block uses only the values inside the observed entries; hence, no errors are accumulated. Test results shows that the proposed method converges rapidly and outperforms some state-of-the-art methods when applied to grayscale image restoration, recommendation systems, and vote networks.

Introduction

Matrix completion is based on partial observations of a matrix to recover missing entries. It has been widely applied to image processing [1], [2], [3], recommendation systems [4], [5], [6], relational networks [7], [8], compressed sensing [9], sensor networks [10], classification [11], and clustering [12], [13]. Most existing matrix completion methods assume that the target matrix has a low-rank structure, as most real-world data (e.g., images and user ratings) are low rank or approximately low rank [14]. In general, these methods can be broadly classified into three categories: rank minimization (RM), nuclear norm minimization (NNM), and matrix factorization.

Consider an incomplete observation matrix $\mathit{A}\in R^{m\times l}$. The set $\mathrm{\Omega }$ denotes the positions of the observed (known) entries in matrix $\mathit{A}$, and entries outside $\mathrm{\Omega }$ are often considered missing. The RM method aims to recover a low-rank matrix, $\mathit{X}$, that exactly matches $\mathit{A}$ over $\mathrm{\Omega }$, and describes matrix completion as the problem of minimizing $\mathit{rank}(\mathit{X})$ subject to ${\mathcal{P}}_{\mathrm{\Omega }}(\mathit{X})={\mathcal{P}}_{\mathrm{\Omega }}(\mathit{A})$, where $\mathit{rank}(\mathit{X})$ is the rank of $\mathit{X}$ and ${\mathcal{P}}_{\mathrm{\Omega }}\left(·\right)$ denotes a projection operator for extracting the entries of the positions in $\mathrm{\Omega }$ (see the “Notation” part at the end of this section). However, considering the noise in real-world data, many studies have attempted to find a low-rank matrix that is sufficiently close to the original incomplete matrix in terms of the observed entries, and have reformulated the above-mentioned RM problem as follows:

$$\underset{\mathit{X}}{\mathrm{m}\mathrm{i}\mathrm{n}}\frac{1}{2}{\Vert {\mathcal{P}}_{\mathrm{\Omega }}\left(\mathit{X}\right)-{\mathcal{P}}_{\mathrm{\Omega }}\left(\mathit{A}\right)\Vert }_F^2\mathrm{s}.\mathrm{t}.rank\left(\mathit{X}\right)\le \tau ,$$

where $\tau$ is a rank constraint and ${\Vert ·\Vert }_F$ is the Frobenius norm (see the “Notation” part at the end of this section). Some studies have proposed solutions for this model. For example, Wang et al. [15] extended the orthogonal matching pursuit method from the vector case to the matrix case and completed the matrix by tuning the rank. Vandereycken [16] proposed using low-rank nonlinear conjugated gradients in Euclidean space to perform optimization on manifolds (LRGeomCG); however, he fixed the minimum rank of the matrix. Chen et al. [17] proposed a first-order fast optimization algorithm (FOA) and further transformed the fixed rank into an unfixed rank. To increase the robustness of the algorithm, Le Pendu et al. [18] added an equality constraint to Model (1), defined an augmented Lagrangian function, and then used the alternating direction method of multipliers [19] to update several variables iteratively. Guo et al. [20] filled the missing data in a tensor using the Frank–Wolfe algorithm under the condition of rank constraint. In these methods, the rank constraint setting is critical and can strongly affect the results. However, adjusting the parameters to obtain the optimal rank constraint is a tedious and time-consuming process. Furthermore, in practical applications, the matrix to be recovered is only approximately low rank, not exactly low rank [21]. Moreover, owing to the discrete nature of the rank function, Model (1) is usually an NP-hard problem [22].

Candès and Recht [23] proved that the RM problem can be surrogated by NNM, a convex relaxation of RM. In NNM, the matrix completion substitutes the nuclear norm regularization for the rank constraint and thus becomes the following optimization problem:

$$\underset{\mathit{X}}{\min }\frac{1}{2}{\Vert {\mathcal{P}}_{\mathrm{\Omega }}\left(\mathit{X}\right)-{\mathcal{P}}_{\mathrm{\Omega }}\left(\mathit{A}\right)\Vert }_F^2+\gamma {\Vert \mathit{X}\Vert }_{\ast },$$

where ${\Vert ·\Vert }_{\ast }$ denotes the nuclear norm (see the “Notation” part at the end of this section), $\gamma {\Vert \mathit{X}\Vert }_{\ast }$ is a regularization term, and $\gamma$ is a regularization parameter. For Model (2), the soft-impute method [24] uses data-specific “sparse plus low-rank” structures to process large-scale data. Yao and Kwok [25] proposed an accelerated soft-impute method based on the power method [26]. As the nuclear norm treats each singular value equally and may cause large singular values to prevail, some researchers have used the nonconvex nuclear norm [21], [27] or the weighted nuclear norm [28] in Model (2). Zhang et al. [29] believed that the nuclear norm does not necessarily approximate the matrix rank well in matrix completion; hence, they proposed the concept of the truncated nuclear norm. The truncated nuclear norm was also used by Hu et al. [30] and Hu et al. [31]. However, adjusting the regularization parameters in these methods is a time-consuming process, and the best way to select the appropriate norm according to the application scenario is a complex problem.

Matrix factorization is a classical method of matrix completion. A matrix, X, can be completed via $\mathit{X}={\mathit{U}}^{m\times r}{\mathit{V}}^{r\times l}$ [32], [33], where r is the rank of $\mathit{X}$. In some studies, X has been decomposed via singular value decomposition (SVD): X = ${\mathit{U}}^{m\times r}{\mathbf{\Sigma }}^{r\times r}{\left({\mathit{V}}^{l\times r}\right)}^T$, where $\mathbf{\Sigma }$ is a diagonal matrix comprising singular values and the column vectors of U and V are orthogonal [34], [35]. Because the value of rank r must be adjusted, these methods also suffer from the same shortcomings as previous RM methods. Rank-one matrix completion (R1MC) uses the sum of several rank-one matrices to approximate the observation matrix, i.e., ${\sum }_{i=1}^N{\alpha }_i{\mathit{u}}_{\mathit{i}}{{\mathit{v}}_i}^T$. Owing to its simple form, R1MC has been widely used in matrix completion [14], [36], [37]. As SVD naturally has the form of R1MC, SVD-based matrix completion is a special case of R1MC. R1MC methods usually adopt an iterative method to ensure completion accuracy, and penalize singular values, including the quantity limit [14] and threshold operation [24], to ensure a low-rank completion matrix. However, existing R1MC methods have the following shortcomings. (i) Partial SVD is required for each iteration; hence, the computational overhead involved in the completion of large-scale matrices is too high. (ii) Each iteration depends on values outside the observed entries calculated by the previous iterations, which may lead to noise accumulation. (iii) Finally, the rank or regularization parameter needs to be adjusted. R1MC is discussed in detail in Section 2.1.

To address the above-mentioned challenges, we propose a novel matrix completion method called IMC, i.e., iterative R1MC via SVD and nuclear norm regularization. This method constructs an R1MC model using nuclear norm regularization and designs an iterative SVD-based solution algorithm according to the model characteristics. The variables to be optimized in the model are divided into several blocks, which are iteratively optimized one by one until convergence. To optimize each block, we propose an efficient solution based on SVD and penalize singular values to ensure a low-rank completion matrix. The main contributions of this study are summarized as follows:

• We propose a new R1MC method that converts the complex optimization model into multistep subproblems with iterative solutions by dividing the variables to be optimized into blocks. Each subproblem uses only the values inside the observed entries without accumulating errors. More importantly, we analyze the convergence of such iterative solutions.

• To optimize several variables in each block, many studies have adopted an iterative approach. By analyzing the characteristics of Lagrangian partial derivatives, we propose a one-time optimization scheme based on SVD without iterative calculation. Moreover, our SVD approach needs to calculate only the maximum singular value and leading singular vectors of a sparse matrix, and the calculation cost is relatively low.

• Penalizing singular values, including the quantity limit and threshold operation, is a common method for ensuring a low-rank completion matrix; however, existing methods require the rank or regularization parameter to be set. Based on the idea of a singular value penalty, we design a nonparametric nonconvex penalty function that can be used in various applications. Some related studies have used other forms of norms as the regularization term, which treat different singular values differently; our penalty function also achieves this effect.

• The proposed method was applied to grayscale image restoration, recommendation systems, and vote networks. The experimental results verified the robustness and good performance of our method, compared to some state-of-the-art methods.

The remainder of this paper is organized as follows. Section 2 introduces the related work and preliminaries. Section 3 describes the proposed method and discusses the theoretical analysis and refinement. Section 4 presents detailed experimental results. Finally, Section 5 concludes the paper and explores the scope for future work.

Notation: In this paper, vectors and matrices are denoted by lowercase and uppercase bold italic letters, respectively (e.g., vector $\mathit{a}\in {\mathrm{ℝ}}^m$ and matrix $\mathit{A}\in {\mathrm{ℝ}}^{m\times l}$). Scalars are represented by lowercase italic letters (e.g., a). The relevant symbols and formulas include ${\Vert \mathit{A}\Vert }_{\ast }={\sum }_i{\sigma }_i$, which is the nuclear norm of matrix $\mathit{A}$, where ${\sigma }_i$ is the singular value of $\mathit{A}$. The formula $〈\mathit{A},\mathit{B}〉={\sum }_{i_1=1}^m{\sum }_{i_2=1}^l{\mathit{A}}_{i_1{i}_2}{\mathit{B}}_{i_1{i}_2}$ denotes the inner product of two matrices $\mathit{A}$ and $\mathit{B}$ of the same dimensions, where ${\mathit{A}}_{i_1{i}_2}$ and ${\mathit{B}}_{i_1{i}_2}$ are the elements of $\mathit{A}$ and $\mathit{B}$, respectively. In addition, ${\Vert \mathit{A}\Vert }_F=\sqrt{〈\mathit{A}\text{,}\mathit{A}〉}$ is the Frobenius norm of matrix $\mathit{A}$. The set $\mathrm{\Omega }$ denotes the positions of the observed (known) entries in an observation matrix $\mathit{A}$, and ${\mathcal{P}}_{\mathrm{\Omega }}\left(·\right)$ denotes a projection operator (i.e., ${\left({\mathcal{P}}_{\mathrm{\Omega }}\left(\mathit{A}\right)\right)}_{i_1{i}_2}={\mathit{A}}_{i_1{i}_2}$ if the position $(i_1,i_2)$ is included in $\mathrm{\Omega }$ and 0 otherwise). The range outside $\mathrm{\Omega }$ is represented by $\stackrel{-}{\mathrm{\Omega }}$. Additional notation is introduced as it occurs.