Max–Min Robust Principal Component Analysis

Abstract

Principal Component Analysis (PCA) is a powerful unsupervised dimensionality reduction algorithm, which uses squared ${\ell }_2$-norm to cleverly connect reconstruction error and projection variance, and those improved PCA methods only consider one of them, which limits their performance. To alleviate this problem, we propose a novel Max–Min Robust Principal Component Analysis via binary weight, which ingeniously combines reconstruction error and projection variance to learn projection matrix more accurately, and uses ${\ell }_2$-norm as evaluation criterion to make the model rotation invariant. In addition, we design binary weight to remove outliers to improve the robustness of model and obtain the ability of anomaly detection. Subsequently, we exploit an efficient iterative optimization algorithm to solve this problem. Extensive experimental results show that our model outperforms related state-of-the-art PCA methods.

Introduction

Principal Component Analysis [1], [2] (PCA) is a very popular unsupervised dimensionality reduction method [3], [4], which is often used in image denoising [5], [6], [7], [8], image compression [9], [10], [11], subspace learning [12], [13], etc. In recent years, with the development of science and technology, it has also been widely used in biology and chemistry, such as cancelable biometrics [14], biometric cryptosystems [15] and chemometrics [16], etc. Therefore, it has an important place in many fields.

PCA learns projection matrix using maximum projection variance or minimum reconstruction error as cost function [17], [18]. To be specific, suppose the data matrix $X=[{\mathit{x}}_1,{\mathit{x}}_2,{\mathit{x}}_3,\dots ,{\mathit{x}}_n]\in {\mathbb{R}}^{d\times n}$, where d and n represent the dimensionality and number of data, respectively. Without loss of generality, the data have been centralized, $i.e$., ${\sum }_{i=1}^n{\mathit{x}}_{\mathit{i}}=0$. Let $W=[{\mathit{w}}_1,{\mathit{w}}_2,{\mathit{w}}_3,\dots ,{\mathit{w}}_k]\in {\mathbb{R}}^{d\times k}$ be a projection matrix, and then the traditional PCA focuses on solving the following optimization problem:

$$\begin{array}{cc}& \underset{W}{\min }\sum _{i=1}^n{||W^{\top }{\mathit{x}}_{\mathit{i}}||}_2^2\Rightarrow \begin{array}{c}\underset{W}{\max }\end{array}\mathrm{Tr}\left(W^{\top }{\mathit{XX}}^{\top }W\right)\\ & \Rightarrow \begin{array}{c}\underset{W}{\min }\end{array}\sum _{i=1}^n{||{\mathit{x}}_{\mathit{i}}-W{W}^{\top }{\mathit{x}}_{\mathit{i}}||}_2^2,s.t.W^{\top }W=I\text{.}\end{array}$$

Since the above objective functions are based on squared ${\ell }_2$-norm, they are equivalent, but because of this, PCA is very sensitive to outliers [19]. Many recent research efforts have been proposed to alleviate this drawback.

In [20], Wright et al. proposed L1PCA, which uses the ${\ell }_1$-norm to minimize the reconstruction error and try to solve the following problem:

$$\underset{W}{\min }\sum _{i=1}^n{||{\mathit{x}}_{\mathit{i}}-W{W}^{\top }{\mathit{x}}_{\mathit{i}}||}_1,s.t.W^{\top }W=I\text{.}$$

In [21], Kwak et al. proposed PCAL1 based on the maximum projected variance, which focuses on solving the following problem:

$${\displaystyle \underset{W}{\min }}{\displaystyle \sum _{i=1}^n}{‖W^{\top }{\mathit{x}}_{\mathit{i}}‖}_1,s.t.W^{\top }W=I\text{.}$$

In [22], Ding et al. proposed R1PCA by applying ${\ell }_1$-norm and ${\ell }_2$-norm to the data and spatial dimensions, respectively, which attempts to optimize the following problem:

$$\underset{W}{\min }\sum _{i=1}^n{||{\mathit{x}}_{\mathit{i}}-W{W}^{\top }{\mathit{x}}_{\mathit{i}}||}_{2,1},s.t.W^{\top }W=I\text{.}$$

In [23], Wang et al. proposed ${\ell }_{2,p}$-PCA using the ${\ell }_2$-norm, which correlates PCA and R1PCA by setting different p values, and then this problem is optimized by the following problem:

$$\underset{W}{\min }\sum _{i=1}^n{||{\mathit{x}}_{\mathit{i}}-W{W}^{\top }{\mathit{x}}_{\mathit{i}}||}_2^p,s.t.W^{\top }W=I\text{.}$$

Similar studies also include RPCA-OM [24], TRPCA [25], KPCA [26] and more [27], [28], [29], [30], [31]. Through the above analysis, although these existing PCA works mitigate the effects of outliers to some extent, they usually focus on the minimum reconstruction error or maximum projection variance without considering the connection between the two. In fact, the traditional PCA has this property, that is, ${\sum }_{i=1}^n{||{\mathit{x}}_{\mathit{i}}-W{W}^{\top }{\mathit{x}}_{\mathit{i}}||}_2^2+{\sum }_{i=1}^n{||W^{\top }{\mathit{x}}_{\mathit{i}}||}_2^2={\sum }_{i=1}^n{||{\mathit{x}}_{\mathit{i}}||}_2^2$, which means that the projection variance and reconstruction error of data are considered at the same time, while those improved methods do not have the above equivalence relationship, resulting in limited model performance.

In this paper, we propose a novel Max–Min Robust Principal Component Analysis via binary weight (MMRPCA), which simultaneously considers reconstruction error and projection variance, so that this model not only obtains strong reconstruction ability, but also makes the data after dimensionality reduction more separable. Furthermore, to improve the robustness of model, we carefully design binary weight to make this model treat normal samples and outliers differently, and eliminate the negative effects of outliers to obtain a more accurate projection matrix. Interestingly, we also find that the ability of anomaly detection can be achieved by binary weight, which is not considered by existing PCA methods. To solve this model, we explore an efficient iterative optimization algorithm and strictly guarantee its convergence. Extensive experimental results show that our proposed method outperforms other methods.

Notations: In this paper, we use uppercase letters, bold lowercase letters and lowercase letters represent matrices, vectors and scalars, respectively. For matrix $Q,{\mathit{q}}_{\mathit{i}}$ represents the i-th column, $Q^{\top }$ is the transpose of Q, $\mathrm{Tr}\left(Q\right)$ is the trace operator of Q, and I denotes identity matrix. For vector $\mathit{r}$, where $\Vert \mathit{r}{\Vert }_{\mathit{1}}$ and $\Vert \mathit{r}{\Vert }_{\mathit{2}}$ are the ${\ell }_1$-norm and ${\ell }_2$-norm of vector $\mathit{r}$, respectively.