Robust semi-supervised data representation and imputation by correntropy based constraint nonnegative matrix factorization

Abstract

Many methods have been proposed recently for high-dimensional data representation to reduce the dimensionality of the data. Matrix Factorization (MF) as an efficient dimension-reduction method is increasingly used in a wide range of applications. However, these methods are often unable to handle data with missing entries. In a Semi-Supervised Learning (SSL) scenario, many commonly used missing value imputation methods, e.g., KNN imputation, cannot utilize the existing information on the labels, which is one of the most discriminative information in the data. Considering the outliers in the observed entries, in this paper, we propose an algorithm called Correntropy based Constraint Nonnegative Matrix Factorization Completion (CCNMF) for simultaneous construction of robust representation and imputation of high-dimensional data in an SSL scenario. Specifically, the Maximum Correntropy Criterion (MCC) is used to construct the model of the CCNMF method to alleviate the negative effects of non-Gaussian noise and outliers in the data. To solve the optimization problem, an iterative algorithm based on a Fenchel Conjugate (FC) and Block Coordinate Update (BCU) framework is proposed. We show that the proposed algorithm can satisfy not only objective sequential convergence but also iterate sequence convergence. The experiments are conducted on the real-world image dataset and community health dataset. In many cases, it is shown that the proposed method outperforms several state-of-the-art methods for both representation and imputation.

Appendices

Appendix: A. Proof Of Theorem 1

Proof

The problem in (19) can be decomposed into n × d independent problems, each involving one element in B and coming in the form

$$ \min\limits_{{\mathcal{P}}_{\Omega}({\mathbf{B}})={\mathcal{P}}_{\Omega}({\mathbf{A}})} \frac{1}{2}{\mathbf{C}}^{2}_{i,j}({\mathbf{B}}_{i,j}-{\mathbf{D}}_{i,j})^{2},\quad \forall i,j. $$

(A.1)

If i,j ∈Ω, due to the constraint ${\mathcal {P}}_{\Omega }({\mathbf {B}})={\mathcal {P}}_{\Omega }({\mathbf {A}})$, ${\mathbf {B}}^{*}_{i,j}$ needs to equal A_i,j. If i,j∉Ω, the problem (A.1) can be written as

$$ \min\limits_{{\mathbf{B}}_{i,j}} \frac{1}{2}{\mathbf{C}}^{2}_{i,j}({\mathbf{B}}_{i,j}-{\mathbf{D}}_{i,j})^{2},\quad \forall i,j, $$

(A.2)

which is a conventional quadratic function, with optimal solution as ${\mathbf {B}}^{*}_{i,j}=D_{i,j}$. Therefore, (20) is optimal solution of (19). □

Appendix: B. The Equivalence of (12c) and (15)

Let ${\mathbf {C}} = \nabla _{{\mathbf {X}}} f(\hat {{\mathbf {X}}}^{t},{\mathbf {Z}}^{t},{\mathbf {P}}^{t+1},{\mathbf {B}}^{t})$ and $L = L_{{\mathbf {X}}}^{t}$, the objective of (12c) can be written as

$$ \begin{array}{@{}rcl@{}} &&\text{Tr}[({\mathbf{X}}-\hat{\mathbf{X}}^{t})]-{\frac{L}{2}}\text{Tr}[({\mathbf{X}}-\hat{\mathbf{X}}^{t})]\\ &=&\text{Tr}[\mathbf{X}^{\top}\mathbf{C}-(\hat{\mathbf{X}}^{t})^{\top}]\\&&-{\frac{L}{2}}\text{Tr}[\mathbf{X}^{\top}\mathbf{X}-2{\mathbf{X}}^{\top}\hat{\mathbf{X}}^{t}+(\hat{\mathbf{X}}^{t})^{\top}\hat{\mathbf{X}}^{t}] \end{array} $$

(B.1)

Eliminating the terms which are independent of X in (B.1), (12c) can be reformulated as follows:

$$ \begin{array}{l} \quad\underset{{\mathbf{X}}\geq 0}{\arg\max} \text{Tr}\left( {\mathbf{X}}^{\top}{\mathbf{C}}-{\frac{L}{2}}{\mathbf{X}}^{\top}{\mathbf{X}} + L {\mathbf{X}}^{\top}\hat{\mathbf{X}}^{t}\right)\\ \Leftrightarrow\underset{{\mathbf{X}}\geq 0}{\arg}{\max} -\frac{L}{2} \text{Tr}\left( {\mathbf{X}}^{\top} {\mathbf{X}}-\frac{2}{L}{\mathbf{X}}^{\top} {\mathbf{C}}+2{\mathbf{X}}^{\top} \hat{{\mathbf{X}}}^{t}\right)\\ \Leftrightarrow\underset{{\mathbf{X}}\geq 0}{\arg}{\min} \text{Tr}\left( {\mathbf{X}}^{\top} {\mathbf{X}}-\frac{2}{L}{\mathbf{X}}^{\top} {\mathbf{C}}+2{\mathbf{X}}^{\top} \hat{\mathbf{X}}^{t}\right)\\ \Leftrightarrow\underset{{\mathbf{X}}\geq 0}{\arg}{\min} \text{Tr}\left( {\mathbf{X}}^{\top} {\mathbf{X}}-2{\mathbf{X}}^{\top} (\hat{{\mathbf{X}}}^{t}+\frac{1}{L}{\mathbf{C}})\right). \end{array} $$

Finally, (12c) can be reformulated as:

$$ \underset{{\mathbf{X}}\geq 0}{\arg}{\min} \frac{1}{2}\|\mathbf{X}-\left( \hat{\mathbf{X}}^{t}+\frac{1}{L}{\mathbf{C}}\right)\|_{F}^{2}, $$

which is the formulation of (15).