A graph optimization method for dimensionality reduction with pairwise constraints

Abstract

Graph is at the heart of many dimensionality reduction (DR) methods. Despite its importance, how to establish a high-quality graph is currently a pursued problem. Recently, a new DR algorithm called graph-optimized locality preserving projections (GoLPP) was proposed to perform graph construction with DR simultaneously in a unified objective function, resulting in an automatically optimized graph rather than pre-specified one as involved in typical LPP. However, GoLPP is unsupervised and can not naturally incorporate supervised information due to a strong sum-to-one constraint of weights of graph in its model. To address this problem, in this paper we give an improved GoLPP model by relaxing the constraint, and then develop a semi-supervised GoLPP (S-GoLPP) algorithm by incorporating pairwise constraint information into its modeling. Interestingly, we obtain a semi-supervised closed-form graph-updating formulation with natural possibility explanation. The feasibility and effectiveness of the proposed method is verified on several publicly available UCI and face data sets.

Appendix: How to solve the weight matrix S = (S ij ) n×n in problem (5)

As seen from Step 2 in Sect. 3.3, given W, computing S = (S _ij ) _n×n in model (4) is equivalent to solving the following optimization problem:

$$\begin{aligned} & \mathop {\hbox{min} }\limits_{{S_{ij} }} \sum\nolimits_{i,j = 1}^{n} {||W^{T} x_{i} - W^{T} x_{j} ||^{2} S_{ij}^{{}} } + \eta \sum\nolimits_{i,j = 1}^{n} {S_{ij} \ln (S_{ij} /\alpha )} \\ & s.t. \, \sum\nolimits_{j = 1}^{n} {S_{ij} } > 0, \quad i = 1, \ldots ,n \\ &S_{ij} = 1, \quad if \, (x_{i} ,x_{j} ) \in M, \quad i,j = 1, \ldots ,n \\ & S_{ij} = 0, \quad if \, (x_{i} ,x_{j} ) \in C, \quad i,j = 1, \ldots ,n \\ & S_{ij} \ge 0, \quad otherwise \\ \end{aligned}$$

(5)

Obviously, from the constraints of (5) we can get that,

$$\begin{aligned} & S_{ij} = 1, \quad if \, (x_{i} ,x_{j} ) \in M, \quad i,j = 1, \ldots ,n; \\ & S_{ij} = 0, \quad if \, (x_{i} ,x_{j} ) \in C, \quad i,j = 1, \ldots ,n. \\ \end{aligned}$$

So, we only solve the weight value S _ij corresponding to the samples without constraints. That is, we just consider the following problem:

$$\begin{aligned} & \mathop {\hbox{min} }\limits_{{S_{ij} }} \sum\nolimits_{i,j = 1}^{n} {||W^{T} x_{i} - W^{T} x_{j} ||^{2} S_{ij}^{{}} + \eta \sum\nolimits_{i,j = 1}^{n} {S_{ij} \ln (S_{ij} /\alpha )} } \\ & s.t. \, \sum\nolimits_{j = 1}^{n} {S_{ij} } > 0, \quad i = 1, \ldots ,n \\ & S_{ij} \ge 0, \quad i,j = 1, \ldots ,n \\ \end{aligned}$$

(6)

where $(x_{i} ,x_{j} ) \notin M\begin{array}{*{20}c} {} \\ \end{array} {\text{and}}\begin{array}{*{20}c} {} \\ \end{array} (x_{i} ,x_{j} ) \notin C$. To optimize S _ij , we establish the lagrangian function as follows:

$$\begin{array}{*{20}c} {L(S_{ij} ,\lambda_{i} ,\mu_{ij} ) = \sum\nolimits_{i,j = 1}^{n} {||W^{T} x_{i} - W^{T} x_{j} ||^{2} S_{ij}^{{}} + \eta \sum\nolimits_{i,j = 1}^{n} {S_{ij} \ln (S_{ij} /\alpha )} } } \\ \end{array} - \sum\nolimits_{i = 1}^{n} {(\lambda_{i} \sum\nolimits_{j = 1}^{n} {S_{ij} ) - } } \sum\nolimits_{i,j = 1}^{n} {(\mu_{ij} S_{ij} )} .$$

By the KKT condition,

$$\lambda_{i} \sum\nolimits_{j = 1}^{n} {S_{ij} = 0,\mu_{ij} S_{ij} = 0} ,$$

so the lagrangian function is simplified as

$$\begin{array}{*{20}c} {L(S_{ij} ) = \sum\nolimits_{i,j = 1}^{n} {||W^{T} x_{i} - W^{T} x_{j} ||^{2} S_{ij}^{{}} + \eta \sum\nolimits_{i,j = 1}^{n} {S_{ij} \ln (S_{ij} /\alpha )} } } \\ \end{array} .$$

Let $\frac{\partial L}{{\partial S_{ij} }} = ||W^{T} x_{i} - W^{T} x_{j} ||^{2} + \eta (\ln (S_{ij} /\alpha ) + 1) = 0$, then we have

$$\begin{array}{*{20}c} {S_{ij} = \alpha \;\exp ( - 1) \cdot \;\exp ( - ||W^{T} x_{i} - W^{T} x_{j} ||^{2} /\eta )} \\ \end{array}$$

(7)

where $(x_{i} ,x_{j} ) \notin M ,\begin{array}{*{20}c} {} \\ \end{array} (x_{i} ,x_{j} ) \notin C,\begin{array}{*{20}c} {} \\ \end{array} \alpha$ is a positive parameter. Intuitively, one expects the weight S _ij approximates 1 when the distance of two samples tends to 0. With this intuition, we set α = e, and obtain

$$\begin{array}{*{20}c} {S_{ij} = \exp ( - ||W^{T} x_{i} - W^{T} x_{j} ||^{2} /\eta )} \\ \end{array} .$$

Lastly, we sum up the solution of problem (5) as follows:

$$\hat{S}_{ij} = \left\{ \begin{array}{ll} 1, & if \, (x_{i} ,x_{j} ) \in M \\ 0, & if \, (x_{i} ,x_{j} ) \in C \\ {\exp \left( - \tfrac{{||W^{T} x_{i} - W^{T} x_{j} ||^{2} }}{\eta }\right)}, & otherwise \\ \end{array} \right. .$$