Constrained feature weighting for semi-supervised learning

Abstract

Semi-supervised feature selection plays a crucial role in semi-supervised classification tasks by identifying the most informative and relevant features while discarding irrelevant or redundant features. Many semi-supervised feature selection approaches take advantage of pairwise constraints. However, these methods either encounter obstacles when attempting to automatically determine the appropriate number of features or cannot make full use of the given pairwise constraints. Thus, we propose a constrained feature weighting (CFW) approach for semi-supervised feature selection. CFW has two goals: maximizing the modified hypothesis margin related to cannot-link constraints and minimizing the must-link preserving regularization related to must-link constraints. The former makes the selected features strongly discriminative, and the latter makes similar samples with selected features more similar in the weighted feature space. In addition, L1-norm regularization is incorporated in the objective function of CFW to automatically determine the number of features. Extensive experiments are conducted on real-world datasets, and experimental results demonstrate the superior effectiveness of CFW compared to that of the existing popular supervised and semi-supervised feature selection methods.

Appendices

Appendices

Appendix A: Proof of Theorem 1

It is well known that a function defined on an open set is convex if and only if its Hessian matrix is positive semi-definite. Therefore, to prove the convexity of the function $J_{R}\left( \textbf{w} \right)$ in (19), we must demonstrate that its Hessian matrix is positive semi-definite.

Because the Laplacian matrix $\textbf{L}^{\mathcal {M}}$ is symmetric and positive semi-definite, $\textbf{Q}=\textbf{F}^T \textbf{L}^{\mathcal {M}} \textbf{F}$ is also symmetric and positive semi-definite. Therefore, we can express the must-link preserving regularization as $J_{R}\left( \textbf{w} \right) =\text {trace} \left( 2\textbf{M}^T \textbf{Q} \textbf{M} \right)$, as shown in (21).

Then, the first partial derivative of $J_{R}\left( \textbf{w} \right)$ with respect to $w_r$ can be calculated as follows:

$$\begin{aligned} \frac{\partial J_{R}\left( \textbf{w} \right) }{\partial w_r} = 4 w_r Q_{rr}, ~~~~r=1,...,d \end{aligned}$$

(A1)

where $Q_{rr}$ is the element in the r-th row and r-th column of $\textbf{Q}$. The second partial derivative of $J_{R}\left( \textbf{w} \right)$ with respect to $w_s$ can be expressed as

$$\begin{aligned} \frac{\partial ^{2} J_{R}\left( \textbf{w}\right) }{\partial w_{r} \partial w_{s}}=\left\{ \begin{array}{ll} 4 Q_{rr}, & \text{ if } r=s \\ 0, & \text{ otherwise } \end{array}\right. \end{aligned}$$

(A2)

Therefore, the Hessian matrix $\textbf{H}$ of $J_{R}\left( \textbf{w}\right)$ is a diagonal matrix, where the diagonal elements are $H_{rr}=4Q_{rr}$.

Since $\textbf{Q}$ is positive semi-definite, we have that $Q_{rr} \ge 0$. As a result, the Hessian matrix $\textbf{H}$ of $J_{R}\left( \textbf{w}\right)$ is positive semi-definite. In other words, $J_{R}\left( \textbf{w} \right)$ is a convex function of $\textbf{w}$ when $\textbf{w} \ge 0$. This concludes the proof.

Appendix B: Proof of Theorem 2

Let $J_{1}\left( \textbf{w}\right) =\log \left( 1+\exp \left( -\textbf{w}^{T} {\textbf{z}}\right) \right)$ and $J_{2}\left( \textbf{w}\right) =\lambda _{1}\Vert \textbf{w}\Vert _{1}$, then the objective function (24) can be rewritten as:

$$\begin{aligned} J\left( \textbf{w}\right) =J_{1}\left( \textbf{w}\right) + \lambda _{1}J_{2}\left( \textbf{w}\right) + \lambda _{2}J_{R}\left( \textbf{w} \right) \end{aligned}$$

(B3)

According to the properties of convex functions, $J\left( \textbf{w}\right)$ is a convex function if and only if $J_{1}\left( \textbf{w}\right)$, $J_{2}\left( \textbf{w}\right)$, and $J_{R}\left( \textbf{w} \right)$ are convex functions. Theorem 1 states that $J_{R}\left( \textbf{w} \right)$ is a convex function. Now, we need to prove that the other two functions are also convex. Following the approach used to prove Theorem 1, we simply need to demonstrate that the Hessian matrices of both $J_{1}\left( \textbf{w}\right)$ and $J_{2}\left( \textbf{w}\right)$ are positive semi-definite.

We start by calculating the first and second partial derivatives of $J_{1}\left( \textbf{w}\right)$ with respect to $\textbf{w}$, as shown below

$$\begin{aligned} \frac{\partial J_{1}\left( \textbf{w}\right) }{\partial \textbf{w}}= -\frac{\exp \left( -\textbf{w}^{T} {\textbf{z}}\right) }{1+\exp \left( -\textbf{w}^{T} {\textbf{z}}\right) } {\textbf{z}} \end{aligned}$$

(B4)

and

$$\begin{aligned} \frac{\partial ^{2} J_{1}\left( \textbf{w}\right) }{\partial ^{2} \textbf{w}}= \frac{\exp \left( -\textbf{w}^{T} {\textbf{z}}\right) }{\left( 1+\exp \left( -\textbf{w}^{T} {\textbf{z}}\right) \right) ^{2}} {\textbf{z}} {\textbf{z}}^{T} \end{aligned}$$

(B5)

Without loss of generality, let $c=\sqrt{\frac{\exp \left( -\textbf{w}^{T} \textbf{z}\right) }{\left( 1+\exp \left( -\textbf{w}^{T} {\textbf{z}}\right) \right) ^{2}}}$. Substituting c into (B5), we have

$$\begin{aligned} \frac{\partial ^{2} J_{1}\left( \textbf{w}\right) }{\partial ^{2} \textbf{w}} =\left( c {\textbf{z}}\right) \left( c {\textbf{z}}\right) ^{T} =\textbf{H}_1 \end{aligned}$$

(B6)

where $\textbf{H}_1$ is the Hessian matrix of $J_{1}(\textbf{w})$. Because $\textbf{H}_1$ can be regarded as the outer product of a column vector $c{\textbf{z}}$ and its own transpose vector $\left( c {\textbf{z}}\right) ^{T}$. So $\textbf{H}_1$ is a matrix of rank 1 with only one non-zero eigenvalue. It can be calculated that the non-zero eigenvalue of matrix $\textbf{H}_1$ is $c^2\left\| \textbf{z}\right\| ^2$, which is greater than 0. In this case, the Hessian matrix of (B6) is positive semi-definite. Therefore, $J_{1}\left( \textbf{w}\right)$ is a convex function.

As for $J_{2}\left( \textbf{w}\right)$, we have

$$\begin{aligned} \frac{\partial J_{2}\left( \textbf{w} \right) }{\partial w_r} = 1, ~~~~ r=1,...,d \end{aligned}$$

(B7)

and

$$\begin{aligned} \frac{\partial ^{2} J_{2}\left( \textbf{w}\right) }{\partial w_{r} \partial w_{s}}=0, ~~~~ r,s=1,...,d. \end{aligned}$$

(B8)

Thus, the Hessian matrix of $J_{2}\left( \textbf{w} \right)$ is a matrix with all zeros, which means that the Hessian matrix of $J_{2}\left( \textbf{w} \right)$ is positive semi-definite. Hence, $J_{2}\left( \textbf{w} \right)$ is a also convex function.

In summary, $J_{1}\left( \textbf{w} \right)$, $J_{2}\left( \textbf{w} \right)$ and $J_{R}\left( \textbf{w} \right)$ are convex functions. Thus, $J\left( \textbf{w} \right)$ is a convex function. This completes the proof.

Appendix C: Proof of Theorem 3

Note that $\textbf{z}(t)$ is updated by $\textbf{w}(t-1)$ in the t-th iteration. When $\textbf{z}(t)$ and $\textbf{w}(t-1)$ are given, $\textbf{w}(t)$ is updated using the gradient descent scheme in (27) and the truncation rule in (28). Consequently, the objective function achieves its minimum $J(\textbf{w}(t) \mid \textbf{z}(t))$ for fixed $\textbf{z}(t)$. In other words,

$$\begin{aligned} J(\textbf{w}(t) \mid \textbf{z}(t)) \le J(\textbf{w}(t-1) \mid \textbf{z}(t)) \end{aligned}$$

(C9)

which completes the proof of Theorem 3.