Sparse optimal discriminant clustering

Abstract

In this manuscript, we reinvestigate an existing clustering procedure, optimal discriminant clustering (ODC; Zhang and Dai in Adv Neural Inf Process Syst 23(12):2241–2249, 2009), and propose to use cross-validation to select the tuning parameter. Furthermore, because in high-dimensional data many of the features may be non-informative for clustering, we develop a variation of ODC, sparse optimal discriminant clustering (SODC), by adding a group-lasso type of penalty to ODC. We also demonstrate that both ODC and SDOC can be used as a dimension reduction tool for data visualization in cluster analysis.

Appendix

Given $\mathbf Y$, the sub-gradient equations for (3) of $W$ is

$$\begin{aligned}&- 2\widetilde{\mathbf{X}}_j^{'} \left( \mathbf{Y}- \sum _{l=1}^p \widetilde{\mathbf{X}}_l w_l\right) + 2\lambda _2 w_{j}\\&\quad + \lambda _1\frac{w_{j}}{\Vert w_{j}\Vert _2} = 0, \quad j=1, \ldots , p. \end{aligned}$$

Let the minimizer of (3) be $\widehat{W}=\left( \widehat{w}_1, \ldots , \widehat{w}_p\right) '$. If

$$\begin{aligned} \left\| \widetilde{\mathbf{X}}_j^{'}\left( \mathbf{Y}- \sum _{l\ne j} \widetilde{\mathbf{X}}_l\widehat{w}_l\right) \right\| _2< \left\| \frac{\lambda _1}{2}\right\| , \end{aligned}$$

then $\widehat{w}_j=0$. (Therefore, $\widehat{w}_j=0$ for any $j$ if $\lambda >\lambda ^{\max }_1=\max _{j}\Vert \widetilde{\mathbf{X}}_j^{'}\mathbf{Y}\Vert$.) Otherwise,

$$\begin{aligned} \widehat{w}_j= \left( \widetilde{\mathbf{X}}_j^{'} \widetilde{\mathbf{X}}_j+\lambda _2+\frac{\lambda _1 }{2\Vert w_{j} \Vert _2}\right) ^{-1}V_j, \end{aligned}$$

where $V_j=\widetilde{\mathbf{X}}_j^{'}(\mathbf{Y}- \sum _{l\ne j} \widetilde{\mathbf{X}}_l\widehat{w}_l)$. Note that $\widetilde{\mathbf{X}}_j^{'} \widetilde{\mathbf{X}}_j$ is actually a diagonal matrix where diagonal terms are sample variances of features. If we conduct standardization on the design matrix at the beginning, we have $\widetilde{\mathbf{X}}_j^{'} \widetilde{\mathbf{X}}_j=I_{k-1}$ then the above equation becomes

$$\begin{aligned} \widehat{w}_j = \left( \frac{2\Vert \widehat{w}_{j}\Vert _{2}}{\lambda _1+ 2(1+\lambda _2)\Vert \widehat{w}_{j}\Vert _{2}}\right) V_{j}. \end{aligned}$$

The Euclidean norm is $\Vert \widehat{w}_{j}\Vert _{2}=\frac{2\Vert V_{j}\Vert _2-\lambda _1}{2(1+\lambda _2)}$. Plugging this norm to the above formula of $\widehat{w}_j$, we get the formula of $\widehat{w}_j$ stated in the theorem.