Distributed learning for supervised multiview feature selection

Abstract

Multiview feature selection technique is specifically designed to reduce the dimensionality of multiview data and has received much attention. Most proposed multiview supervised feature selection methods suffer from the problem of efficiently handling the large-scale and high-dimensional data. To address this, this paper designs an efficient supervised multiview feature selection method for multiclass problems by combining the distributed optimization method in the Alternating Direction Method of Multipliers (ADMM). Specifically, the distributed strategy is reflected in two aspects. On the one hand, a sample-partition based distributed strategy is adopted, which calculates the loss term of each category individually. On the other hand, a view-partition based distributed strategy is used to explore the consistent and characteristic information of views. We adopt the individual regularization on each view and the common loss term which is obtained by fusing different views to jointly share the label matrix. Benefited from the distributed framework, the model can realize a distributed solution for the transformation matrix and reduce the complexity for multiview feature selection. Extensive experiments have demonstrated that the proposed method achieves a great improvement on training time, and the comparable or better performance compared to several state-of-the-art supervised feature selection algorithms.

Appendix

Theorem 2

Equations (16)-(19) is equivalent to (20)-(23).

Proof

• 1) For fixed i, the solution in (17) is obtained by updating V matrices $\mathbf {Z}_{iv}\in \mathbb {R}^{n_{i}\times C}(v=1,2...V)$ with totally n_iCV variables. Now, we will carry it out by solving the (21) with only n_iC variables.

  Let $\overline {\mathbf {Z}_{i}}=\frac {1}{V}{\sum }_{v=1}^{V}\mathbf {Z}_{iv}$, the (17) can be rewritten as

  $$ \begin{array}{@{}rcl@{}} \underset{\overline{\mathbf{Z}_{i}},\mathbf{Z}_{iv}}\min &&tr\left[\left( V\overline{\mathbf{Z}_{i}}-\mathbf{Y}_{i}\right)^{T}{\mathbf{F}_{i}}^{(k)} \left( V\overline{\mathbf{Z}_{i}}-\mathbf{Y}_{i}\right)\right]\\ &&\qquad+\frac{\rho}{2}\sum\limits_{v=1}^{V} \|\mathbf{Z}_{iv}-\mathbf{X}_{iv}{\mathbf{W}_{v}}^{(k+1)}+\frac{1}{\rho}{\mathbf{P}_{iv}}^{(k)}\|_{F}^{2}\\ s.t. && \overline{\mathbf{Z}_{i}}=\frac{1}{V}\sum\limits_{v=1}^{V}\mathbf{Z}_{iv} \end{array} $$

  (34)

  Minimizing over Z_iv(v = 1, 2,...,V ) with $\overline {\mathbf {Z}_{i}}$ fixed has the solution

  $$ \mathbf{Z}_{iv}-\mathbf{X}_{iv}{\mathbf{W}_{v}}^{(k+1)}+\frac{1}{\rho}{\mathbf{P}_{iv}}^{(k)}=0 $$

  (35)

  Let ${\overline {\mathbf {X}_{i}\mathbf {W}}}=\frac {1}{V}{\sum }_{v=1}^{V} \mathbf {X}_{iv}{\mathbf {W}_{v}}, {\overline {\mathbf {P}_{i}}}=\frac {1}{V}{\sum }_{v=1}^{V} {\mathbf {P}_{iv}}$. Then, we have

  $$ \overline{\mathbf{Z}_{i}}-\overline{\mathbf{X}_{i}\mathbf{W}}^{(k+1)}+\frac{1}{\rho}\overline{\mathbf{P}_{i}}^{(k)}=0 $$

  (36)

  Combining (35) and (36), we have

  $$ \mathbf{Z}_{iv} = \mathbf{X}_{iv}{\mathbf{W}_{v}}^{(k+1)} - \frac{1}{\rho}{\mathbf{P}_{iv}}^{(k)} + \overline{\mathbf{Z}_{i}}-\overline{\mathbf{X}_{i}\mathbf{W}}^{(k+1)}+\frac{1}{\rho}\overline{\mathbf{P}_{i}}^{(k)} $$

  (37)

  Substituting (37) into (34),we get the following equation.

  $$ \begin{array}{@{}rcl@{}} \underset{\overline{\mathbf{Z}_{i}}} \min && tr\left[\left( V\overline{\mathbf{Z}_{i}}-\mathbf{Y}_{i}\right)^{T}{\mathbf{F}_{i}}^{(k)}\left( V\overline{\mathbf{Z}_{i}}-\mathbf{Y}_{i}\right)\right]\\ &&+\frac{\rho V}{2}\|\overline{\mathbf{Z}_{i}}-\overline{\mathbf{X}_{i}\mathbf{W}}^{(k+1)}+\frac{1}{\rho} {\overline{\mathbf{P}_{i}}}^{(k)}\|_{F}^{2} \end{array} $$

  (38)

  Based on the above derivation, we can see that solving (17) is equivalent to solving (21).

• 2) Similarly, for fixed i, the P_iv-update in (19) can be carried out by solving the (23) with only n_iC variables. Replacing Z_iv in (37) with ${\mathbf {Z}_{iv}}^{(k+1)}$ obtains

  $$ \begin{array}{@{}rcl@{}} {\mathbf{Z}_{iv}}^{(k+1)}&=&\mathbf{X}_{iv}{\mathbf{W}_{v}}^{(k+1)}-\frac{1}{\rho}{\mathbf{P}_{iv}}^{(k)}+\overline{\mathbf{Z}_{i}}^{(k+1)}\\ &&-\overline{\mathbf{X}_{i}\mathbf{W}}^{(k+1)} +\frac{1}{\rho}\overline{\mathbf{P}_{i}}^{(k)} \end{array} $$

  (39)

  Substituting (39) into (19) gives

  $$ {\mathbf{P}_{iv}}^{(k+1)}=\rho\left( \overline{\mathbf{Z}_{i}}^{(k+1)}-\overline{\mathbf{X}_{i}\mathbf{W}}^{(k+1)}\right)+\overline{\mathbf{P}_{i}}^{(k)} $$

  (40)

  (40) shows that the variables ${\mathbf {P}_{iv}}^{(k+1)}(v=1,2,...,V)$ are all equal for fixed i. So we can get ${\mathbf {P}_{iv}}^{(k+1)}=\frac {1}{V}{\sum }_{v=1}^{V} {\mathbf {P}_{iv}}^{(k+1)}=\overline {\mathbf {P}_{i}}^{(k+1)}$. Then, we have

  $$ \overline{\mathbf{P}_{i}}^{k+1}=\rho\left( \overline{\mathbf{Z}_{i}}^{(k+1)}-\overline{\mathbf{X}_{i}\mathbf{W}}^{(k+1)}\right)+\overline{\mathbf{P}_{i}}^{(k)} $$

  (41)

  Based on the above derivation, we can see that solving (19) is equivalent to solving (23).

• 3) By replacing Z_iv in (37) with ${\mathbf {Z}_{iv}}^{(k)}$ and substituting ${\mathbf {Z}_{iv}}^{(k)}$ into (16), we have

  $$ \begin{array}{@{}rcl@{}} {\mathbf{W}_{v}}^{(k+1)} &:=& \underset{\mathbf{W}_{v}}\arg \min \left( \lambda\|\mathbf{W}_{v}\|_{2,1} + \frac{\rho}{2}\sum\limits_{i=1}^{C}\|\mathbf{X}_{iv}\mathbf{W}_{v}\right.\\ &&\qquad\qquad \left. -\left( \vphantom{\frac{1}{\rho}}\mathbf{X}_{iv}{\mathbf{W}_{v}}^{(k)}+ \overline{\mathbf{Z}_{i}}^{(k)} - \overline{\mathbf{X}_{i}\mathbf{W}}^{(k)}\right.\right.\\ &&\qquad\qquad\left.\left. +\frac{1}{\rho}\overline{\mathbf{P}_{i}}^{(k)}\right)\|_{F}^{2}\vphantom{\sum\limits_{i=1}^{C}}\right) \end{array} $$

  (42)

  Based on the above derivation, we can see that solving (16) is equivalent to solving (20).

• 4) According to $\overline {\mathbf {Z}_{i}}=\frac {1}{V}{\sum }_{v=1}^{V}\mathbf {Z}_{iv}$, we can see that solving (18) is equivalent to solving (22).

□