DC formulations and algorithms for sparse optimization problems

Abstract

We propose a DC (Difference of two Convex functions) formulation approach for sparse optimization problems having a cardinality or rank constraint. With the largest-k norm, an exact DC representation of the cardinality constraint is provided. We then transform the cardinality-constrained problem into a penalty function form and derive exact penalty parameter values for some optimization problems, especially for quadratic minimization problems which often appear in practice. A DC Algorithm (DCA) is presented, where the dual step at each iteration can be efficiently carried out due to the accessible subgradient of the largest-k norm. Furthermore, we can solve each DCA subproblem in linear time via a soft thresholding operation if there are no additional constraints. The framework is extended to the rank-constrained problem as well as the cardinality- and the rank-minimization problems. Numerical experiments demonstrate the efficiency of the proposed DCA in comparison with existing methods which have other penalty terms.

A Proofs for Derivation of Exact Penalty Parameter Values

1.1 A.1  Proof of Theorem 3

To prove the claim of Theorem 3, we use the following lemma:

Lemma 1

If $f:\mathbb R^{n}\rightarrow \mathbb R$ is L-smooth, then for any $\varvec{x},\varvec{y}\in \mathbb R^{n}$,

$$\begin{aligned} f(\varvec{y})\le f(\varvec{x})+\nabla f(\varvec{x})^{\top }(\varvec{y}-\varvec{x})+\frac{L}{2}\Vert \varvec{y}-\varvec{x}\Vert _{2}^{2}. \end{aligned}$$

For simplicity of notation, we use $\varvec{w}^{\star }$ instead of $\varvec{w}_\rho ^{\star }$ for an optimal solution of (6) with some $\rho$. Assume by contradiction that $\Vert \varvec{w}^{\star }\Vert _{0}>K$, which implies $|w_{(K+1)}^{\star }| > 0$. We consider ${\tilde{\varvec{w}}}:=\varvec{w}^{\star }-w_{i}^{\star }\varvec{e}_{i}$, where i is the index of the $(K+1)$-st largest element in absolute value and $\varvec{e}_{i}$ denotes the unit vector in the i-th coordinate direction. Note that ${\tilde{\varvec{w}}}$ is a feasible solution to the given problem. Then from Lemma 1 and $\Vert \varvec{w}^{\star }\Vert _{2}\le C$, we have

$$\begin{aligned}&f(\varvec{w}^{\star })+\rho (\Vert \varvec{w}^{\star }\Vert _{1}-|||\varvec{w}^{\star }|||_{K})-f({\tilde{\varvec{w}}})-\rho (\Vert {\tilde{\varvec{w}}}\Vert _{1}-|||{\tilde{\varvec{w}}}|||_{K})\\&\quad =f(\varvec{w}^{\star })-f(\varvec{w}^{\star }-w_{i}^{\star }\varvec{e}_{i})+\rho | w_{i}^{\star }|\\&\quad \ge \nabla f(\varvec{w}^{\star })^{\top }w_{i}^{\star }\varvec{e}_{i}-\frac{L}{2}w_{i}^{\star 2} +\rho | w_{i}^{\star }|\\&\quad \ge |w_{i}^{\star }|\left( \rho -\Vert \nabla f(\varvec{w}^{\star })\Vert _{2}-\frac{1}{2}LC\right) . \end{aligned}$$

By using the Lipschitz continuity of $\nabla f$ again, we have

$$\begin{aligned} \Vert \nabla f(\varvec{w}^{\star })\Vert _{2}&\le \Vert \nabla f(\varvec{0})\Vert _{2}+\Vert \nabla f(\varvec{w}^{\star })-\nabla f(\varvec{0})\Vert _{2}\le \Vert \nabla f(\varvec{0})\Vert _{2}+LC. \end{aligned}$$

Thus we have

$$\begin{aligned}&f(\varvec{w}^{\star })+\rho (\Vert \varvec{w}^{\star }\Vert _{1}-|||\varvec{w}^{\star }|||_{K})-{f({\tilde{\varvec{w}}})}-\rho (\Vert {\tilde{\varvec{w}}}\Vert _{1}-|||{\tilde{\varvec{w}}}|||_{K})\\&\quad \ge |w_{i}^{\star }|\left( \rho -\Vert \nabla f(\varvec{0})\Vert _{2}-\frac{3}{2}LC\right) >0, \end{aligned}$$

which contradicts the optimality of $\varvec{w}^\star$. $\square$

1.2 A.2  Proof of Corollary 3

Let $\varvec{w}^\star$ be an optimal solution of (6) and assume by contradiction that $\Vert \varvec{w}^{\star }\Vert _{0}>K$. We consider a feasible solution ${\tilde{\varvec{w}}}:=\varvec{w}^{\star }-w_{i}^{\star }(\varvec{e}_{i} - \varvec{e}_j)$, where i and j are the indices of the $(K+1)$-st and K-th largest elements in absolute value and $\varvec{e}_{i}$ and $\varvec{e}_j$ denote the unit vectors in the corresponding coordinate directions. Then from Lemma 1, we have

$$\begin{aligned}&f(\varvec{w}^{\star })+\rho (\Vert \varvec{w}^{\star }\Vert _{1}-|||\varvec{w}^{\star }|||_{K})-f({\tilde{\varvec{x}}})-\rho (\Vert {\tilde{\varvec{w}}}\Vert _{1}-|||{\tilde{\varvec{w}}}|||_{K})\\&\quad =f(\varvec{w}^{\star })-f(\varvec{w}^{\star }-w_{i}^{\star }(\varvec{e}_{i}-\varvec{e}_j))+\rho | w_{i}^{\star }|\\&\quad \ge \nabla f(\varvec{w}^{\star })^{\top }w_{i}^{\star }(\varvec{e}_{i}-\varvec{e}_j)-\frac{L}{2}w_{i}^{\star 2}\Vert \varvec{e}_i-\varvec{e}_j\Vert _2^2 +\rho | w_{i}^{\star }|\\&\quad \ge |w_{i}^{\star }|\left( \rho -\Vert \nabla f(\varvec{w}^{\star })\Vert _{2}\Vert \varvec{e}_i-\varvec{e}_j\Vert _2-L|w_i^\star |\right) \\&\quad \ge |w_{i}^{\star }|\left( \rho -\sqrt{2}\Vert \nabla f(\varvec{w}^{\star })\Vert _{2}-L\right) . \end{aligned}$$

Then in the same way as in the proof of Theorem 3, we have

$$\begin{aligned} f(\varvec{w}^{\star })+\rho (\Vert \varvec{w}^{\star }\Vert _{1}-|||\varvec{w}^{\star }|||_{K})-{f({\tilde{\varvec{w}}})}-\rho (\Vert {\tilde{\varvec{w}}}\Vert _{1}-|||{\tilde{\varvec{w}}}|||_{K})>0, \end{aligned}$$

which contradicts the optimality of $\varvec{w}^\star$. $\square$

1.3 A.3  Proof of Theorem 4

For simplicity of notation, we use $\varvec{w}^{\star }$ instead of $\varvec{w}_\rho ^{\star }$ for an optimal solution of (6) with some $\rho$. Assume that $\Vert \varvec{w}^{\star }\Vert _{0}>K$ and construct a feasible solution $\tilde{\varvec{w}}:=\varvec{w}^{\star }-w^{\star }_{j}\varvec{e}_{j}$, where j is the index of the $(K+1)$-st largest element of $(|w^{\star }_{1}|,\ldots ,|w^{\star }_{n}|)^{\top }$. Then, from the Cauchy–Schwarz inequality, we have

$$\begin{aligned}&\frac{1}{2}(\varvec{w}^{\star })^{\top }\varvec{Q}\varvec{w}^{\star }+\varvec{q}^{\top }\varvec{w}^{\star }+\rho (\Vert \varvec{w}^{\star }\Vert _{1}-|||\varvec{w}^{\star }|||_{K}) \\&\qquad -\frac{1}{2}\varvec{\tilde{w}}^{\top }\varvec{Q}\varvec{\tilde{w}}-\varvec{q}^{\top }\varvec{\tilde{w}}+\rho (\Vert {\tilde{\varvec{w}}}\Vert _{1}-|||{\tilde{\varvec{w}}}|||_{K})\\&\quad = {w}^{\star }_{j}\varvec{e}_{j}^{\top }\varvec{Q}\varvec{w}^{\star }-\frac{1}{2}{w^{\star }_{j}}^{2}Q_{jj}+w^{\star }_{j}q_{j}+\rho |w^{\star }_{j}|\\&\quad \ge -|{w}^{\star }_{j}|\Vert \varvec{Q}\varvec{e}_{j}\Vert _{2}\Vert \varvec{w}^{\star }\Vert _{2}-\frac{1}{2}|w^{\star }_{j}|\Vert \varvec{w}^{\star }\Vert _{2}|Q_{jj}| \\&\qquad -|q_{j}||w^{\star }_{j}|+\rho |w^{\star }_{j}| ~~ \left( \because |w^{\star }_j|\le \Vert \varvec{w}^{\star }\Vert _2\right) \\&\quad \ge |{w}^{\star }_{j}|\left( \rho -C\Vert \varvec{Q}\varvec{e}_{j}\Vert _{2}-C|Q_{jj}|/2-|q_{j}|\right) >0, \end{aligned}$$

which contradicts the optimality of $\varvec{w}^{\star }$. $\square$

1.4 A.4  Proof of Corollary 4

Let $\varvec{w}^{\star }$ be an optimal solution of (6) and assume by contradiction that $\Vert \varvec{w}^{\star }\Vert _{2}> 2 \Vert \varvec{q}\Vert _{2}/\lambda _{\min }(\varvec{Q})$. We have

$$\begin{aligned} f(\varvec{w}^{\star })= & {} \frac{1}{2}(\varvec{w}^{\star })^{\top }\varvec{Q}\varvec{w}^{\star }+\varvec{q}^{\top }\varvec{w}^{\star }+\rho (\Vert \varvec{w}^{\star }\Vert _{1}-|||\varvec{w}^{\star }|||_{K}) \\\ge & {} \frac{1}{2}\lambda _{\min }(\varvec{Q})\Vert \varvec{w}^{\star }\Vert _{2}^{2}-\Vert \varvec{q}\Vert _{2}\Vert \varvec{w}^{\star }\Vert _{2}>0. \end{aligned}$$

On the other hand, since $f(\varvec{0})=0$, the optimal value of (6) must be nonpositive. Hence it suffices to consider the case $\Vert \varvec{w}^{\star }\Vert _{2}\le 2 \Vert \varvec{q}\Vert _{2}/\lambda _{\min }(\varvec{Q})$. Applying $C=2 \Vert \varvec{q}\Vert _{2}/{\lambda }_{\min }(\varvec{Q})$ to Theorem 4, we have the desired result. $\square$

1.5 A.5  Proof of Theorem 5

For simplicity of notation, we use $\varvec{W}^{\star }$ instead of $\varvec{W}_\rho ^{\star }$. Assume by contradiction that $\mathrm {rank}(\varvec{W}^{\star })>K$ and construct a feasible solution $\varvec{\tilde{W}}:=\varvec{W}^{\star }-\varvec{W}^{\star }\varvec{y}_{K+1}\varvec{y}_{K+1}^{\top }$, where $\varvec{y}_{K+1}$ is the $(K+1)$-st leading eigenvector of $\varvec{W}^{\star \top }\varvec{W}^{\star }$ with $\Vert \varvec{y}_{K+1}\Vert _{2}=1$. Then from the quadratic upper bound from the Lipschitz continuity, we have

$$\begin{aligned}&f(\varvec{W}^{\star })+\rho (\Vert \varvec{W}^{\star }\Vert _*-|||\varvec{W}^{\star }|||_{K})-f(\varvec{\tilde{W}})-\rho (\Vert \varvec{\tilde{W}}\Vert _*-|||\varvec{\tilde{W}}|||_{K})\\&\quad =f(\varvec{W}^{\star })-f(\varvec{W}^{\star }-\varvec{W}^{\star }\varvec{y}_{K+1}\varvec{y}_{K+1}^{\top })+\rho \sigma _{K+1}\\&\quad \ge \nabla f(\varvec{W}^{\star })\bullet (\varvec{W}^{\star }\varvec{y}_{K+1}\varvec{y}_{K+1}^{\top })-\frac{L}{2}\Vert \varvec{W}^{\star }\varvec{y}_{K+1}\varvec{y}_{K+1}^{\top }\Vert _{\mathrm {F}}^{2}+\rho \sigma _{K+1}\\&\quad \ge \rho \sigma _{K+1}-\Vert \nabla f(\varvec{W}^{\star })\Vert _{\mathrm {F}}\Vert \varvec{W}^{\star }\varvec{y}_{K+1}\varvec{y}_{K+1}^{\top }\Vert _{\mathrm {F}}-\frac{L}{2}\sigma _{K+1}^{2}\\&\quad \ge \rho \sigma _{K+1}-\sigma _{K+1}(\Vert \nabla f(\varvec{O})\Vert _{\mathrm {F}}+LC)-\frac{LC}{2}\sigma _{K+1}\\&\quad \ge \sigma _{K+1}\left( \rho -\Vert \nabla f(\varvec{O})\Vert _{\mathrm {F}}-\frac{3}{2}LC\right) >0, \end{aligned}$$

which contradicts the optimality of $\varvec{W}^\star$. $\square$

1.6 A.6  Exact penalty parameter for the MIP-based formulation

Theorem 6

Let $(\varvec{w}^\star ,\varvec{u}^\star )$ be an arbitrary optimal solution of (27). Then $\varvec{w}^\star$ is optimal to the corresponding cardinality-constrained problem if

$$\begin{aligned} \rho > \max _{i}\left\{ \frac{8\Vert \varvec{q}\Vert _{2}}{\lambda _{\min }(\varvec{Q})}\left( |q_{i}|+\frac{2\Vert \varvec{Q}\varvec{e}_{i}\Vert _{2}\Vert \varvec{q}\Vert _{2}+\Vert \varvec{q}\Vert _{2}Q_{ii}}{\lambda _{\min }(\varvec{Q})}\right) \right\} . \end{aligned}$$

Proof

In the same manner as in the proofs of Corollary 4, $\varvec{w}^{\star }$ is bounded by $\Vert \varvec{w}^{\star }\Vert _{2}\le 2\Vert \varvec{q}\Vert _{2}/\lambda _{\min }(\varvec{Q})$. Assume by contradiction that $\varvec{1}^{\top }\varvec{u}^{\star }-(\varvec{u}^{\star })^{\top }\varvec{u}^{\star }>0$.

• If there exists j such that $0<u^{\star }_{j}\le 1/2$, by constructing a feasible solution $(\tilde{\varvec{w}},\tilde{\varvec{u}})=(\varvec{w}^{\star }-w^{\star }_{j}\varvec{e}_{j},\varvec{u}^{\star }-u^{\star }_{j}\varvec{e}_{j})$, we have

  $$\begin{aligned}&\frac{1}{2}(\varvec{w}^{\star })^{\top }\varvec{Q}\varvec{w}^{\star }+\varvec{q}^{\top }\varvec{w}^{\star }-\rho (\varvec{u}^{\star })^{\top }\varvec{u}^\star -\frac{1}{2}\varvec{\tilde{w}}^{\top }\varvec{Q}\varvec{\tilde{w}}-\varvec{q}^{\top }\varvec{\tilde{w}}+\rho \varvec{\tilde{u}}^{\top }\varvec{\tilde{u}}\\&\quad =w^{\star }_{j}\varvec{e}_{j}^{\top }\varvec{Q}\varvec{w}^{\star }-\frac{1}{2}{w^{\star }_{j}}^{2}Q_{jj}+w^{\star }_{j}q_{j}+\rho (u^{\star }_{j}-(u^{\star }_{j})^{2})\\&\quad \ge w^{\star }_{j}\varvec{e}_{j}^{\top }\varvec{Q}\varvec{w}^{\star }-\frac{1}{2}{w^{\star }_{j}}^{2}Q_{jj}+w^{\star }_{j}q_{j}+\frac{1}{2}\rho u^{\star }_{j}\quad (\because x-x^{2}> x/2 \ \text{ for } \ 0<x<1/2)\\&\quad \ge \frac{1}{2}\rho u^{\star }_{j}-|w^{\star }_{j}|\Vert \varvec{Q}\varvec{e}_{j}\Vert _{2}\Vert \varvec{w}^{\star } \Vert _{2}-\frac{1}{2}|w^{\star }_{j}|\Vert \varvec{w}^{\star }\Vert _{2}Q_{jj}-|w_{j}||q_{j}|\\&\quad \ge \frac{\rho }{2M}|w^{\star }_{j}|-\frac{2|w^{\star }_{j}|\Vert \varvec{Q}\varvec{e}_{j}\Vert _{2}\Vert \varvec{q}\Vert _{2}}{\lambda _{\min }(\varvec{Q})}-\frac{|w^{\star }_{j}|\Vert \varvec{q}\Vert _{2}Q_{jj}}{\lambda _{\min }(\varvec{Q})}-|w_{j}||q_{j}|\\&\quad \ge \frac{|w^{\star }_{j}|}{2M}\left[ \rho -\frac{8\Vert \varvec{q}\Vert _{2}}{\lambda _{\min }(\varvec{Q})}\left( |q_{j}|+\frac{2\Vert \varvec{Q}\varvec{e}_{j}\Vert _{2}\Vert \varvec{q}\Vert _{2}+\Vert \varvec{q}\Vert _{2}Q_{jj}}{\lambda _{\min }(\varvec{Q})}\right) \right] >0. \end{aligned}$$

• Otherwise, if $u^{\star }_{j}\notin \{0,1\}$, then $1/2< u^{\star }_{j}<1$.

  • Case: $\varvec{1}^{\top }\varvec{u}^{\star }<K$. We can take $\varepsilon >0$ such that $\varvec{1}^{\top }\varvec{u}^{\star }+\varepsilon \le K$ and $u^{\star }_{j}+\varepsilon \le 1$. Then by putting $(\tilde{\varvec{w}},\tilde{\varvec{u}})=(\varvec{w}^{\star },\varvec{u}^{\star }+\varepsilon \varvec{e}_{j})$, we have

    $$\begin{aligned}&f(\varvec{w}^{\star },\varvec{u}^{\star })-f(\tilde{\varvec{w}},\tilde{\varvec{u}}) \\&\quad =\rho [\varvec{1}^{\top }\varvec{u}^{\star }-(\varvec{u}^{\star })^{\top }\varvec{u}^{\star }-\varvec{1}^{\top }(\varvec{u}^{\star }+\varepsilon \varvec{e}_{j})+(\varvec{u}^{\star }+\varepsilon \varvec{e}_{j})^{\top }(\varvec{u}^{\star }+\varepsilon \varvec{e}_{j})]\\&\quad =\rho \left( -\varepsilon +2\varepsilon u^{\star }_j+\varepsilon ^{2}\right) >0. \end{aligned}$$

  • Case: $\varvec{1}^{\top }\varvec{u}^{\star }=K$. There exists $i\ (\ne j)$ such that $1/2< u^{\star }_{i}<1$. Assume $u^{\star }_{i}\ge u^{\star }_{j}$ without loss of generality. If we choose $\varepsilon >0$ such that $u^{\star }_{i}+\varepsilon \le 1$ and $u^{\star }_{j}-\varepsilon \ge 1/2$, a solution $(\tilde{\varvec{w}},\tilde{\varvec{u}})=(\varvec{w}^{\star },\varvec{u}^{\star }+\varepsilon \varvec{e}_{i}-\varepsilon \varvec{e}_{j})$ is feasible, since $M(u^{\star }_{j}-\varepsilon )\ge M/2\ge |w_{j}^\star |$. Then we have

    $$\begin{aligned} f(\varvec{w}^{\star },\varvec{u}^{\star })-f(\tilde{\varvec{w}},\tilde{\varvec{u}})&=\rho (-(\varvec{u}^{\star })^{\top }\varvec{u}^{\star }+(\varvec{u}^{\star }+\varepsilon \varvec{e}_{i}-\varepsilon \varvec{e}_{j})^{\top }(\varvec{u}^{\star }+\varepsilon \varvec{e}_{i}-\varepsilon \varvec{e}_{j}))\\&=\rho (2\varepsilon u^{\star }_{i}-2\varepsilon u^{\star }_{j}+2\varepsilon ^{2})>0. \end{aligned}$$

$\square$