Open issues and recent advances in DC programming and DCA

Abstract

DC (difference of convex functions) programming and DC algorithm (DCA) are powerful tools for nonsmooth nonconvex optimization. This field was created in 1985 by Pham Dinh Tao in its preliminary state, then the intensive research of the authors of this paper has led to decisive developments since 1993, and has now become classic and increasingly popular worldwide. For 35 years from their birthday, these theoretical and algorithmic tools have been greatly enriched, thanks to a lot of their applications, by researchers and practitioners in the world, to model and solve nonconvex programs from many fields of applied sciences. This paper is devoted to key open issues, recent advances and trends in the development of these tools to meet the growing need for nonconvex programming and global optimization. We first give an outline in foundations of DC programming and DCA which permits us to highlight the philosophy of these tools, discuss key issues, formulate open problems, and bring relevant answers. After outlining key open issues that require deeper and more appropriate investigations, we will present recent advances and ongoing works in these issues. They turn around novel solution techniques in order to improve DCA’s efficiency and scalability, a new generation of algorithms beyond the standard framework of DC programming and DCA for large-dimensional DC programs and DC learning with Big data, as well as for broader classes of nonconvex problems beyond DC programs.

Appendices

Appendix A Convergence of Standard DCA

Let X $\subset \mathbb {R}^n$ and Y $\subset \mathbb {R}^n$ be two nonempty convex sets, $\rho _{i}$ and $\rho _{i}^{*},(i=1,2)$ be real nonnegative numbers such that $0 \le \rho _{i}<\rho (f_{i},X)$ (resp. $0 \le \rho _{i}^{*}<\rho (f_{i}^{*},Y)$) where $\rho _{i}=0$ (resp. $\rho _{i}^{*}=0$) if $\rho (f_{i},X)=0$ (resp. $\rho (f_{i}^{*},Y)=0$) and $\rho _{i}$ (resp. $\rho _{i}^{*}$) may take the value $\rho (f_{i},X)$ (resp. $\rho (f_{i}^{*},Y)$) if it is attained. We next set $f_{1}=g$ and $f_{2}=h$. Also let $dx^{k}:=x^{k+1}-x^{k}$ and $dy^{k}:=y^{k+1}-y^{k}$.

Theorem 11

Let $X \subset \mathbb {R}^n$ and $Y \subset \mathbb {R}^n$ be two nonempty convex sets containing the sequences $\{x^{k}\}$ and $\{y^{k}\}$ generated by DCA, respectively, and $dx^{k}:=x^{k+1}-x^{k}, dy^{k}:=y^{k+1}-y^{k}$. The DCA is a descent method without linesearch but with global convergence, which enjoys the following key properties:

1 For the primal DC program $(P_{dc})$

The decrease of the sequence $\{(g-h)(x^{k})\}$ is expressed by $(g-h)(x^{k+1}) \le (h^{*}-g^{*})(y^{k})-\dfrac{\rho _{2}}{2}\Vert dx^{k}\Vert ^{2} \le (g-h)(x^{k})-\dfrac{\rho _{1}+\rho _{2}}{2}\Vert dx^{k}\Vert ^{2}, \forall k$ where the equality

$$\begin{aligned} (g-h)(x^{k+1})=(g-h)(x^{k}) \end{aligned}$$

(A1)

is verified if and only if $x^{k}\in \partial g^{*}(y^{k})$, $y^{k}\in \partial h(x^{k+1})$ and $(\rho _{1}+\rho _{2})dx^{k}=0.$

In this case, one obtains the following main statements:

1.1 $x^{k},x^{k+1}$ are DC-critical points of $g-h$ satisfying $y^{k}\in (\partial g(x^{k})\cap \partial h(x^{k}))$ and $y^{k}\in (\partial g(x^{k+1})\cap \partial h(x^{k+1})),$

1.2 $y^{k}$ is a DC critical point of $h^{*}-g^{*}$ and $[x^{k},x^{k+1}]\subset (\partial g^{*}(y^{k})\cap \partial h^{*}(y^{k})),$

1.3 If $\rho _{1}+\rho _{2}>0$, then $x^{k+1}=x^{k}$, $y^{k}=y^{k-1}$ if $\rho _{1}^{*}>0$ and $y^{k+1}=y^{k}$ if $\rho _{2}^{*}>0$.

Furthermore, if g or h is strictly convex on X, then $x^{k+1}=x^{k}$.

In such a case (A1), the DCA terminates at the $k^{th}$ iteration (finite convergence of DCA).

2 For the dual DC program $(D_{dc})$

Similarly, the DC duality provides the dual DC program $(D_{dc})$ with

$$\begin{aligned} (h^{*}-g^{*})(y^{k+1}) \le (g-h)(x^{k+1})-\dfrac{\rho _{1}^{*}}{2}\Vert dy^{k}\Vert ^{2} \le (h^{*}-g^{*})(y^{k})-\dfrac{\rho _{1}^{*}+\rho _{2}^{*}}{2}\Vert dy^{k}\Vert ^{2}. \end{aligned}$$

The equality

$$\begin{aligned} (h^{*}-g^{*})(y^{k+1})=(h^{*}-g^{*})(y^{k}) \end{aligned}$$

(A2)

occurs if and only if $x^{k+1}\in \partial g^{*}(y^{k+1}),y^{k}\in \partial h(x^{k+1})$ and $(\rho _{1}^{*}+\rho _{2}^{*})dy^{k}=0$.

In this case, the following properties hold:

2.1 The equality $(h^{*}-g^{*})(y^{k+1})=(g-h)(x^{k+1})$ holds and $y^{k},y^{k+1}$ are the DC critical points of $h^{*}-g^{*}$ with $x^{k+1}\in (\partial g^{*}(y^{k})\cap h^{*}(y^{k}))$ and $x^{k+1}\in (\partial g^{*}(y^{k+1})\cap \partial h^{*}(y^{k+1})),$

2.2 $x^{k+1}$ is a DC critical point of $g-h$ and $[y^{k},y^{k+1}]\subset (\partial g(x^{k+1})\cap \partial h(x^{k+1})),$

2.3 $y^{k+1}=y^{k}$ if $\rho _{1}^{*}+\rho _{2}^{*}>0,x^{k+1}=x^{k}$ if $\rho _{2}>0$ and $x^{k+2}=x^{k+1}$ if $\rho _{1}>0$.

Furthermore, if $g^{*}$ or $h^{*}$ is strictly convex on $\mathbb {R}^n$, then $y^{k+1}=y^{k}.$

As for 1.3, in the case (A2), the DCA terminates at the $k^{th}$ iteration (finite convergence of DCA).

3. If $\rho _{1}+\rho _{2}>0$ then the primal DC series $\{\Vert x^{k+1}-x^{k}\Vert ^{2}\}$ converges with its limit bounded above by

$$\begin{aligned} \frac{\rho _{1}+\rho _{2}}{2}\sum \limits _{l=0}^{k}\Vert x^{l+1}-x^{l}\Vert ^{2}&\le (g-h)(x^{0})-(g-h)(x^{k+1}) \nonumber \\&\le (g-h)(x^{0})-\beta \le (g-h)(x^{0})-\alpha , \forall k. \end{aligned}$$

(A3)

Dually, if $\rho _{1}^{*}+\rho _{2}^{*}>0$, then the dual DC series $\{\Vert y^{k+1}-y^{k}\Vert ^{2}\})$ is convergent with its limit bounded above by

$$\begin{aligned} \frac{\rho _{1}^{*}+\rho _{2}^{*}}{2}\sum \limits _{l=0}^{k}\Vert y^{l+1}-y^{l}\Vert ^{2}&\le (h^{*}-g^{*})(y^{0})-(h^{*}-g^{*})(y^{k+1})\le (h^{*}-g^{*})(y^{0})-\beta \nonumber \\&\le (h^{*}-g^{*})(y^{0})-\alpha ,~\forall k. \end{aligned}$$

(A4)

4. If $\alpha$ is finite, the sequences $\{(g-h)(x^{k})\}$, $\{(h^{*}-g^{*})(y^{k})\}$ decrease and converge to the same limit $\beta \ge \alpha$: $\lim _{k\rightarrow +\infty }(g-h)(x^{k})=\lim _{k\rightarrow +\infty }(h^{*}-g^{*})(y^{k})=\beta .$

5. If $\alpha$ is finite and the sequences $\{x^{k}\}$ and $\{y^{k}\}$ are bounded, then for every limit point $x^{*}$ of $\{x^{k}\}$ (resp. $y^{*}$ of $\{y^{k}\}$) there exists a limit point $y^{*}$ of $\{y^{k}\}$ (resp. $x^{*}$ of $\{x^{k}\}$) such that $(x^{*},y^{*})\in [\partial g^{*}(y^{*})\cap \partial h^{*}(y^{*})]\times [\partial g(x^{*})\cap \partial h(x^{*})]$ and $(g-h)(x^{*})=(h^{*}-g^{*})(y^{*})=\beta \ge \alpha .$ Such a point $x^{*}$(resp. $y^{*})$ is DC critical point of $g-h$ (resp. $h^{*}-g^{*}$).

6. DCA’s complexity for primal and dual DC programs

Let $x^{*}$be a DC critical point of $g-h$ defined as a limit point of the sequence $\{x^{k}\}$ computed by the primal DCA. Then, from (A3), one deduces $f(x^{*})=(g-h)(x^{*})=\beta :=\lim _{k\rightarrow +\infty }f(x^{k})=\lim _{k\rightarrow +\infty }(g-h)(x^{k})$ and $\frac{\rho _{1}+\rho _{2}}{2}(k+1)\min \{\Vert x^{l+1}-x^{l}\Vert ^{2}:l=0,\ldots ,k\}\le [f(x^{0})-f(x^{*})].$ Moreover, if $\rho _{1}+\rho _{2}>0$, then $\min \{\Vert x^{l+1}-x^{l}\Vert :l=0,\ldots ,k\}\le \frac{2^{1/2}[f(x^{0})-f(x^{*})]^{1/2}}{(\rho _{1}+\rho _{2})^{1/2}(k+1)^{1/2}}.$

Likewise, by using the same reasoning for the sequence $\{y^{k}\}$ via (A4), we get the similar results for dual DCA: $(h^{*}-g^{*})(y^{*})=\beta :=\lim _{k\rightarrow +\infty }(h^{*}-g^{*})(y^{k})$ and $\frac{\rho _{1}^{*}+\rho _{2}^{*}}{2}\sum \limits _{k=0}^{k}\Vert y^{l+1}-y^{l}\Vert ^{2} \le (h^{*}-g^{*})(y^{0})-(h^{*}-g^{*})(y^{k+1}) \le (h^{*}-g^{*})(y^{0})-\beta \le (h^{*}-g^{*})(y^{0})-\alpha ,~\forall k.$ Hence, the inequality $\rho _{1}^{*}+\rho _{2}^{*}>0$ gives

$\min \{\Vert y^{l+1}-y^{l}\Vert :l=0,\ldots ,k\}\le \frac{2^{1/2}[(h^{*}-g^{*})(y^{0})-(h^{*}-g^{*})(y^{*})]^{1/2}}{(\rho _{1}^{*}+\rho _{2}^{*})^{1/2}(k+1)^{1/2}}.$

Therefore, both primal and dual DCA have a complexity $O(1/\sqrt{k})$.

Appendix B Global convergence of GDCA1

Denote by $I(x):=\left\{ i\in \{1,\ldots ,m\}: ~ f_{i}(x)=p(x)\right\}$. We say that the extended Mangasarian-Fromowitz constraint qualification (EMFCQ) is satisfied at $x^{*}\in E$ with $I(x^{*})\not =\emptyset$ if

$$\begin{aligned} \begin{array}{ll} (\text {MFCQ}) &{} \text{ there } \text{ is } \text{ a } \text{ vector } d\in \text{ cone }(C-\{x^*\}) \text{(the } \text{ cone } \text{ hull } \text{ of } C-\{x^*\}) \\ &{} \text {such that }f_{i}^{\uparrow }(x^{*},d)<0 \text{ for } \text{ all } i\in I(x^{*}). \end{array} \end{aligned}$$

When $f_{i}^{\prime }s$ are continuously differentiable, then $f_{i}^{\uparrow }(x^{*},d)=\langle \nabla f(x^{*}),d\rangle .$ Therefore, (EMFCQ) becomes the well-known Mangasarian-Fromowitz constraint qualification. It is well known that if the (extended) Mangasarian-Fromowitz constraint qualification is satisfied at a local minimizer $x^{*}$ of problem (6) then the KKT first order necessary condition (7) holds (see [160, 161]). In the global convergence theorem, we make use of the following assumption:

Assumption 3

The (extended) Mangasarian-Fromowitz constraint qualification (EMFCQ) is satisfied at any $x\in {{\mathbb {R}}}^{n}$ with $p(x)\ge 0.$

When $f_{i}$, $i=1,\ldots ,m,$ are all convex functions, then it is obvious that this assumption is satisfied under the $f_{i}(x)<0$ for all $i=1,\ldots ,m.$

Theorem 12

Suppose that $C\subseteq {{\mathbb {R}}}^{n}$ is a nonempty closed convex set and $f_{i}$, $i=1,\ldots ,m$ are DC functions on C. Suppose further that Assumptions 1–3 are verified. Let $\delta >0,$ $\beta _{1}>0$ be given. Let $\{x^{k}\}$ be a sequence generated by GDCA1. Then GDCA1 either stops, after finitely many iterations, at a KKT point $x^{k}$ for problem (6) or generates an infinite sequence $\{x^{k}\}$ of iterates such that $\lim _{k\rightarrow \infty }\Vert x^{k+1}-x^{k}\Vert =0$ and every limit point $x^{\infty }$ of the sequence $\{x^{k}\}$ is a KKT point of problem (6).

Appendix C Global convergence of GDCA2

Recall, as defined in the preceding section, that $\varphi _{k}(x):=f_{0}(x)+\beta _{k}p^{+}(x).$ The following lemma is needed to investigate the convergence of GDCA2.

Lemma 13

The sequence $(x^{k},t^{k})$ generated by GDCA2 satisfies the following inequality $\varphi _{k}(x^{k})-\varphi _{k}(x^{k+1})\ge \frac{\rho }{2}\Vert x^{k+1}-x^{k}\Vert ^{2}$, for all $k=1,2,\ldots$ where, $\rho :=\rho (g_{0},C)+\rho (h_{0},C)+\min \{\rho (g_{i},C): ~ i=1,\ldots ,m\}.$

Theorem 14

Suppose that $C\subseteq {{\mathbb {R}}}^{n}$ is a nonempty closed convex set and $f_{i}, i=1,\ldots ,m,$ are DC functions on C such that Assumptions 1 and 3 are verified. Suppose further that for each $i=0,\ldots ,m,$ either $g_{i}$ or $h_{i}$ is differentiable on C and that $\rho :=\rho (g_{0},C)+\rho (h_{0},C)+\min \{\rho (g_{i},C): i=1,\ldots ,m\}>0.$ Let $\delta _{1},\delta _{2}>0,$ $\beta _{1}>0$ be given. Let $\{x^{_{k}}\}$ be a sequence generated by GDCA2. Then GDCA2 either stops, after finitely many iterations, at a KKT point $x^{k}$ for problem (6) or generates an infinite sequence $\{x^{k}\}$ of iterates such that $\lim _{k\rightarrow \infty }\Vert x^{k+1}-x^{k}\Vert =0$ and every limit point $x^{\infty }$ of the sequence $\{x^{k}\}$ is a KKT point of problem (6).

Note that, as shown in Theorems 12 and 14, the penalty parameter $\beta _{k}$ is constant when k is sufficiently large. Observing from the proof of these convergence theorems, the sequence $\{\varphi (x^{k})\}$ of values of the function $\varphi (x)=f_{0}(x)+\beta _{k}p^{+}(x)$ along with the sequence $\{x^{k}\}$ generated by GDCA1 and GDCA2 is decreasing. These results remain valid if we replace, in (11), the variable t by $t_{i}$ for $i=1,\ldots ,m$ and the function $\beta _{k}t$ by $\beta _{k}\sum _{i=1}^{m}t_{i}.$