An Accelerated Coordinate Gradient Descent Algorithm for Non-separable Composite Optimization

Abstract

Coordinate descent algorithms are popular in machine learning and large-scale data analysis problems due to their low computational cost iterative schemes and their improved performances. In this work, we define a monotone accelerated coordinate gradient descent-type method for problems consisting of minimizing $f+g$, where f is quadratic and g is nonsmooth and non-separable and has a low-complexity proximal mapping. The algorithm is enabled by employing the forward–backward envelope, a composite envelope that possess an exact smooth reformulation of $f+g$. We prove the algorithm achieves a convergence rate of $O(1/k^{1.5})$ in terms of the original objective function, improving current coordinate descent-type algorithms. In addition, we describe an adaptive variant of the algorithm that backtracks the spectral information and coordinate Lipschitz constants of the problem. We numerically examine our algorithms on various settings, including two-dimensional total-variation-based image inpainting problems, showing a clear advantage in performance over current coordinate descent-type methods.

Appendix A: Proof of Theorem 4.1

Throughout the proof, we use the notation: $\mathbf{L}= \hbox {diag}(\{L_i\}_{i=1}^n)$ and denote the $\mathbf{L}$-norm and the $\mathbf{L}$-inner product by

$$ \langle \mathbf{x},\mathbf{y}\rangle _{\mathbf{L}} \equiv \textstyle \sum _{i=1}^n L_i x_i y_i; \; \Vert \mathbf{x}\Vert _{\mathbf{L}} \equiv \sqrt{\langle \mathbf{x},\mathbf{x}\rangle _{\mathbf{L}}} =\sqrt{\textstyle \sum _{i=1}^n L_i x_i^2}. \; $$

By the definition of step 4 and the block descent lemma [3, Lemma 11.8], it follows that

$$ H({\tilde{\mathbf{x}}}^{k+1}) \le H(\mathbf{y}^k)+\nabla _{i_k} H(\mathbf{y}^k)({\tilde{x}}^{k+1}_{i_k}-y^k_{i_k})+\frac{L_{i_k}}{2}({\tilde{x}}^{k+1}_{i_k}-y^k_{i_k})^2$$

Taking the expectation with respect to $i_{k}$, and recalling that ${\tilde{\mathbf{x}}}^{k+1} = \mathbf{y}^k-\frac{1}{L_{i_k}} \nabla _{i_k} H(\mathbf{y}^k)\mathbf{e}_{i_k}$, we obtain

$$\begin{aligned} {\mathbb {E}}_{i_k} H({\tilde{\mathbf{x}}}^{k+1}) \le H(\mathbf{y}^k)+ \nabla H(\mathbf{y}^k)^T(\mathbf{s}^{k+1}-\mathbf{y}^k)+\frac{n}{2} \Vert \mathbf{s}^{k+1}-\mathbf{y}^k\Vert _{\mathbf{L}}^2,\nonumber \\\end{aligned}$$

(7.1)

where $\mathbf{s}^{k+1}= \mathbf{y}^k- \frac{1}{n} \mathbf{L}^{-1}\nabla H(\mathbf{y}^k).$ Define

$$\begin{aligned} \mathbf{t}^{k+1}\equiv & {} \mathbf{z}^k- \frac{1}{n\theta ^k } \mathbf{L}^{-1}\nabla H(\mathbf{y}^k)\nonumber \\= & {} \displaystyle \mathop {\mathrm{argmin}}_{\mathbf{y}} \left\{ \nabla H(\mathbf{y}^k)^T(\mathbf{y}-\mathbf{z}^k)+\frac{n\theta ^k }{2} \Vert \mathbf{y}-\mathbf{z}^k\Vert _{\mathbf{L}}^2 \right\} . \end{aligned}$$

(7.2)

Obviously, $\mathbf{s}^{k+1}-\mathbf{y}^k = \theta ^k (\mathbf{t}^{k+1}-\mathbf{z}^k)$. Thus, by (7.1) and the fact that $H(\mathbf{x}^{k+1})\le H({\tilde{\mathbf{x}}}^{k+1})$ (step 5), it follows that

$$\begin{aligned} {\mathbb {E}}_{i_k} H(\mathbf{x}^{k+1})\le & {} {\mathbb {E}}_{i_k} H({\tilde{\mathbf{x}}}^{k+1}) \le H(\mathbf{y}^k)+ \nabla H(\mathbf{y}^k)^T(\mathbf{s}^{k+1}-\mathbf{y}^k)+\frac{n}{2} \Vert \mathbf{s}^{k+1}-\mathbf{y}^k\Vert _{\mathbf{L}}^2\nonumber \\= & {} H(\mathbf{y}^k)+\theta ^k \left[ \nabla H(\mathbf{y}^k)^T(\mathbf{t}^{k+1}-\mathbf{z}^k)+\frac{n\theta ^k }{2} \Vert \mathbf{t}^{k+1}-\mathbf{z}^k\Vert _{\mathbf{L}}^2 \right] . \end{aligned}$$

(7.3)

By Tseng’s three-points property [37, Property 1] and the relation (7.2), we have

$$\begin{aligned} \nabla H(\mathbf{y}^k)^T(\mathbf{x}^*-\mathbf{z}^k) +\frac{n\theta ^k}{2} \Vert \mathbf{x}^*-\mathbf{z}^k\Vert _{\mathbf{L}}^2- \nabla H(\mathbf{y}^k)^T(\mathbf{t}^{k+1}-\mathbf{z}^k)\nonumber \\ -\frac{n\theta ^k}{2} \Vert \mathbf{t}^{k+1}-\mathbf{z}^k\Vert _{\mathbf{L}}^2 \ge \frac{n\theta ^k}{2}\Vert \mathbf{x}^*-\mathbf{t}^{k+1}\Vert _{\mathbf{L}}^2. \end{aligned}$$

(7.4)

Combining the above with (7.3) yields

$$\begin{aligned} {\mathbb {E}}_{i_k} H(\mathbf{x}^{k+1})\le & {} H(\mathbf{y}^k)+\theta ^k \left[ \nabla H(\mathbf{y}^k)^T(\mathbf{x}^*-\mathbf{z}^k)+\frac{n\theta ^k}{2} \Vert \mathbf{x}^*-\mathbf{z}^k\Vert _{\mathbf{L}}^2 -\frac{n\theta ^k}{2} \Vert \mathbf{x}^*-\mathbf{t}^{k+1}\Vert _{\mathbf{L}}^2 \right] \nonumber \\= & {} H(\mathbf{y}^k)+\theta ^k \left[ \nabla H(\mathbf{y}^k)^T(\mathbf{x}^*-\mathbf{z}^k)+\frac{n^2\theta ^k}{2} \Vert \mathbf{x}^*-\mathbf{z}^k\Vert _{\mathbf{L}}^2 -\frac{n^2\theta ^k}{2} {\mathbb {E}}_{i_k}\Vert \mathbf{x}^*-\mathbf{z}^{k+1}\Vert _{\mathbf{L}}^2 \right] ,\nonumber \\ \end{aligned}$$

(7.5)

where the equality follows by the following argument:

$$\begin{aligned} \Vert \mathbf{x}^*-\mathbf{z}^k\Vert _{\mathbf{L}}^2-\Vert \mathbf{x}^*-\mathbf{t}^{k+1}\Vert _{\mathbf{L}}^2= & {} {2}\langle \mathbf{t}^{k+1}-\mathbf{z}^k, \mathbf{x}^*-\mathbf{z}^k\rangle _{\mathbf{L}} -\Vert \mathbf{t}^{k+1}-\mathbf{z}^k\Vert _{\mathbf{L}}^2\\ {}= & {} {2}n {\mathbb {E}}_{i_k}\langle \mathbf{z}^{k+1}-\mathbf{z}^k, \mathbf{x}^*-\mathbf{z}^k\rangle _{\mathbf{L}} -n{\mathbb {E}}_{i_k}\Vert \mathbf{z}^{k+1}-\mathbf{z}^k\Vert _{\mathbf{L}}^2 \\= & {} n{\mathbb {E}}_{i_k} (\Vert \mathbf{x}^*-\mathbf{z}^k\Vert _{\mathbf{L}}^2-\Vert \mathbf{x}^*-\mathbf{z}^{k+1}\Vert _{\mathbf{L}}^2) \end{aligned}$$

Now, using the update formula in step 2, we have

$$\begin{aligned} \nabla H(\mathbf{y}^k)^T(\theta ^k\mathbf{x}^*- \theta ^k \mathbf{z}^k)= & {} \nabla H(\mathbf{y}^k)^T(\theta ^k\mathbf{x}^*-\mathbf{y}^k+(1-\theta ^k)\mathbf{x}^k) \nonumber \\= & {} \theta ^k \nabla H(\mathbf{y}^k)^T(\mathbf{x}^*-\mathbf{y}^k)+(1-\theta ^k) \nabla H(\mathbf{y}^k)^T(\mathbf{x}^k-\mathbf{y}^k).\nonumber \\ \end{aligned}$$

(7.6)

Thus, combining (7.5) and (7.6) along with the gradient inequality, the following is implied:

$$\begin{aligned} {\mathbb {E}}_{i_k} H(\mathbf{x}^{k+1}) \le (1-\theta ^k)H(\mathbf{x}^k)+\theta ^k H(\mathbf{x}^*)+\frac{n^2(\theta ^k)^2}{2} \Vert \mathbf{x}^*\nonumber \\ -\mathbf{z}^k\Vert _{\mathbf{L}}^2 -\frac{n^2 (\theta ^k)^2}{2} {\mathbb {E}}_{i_k}\Vert \mathbf{x}^*-\mathbf{z}^{k+1}\Vert _{\mathbf{L}}^2, \end{aligned}$$

(7.7)

which is the same as

$$\begin{aligned} {\mathbb {E}}_{i_k} H(\mathbf{x}^{k+1})-H(\mathbf{x}^*)\le & {} (1-\theta ^k)(H(\mathbf{x}^k)-H(\mathbf{x}^*))+\frac{n^2(\theta ^k)^2}{2} \Vert \mathbf{x}^*\nonumber \\&\quad -\, \mathbf{z}^k\Vert _{\mathbf{L}}^2 -\frac{n^2 (\theta ^k)^2}{2} {\mathbb {E}}_{i_k}\Vert \mathbf{x}^*-\mathbf{z}^{k+1}\Vert _{\mathbf{L}}^2. \end{aligned}$$

(7.8)

Taking expectation over $\xi _{k-1}$ leads to

$$\begin{aligned} {\mathbb {E}}_{\xi _k} H(\mathbf{x}^{k+1})-H(\mathbf{x}^*)\le & {} (1-\theta ^k)({\mathbb {E}}_{\xi _{k-1}}H(\mathbf{x}^k)-H(\mathbf{x}^*))\nonumber \\&+\frac{n^2(\theta ^k)^2}{2} {\mathbb {E}}_{\xi _{k-1}} \Vert \mathbf{x}^*-\mathbf{z}^k\Vert _{\mathbf{L}}^2 -\frac{n^2 (\theta ^k)^2}{2} {\mathbb {E}}_{\xi _k}\Vert \mathbf{x}^*-\mathbf{z}^{k+1}\Vert _{\mathbf{L}}^2.\nonumber \\ \end{aligned}$$

(7.9)

Denoting $e_k \equiv {\mathbb {E}}_{\xi _{k-1}} H(\mathbf{x}^{k})-H(\mathbf{x}^*)$ and $\Delta _k \equiv \frac{n^2}{2}{\mathbb {E}}_{\xi _{k-1}} \Vert \mathbf{x}^*-\mathbf{z}^k\Vert _{\mathbf{L}}^2$, we can rewrite (7.9) as

$$ e_{k+1}\le (1-\theta ^k)e_k+(\theta ^k)^2\Delta _k-(\theta ^k)^2\Delta _{k+1}.$$

Dividing the inequality by $(\theta ^k)^2$ yields

$$ \frac{1}{(\theta ^k)^2} e_{k+1}\le \frac{1-\theta ^k}{(\theta ^k)^2}e_k+\Delta _k-\Delta _{k+1}.$$

By the definition of the sequence $\theta ^k$ (Step 6), the above is the same as

$$ \frac{1}{(\theta ^k)^2} e_{k+1}\le \frac{1}{(\theta ^{k-1})^2}e_k+\Delta _k-\Delta _{k+1},$$

and hence,

$$ \frac{1}{(\theta ^k)^2} e_{k+1}+\Delta _{k+1}\le \frac{1}{(\theta ^{k-1})^2}e_k+\Delta _k.$$

Since $\theta ^0=1$ the above inequality results in that $\frac{1}{(\theta ^{k-1})^2} e_{k}\le \Delta _0,$ which by the facts that $\Delta _0 = \frac{n^2}{2}\Vert \mathbf{x}^*-\mathbf{x}^0\Vert _{\mathbf{L}}^2$ and $\theta ^k \le \frac{2}{k+2}$ (see [37]) leads to the desired result (4.3).