An Inertial Parallel and Asynchronous Forward–Backward Iteration for Distributed Convex Optimization

Abstract

Two characteristics that make convex decomposition algorithms attractive are simplicity of operations and generation of parallelizable structures. In principle, these schemes require that all coordinates update at the same time, i.e., they are synchronous by construction. Introducing asynchronicity in the updates can resolve several issues that appear in the synchronous case, like load imbalances in the computations or failing communication links. However, and to the best of our knowledge, there are no instances of asynchronous versions of commonly known algorithms combined with inertial acceleration techniques. In this work, we propose an inertial asynchronous and parallel fixed-point iteration, from which several new versions of existing convex optimization algorithms emanate. Departing from the norm that the frequency of the coordinates’ updates should comply to some prior distribution, we propose a scheme, where the only requirement is that the coordinates update within a bounded interval. We prove convergence of the sequence of iterates generated by the scheme at a linear rate. One instance of the proposed scheme is implemented to solve a distributed optimization load sharing problem in a smart grid setting, and its superiority with respect to the nonaccelerated version is illustrated.

Appendices

Appendices

1.1 Proof of Lemma 5.1

Proof

Using Assumption 1 and Eq. (7), we can start by rewriting $y_B=(T_B x^i_\mathrm {read})[i]$.

$$\begin{aligned} (T_B x^i_\mathrm {read})[i]&= x^i_\mathrm {read}[i] - \gamma (Bx^i_\mathrm {read})[i] \\&= x_{k}[i] - a_k^i[i] - \gamma (Bx_{k})[i] + \gamma (Bx_{k})[i] - \gamma (Bx^i_\mathrm {read})[i] \\&= (T_Bx_k)[i] + \gamma ((Bx_{k})[i]-(Bx^i_\mathrm {read})[i]) - a_k^i[i] \\&= (T_Bx_k)[i] + d_k^i[i], \end{aligned}$$

where $d_k[i]=\gamma \left( (Bx_{k})[i]-(Bx^i_\mathrm {read})[i]\right) - a_k^i[i]$.

Similarly, we have from Eq. (7) that

$$\begin{aligned} \beta ( y_\mathrm {write} - y_\mathrm {write}^\mathrm {prev} )&= \beta ( x_k[i] - b_k^i[i] - x_k[i] + c_k^i[i] ) \\&= \beta (c_k^i[i] - b_k^i[i]). \end{aligned}$$

Using the above relations, a coordinate update of iteration (6) can be expressed as

$$\begin{aligned} x_{k+1}[i] = x_k[i] + \eta (T_{A_{i}}( (T_B x_k)[i] + d_k[i] + \beta (c_k^i[i] - b_k^i[i])) - x_k[i]), \end{aligned}$$

or, equivalently, as

$$\begin{aligned} x_{k+1}[i] = x_k[i] + \eta ( T_{A_{i}}((T_B x_k)[i]) - x_k[i] + e_k[i]), \end{aligned}$$

with $e_k[i] = T_{A_{i}}((T_Bx_k)[i] + d_k[i] + \beta (c_k^i[i]-b_k^i[i])) - T_{A_{i}}((T_Bx_k)[i])$, which concludes the proof. $\square$

1.2 Proof of Lemma 5.2

Proof

Squaring (9) we get:

$$\begin{aligned} \Vert x_{k+1}-x_*\Vert ^2 = \Vert x_{k}-x_*\Vert ^2 -2\eta \langle x_k-x_*, Sx_k-e_k \rangle +\eta ^2\Vert Sx_k-e_k\Vert ^2. \end{aligned}$$

(22)

Let us now analyze the second and third term in (22).

• Bound $-2\eta \langle x_k-x_*, Sx_k-e_k \rangle$: We will upper-bound the resulting inner product terms. In order to do so, we use both the cocoercivity and the quasi-strong monotonicity of S, the former proven in ‘Appendix A.6,’ and the latter holding from Assumption 1. Since S is 1 / 2-cocoercive, we have that

  $$\begin{aligned} \langle x_k-x_*,Sx_k\rangle \ge \frac{1}{2}\Vert Sx_k\Vert ^2. \end{aligned}$$

  From the quasi-$\nu$-strong monotonicity of S, we have:

  $$\begin{aligned} \langle x_k-x_*,Sx_k\rangle \ge \nu \Vert x_k-x_*\Vert ^2. \end{aligned}$$

  Putting these two together, we get that

  $$\begin{aligned} -2\eta \langle x_k-x_*,Sx_k\rangle \le -\eta \nu {\mathbf {dist} }_k^2 -\frac{\eta }{2}\Vert Sx_k\Vert ^2. \end{aligned}$$

  (23)

  For the second inner product term involving the error, we can easily derive the bound

  $$\begin{aligned} 2\eta \langle x_k-x_*,e_k\rangle \le 2\eta {\mathbf {dist} }_k\Vert e_k\Vert . \end{aligned}$$

  (24)

  Equations (23) and (24) result in the bound

  $$\begin{aligned} -2\eta \langle x_k-x_*, Sx_k-e_k\rangle \le -\eta \nu {\mathbf {dist} }_k^2-\frac{\eta }{2}\Vert Sx_k\Vert ^2 + 2\eta {\mathbf {dist} }_k\Vert e_k\Vert . \end{aligned}$$

  (25)

• Bound $\eta ^2\Vert Sx_k-e_k\Vert ^2$: By developing the square, we have that

  $$\begin{aligned} \eta ^2\Vert Sx_k-e_k\Vert ^2&= \eta ^2 (\Vert Sx_k\Vert ^2 -2\langle Sx_k,e_k\rangle + \Vert e_k\Vert ^2). \end{aligned}$$

  (26)

  The inner product term in (26) can be bounded by employing Young’s inequality¹ as follows:

  $$\begin{aligned} -\,2\langle Sx_k,e_k\rangle&\le 2\Vert Sx_k\Vert \Vert e_k\Vert \nonumber \\&\le 2\left( \frac{\delta }{2}\Vert Sx_k\Vert ^2+\frac{1}{2\delta }\Vert e_k\Vert ^2\right) \nonumber \\&= \delta \Vert Sx_k\Vert ^2 + \frac{1}{\delta }\Vert e_k\Vert ^2, \end{aligned}$$

  (27)

  for any $\delta >0$. Putting together (26) and (27), we get the bound:

  $$\begin{aligned} \eta ^2\Vert Sx_k-e_k\Vert ^2 \le \eta ^2(1+\delta )\Vert Sx_k\Vert ^2 + \eta ^2\frac{(\delta +1)}{\delta }\Vert e_k\Vert ^2. \end{aligned}$$

  (28)

Using (25) and (28), inequality (22) can be written as

$$\begin{aligned} {\mathbf {dist} }_{k+1}^2\le & {} (1-\eta \nu ){\mathbf {dist} }_k^2 +\eta \left( -\frac{1}{2}+\eta (1+\delta )\right) \Vert Sx_k\Vert ^2\nonumber \\&+2\eta {\mathbf {dist} }_k\Vert e_k\Vert +\eta ^2\frac{(\delta +1)}{\delta }\Vert e_k\Vert ^2 . \end{aligned}$$

(29)

The second term in the sum can be eliminated by assuming that

$$\begin{aligned} -\frac{1}{2}+\eta (1+\delta )<0\Rightarrow \eta <\frac{1}{2(1+\delta )}, \end{aligned}$$

(30)

which gives rise to the inequality

$$\begin{aligned} {\mathbf {dist} }_{k+1}^2 \le (1-\eta \nu ){\mathbf {dist} }_k^2 +2\eta {\mathbf {dist} }_k\Vert e_k\Vert +\eta ^2\frac{(\delta +1)}{\delta }\Vert e_k\Vert ^2 . \end{aligned}$$

(31)

The complicating term on the right-hand side can be eliminated by using once more Young’s inequality, i.e.,

$$\begin{aligned} 2\eta {\mathbf {dist} }_k\Vert e_k\Vert&\le 2\eta \left( \frac{\epsilon }{2}{\mathbf {dist} }_k^2+\frac{1}{2\epsilon }\Vert e_k\Vert ^2\right) \\&= \eta \epsilon {\mathbf {dist} }_k^2+\frac{\eta }{\epsilon }\Vert e_k\Vert ^2. \end{aligned}$$

1.3 Proof of Lemma 5.3

Proof

We will bound the error term $\Vert e_k\Vert$ componentwise. For some arbitrary $i\in \{1,\ldots ,N\}$ and $k\in \mathbb {N}$, we have from (13) that

$$\begin{aligned} \Vert e_k[i]\Vert \le \Vert d_k[i]\Vert + \beta (\Vert c_k^i[i]\Vert +\Vert b_k^i[i]\Vert ). \end{aligned}$$

Consequently,

$$\begin{aligned} \Vert e_k\Vert \le (1+\gamma L)N\underset{1\le i\le N}{\max }\Vert a_k^i\Vert + \beta N\underset{1\le i\le N}{\max }(\Vert c_k^i\Vert +\Vert b_k^i\Vert ). \end{aligned}$$

(32)

The first term can be recovered by using the 1 / L-cocoercivity of B (Assumption 1) in (11), as well as the inequality $\Vert a_k^i[i]\Vert \le \Vert a_k^i\Vert$, while the second term follows from the inequality

$$\begin{aligned} \Vert c_k^i[i]\Vert +\Vert b_k^i[i]\Vert \le \Vert c_k^i\Vert +\Vert b_k^i\Vert . \end{aligned}$$

• We want to bound the two summands of (32). Let us start with bounding $\Vert a_k^i\Vert$, for which we have for all i:

  $$\begin{aligned} \underset{1\le i\le N}{\max }\Vert a_k^i\Vert&\le \sum _{m=k-2\tau }^{k-1}\Vert x_{m+1}-x_m\Vert \nonumber \\&= \eta \sum _{m=k-2\tau }^{k-1}\Vert e_m-Sx_m\Vert \nonumber \\&\le \eta \left( \sum _{m=k-2\tau }^{k-1}\Vert e_m\Vert +\sum _{m=k-2\tau }^{k-1}\Vert Sx_m\Vert \right) \nonumber \\&\le \eta \sum _{m=k-2\tau }^{k-1}(\Vert d_m\Vert + \beta \Vert c_m-b_m\Vert + \Vert Sx_m\Vert ). \end{aligned}$$

  (33)

  The first inequality follows from the definitions of $a_k^i$ in (7), the first equality from (9), while the last two inequalities from the triangle inequality and (13).

• Bound $\Vert c_k-b_k\Vert$: Following the same process as in (33), we have that

  $$\begin{aligned} \underset{1\le i\le N}{\max }(\Vert c_k^i\Vert +\Vert b_k^i\Vert )&\le \sum _{m=k-3\tau }^{k-1}\Vert x_{m+1}-x_m\Vert + \sum _{m=k-2\tau }^{k-1}\Vert x_{m+1}-x_m\Vert \nonumber \\&\le \eta \left( \sum _{m=k-3\tau }^{k-1}(\Vert d_m\Vert + \beta \Vert c_m-b_m\Vert + \Vert Sx_m\Vert ) \right. \nonumber \\&\left. \quad +\,\sum _{m=k-2\tau }^{k-1}(\Vert d_m\Vert + \beta \Vert c_m-b_m\Vert + \Vert Sx_m\Vert )\right) . \end{aligned}$$

  (34)

Finally, (33), and (34) can be bounded by means of the quantity ($\varSigma$), and by substituting (33) to (32), the result follows.

1.4 Proof of Lemma 5.4

Proof

Let us start by bounding the quantities involved in ($\varSigma$), namely $\Vert b_k\Vert$, $\Vert c_k\Vert$ and $\Vert d_k\Vert$ with respect to the maximum distance from the optimizer. The following inequalities hold:

$$\begin{aligned} \Vert b_k\Vert&\le N\underset{1\le i\le N}{\max }\Vert b_k^i[i]\Vert \le N\underset{1\le i\le N}{\max }\Vert b_k^i\Vert \nonumber \\ \Vert c_k\Vert&\le N\underset{1\le i\le N}{\max }\Vert c_k^i[i]\Vert \le N\underset{1\le i\le N}{\max }\Vert c_k^i\Vert \nonumber \\ \Vert d_k\Vert&\le (1+\gamma L)N\underset{1\le i\le N}{\max }\Vert a_k^i\Vert . \end{aligned}$$

(35)

Note, also, that $\forall \;i=1,\ldots ,N$ and for $l_i\in \{1,\ldots ,K\}$ holds

$$\begin{aligned} \Vert x_k-x_{k-l_i}\Vert&= \Vert x_k-x_*+x_*-x_{k-l_i}\Vert \\&\le \Vert x_k-x_*\Vert + \Vert x_*-x_{k-l_i}\Vert . \end{aligned}$$

Since the first inequality holds $\forall \;i=1,\ldots ,N$, by denoting $i_*={{\text { argmax}}}_{{i\in \{1,\ldots ,N\}}}\Vert x_k-x_{k-l_i}\Vert$, we get

$$\begin{aligned} \underset{1\le i\le N}{\max }\Vert x_k-x_{k-l_i}\Vert&= \Vert x_k-x_{k-l_{i_*}}\Vert \\&\le (\Vert x_k-x_*\Vert + \Vert x_*-x_{k-l_{i_*}}\Vert ) . \end{aligned}$$

From the definition of $a_k^i$ in (7), we have that

$$\begin{aligned} \underset{1\le i\le N}{\max }\Vert a_k^i\Vert&\le \underset{1\le i\le N}{\max }\Vert x_k-x_{k-l_i}\Vert ,\quad l_i\in \{1,\ldots ,2\tau \} \nonumber \\&\le \left( \Vert x_k-x_*\Vert + \Vert x_*-x_{k-l_{i_*}}\Vert \right) \nonumber \\&\le \left( \Vert x_k-x_*\Vert + \underset{k-2\tau \le m \le k-1}{\max }{\mathbf {dist} }_m\right) \nonumber \\&\le 2\underset{k-2\tau \le m \le k}{\max }{\mathbf {dist} }_m, \end{aligned}$$

(36)

for some $l_i\in \{1,\ldots ,2\tau \}$.

Substituting in (35), and following developments similar to (36), we conclude that

$$\begin{aligned} \Vert b_k\Vert&\le 2N\underset{k-2\tau \le m \le k}{\max }{\mathbf {dist} }_m\nonumber \\ \Vert c_k\Vert&\le 2N\underset{k-3\tau \le m \le k}{\max }{\mathbf {dist} }_m \nonumber \\ \Vert d_k\Vert&\le 2(1+\gamma L)N\underset{k-2\tau \le m \le k}{\max }{\mathbf {dist} }_m \end{aligned}$$

(37)

Using (37), the sums can be easily bounded as shown below.

$$\begin{aligned} \sum _{m=k-K}^{k-1}\Vert a_m\Vert&\le 2NK\underset{k-K-2\tau \le j \le k-1}{\max }{\mathbf {dist} }_j \nonumber \\ \sum _{m=k-K}^{k-1}\Vert b_m\Vert&\le 2NK\underset{k-K-2\tau \le j \le k-1}{\max }{\mathbf {dist} }_j \nonumber \\ \sum _{m=k-K}^{k-1}\Vert c_m\Vert&\le 2NK\underset{k-K-3\tau \le j \le k-1}{\max }{\mathbf {dist} }_j \nonumber \\ \sum _{m=k-K}^{k-1}\Vert d_m\Vert&\le 2(1+\gamma L)NK\underset{k-K-2\tau \le j \le k-1}{\max }{\mathbf {dist} }_j \nonumber \\ \sum _{m=k-K}^{k-1}\Vert Sx_m\Vert&\le 2K\underset{k-K \le j \le k-1}{\max }{\mathbf {dist} }_j, \end{aligned}$$

(38)

the last inequality following from Corollary 4.1.

From the definition of $\varSigma _K(k)$ in ($\varSigma$) and from (38), by introducing

$$\begin{aligned} Y:=1+\gamma L+2\beta , \end{aligned}$$

we have that

$$\begin{aligned} \varSigma _K(k)&\le \sum _{m=k-K}^{k-1}(\Vert d_m\Vert + \beta \Vert c_m\Vert + \beta \Vert b_m\Vert + \Vert Sx_m\Vert ) \\&\le \underbrace{2K(YN+1)}_{W(K)}\underset{k-K-3\tau \le j \le k-1}{\max }{\mathbf {dist} }_j \end{aligned}$$

Since ($\varSigma$) is bounded, we can accordingly bound (13a) and (13b):

$$\begin{aligned} \Vert c_k-b_k\Vert&\le \eta N2\varSigma _{3\tau }(k) \nonumber \\&\le \eta N2W(3\tau )\underset{k-6\tau \le j \le k-1}{\max }{\mathbf {dist} }_j \nonumber \\&= \eta 2(3\tau )N(YN+1) \underset{k-6\tau \le j \le k-1}{\max }{\mathbf {dist} }_j \end{aligned}$$

(39a)

$$\begin{aligned} \Vert d_k\Vert&\le \eta N(1+\gamma L)\varSigma _{2\tau }(k) \nonumber \\&\le \eta N(1+\gamma L)W(2\tau )\underset{k-5\tau \le j \le k-1}{\max }{\mathbf {dist} }_j \nonumber \\&= \eta (1+\gamma L)4\tau N(YN+1)\underset{k-5\tau \le j \le k-1}{\max }{\mathbf {dist} }_j. \end{aligned}$$

(39b)

Using (39a) and (39b), $\Vert e_k\Vert$ from (13) can be bounded as

$$\begin{aligned} \Vert e_k\Vert&\le \Vert d_k\Vert + \beta \Vert c_k-b_k\Vert \nonumber \\&\le \eta \underbrace{N(YN+1)(4\tau (1+\gamma L)+6\beta \tau )}_{X}\underset{k-6\tau \le j \le k-1}{\max }{\mathbf {dist} }_j, \end{aligned}$$

(40)

where we bounded the quantities with the maximum delay that appeared in (39a) and (39b).

1.5 Proof of Theorem 5.1

Proof

Note that, (16) simplifies to

$$\begin{aligned} \eta ^2X^2\left( \frac{1}{\epsilon }+\frac{\eta (1+\delta )}{\delta }\right) < \nu -\epsilon . \end{aligned}$$

As a result, we need to find parameters $\eta ,\beta ,\gamma ,\delta ,\epsilon$ such that the following set of inequalities is satisfied:

$$\begin{aligned}&\eta ^2X^2\left( \frac{1}{\epsilon }+\frac{\eta (1+\delta )}{\delta }\right)< \nu -\epsilon ,\nonumber \\&Y = 1+\gamma L+2\beta ,\nonumber \\&X = N(YN+1)(4\tau (1+\gamma L)+6\beta \tau ),\nonumber \\&\delta>0,\nonumber \\&\epsilon>0,\nonumber \\&\beta >0,\nonumber \\&0<\gamma<\gamma _{\max },\nonumber \\&0<\rho <\frac{1}{2(1+\delta )}. \end{aligned}$$

(41)

The upper-bound $\gamma _{\max }$ ensures that the stepsize $\gamma$ is admissible (a possible option is, e.g., $\gamma _{\max }=2/L$ as proven in ‘Appendix A.6’). We start by noting that the values of $\delta$ and $\epsilon$ are irrelevant as long as they are positive. To this end, we can start by choosing $\epsilon$ such that $\nu -\epsilon >0$. From the inequality $\eta <1/(2(1+\delta ))$, it follows that

$$\begin{aligned} \frac{1}{\epsilon } + \frac{\eta (1+\delta )}{\delta } < \frac{2\delta +\epsilon }{2\delta \epsilon }, \end{aligned}$$

thus having

$$\begin{aligned} \eta ^2 < \frac{2\delta \epsilon (\nu -\epsilon )}{X^2(2\delta +\epsilon )}, \end{aligned}$$

from which the result follows. $\square$

1.6 Proof of Corollary 4.1.

Proof

• From Assumption 1, B is 1 / L-cocoercive, which means that $\gamma B$ is $1/\gamma L$-cocoercive. It follows from [24, Proposition 4.13] that $T_B=I-\gamma B$ is $\gamma L/2$-averaged. From [7, Proposition 4.25 (i)], it follows that $T_B$ is nonexpansive provided that $\gamma <2/L$. Finally, we conclude that T is nonexpansive as the composition of nonexpansive operators.

• From [7, Proposition 4.33], we have that S is 1 / 2-cocoercive if and only if T is nonexpansive, which is proven above.

• The claim is proven in [4, Proposition 2] for the case of the proximal gradient method. The proof below is essentially the same generalized for an operator T. From [7, Example 22.5], we have that if T is c-Lipschitz continuous for some $c\in [0,1)$ then $I-T$ is $(1-c)$-strongly monotone. Let us then prove that T is indeed Lipschitz continuous. For any $x\in \mathcal {H}$ and $x_*\in {\text {fix}} T$, it holds that:

  $$\begin{aligned} \Vert T_Bx-T_Bx_*\Vert ^2&= \Vert x-x_*\Vert ^2-2\gamma \langle x-x_*,Bx-Bx_*\rangle +\gamma ^2\Vert Bx-Bx_*\Vert ^2 \nonumber \\&\le \Vert x-x_*\Vert ^2-\gamma (2-\gamma L)\langle x-x_*,Bx-Bx_*\rangle \nonumber \\&\le \Vert x-x_*\Vert ^2-\mu \gamma (2-\gamma L)\Vert x-x_*\Vert ^2 \nonumber \\&= (1-2\gamma \mu +\mu \gamma ^2L)\Vert x-x_*\Vert ^2, \end{aligned}$$

  where the first inequality follows from the 1 / L-cocoercivity of B, while the second one from the $\mu$-strong monotonicity of B.

  Thus, $\Vert T_Bx-T_Bx_*\Vert \le \sqrt{(1-2\gamma \mu +\mu \gamma ^2L)}\Vert x-x_*\Vert$ and since $T_A$ is nonexpansive, we have that $\Vert Tx-Tx_*\Vert \le \sqrt{(1-2\gamma \mu +\mu \gamma ^2L)}\Vert x-x_*\Vert$. Finally, S is quasi-$\nu$-strongly monotone with $\nu =1-\sqrt{(1-2\gamma \mu +\mu \gamma ^2L)}$ for $\gamma <2/L$.

$\square$

1.7 Proof of Lemma 6.1.

Proof

Since $T_A=\mathbf{prox}_{\gamma g}$, it is firmly nonexpansive from [7, Proposition 12.27], and 1 / 2-averaged from [7, Proposition 4.2]. In addition, and as we saw in ‘Appendix A.6,’ $T_B=I-\gamma \nabla f$ is $\gamma L/2$-averaged from [24, Proposition 4.13]. Since $T=T_AT_B$, the composition is $\alpha$-averaged from [7, Proposition 4.24], with $\alpha =\max \left\{ \frac{2}{3},\frac{2}{1+\frac{2}{\gamma L}}\right\}$. Making use of [7, Proposition 4.25 (iii)], we have that for all $x\in {\mathbb {R} }^n,y\in {\mathbb {R} }^n$:

$$\begin{aligned}&\frac{1-\alpha }{\alpha }\Vert Sx-Sy\Vert ^2 +\Vert Tx-Ty\Vert ^2 \le \Vert x-y\Vert ^2 \\&(1-\alpha )\Vert Sx-Sy\Vert ^2 +\alpha \Vert (I-S)x-(I-S)y\Vert ^2 \le \alpha \Vert x-y\Vert ^2 \\&(1-\alpha )\Vert Sx-Sy\Vert ^2 +\alpha (\Vert x-y\Vert ^2-2\langle x-y,Sx-Sy\rangle +\Vert Sx-Sy\Vert ^2) \le \alpha \Vert x-y\Vert ^2 \end{aligned}$$

After performing the simplifications, we end up having

$$\begin{aligned} \langle x-y,Sx-Sy\rangle \ge \frac{1}{2\alpha }\Vert Sx-Sy\Vert ^2, \end{aligned}$$

which concludes the proof. $\square$