On the Information-Adaptive Variants of the ADMM: An Iteration Complexity Perspective

Abstract

Designing algorithms for an optimization model often amounts to maintaining a balance between the degree of information to request from the model on the one hand, and the computational speed to expect on the other hand. Naturally, the more information is available, the faster one can expect the algorithm to converge. The popular algorithm of ADMM demands that objective function is easy to optimize once the coupled constraints are shifted to the objective with multipliers. However, in many applications this assumption does not hold; instead, often only some noisy estimations of the gradient of the objective—or even only the objective itself—are available. This paper aims to bridge this gap. We present a suite of variants of the ADMM, where the trade-offs between the required information on the objective and the computational complexity are explicitly given. The new variants allow the method to be applicable on a much broader class of problems where only noisy estimations of the gradient or the function values are accessible, yet the flexibility is achieved without sacrificing the computational complexity bounds.

Appendix

1.1 Proof of Proposition 2.3

Here we will prove Proposition 2.3. Before proceeding, let us present some technical lemmas without proof.

Lemma 7.1

Suppose function f is smooth and its gradient is Lipschitz continuous, i.e. (2) holds, then we have

$$\begin{aligned} f(x) \le f(y) + \nabla f(y)^{\top }(x-y) + \frac{L}{2}\Vert x - y\Vert ^2. \end{aligned}$$

(93)

Lemma 7.2

Suppose function f is smooth and convex, and its gradient is Lipschitz continuous with the constant L, i.e. (2) holds. Then we have

$$\begin{aligned} (x-y)^{\top }\nabla f(z) \le f(x)-f(y)+\frac{L}{2}\Vert z-y\Vert ^2 . \end{aligned}$$

(94)

Furthermore, if f is $\kappa$-strongly convex, we have

$$\begin{aligned} (x-y)^{\top }\nabla f(z) \le f(x)-f(y)+\frac{L}{2}\Vert z-y\Vert ^2-\frac{\kappa }{2}\Vert x-z\Vert ^2 . \end{aligned}$$

(95)

Lemma 7.1 is also known as the descent lemma which is well known; one can find its proof in e.g. [1]. Lemma 7.2 is similar to Fact 1 in [11] which follows from the (strong) convexity of f and Lemma 7.1.

Proof of (20)  in Proposition 2.3

Proof

First, by the optimality condition of the two subproblems in SGADM, we have

$$\begin{aligned}&(y-y^{k+1})^{\top }\left( \partial g(y^{k+1})-B^{\top }\left( \lambda ^k-\gamma (Ax^k+By^{k+1}-b)\right) -H(y^k-y^{k+1})\right) \\&\quad \ge 0, \quad \forall y\in \mathcal {Y}, \end{aligned}$$

and

$$\begin{aligned}&(x-x^{k+1})^{\top }\left( x^{k+1}-\left( x^k-\alpha _k \left( G(x^k,\xi ^{k+1})-A^{\top }(\lambda ^k-\gamma (Ax^k+By^{k+1}-b))\right) \right) \right) \\&\quad \ge 0, \quad \forall x\in \mathcal {X}, \end{aligned}$$

where $\partial g(y)$ is a subgradient of g at y. Using $\tilde{\lambda }^k=\lambda ^k-\gamma (Ax^k+By^{k+1}-b)$ and the definition of $\tilde{w}^k$ in (14), the above two inequalities are equivalent to

$$\begin{aligned} (y-\tilde{y}^{k})^{\top }\left( \partial g(\tilde{y}^{k})-B^{\top }\tilde{\lambda }^k-H(y^k-y^{k+1})\right) \ge 0, \quad \forall y\in \mathcal {Y}, \end{aligned}$$

(96)

and

$$\begin{aligned} (x-\tilde{x}^{k})^{\top }\left( \alpha _k\left( G(x^k,\xi ^{k+1})-A^{\top }\tilde{\lambda }^k\right) -(x^k-\tilde{x}^k)\right) \ge 0, \quad \forall x\in \mathcal {X}. \end{aligned}$$

(97)

Moreover,

$$\begin{aligned} (A\tilde{x}^k+B\tilde{y}^k-b)-\left( -A(x^k-\tilde{x}^k)+\frac{1}{\gamma }\left( \lambda ^k-\tilde{\lambda }^k\right) \right) =0 . \end{aligned}$$

Thus

$$\begin{aligned} (\lambda -\tilde{\lambda }^k)^{\top }(A\tilde{x}^k+B\tilde{y}^k-b) =\left( \lambda -\tilde{\lambda }^k\right) ^{\top }\left( -A\left( x^k-\tilde{x}^k\right) +\frac{1}{\gamma }\left( \lambda ^k-\tilde{\lambda }^k\right) \right) . \end{aligned}$$

(98)

By the convexity of g(y) and (96),

$$\begin{aligned} g(y)-g(\tilde{y}^k)+(y-\tilde{y}^{k})^{\top }\left( -B^{\top }\tilde{\lambda }^k\right) \ge (y-\tilde{y}^{k})^{\top }H(y^k-\tilde{y}^{k}), \quad \forall y\in \mathcal {Y}. \end{aligned}$$

(99)

Since $\delta _{k+1}=G(x^{k},\xi ^{k+1})-\nabla f(x^{k})$, and by (97) we have

$$\begin{aligned} (x-\tilde{x}^{k})^{\top }\left( \alpha _k(\nabla f(x^k)-A^{\top }\tilde{\lambda }^k)+\alpha _k\delta _{k+1}-(x^k-\tilde{x}^k)\right) \ge 0, \quad \forall x\in \mathcal {X}\end{aligned}$$

which leads to

$$\begin{aligned}&(x-\tilde{x}^{k})^{\top }\left( \alpha _k(\nabla f(x^k)-A^{\top }\tilde{\lambda }^k)\right) \\&\quad \ge (x-\tilde{x}^k)^{\top } \left( x^k-\tilde{x}^k\right) -\,\alpha _k\left( x-\tilde{x}^k\right) ^{\top }\delta _{k+1}, \quad \forall x\in \mathcal {X}. \end{aligned}$$

Using (94), the above further leads to

$$\begin{aligned}&\alpha _k(f(x)-f(\tilde{x}^k))+(x-\tilde{x}^{k})^{\top }(-\alpha _k A^{\top }\tilde{\lambda }^k) \nonumber \\&\quad \ge (x-\tilde{x}^k)^{\top }\left( x^k-\tilde{x}^k\right) -\alpha _k \left( x-\tilde{x}^k\right) ^{\top }\delta _{k+1}-\frac{\alpha _kL}{2}\Vert x^k-\tilde{x}^k\Vert ^2, \quad \forall x\in \mathcal {X}.\nonumber \\ \end{aligned}$$

(100)

Furthermore,

$$\begin{aligned} (x-\tilde{x}^k)^{\top }\delta _{k+1}= & {} (x-x^k)^{\top }\delta _{k+1}+(x^k-\tilde{x}^k)^{\top }\delta _{k+1} \nonumber \\\le & {} (x-x^k)^{\top }\delta _{k+1}+\frac{\eta _k}{2}\Vert x^k-\tilde{x}^k\Vert ^2+\frac{\Vert \delta _{k+1}\Vert ^2}{2\eta _k}. \end{aligned}$$

(101)

Substituting (101) in (100), and dividing both sides by $\alpha _k$, we get

$$\begin{aligned}&f(x)-f(\tilde{x}^k)+(x-\tilde{x}^{k})^{\top }(-A^{\top }\tilde{\lambda }^k) \nonumber \\&\quad \ge \frac{(x-\tilde{x}^k)^{\top }(x^k-\tilde{x}^k)}{\alpha _k} -(x-x^k)^{\top }\delta _{k+1}-\frac{\Vert \delta _{k+1}\Vert ^2}{2\eta _k}-\frac{\eta _k+L}{2}\Vert x^k-\tilde{x}^k\Vert ^2.\qquad \end{aligned}$$

(102)

Finally, (20) follows by summing (102), (99), and (98). $\square$

Now we show the second statement in Proposition 2.3.

Proof of (21)  in Proposition 2.3

Proof

First, by (15), we have $P(w^k-\tilde{w}^k)=(w^k-w^{k+1})$, and so

$$\begin{aligned} (w-\tilde{w}^k)^{\top } Q_k(w^k-\tilde{w}^k)=(w-\tilde{w}^k)^{\top } M_kP(w^k-\tilde{w}^k)=(w-\tilde{w}^k)^{\top } M_k(w^k-w^{k+1}) . \end{aligned}$$

Applying the identity

$$\begin{aligned} (a-b)^{\top }M_k(c-d)=\frac{1}{2}\left( \Vert a-d\Vert _{M_k}^2-\Vert a-c\Vert _{M_k}^2\right) +\frac{1}{2}\left( \Vert c-b\Vert _{M_k}^2-\Vert d-b\Vert _{M_k}^2\right) \end{aligned}$$

to the term $(w-\tilde{w}^k)^{\top }M_k(w^k-w^{k+1})$, we obtain

$$\begin{aligned}&(w-\tilde{w}^k)^{\top }M_k(w^k-w^{k+1}) \nonumber \\&\quad = \frac{1}{2}\left( \Vert w-w^{k+1}\Vert _{M_k}^2-\Vert w-w^k\Vert _{M_k}^2\right) +\frac{1}{2}\left( \Vert w^k-\tilde{w}^k\Vert _{M_k}^2-\Vert w^{k+1}-\tilde{w}^k\Vert _{M_k}^2\right) .\nonumber \\ \end{aligned}$$

(103)

Using (15) again, we have

$$\begin{aligned}&\Vert w^k-\tilde{w}^k\Vert _{M_k}^2-\Vert w^{k+1}-\tilde{w}^k\Vert _{M_k}^2\nonumber \\&\quad =\Vert w^k-\tilde{w}^k\Vert _{M_k}^2-\Vert (w^k-\tilde{w}^k)-(w^k-w^{k+1})\Vert _{M_k}^2\nonumber \\&\quad =\Vert w^k-\tilde{w}^k\Vert _{M_k}^2-\Vert (w^k-\tilde{w}^k)-P(w^k-\tilde{w}^{k})\Vert _{M_k}^2\nonumber \\&\quad =(w^k-\tilde{w}^k)^{\top }(2M_kP-P^{\top }M_kP)(w^k-\tilde{w}^k). \end{aligned}$$

(104)

Note that $Q_k=M_kP$ and the definition of those matrices (see (13)), we have

$$\begin{aligned} 2M_kP-P^{\top }M_kP=2Q_k-P^{\top }Q_k=\left( \begin{array}{ccc} H &{} 0 &{} 0 \\ 0 &{} \frac{1}{\alpha _k}I_{n_x}-\gamma A^{\top }A &{} A^{\top } \\ 0 &{} -A &{} \frac{1}{\gamma }I_m \\ \end{array} \right) . \end{aligned}$$

As a result,

$$\begin{aligned}&(w^k-\tilde{w}^k)^{\top }(2M_kP-P^{\top }M_kP)(w^k-\tilde{w}^k)\nonumber \\&\quad = \Vert y^k-\tilde{y}^k\Vert _H^2+\frac{1}{\gamma }\Vert \lambda ^k-\tilde{\lambda }^k\Vert ^2+(x^k-\tilde{x}^k)^{\top }\left( \frac{1}{\alpha _k}I_{n_x}-\gamma A^{\top }A\right) (x^k-\tilde{x}^k) \nonumber \\&\quad \ge (x^k-\tilde{x}^k)^{\top }\left( \frac{1}{\alpha _k}I_{n_x}-\gamma A^{\top }A\right) (x^k-\tilde{x}^k). \end{aligned}$$

(105)

Combining (105), (104), and (103), the desired inequality (21) follows. $\square$

1.2 Proof of Proposition 3.1

We first show the first part of Proposition 3.1.

Proof of (49)  in Proposition 3.1

Proof

by the optimality condition of the two subproblems in SGALM, we have

$$\begin{aligned}&(y-y^{k+1})^{\top }\left( y^{k+1}-y^k+\beta _k\left( S_g(y^k,\zeta ^{k+1})-B^{\top }(\lambda ^k-\gamma (Ax^k+By^{k}-b))\right) \right) \\&\quad \ge 0, \quad \forall y\in \mathcal {Y}, \end{aligned}$$

and also

$$\begin{aligned}&(x-x^{k+1})^{\top }\left( x^{k+1}-x^k+\alpha _k \left( S_f(x^k,\xi ^{k+1})-A^{\top }(\lambda ^k-\gamma (Ax^k+By^{k+1}-b))\right) \right) \\&\quad \ge 0, \quad \forall x\in \mathcal {X}. \end{aligned}$$

Using $\tilde{\lambda }^k=\lambda ^k-\gamma (Ax^k+By^{k+1}-b)$ and the definition of $\tilde{w}^k$, the above two inequalities are equivalent to

$$\begin{aligned} (y-\tilde{y}^{k})^{\top }\left( \beta _k\left( S_g(y^k,\zeta ^{k+1})-B^{\top }\tilde{\lambda }^k\right) -(I_{n_y}-\beta _k\gamma B^{\top }B)(y^k-\tilde{y}^k)\right) \ge 0, \quad \forall y\in \mathcal {Y}, \end{aligned}$$

(106)

and

$$\begin{aligned} (x-\tilde{x}^{k})^{\top }\left( \alpha _k(S_f(x^k,\xi ^{k+1})-A^{\top }\tilde{\lambda }^k)-(x^k-\tilde{x}^k)\right) \ge 0, \quad \forall x\in \mathcal {X}. \end{aligned}$$

(107)

Also,

$$\begin{aligned} (\lambda -\tilde{\lambda }^k)^{\top }(A\tilde{x}^k+B\tilde{y}^k-b)=(\lambda -\tilde{\lambda }^k)^{\top } \left( -A(x^k-\tilde{x}^k)+\frac{1}{\gamma }(\lambda ^k-\tilde{\lambda }^k)\right) . \end{aligned}$$

(108)

Since $\delta ^f_{k+1}=S_f(x^{k},\xi ^{k+1})-\nabla f(x^{k})$ and using (107), similar to (100) and (101) we have

$$\begin{aligned}&f(x)-f(\tilde{x}^k)+(x-\tilde{x}^{k})^{\top }(-A^{\top }\tilde{\lambda }^k) \nonumber \\&\quad \ge \frac{(x-\tilde{x}^k)^{\top }(x^k-\tilde{x}^k)}{\alpha _k} -(x-x^k)^{\top }\delta ^f_{k+1}-\frac{\Vert \delta ^f_{k+1}\Vert ^2}{2\eta _k}-\frac{\eta _k+L}{2}\Vert x^k-\tilde{x}^k\Vert ^2.\qquad \quad \end{aligned}$$

(109)

Similarly, since $\delta ^g_{k+1}=S_g(y^{k},\zeta ^{k+1})-\nabla g(y^{k})$ and using (106), we also have

$$\begin{aligned}&g(y)-g(\tilde{y}^k)+(y-\tilde{y}^{k})^{\top }(-B^{\top }\tilde{\lambda }^k) \nonumber \\&\quad \ge (y-\tilde{y}^k)^{\top }\left( \frac{1}{\beta _k}I_{n_y}-\gamma B^{\top }B\right) (y^k-\tilde{y}^k) \nonumber \\&\qquad -(y-y^k)^{\top }\delta ^g_{k+1}-\frac{\Vert \delta ^g_{k+1}\Vert ^2}{2\eta _k}-\frac{\eta _k+L}{2}\Vert y^k-\tilde{y}^k\Vert ^2. \end{aligned}$$

(110)

Finally, (49) follows by summing (110), (109), and (108). $\square$

Notice that $\hat{Q}_k=\hat{M}_kP$ and

$$\begin{aligned} 2M_kP-P^{\top }M_kP= & {} \left( \begin{array}{ccc} H_k &{} 0 &{} 0 \\ 0 &{} \frac{1}{\alpha _k}I_{n_x}-\gamma A^{\top }A &{} A^{\top } \\ 0 &{} -A &{} \frac{1}{\gamma }I_m \\ \end{array} \right) \\= & {} \left( \begin{array}{ccc} \frac{1}{\alpha _k}I_{n_x}-\gamma B^{\top }B &{} 0 &{} 0 \\ 0 &{} \frac{1}{\alpha _k}I_{n_x}-\gamma A^{\top }A &{} A^{\top } \\ 0 &{} -A &{} \frac{1}{\gamma }I_m \\ \end{array} \right) . \end{aligned}$$

Inequality (50) in Proposition 3.1 follows similarly as the derivation of (21) in Proposition 2.3.

1.3 Properties of the Smoothing Function

In this subsection, we will prove Lemma 4.1. Before that, we need some technical preparations which are summarized in the following lemma.

Lemma 7.3

Let $\alpha (n)$ be the volume of the unit ball in $\mathbf {R}^n$, and $\beta (n)$ be the surface area of the unit sphere in $\mathbf {R}^n$. We also denote B, and $S_p$, to be the unit ball and unit sphere respectively.

• (a) If $M_p$ is defined as $M_p=\frac{1}{\alpha (n)}\int _{v \in B}\Vert v\Vert ^pdv$, we have

  $$\begin{aligned} M_p=\frac{n}{n+p}. \end{aligned}$$

  (111)

• (b) Let I be the identity matrix in $\mathbf {R}^{n\times n}$, then

  $$\begin{aligned} \int _{S_p}vv^{\top }dv=\frac{\beta (n)}{n}I. \end{aligned}$$

  (112)

Proof

For (a), we can directly compute $M_p$ by using the polar coordinates,

$$\begin{aligned} M_p=\frac{1}{\alpha (n)}\int _{B}\Vert v\Vert ^pdv=\frac{1}{\alpha (n)}\int _{0}^1\int _{S_p}r^p r^{n-1}drd\theta =\frac{1}{n+p}\frac{\beta (n)}{\alpha (n)}=\frac{n}{n+p}. \end{aligned}$$

For (b), Let $V=vv^{\top }$, then we know that $V_{ij}=v_i v_j$. Therefore, if $i\ne j$, by the symmetry of the unit sphere $S_p$ (i.e. if $v\in S_p,v=(v_1,v_2,\ldots ,v_n)$, then $w\in S_p$ for all $w=(\pm v_1,\pm v_2,\ldots ,\pm v_n)$), we have

$$\begin{aligned} \int _{S_p}V_{ij}dv=\int _{S_p}v_i v_jdv=\int _{S_p}-v_i v_jdv=\int _{S_p}-V_{ij}dv. \end{aligned}$$

Thus, we obtain $\int _{S_p}V_{ij}dv=0$.

If $i=j$, we know that $V_{ii}=v_i^2$. Since we already know that

$$\begin{aligned} \int _{S_p}(v_1^2+v_2^2+\ldots +v_n^2)dv=\int _{S_p}\Vert v\Vert ^2dv=\beta (n). \end{aligned}$$

Then, by symmetry, we have

$$\begin{aligned} \int _{S_p}v_1^2dv=\int _{S_p}v_2^2dv=\ldots =\int _{S_p}v_n^2dv=\frac{\beta (n)}{n}. \end{aligned}$$

Thus we also have $\int _{S_p}V_{ii}^2dv=\frac{\beta (n)}{n}$, for $i=1,2,\ldots ,n$. Therefore, $\int _{S_p}vv^{\top }dv=\frac{\beta (n)}{n}I$. $\square$

By the next three propositions, the part (b) of Lemma 4.1 is shown; for part (a) and (c) of Lemma 4.1, the proof can be found in [48].

Proposition 7.4

If $f\in C^1_L(\mathbf {R}^n)$, then

$$\begin{aligned} |f_\mu (x)-f(x)|\le \frac{L\mu ^2}{2}. \end{aligned}$$

(113)

Proof

Since $f\in C^1_L(\mathbf {R}^n)$, we have

$$\begin{aligned} |f_\mu (x)-f(x)|= & {} \left| \frac{1}{\alpha (n)}\int _{B}f(x+\mu v)dv-f(x)\right| \\= & {} \left| \frac{1}{\alpha (n)}\int _{B}(f(x+\mu v)-f(x)-\nabla f(x)^{\top }\mu v)dv\right| \\\le & {} \int _{B}\left| (f(x+\mu v)-f(x)-\nabla f(x)^{\top }\mu v)\right| dv \\\le & {} \int _{B}\frac{L\mu ^2}{2}\Vert v\Vert ^2dv \\{\mathop {=}\limits ^{(111)}}&\frac{L\mu ^2}{2}\frac{n}{n+2}\le \frac{L\mu ^2}{2} . \end{aligned}$$

$\square$

Proposition 7.5

If $f\in C^1_L(\mathbf {R}^n)$, then

$$\begin{aligned} \Vert \nabla f_\mu (x)-\nabla f(x)\Vert \le \frac{\mu nL}{2} . \end{aligned}$$

(114)

Proof

$$\begin{aligned}&\Vert \nabla f_\mu (x)-\nabla f(x)\Vert \\&\quad =\left\| \frac{1}{\beta (n)}\left[ \frac{n}{\mu }\int _{S_p}f(x+\mu v)vdv\right] -\nabla f(x)\right\| \\&\quad {\mathop {=}\limits ^{(112)}}\left\| \frac{1}{\beta (n)}\left[ \frac{n}{\mu }\int _{S_p}f(x+\mu v)vdv-\int _{S_p}\frac{n}{\mu }f(x)vdv-\int _{S_p}\frac{n}{\mu }\langle \nabla f(x),\mu v\rangle vdv\right] \right\| \\&\quad \le \frac{n}{\beta (n)\mu }\int _{S_p}|f(x+\mu v)-f(x)-\langle \nabla f(x),\mu v\rangle |\Vert v\Vert dv \\&\quad \le \frac{n}{\beta (n)\mu }\frac{L\mu ^2}{2}\int _{S_p}\Vert v\Vert ^3dv=\frac{\mu n L}{2} . \end{aligned}$$

$\square$

Proposition 7.6

If $f\in C^1_L(\mathbf {R}^n)$, and the ${\mathcal {SZO}}$ defined as $g_\mu (x)=\frac{n}{\mu }[f(x+\mu v)-f(x)]v$, then we have

$$\begin{aligned} \mathbf{\mathsf E}_v\left[ \Vert g_\mu (x)\Vert ^2\right] \le 2n\Vert \nabla f(x)\Vert ^2+\frac{\mu ^2}{2}L^2n^2 . \end{aligned}$$

(115)

Proof

$$\begin{aligned} \mathbf{\mathsf E}_v[\Vert g_\mu (x)\Vert ^2]= & {} \frac{1}{\beta (n)}\int _{S_p}\frac{n^2}{\mu ^2}|f(x+\mu v)-f(x)|^2\Vert v\Vert ^2dv \\= & {} \frac{n^2}{\beta (n)\mu ^2}\int _{S_p}\left[ f(x+\mu v)-f(x)-\langle \nabla f(x),\mu v\rangle +\langle \nabla f(x),\mu v\rangle \right] ^2dv \\\le & {} \frac{n^2}{\beta (n)\mu ^2}\int _{S_p} \left[ 2\left( f(x+\mu v)-f(x)-\langle \nabla f(x),\mu v\rangle \right) ^2\right. \\&\left. +\,2\left( \langle \nabla f(x),\mu v\rangle \right) ^2\right] dv \\\le & {} \frac{n^2}{\beta (n)\mu ^2}\left[ \int _{S_p}2\left( \frac{L\mu ^2}{2}\Vert v\Vert ^2\right) ^2dv+2\mu ^2\int _{S_p}\nabla f(x)^{\top }vv^{\top }\nabla f(x)dv\right] \\&{\mathop {=}\limits ^{(112)}}\frac{n^2}{\beta (n)\mu ^2}\left[ \frac{L^2\mu ^4}{2}\beta (n)+2\mu ^2\frac{\beta (n)}{n}\Vert \nabla f(x)\Vert ^2\right] \\= & {} 2n\Vert \nabla f(x)\Vert ^2+\frac{\mu ^2}{2}L^2n^2 . \end{aligned}$$

$\square$