An elementary approach to tight worst case complexity analysis of gradient based methods

Abstract

This work presents a novel analysis that allows to achieve tight complexity bounds of gradient-based methods for convex optimization. We start by identifying some of the pitfalls rooted in the classical complexity analysis of the gradient descent method, and show how they can be remedied. Our methodology hinges on elementary and direct arguments in the spirit of the classical analysis. It allows us to establish some new (and reproduce known) tight complexity results for several fundamental algorithms including, gradient descent, proximal point and proximal gradient methods which previously could be proven only through computer-assisted convergence proof arguments.

Appendix A

In the following lemma we establish a useful relation between $\lambda _n$ and $T_n$.

Lemma 13

For a given positive integer k, let $\{t_n\}_{n=0}^{k-1}$ be the sequence defined by the recurrence relation (4.4). Then for any $n=0,1,\dots ,k-1$ it holds that

$$\begin{aligned} \lambda _n = \frac{LT_n}{2+LT_n}. \end{aligned}$$

(A.1)

Proof

Recall that according to (4.4), $Lt_0=\sqrt{2}$ and that $Lt_n$ is the positive root of

$$\begin{aligned} 0= & {} (Lt_n)^2+(LT_{n-1})Lt_n-2(LT_{n-1}+1)\\= & {} (Lt_n-1)^2+(LT_{n-1}+2)(Lt_n-1)-(LT_{n-1}+1), \end{aligned}$$

for any $n=1,2,\dots ,k-1$. Thus, $\lambda _0=Lt_0-1=\sqrt{2}-1$ and $\lambda _n=Lt_n-1$ is the positive root of

$$\begin{aligned} \lambda _n^2+(LT_{n-1}+2)\lambda _n-(LT_{n-1}+1)=0, \end{aligned}$$

for any $n=1,2,\dots ,k-1$. The last quadratic equation can be written as

$$\begin{aligned} \lambda _n\left( 2+LT_{n-1}+(\lambda _n+1)\right) = LT_{n-1}+\lambda _n+1. \end{aligned}$$

Using the relation $LT_n=LT_{n-1}+Lt_n=LT_{n-1}+\lambda _n+1$ we can write the above as

$$\begin{aligned} \lambda _n = \frac{LT_{n-1}+(\lambda _n+1)}{2+LT_{n-1}+(\lambda _n+1)} = \frac{LT_n}{2+LT_n}. \end{aligned}$$

The above holds for $n=0,1,\dots ,k-1$ (also for $n=0$) since

$$\begin{aligned} \lambda _0 = \sqrt{2}-1 =(\sqrt{2}-1)\frac{(\sqrt{2}+1)\sqrt{2}}{2+\sqrt{2}} =\frac{(2-1)\sqrt{2}}{2+\sqrt{2}} = \frac{LT_0}{2+LT_0}. \end{aligned}$$

$\square$

We are now ready to prove Lemma 6.

Proof of Lemma 6

We begin by proving that $\rho _n=0$ for all $n=0,1,\dots ,k-1$. Since $\lambda _0+1>0$ in order to examine whether $\rho _0=(\lambda _0+1)(\tau (\lambda _0)-\lambda _0)=0$ it is enough to verify that $\lambda _0=\tau (\lambda _0)$. The latter relation holds for $\lambda _0=\sqrt{2}-1$. Indeed,

$$\begin{aligned}&\sqrt{2}-1=\lambda _0=\tau (\lambda _0)=1-\frac{2\lambda _0^2}{1-\lambda _0} = 1-\frac{2(\sqrt{2}-1)^2}{2-\sqrt{2}}\\&\quad =1-\frac{\sqrt{2}(\sqrt{2}-1)^2}{\sqrt{2}-1}=\sqrt{2}-1. \end{aligned}$$

To show that $\rho _n=0$ for all $n=1,2,\dots ,k-1$ we first observe that due to Lemma 13

$$\begin{aligned} \lambda _{n-1} = \frac{LT_{n-1}}{2+LT_{n-1}}=\frac{LT_{n}-(\lambda _n+1)}{2+LT_{n}-(\lambda _n+1)}=\frac{LT_{n}\lambda _n-1}{LT_{n}\lambda _n+1}. \end{aligned}$$

Hence,

$$\begin{aligned} 1+\lambda _{n-1}= \frac{LT_{n}-\lambda _n+1+LT_{n}-\lambda _n-1}{LT_{n}-\lambda _n+1} = \frac{2(LT_n-\lambda _n)}{LT_{n}-\lambda _n+1}, \end{aligned}$$

and

$$\begin{aligned} 1-\lambda _{n-1}= \frac{LT_{n}-\lambda _n+1-\left( LT_{n}-\lambda _n-1\right) }{LT_{n}-\lambda _n+1} = \frac{2}{LT_{n}-\lambda _n+1}, \end{aligned}$$

and thus

$$\begin{aligned} \tau ^+(\lambda _{n-1})=1+\frac{2\lambda _{n-1}}{1-\lambda _{n-1}}=\frac{1+\lambda _{n-1}}{1-\lambda _{n-1}} = LT_n-\lambda _n. \end{aligned}$$

(A.2)

Using the above we can write for all $n=1,2,\dots ,k-1$

$$\begin{aligned} \begin{array}{rl} \rho _n &{}= LT_n\tau (\lambda _n)+LT_{n-1}\tau ^{+}(\lambda _{n-1})-(\lambda _n+1)\lambda _n\\ &{}=LT_n\left( 1-\frac{2\lambda _n^2}{1-\lambda _n}\right) +\left[ LT_n-(\lambda _n+1)\right] (LT_n-\lambda _n)-(\lambda _n+1)\lambda _n\\ &{}=LT_n\left( 1-\frac{2\lambda _n^2}{1-\lambda _n}\right) +LT_n\left[ LT_n-(2\lambda _n+1)\right] =LT_n\left[ LT_n-\frac{2\lambda _n^2+2\lambda _n(1-\lambda _n)}{1-\lambda _n} \right] \\ &{}=LT_n\left[ LT_n-\frac{2\lambda _n}{1-\lambda _n}\right] \overset{(A.1)}{=}LT_n\left[ LT_n-\frac{2LT_n/(2+LT_n)}{2/(2+LT_n)}\right] =0. \end{array} \end{aligned}$$

Finally, the alleged value of $\rho _k$ is established due to Lemma 13 as follows

$$\begin{aligned} \rho _k=LT_{k-1}\tau ^+(\lambda _{k-1})\overset{(A.2)}{=}LT_{k-1}(LT_k-\lambda _k)=LT_{k-1}(LT_{k-1}+1), \end{aligned}$$

which completes the proof.$\square$