Random Gradient-Free Minimization of Convex Functions

Abstract

In this paper, we prove new complexity bounds for methods of convex optimization based only on computation of the function value. The search directions of our schemes are normally distributed random Gaussian vectors. It appears that such methods usually need at most n times more iterations than the standard gradient methods, where n is the dimension of the space of variables. This conclusion is true for both nonsmooth and smooth problems. For the latter class, we present also an accelerated scheme with the expected rate of convergence $O\Big ({n^2 \over k^2}\Big )$, where k is the iteration counter. For stochastic optimization, we propose a zero-order scheme and justify its expected rate of convergence $O\Big ({n \over k^{1/2}}\Big )$. We give also some bounds for the rate of convergence of the random gradient-free methods to stationary points of nonconvex functions, for both smooth and nonsmooth cases. Our theoretical results are supported by preliminary computational experiments.

Appendix: Proofs of Statements of Sect. 2

Proof of Lemma 1

Denote $\psi (p) = \ln M_p$. This function is convex in p. Let us represent $p = (1-\alpha )\cdot 0 + \alpha \cdot 2$ (thus, $\alpha = {p \over 2}$). For $p \in [0,2]$, we have $\alpha \in [0,1]$. Therefore,

$$\begin{aligned} \begin{array}{rcl} \psi (p)\le & {} (1-\alpha ) \psi (0) + \alpha \psi (2) \; \mathop {=}\limits ^{(14)} \; {p \over 2} \ln n. \end{array} \end{aligned}$$

This is the upper bound (16). If $p \ge 2$, then $\alpha \ge 1$, and $\alpha \psi (2)$ becomes a lower bound for $\psi (p)$. It remains to prove the upper bound in (17).

Let us fix some $\tau \in (0,1)$. Note that for any $t \ge 0$ we have

$$\begin{aligned} \begin{array}{rcl} t^p \mathrm{e}^{-{\tau \over 2} t^2}\le & {} \left( p \over \tau e\right) ^{p/2}. \end{array} \end{aligned}$$

(80)

Therefore,

$$\begin{aligned} \begin{array}{rcl} M_p &{} = &{} {1 \over \kappa }\int \limits _E \Vert u \Vert ^p \mathrm{e}^{-{1 \over 2}\Vert u \Vert ^2} \mathrm{d}u \; = \; {1 \over \kappa } \int \limits _E \Vert u \Vert ^p \mathrm{e}^{-{\tau \over 2} \Vert u \Vert ^2} \mathrm{e}^{-{1-\tau \over 2} \Vert u \Vert ^2} \mathrm{d}u\\ \\ &{} \mathop {\le }\limits ^{(80)} &{} {1 \over \kappa } \left( p \over \tau e\right) ^{p/2} \int \limits _E \mathrm{e}^{-{1-\tau \over 2} \Vert u \Vert ^2} \mathrm{d}u\; = \; \left( p \over \tau e\right) ^{p/2} {1 \over (1-\tau )^{n/2}}. \end{array} \end{aligned}$$

The minimum of the right-hand side in $\tau \in (0,1)$ is attained at $\tau = {p \over p+n}$. Thus,

$$\begin{aligned} \begin{array}{rcl} M_p\le & {} \left( p \over e\right) ^{p/2} \left( 1 + {n \over p}\right) ^{p/2} \left( 1 + {p \over n}\right) ^{n/2} \; \le \; (p+n)^{p/2}. \end{array} \end{aligned}$$

$\square$

Proof of Theorem 1

Indeed, for any $x \in E$ we have $f_{\mu }(x) - f(x) = {1 \over \kappa } \int \limits _E [ f(x+\mu u) - f(x)] \mathrm{e}^{-{1 \over 2}\Vert u \Vert ^2} \mathrm{d}u$. Therefore, if $f \in C^{0,0}(E)$, then

$$\begin{aligned} \begin{array}{rcl} |f_{\mu }(x) - f(x) | &{} \le &{} {1 \over \kappa } \int \limits _E | f(x+\mu u) - f(x)| \mathrm{e}^{-{1 \over 2}\Vert u \Vert ^2} \mathrm{d}u \\ \\ &{} \le &{} {\mu L_0(f) \over \kappa } \int \limits _E \Vert u \Vert \mathrm{e}^{-{1 \over 2}\Vert u \Vert ^2} \mathrm{d}u \; \mathop {\le }\limits ^{(16)} \; \mu L_0(f) n^{1/2}. \end{array} \end{aligned}$$

Further, if f is differentiable at x, then

$$\begin{aligned} \begin{array}{rcl} f_{\mu }(x) - f(x)= & {} {1 \over \kappa } \int \limits _E [ f(x+\mu u) - f(x) - \mu \langle \nabla f(x), u \rangle ] \mathrm{e}^{-{1 \over 2}\Vert u \Vert ^2} \mathrm{d}u. \end{array} \end{aligned}$$

Therefore, if $f \in C^{1,1}(E)$, then

$$\begin{aligned} \begin{array}{rcl} |f_{\mu }(x) - f(x) |&\mathop {\le }\limits ^{(6)}&{\mu ^2 L_1(f) \over 2\kappa } \int \limits _E \Vert u \Vert ^2 \mathrm{e}^{-{1 \over 2}\Vert u \Vert ^2} \mathrm{d}u \; \mathop {=}\limits ^{(14)} \; {\mu ^2 L_1(f) \over 2} n. \end{array} \end{aligned}$$

Finally, if f is twice differentiable at x, then

$$\begin{aligned} \begin{array}{c} {1 \over \kappa } \int \limits _E [ f(x+\mu u) - f(x) - \mu \langle \nabla f(x), u \rangle - {\mu ^2 \over 2} \langle \nabla ^2 f(x) u, u \rangle ] \mathrm{e}^{-{1 \over 2}\Vert u \Vert ^2} \mathrm{d}u\\ \\ \mathop {=}\limits ^{(13)} \; f_{\mu }(x) - f(x) - {\mu ^2 \over 2} \langle \nabla ^2 f(x), B^{-1} \rangle . \end{array} \end{aligned}$$

Therefore, if $f \in C^{2,2}(E)$, then

$$\begin{aligned}&|f_{\mu }(x) - f(x) - {\mu ^2 \over 2} \langle \nabla ^2 f(x), B^{-1} \rangle | \; \mathop {\le }\limits ^{(7)} \; {\mu ^3 L_2(f) \over 6\kappa } \int \limits _E \Vert u \Vert ^3 \mathrm{e}^{-{1 \over 2}\Vert u \Vert ^2} \mathrm{d}u \\&\mathop {=}\limits ^{(17)} \; {\mu ^3 L_1(f) \over 6} (n+3)^{3/2}. \end{aligned}$$

$\square$

Proof of Lemma 2

Indeed, for all x and y in E, we have

$$\begin{aligned} \begin{array}{rcl} \Vert \nabla f_{\mu }(x) - \nabla f_{\mu }(y)\Vert _* &{} \mathop {\le }\limits ^{(21)} &{} {1 \over \kappa \mu } \int \limits _E |f(x + \mu u) - f(y+\mu u)| \, \Vert u \Vert \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2} \mathrm{d}u\\ \\ &{} \le &{} {1 \over \kappa \mu } L_0(f) \int \limits _E \Vert u \Vert \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2} \mathrm{d}u \cdot \Vert x - y \Vert . \end{array} \end{aligned}$$

It remains to apply (16). $\square$

Proof of Theorem 2

Let $\mu >0$. Since $f_{\mu }$ is convex, for all x and $y \in E$ we have

Taking now the limit as $\mu \rightarrow 0$, we prove the statement for $\mu = 0$. $\square$

Proof of Lemma 3

Indeed, for function $f \in C^{1,1}(E)$, we have

$$\begin{aligned} \begin{array}{rcl} \Vert \nabla f_{\mu }(x) - \nabla f(x) \Vert _* &{} \mathop {=}\limits ^{(25)} &{} \Big \Vert {1 \over \kappa } \int \limits _E \left( {f(x+\mu u ) - f(x) \over \mu } - \langle \nabla f(x), u \rangle \right) Bu \, \mathrm{e}^{-{1 \over 2}\Vert u \Vert ^2} \mathrm{d}u \Big \Vert _* \\ \\ &{} \le &{} {1 \over \kappa \mu } \int \limits _E |f(x+\mu u ) - f(x) - \mu \langle \nabla f(x), u \rangle | \, \Vert u \Vert \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2}\mathrm{d}u\\ \\ &{} \mathop {\le }\limits ^{(6)} &{} {\mu L_1(f) \over 2 \kappa } \int \limits _E \Vert u \Vert ^3 \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2}\mathrm{d}u \; \mathop {\le }\limits ^{(17)} \; {\mu \over 2 } L_1(f) (n+3)^{3/2}. \end{array} \end{aligned}$$

Let $f \in C^{2,2}(E)$. Denote $a_u(\tau ) = f(x+\tau u) - f(x) - \tau \langle \nabla f(x), u \rangle - {\tau ^2 \over 2} \langle \nabla ^2 f(x) u, u \rangle$. Then, $|a_u(\pm \mu )| \mathop {\le }\limits ^{(7)} {\mu ^3 \over 6} L_2(f) \Vert u \Vert ^3$. Since

we have

$$\begin{aligned} \Vert \nabla f_{\mu }(x) - \nabla f(x) \Vert _*\le & {} {1 \over 2\kappa \mu } \int \limits _E |f(x+\mu u ) - f(x-\mu u) - 2 \mu \langle \nabla f(x), u \rangle | \, \Vert u \Vert \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2}\mathrm{d}u\\ \\= & {} {1 \over 2\kappa \mu } \int \limits _E |a_u(\mu )-a_u(-\mu )| \, \Vert u \Vert \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2}\mathrm{d}u \\ \\\le & {} {\mu ^2 L_2(f) \over 6 \kappa } \int \limits _E \Vert u \Vert ^4 \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2}\mathrm{d}u \; \mathop {\le }\limits ^{(17)} \; {\mu ^2 \over 6 } L_2(f) (n+4)^{2}. \end{aligned}$$

$\square$

Proof of Lemma 4

Indeed,

$$\begin{aligned} \begin{array}{l} \Vert \nabla f(x) \Vert ^2_* \; \mathop {=}\limits ^{(13)} \; \Vert {1 \over \kappa } \int \limits _E \langle \nabla f(x) , u \rangle Bu \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2}\mathrm{d}u \Vert _*^2\\ \\ = \; \Vert {1 \over \kappa \mu } \int \limits _E \left( [f(x+\mu u) - f(x)] - [f(x+\mu u) - f(x) - \mu \langle \nabla f(x) , u \rangle ]\right) Bu \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2}\mathrm{d}u \Vert _*^2\\ \\ \mathop {\le }\limits ^{(26)} \; 2 \Vert \nabla f_{\mu }(x) \Vert _*^2 + {2 \over \mu ^2} \Vert {1 \over \kappa } \int \limits _E [f(x+\mu u) - f(x) - \mu \langle \nabla f(x) , u \rangle ] Bu \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2}\mathrm{d}u \Vert _*^2\\ \\ \le \; 2 \Vert \nabla f_{\mu }(x) \Vert _*^2 + {2 \over \mu ^2 \kappa } \int \limits _E [f(x+\mu u) - f(x) - \mu \langle \nabla f(x) , u \rangle ]^2 \Vert u\Vert ^2 \mathrm{e}^{-{1 \over 2} \Vert u \Vert ^2}\mathrm{d}u\\ \\ \mathop {\le }\limits ^{(6)} \; 2 \Vert \nabla f_{\mu }(x) \Vert _*^2 + {\mu ^2 \over 2} L_1^2(f) M_6. \end{array} \end{aligned}$$

It remains to use inequality (17). $\square$