General multilevel adaptations for stochastic approximation algorithms of Robbins–Monro and Polyak–Ruppert type

Abstract

In this article we present and analyse new multilevel adaptations of classical stochastic approximation algorithms for the computation of a zero of a function $f:D \rightarrow {{\mathbb {R}}}^d$ defined on a convex domain $D\subset {{\mathbb {R}}}^d$, which is given as a parameterised family of expectations. The analysis of the error and the computational cost of our method is based on similar assumptions as used in Giles (Oper Res 56(3):607–617, 2008) for the computation of a single expectation. Additionally, we essentially only require that f satisfies a classical contraction property from stochastic approximation theory. Under these assumptions we establish error bounds in pth mean for our multilevel Robbins–Monro and Polyak–Ruppert schemes that decay in the computational time as fast as the classical error bounds for multilevel Monte Carlo approximations of single expectations known from Giles (Oper Res 56(3):607–617, 2008). Our approach is universal in the sense that having multilevel implementations for a particular application at hand it is straightforward to implement the corresponding stochastic approximation algorithm.

Appendix

Let $(\Omega ,{\mathcal {F}},{P})$ be a probability space endowed with a filtration $({\mathcal {F}}_n)_{n\in {{\mathbb {N}}}_0}$ and let $\Vert \cdot \Vert$ denote a Hilbert space norm on ${{\mathbb {R}}}^d$.

In this section we provide pth mean estimates for an adapted d-dimensional dynamical system $(\zeta _n)_{n\in {{\mathbb {N}}}_0}$ with the property that for each $n\in {{\mathbb {N}}}$, $\zeta _n$ is a zero-mean perturbation of a previsible proposal $\xi _n$ being comparable in size to $\zeta _{n-1}$. More formally, we assume that there exist a previsible d-dimensional process $(\xi _n)_{n\in {{\mathbb {N}}}}$, a d-dimensional martingale $(M_n)_{n\in {{\mathbb {N}}}_0}$ with $M_0=\zeta _0$ and a constant $c\ge 0$ such that for all $n\in {{\mathbb {N}}}$

$$\begin{aligned} \begin{aligned} \zeta _{n}&= \xi _{n}+\Delta M_n,\\ \Vert \xi _n\Vert&\le \Vert \zeta _{n-1}\Vert \vee c, \end{aligned} \end{aligned}$$

(106)

where $\Delta M_n = M_n-M_{n-1}$. Note that necessarily $\xi _n={{\mathbb {E}}}[\zeta _n|{\mathcal {F}}_{n-1}]$.

Theorem 5.1

Assume that $(\zeta _n)_{n\in {{\mathbb {N}}}_0}$ is an adapted d-dimensional process, which satisfies  (106), and let $p\in [1,\infty )$. Then there exists a constant $\kappa \in (0,\infty )$, which only depends on p, such that for every $n\in {{\mathbb {N}}}_0$,

$$\begin{aligned} {{\mathbb {E}}}\left[ \max _{0\le k\le n}\Vert \zeta _k\Vert ^p\right] \le \kappa \, \bigl ( {{\mathbb {E}}}\bigl [ [M]_n^{p/2}\bigr ] + c^p\bigr ), \end{aligned}$$

where

$$\begin{aligned} {}[M]_n=\sum _{k=1}^{n} \Vert \Delta M_k\Vert ^2 +\Vert M_0\Vert ^2. \end{aligned}$$

Proof

Fix $p\in [1,\infty )$.

We first consider the case where $c=0$. Recall that by the Burkholder-Davis-Gundy inequality there exists a constant $\bar{\kappa }>0$ depending only on p such that for every d-dimensional martingale $(M_n)_{n\in {{\mathbb {N}}}_0}$,

$$\begin{aligned} {{\mathbb {E}}}\bigl [\max _{0\le k\le n}\Vert M_k\Vert ^p\bigr ]\le {\bar{\kappa }}\, {{\mathbb {E}}}\bigl [ [M]_n^{p/2}\bigr ]. \end{aligned}$$

We fix a time horizon $T\in {{\mathbb {N}}}_0$ and prove the statement of the theorem with $\kappa = {\bar{\kappa }}$ by induction: we say that the statement holds up to time $t\in \{0,\dots ,T\}$, if for every d-dimensional adapted process $(\zeta _n)_{n\in {{\mathbb {N}}}_0}$, for every d-dimensional previsible process $(\xi _n)_{n\in {{\mathbb {N}}}}$ and for every d-dimensional martingale $(M_n)_{n\in {{\mathbb {N}}}_0}$ with

one has

$$\begin{aligned} {{\mathbb {E}}}\left[ \max _{0\le n\le T}\Vert \zeta _n\Vert ^p\right] \le {\bar{\kappa }}\, {{\mathbb {E}}}\bigl [ [M]_T^{p/2}\bigr ]. \end{aligned}$$

Clearly, the statement is satisfied up to time 0 as a consequence of the Burkholder–Davis–Gundy inequality. Next, suppose that the statement is satisfied up to time $t\in \{0,\dots ,T-1\}$. Let $(\zeta _n)_{n\in {{\mathbb {N}}}_0}$ be a d-dimensional adapted process, $(\xi _n)_{n\in {{\mathbb {N}}}}$ be a d-dimensional previsible process and $(M_n)_{n\in {{\mathbb {N}}}_0}$ be a d-dimensional martingale satisfying property ($C_{t+1}$). Consider any ${\mathcal {F}}_{t}$-measurable random orthonormal transformation U on $({{\mathbb {R}}}^d,\Vert \cdot \Vert )$ and put

$$\begin{aligned} \zeta ^U_n={\left\{ \begin{array}{ll} \zeta _n,&{}\quad \text { if }n\le t,\\ \zeta _{t}+U(M_{n}-M_{t}), &{}\quad \text { if }n>t\end{array}\right. } \end{aligned}$$

as well as

$$\begin{aligned} M^U_n={\left\{ \begin{array}{ll} M_n,&{}\quad \text { if }n\le t,\\ M_{t}+ U(M_{n}-M_{t}), &{}\quad \text { if }n>t.\end{array}\right. } \end{aligned}$$

Then it is easy to check that $(M^U_n)_{n\in {{\mathbb {N}}}_0}$ is a martingale with $[M^U]_n = [M]_n$ for all $n\in {{\mathbb {N}}}$. Furthermore, $(\zeta ^U_n)_{n\in {{\mathbb {N}}}_0}$ is adapted and the triple $(\zeta ^U,\xi , M^U)$ satisfies property ($C_t$). Hence, by the induction hypothesis,

$$\begin{aligned} {{\mathbb {E}}}\bigl [\max _{0\le n\le T}\Vert \zeta ^U_n\Vert ^p\bigr ] \le {\bar{\kappa }}\, {{\mathbb {E}}}\bigl [ [M^U]_T^{p/2}\bigr ] ={\bar{\kappa }}\, {{\mathbb {E}}}\bigl [ [M]_T^{p/2}\bigr ]. \end{aligned}$$

(107)

Note that for any such random orthonormal transformation U, the norm of the random variable $\zeta _n^U$ is the same as the norm of the variable ${\bar{\zeta }}_n^U$ given by

$$\begin{aligned} {\bar{\zeta }}^U_n={\left\{ \begin{array}{ll} \zeta _n,&{}\quad \text { if }n\le t,\\ U^* \zeta _{t}+ M_{n}-M_{t}, &{}\quad \text { if }n>t,\end{array}\right. } \end{aligned}$$

whence

$$\begin{aligned} {{\mathbb {E}}}\left[ \max _{0\le n\le T}\Vert {\bar{\zeta }}^U_n\Vert ^p\right] = {{\mathbb {E}}}\left[ \max _{0\le n\le T}\Vert \zeta ^U_n\Vert ^p\right] . \end{aligned}$$

(108)

Clearly, we can choose an ${\mathcal {F}}_{t}$-measurable random orthonormal transformation U on $({{\mathbb {R}}}^d,\Vert \cdot \Vert )$ such that

$$\begin{aligned} U^* \zeta _t = \frac{\Vert \zeta _t\Vert }{\Vert \xi _{t+1}\Vert } \xi _{t+1} \end{aligned}$$

holds on $\{\xi _{t+1}\ne 0\}$. Let

$$\begin{aligned} \alpha = \frac{\Vert \xi _{t+1}\Vert +\Vert \zeta _t\Vert }{2\Vert \zeta _t\Vert }\cdot 1_{\{\zeta _t\ne 0\}}. \end{aligned}$$

Then $\alpha$ is ${\mathcal {F}}_{t}$-measurable and takes values in [0, 1] since $\Vert \xi _{t+1}\Vert \le \Vert \zeta _t\Vert$. Moreover, we have $\xi _{t+1} = \alpha U^* \zeta _t + (1-\alpha ) (-U)^* \zeta _t$ so that by property ($C_{t+1}$) of the triple $(\zeta ,\xi ,M)$,

$$\begin{aligned} \zeta _n= \xi _{t+1} + M_n-M_t = \alpha {\bar{\zeta }}^U_n+(1-\alpha ) {\bar{\zeta }}_n^{-U} \end{aligned}$$

for $n= t+1,\dots ,T$. Note that $\zeta _n=\zeta ^U_n=\zeta _n^{-U}$ for $n=0,\dots ,t$. By convexity of $\Vert \cdot \Vert ^p$ we thus obtain

$$\begin{aligned} \max _{0\le n\le T}\Vert {\bar{\zeta }}^U_n\Vert ^p&= \max _{0\le n\le T}\Vert \alpha {\bar{\zeta }}^U_n+ (1-\alpha ) {\bar{\zeta }}^{-U}_n\Vert ^p \\&\le \alpha \max _{0\le n\le T}\Vert {\bar{\zeta }}^U_n\Vert ^p + (1-\alpha )\max _{0\le n\le T}\Vert {\bar{\zeta }}^{-U}_n\Vert ^p. \end{aligned}$$

Hence

$$\begin{aligned} {{\mathbb {E}}}\left[ \max _{0\le n\le T}\Vert \zeta _{n}\Vert ^p|{\mathcal {F}}_{t}\right]&\le \alpha {{\mathbb {E}}}\left[ \max _{0\le n\le T}\Vert {\bar{\zeta }}^U_n\Vert ^p|{\mathcal {F}}_{t}\right] + (1-\alpha ){{\mathbb {E}}}\left[ \max _{0\le n\le T}\Vert {\bar{\zeta }}^{-U}_n\Vert ^p|{\mathcal {F}}_{t}\right] \\&\le {{\mathbb {E}}}\left[ \max _{0\le n\le T}\Vert {\bar{\zeta }}^{U'}_n\Vert ^p|{\mathcal {F}}_{t}\right] , \end{aligned}$$

where $U'$ is the ${\mathcal {F}}_{t}$-measurable random orthonormal transformation given by

$$\begin{aligned} U'(\omega )= {\left\{ \begin{array}{ll} U(\omega )&{}\quad \text { if }\omega \in \bigl \{ {{\mathbb {E}}}[\max _{0\le n\le T}\Vert {\bar{\zeta }}^U_n\Vert ^p|{\mathcal {F}}_{t}]\ge {{\mathbb {E}}}[\max _{0\le n\le T}\Vert {\bar{\zeta }}^{- U}_n\Vert ^p|{\mathcal {F}}_{t}]\bigr \},\\ -U(\omega ) &{}\quad \text { otherwise}. \end{array}\right. } \end{aligned}$$

Applying (107) and (108) with $U=U'$ finishes the induction step.

Next, we consider the case of $c > 0$. Suppose that $\zeta ,\xi$ and M are as stated in the theorem. For $n\in {{\mathbb {N}}}$ we put

$$\begin{aligned} {\tilde{\xi }}_n = (1-c/\Vert \xi _n\Vert )_+ \cdot \xi _n \end{aligned}$$

and

$$\begin{aligned} {\tilde{\zeta }}_n = {\tilde{\xi }}_n + \Delta M_n. \end{aligned}$$

Furthermore, let ${\tilde{\zeta }}_0=\zeta _0=M_0$. We will show that the triple $({\tilde{\zeta }},{\tilde{\xi }},M)$ satisfies (106) with $c=0$. Clearly, $({\tilde{\zeta }}_n)_{n\in {{\mathbb {N}}}_0}$ is adapted and $(\tilde{\xi }_n)_{n\in {{\mathbb {N}}}}$ is previsible. Moreover, one has for $n\in {{\mathbb {N}}}$ on $\{\Vert \xi _n\Vert \ge c\}$ that

$$\begin{aligned} \Vert {\tilde{\xi }}_n\Vert&= \Vert \xi _n\Vert -c\le \Vert \zeta _{n-1}\Vert -c=\Vert \tilde{\zeta }_{n-1}+\xi _{n-1}-{\tilde{\xi }}_{n-1}\Vert -c\\&\le \Vert {\tilde{\zeta }}_{n-1}\Vert +\Vert \xi _{n-1}-{\tilde{\xi }}_{n-1}\Vert -c = \Vert {\tilde{\zeta }}_{n-1}\Vert \end{aligned}$$

and on $\{\Vert \xi _n\Vert < c\}$ that $\Vert {\tilde{\xi }}_n\Vert =0\le \Vert \tilde{\zeta }_{n-1}\Vert$. We may thus apply Theorem 5.1 with $c=0$ to obtain that for every $n\in {{\mathbb {N}}}$,

$$\begin{aligned} {{\mathbb {E}}}\left[ \max _{0\le k \le n}\Vert {\tilde{\zeta }}_n\Vert ^p\right] \le \bar{\kappa }\, {{\mathbb {E}}}\bigl [ [ M]_n^{p/2}\bigr ]. \end{aligned}$$

Since for every $n\in {{\mathbb {N}}}$,

$$\begin{aligned} \Vert \zeta _n\Vert ^p = \Vert {\tilde{\zeta }}_n + \xi _n-{\tilde{\xi }}_n\Vert ^p \le 2^p(\Vert {\tilde{\zeta }}_n\Vert ^p + c^p), \end{aligned}$$

we conclude that

$$\begin{aligned} {{\mathbb {E}}}\left[ \max _{0\le k \le n}\Vert \zeta _n\Vert ^p\right] \le 2^p\bigl (\bar{\kappa }\, {{\mathbb {E}}}\bigl [ [ M]_n^{p/2}\bigr ] + c^p\bigr ) \le 2^p({\bar{\kappa }} \vee 1) \cdot \bigl ( {{\mathbb {E}}}\bigl [ [ M]_n^{p/2}\bigr ] + c^p\bigr ), \end{aligned}$$

which completes the proof. $\square$