Variance reduction for root-finding problems

Abstract

Minimizing finite sums of smooth and strongly convex functions is an important task in machine learning. Recent work has developed stochastic gradient methods that optimize these sums with less computation than methods that do not exploit the finite sum structure. This speedup results from using efficiently constructed stochastic gradient estimators, which have variance that diminishes as the algorithm progresses. In this work, we ask whether the benefits of variance reduction extend to fixed point and root-finding problems involving sums of nonlinear operators. Our main result shows that variance reduction offers a similar speedup when applied to a broad class of root-finding problems. We illustrate the result on three tasks involving sums of n nonlinear operators: averaged fixed point, monotone inclusions, and nonsmooth common minimizer problems. In certain “poorly conditioned regimes,” the proposed method offers an n-fold speedup over standard methods.

Appendices

Appendix

Convergence of candidate algorithm (1.5)

Lemma A.1

Suppose that $S :{\mathcal {H}}\rightarrow {\mathcal {H}}$ satisfies conditions (1.3) and (1.4). Let $x^0 \in {\mathcal {H}}$, let $\lambda = n\min _i\{\beta _i \}$ and consider the following iteration:

$$\begin{aligned} x^{k+1} = x^k - \lambda S(x^k). \end{aligned}$$

Then for all $k \in {\mathbb {N}}$, we have

$$\begin{aligned} \mathrm {dist}(x^{k+1}, {{\,\mathrm{zer}\,}}(S)) \le \left( 1 - \min \{\beta _i\}\mu \right) \mathrm {dist}(x^k, {{\,\mathrm{zer}\,}}(S)). \end{aligned}$$

Consequently, after at most

$$\begin{aligned} \left\lceil \frac{\log (\mathrm {dist}^2(x^0, {{\,\mathrm{zer}\,}}(S))/\varepsilon )}{\mu \min _i\{\beta _i\}}\right\rceil \end{aligned}$$

iterations, the point $x^k$ satisfies $\mathrm {dist}^2(x^k, {{\,\mathrm{zer}\,}}(S)) \le \varepsilon$.

Proof

For all $k \in {\mathbb {N}}$ define ${\bar{x}}^k := \mathrm {proj}_{{{\,\mathrm{zer}\,}}(S)}(x^k)$ and observe that

$$\begin{aligned} \Vert x^{k+1} - {\bar{x}}^k\Vert ^2&= \Vert x^k - {\bar{x}}^k\Vert ^2 - 2\lambda \langle S(x^k), x^{k} - {\bar{x}}^k\rangle + \lambda ^2\Vert S(x^k)\Vert ^2. \end{aligned}$$

(A.1)

By Jensen’s inequality, we can upper bound the term $\Vert S(x^k)\Vert ^2$:

$$\begin{aligned} \Vert S(x^k)\Vert ^2&\le \frac{1}{n}\sum _{i=1}^n \Vert S_i(x^k) - S_i({\bar{x}}^k)\Vert ^2 \\&\le \frac{1}{n\min _i\{\beta _i\} } \sum _{i=1}^n \beta _i \Vert S_i(x^k) - S_i({\bar{x}}^k)\Vert ^2\\&\le \frac{1}{\min _i\{\beta _i\} } \langle S(x^k), x^k - {\bar{x}}^k\rangle , \end{aligned}$$

where the final inequality follows from (1.3). Using this estimate in (A.1), we find that

$$\begin{aligned} \Vert x^{k+1} - {\bar{x}}^k\Vert ^2&\le \Vert x^k - {\bar{x}}^k\Vert ^2 - \lambda \left( 2 - \frac{1}{\min _i\{\beta _i\}}\lambda \right) \langle S(x^k), x^{k} - {\bar{x}}^k\rangle \\&\le (1-\min \{\beta _i\}\mu )\Vert x^k - {\bar{x}}^k\Vert ^2 = (1-\min \{\beta _i\}\mu )\mathrm {dist}(x^{k}, {\mathrm {zer}(S)}), \end{aligned}$$

where the final inequality follows from (1.4). To conclude the proof, note that

$$\begin{aligned} \mathrm {dist}(x^{k+1}, {\mathrm {zer}(S)}) \le \Vert x^{k+1} - {\bar{x}}^k\Vert ^2 \le (1-\min \{\beta _i\}\mu )\mathrm {dist}(x^{k}, {\mathrm {zer}(S)}), \end{aligned}$$

as desired. $\square$