Stochastic Regularized Newton Methods for Nonlinear Equations

Abstract

In this paper, we study stochastic regularized Newton methods to find zeros of nonlinear equations, whose exact function information is normally expensive to calculate but approximations can be easily accessed via calls to stochastic oracles. To handle the potential singularity of Jacobian approximations, we compute a regularized Newton step at each iteration. Then we take a unit step if it can be accepted by an inexact line search condition, and a preset step otherwise. We investigate the global convergence properties and the convergence rate of the proposed algorithm with high probability. We also propose a stochastic regularized Newton method incorporating a variance reduction technique and establishing the corresponding sample complexities in terms of total numbers of stochastic oracle calls to find an approximate solution. Finally, we report some numerical results and demonstrate the promising performances of the two proposed algorithms.

Appendix

The following lemma refers to Theorem 1.6 in [34] and is about concentration inequalities for vector- and matrix-valued martingales, respectively.

Lemma 5.1

Let $({\mathcal {U}}_k)^m_{k=0}$ be a given filtration of the $\sigma$-algebra ${\mathcal {F}}$.

(i):

Let $(X_k)^m_{k=1}, X_k: \Omega \rightarrow {\mathbb {R}}^n$, be a family of random vectors, satisfying $X_k\in {{\mathcal {U}}_k}$ and $\sigma \in {{\mathbb {R}}^m}$ be a given vector with $\sigma _k\ne 0$, $k=1,\ldots ,m$. Suppose that ${\mathbb {E}}[X_k\ |{\mathcal {U}}_{k-1}]=0$, and ${\mathbb {E}}[\Vert X_k\Vert ^2\ |{\mathcal {U}}_{k-1}]\le \sigma _k^2$ almost everywhere for all $k\in {[m]}$. Then it holds

$$\begin{aligned} {\mathbb {E}}[\Vert \sum ^{m}_{k=1}X_k\Vert ^2\ |{\mathcal {U}}_0]\le \Vert \sigma \Vert ^2,\ {\mathbb {P}}(\Vert \sum ^{m}_{k=1}X_k\Vert \ge \tau \Vert \sigma \Vert \ |{\mathcal {U}}_0)\le \tau ^{-2},\ \ \forall \tau >0 \end{aligned}$$

almost everywhere.

(ii):

Let $(X_k)^m_{k=1}$, $X_k: \Omega \rightarrow {\mathbb {R}}^{d_1\times d_2}$, be a sequence of random matrices satisfying $X_k\in {{\mathcal {U}}_k}$. Suppose that ${\mathbb {E}}[X_k\ |{\mathcal {U}}_{k-1}]=0$, and there exists a positive constant R such that $\Vert X_k\Vert \le R$ almost everywhere for all $k\in {[m]}$. Define $\nu ^2=\max \{\Vert \sum _{k=1}^m {\mathbb {E}}(X_kX_k^T)\Vert ,\Vert \sum _{k=1}^m {\mathbb {E}}(X_k^TX_k)\Vert \}$. Then it holds

$$\begin{aligned} {\mathbb {P}}(\Vert \sum ^{m}_{k=1}X_k\Vert \ge t\ |{\mathcal {U}}_0)\le (d_1+d_2)\cdot \exp { (\frac{-t^2/2}{\nu ^2+Rt/3} )},\ \ \forall t>0 \end{aligned}$$

almost everywhere.