Towards explicit superlinear convergence rate for SR1

Abstract

We study the convergence rate of the famous Symmetric Rank-1 (SR1) algorithm, which has wide applications in different scenarios. Although it has been extensively investigated, SR1 still lacks a non-asymptotic superlinear rate compared with other quasi-Newton methods such as DFP and BFGS. In this paper, we address the aforementioned issue to obtain the first explicit non-asymptotic rates of superlinear convergence for the vanilla SR1 methods with a correction strategy that is used to achieve numerical stability. Specifically, the vanilla SR1 with the correction strategy achieves the rate of the form $\left( \frac{2n\ln (4\varkappa )}{k}\right) ^{k/2}$ for general smooth strongly-convex functions where k is the iteration counter, $\varkappa$ is the condition number of the objective function, and n is the dimensionality of the problem. Furthermore, the vanilla SR1 algorithm enjoys a little faster convergence rate and can find the optima of the quadratic objective function at most n steps.

Appendices

Useful lemmas

Lemma 10

Let approximate Hessians be updated recursively as $G_{i+1} \triangleq \mathrm {SR1}(A, G_i, u_i)$ (by Eqn. (5)) with an arbitrary $u_i \in {\mathbb {R}}^n, \forall i \ge 0$. For each $k \ge 0$, if $u_i^\top (G_i - A) u_i > 0$ holds for all $0 \le i \le k$, then vectors $u_0,\dots ,u_k$ are linearly independent. Furthermore, it holds that

$$\begin{aligned} Au_i = G_k u_i, \text{ for } \text{ all } 0\le i\le k-1. \end{aligned}$$

(65)

Proof

Denote $R_i = G_i - A, \forall i \ge 0$. By the update of SR1, we can obtain that for all $i \ge 0$,

$$\begin{aligned} R_{i+1} = {\left\{ \begin{array}{ll} R_{i} - \frac{R_{i}u_{i}u_{i}^\top R_{i}}{u_{i}^\top R_{i} u_{i}} &{} \text { if } R_{i}u_{i} \ne 0, \\ R_{i} &{} \text { otherwise}. \end{array}\right. } \end{aligned}$$

Thus, we get

$$\begin{aligned} \mathrm {Ker}(R_{i}) \subseteq \mathrm {Ker}(R_{i+1}), \text { and } u_{i} \in \mathrm {Ker}(R_{i+1}), \forall i \ge 0, \end{aligned}$$

(66)

where $\mathrm {Ker}(R_i)$ is the null space of $R_i$.

Next, we prove $u_0,\dots , u_k$ are linearly independent provided that $u_{i}^\top (G_{i} - A) u_{i} > 0, \forall i\le k$ by induction. First, for $k = 0$, the result holds trivially. Then, we assume that $u_0,\dots , u_{k}$ are linearly independent for some $k \ge 0$, and we will show that $u_0,\dots , u_{k+1}$ are linearly independent provided that $u_{k+1}^\top (G_{k+1} - A) u_{k+1} > 0$. We try to prove the result by contradiction and assume that $u_{k+1}$ can be represented as

$$\begin{aligned} u_{k+1} = \alpha _0 u_0 + \alpha _1 u_1 +\dots +\alpha _{k} u_{k}, \end{aligned}$$

where scalars $\alpha _0, \dots , \alpha _{k}$ are not all zero. Applying Eq. (66), we have $R_{k+1} u_i=0, \forall 0 \le i < k+1$. Then we further obtain that $R_{k+1} u_{k+1} = R_{k+1} (\alpha _0 u_0 + \dots + \alpha _{k} u_{k}) = 0 + \dots + 0 = 0$. This contradicts the assumption that $u_{k+1}^\top R_{k+1} u_{k+1} = u_{k+1}^\top (G_{k+1} - A) u_{k+1} > 0$. Thus, $u_0,\dots , u_{k+1}$ are linearly independent, and we finish the induction.

Finally, Eq. (65) can be immediately obtained by observing that $u_i \in \mathrm {Ker}(R_{i+1}) \subseteq \mathrm {Ker}(R_k)$ for all $i = 0, \dots , k-1$ based on Eq. (66). $\square$

Upper bound of M for logistic regression

The logistic regression is defined as follows:

$$\begin{aligned} f(x) = \frac{1}{m}\sum _{i=1}^{m} \log [1+\exp (-b_i a_i^\top x)] + \frac{\gamma }{2}\Vert x\Vert ^2, \end{aligned}$$

where $a_i \in {\mathbb {R}}^{n}$ is the i-th input vector, $b_i\in \{-1,1\}$ is the corresponding label, and $\gamma \ge 0$ is the regularization parameter. Accordingly, for all $x,h \in {\mathbb {R}}^n$, we have that

$$\begin{aligned} h^\top \nabla ^2 f(x) h= & {} \frac{1}{m}\sum _{i=1}^{m} p_i(x)(1-p_i(x)) (a_i^\top h)^2 + \gamma \left\| h\right\| ^2, \\ \; p_i(x)\triangleq & {} \frac{1}{1+\exp (-b_i a_i^\top x)} \in (0, 1), \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} D^3 f(x)[h,h,h]&= \frac{1}{m}\sum _{i=1}^{m} (1-2p_i(x)) \left( [p_i'(x)]^\top h\right) (b_i a_i^\top h)^2 \\&{\mathop {=}\limits ^{(i)}} \frac{1}{m}\sum _{i=1}^{m} (1-2p_i(x)) p_i(x)(1-p_i(x)) (b_i a_i^\top h)^3. \end{aligned} \end{aligned}$$

(67)

$D^3 f(x)[h,h,h] {=} \left. \frac{d^3}{d t^3} f(x+t h)\right| _{t=0}$ is the third derivative of f along the direction h and $(i)$ uses the derivation

$$\begin{aligned} p_i'(x) = \frac{\exp (-b_i \cdot a_i^\top x)}{\left( 1+\exp (-b_i \cdot a_i^\top x)\right) ^2} \cdot b_i a_i = p_i(x)(1-p_i(x))b_i a_i. \end{aligned}$$

Since $|t(1-t)(1-2t)| \le \frac{\sqrt{3}}{18}, \forall t\in (0,1)$ and the equality holds when $t = \frac{1}{2} - \frac{1}{2\sqrt{3}}$, we further obtain

$$\begin{aligned} D^3 f(x)[h,h,h] {\mathop {\le }\limits ^{(67)}} \frac{\sqrt{3}}{18m}\sum _{i=1}^{m} \left\| b_i a_i\right\| ^3 \cdot \left\| h\right\| ^3 = \frac{\sqrt{3}}{18m}\sum _{i=1}^{m} \left\| a_i\right\| ^3 \cdot \left\| h\right\| ^3. \end{aligned}$$

(68)

By Eq. (68), f(x) has $L'$-Lipschitz Hessians with $L'\triangleq \frac{\sqrt{3}}{18m}\sum _{i=1}^{m} \left\| a_i\right\| ^3$.

Finally, combining with f(x) being $\gamma$-strongly convex, we could get that given $\forall x,y, z, w\in {\mathbb {R}}^n$, it holds that

$$\begin{aligned} \nabla ^2 f(x) -\nabla ^2 f(y)\preceq & {} L' \cdot \left\| x-y\right\| \cdot I \preceq L' \cdot \frac{\left\| x-y\right\| _{z}}{\sqrt{\gamma }} \cdot \frac{\nabla ^2 f(w)}{\gamma } \\= & {} \frac{L'}{\gamma ^{3/2}} \cdot \left\| x-y\right\| _{z} \cdot \nabla ^2 f(w), \end{aligned}$$

where the second inequality is because of f(x) is $\gamma$-strongly convex. Hence, f is M-strongly self-concordant with

$$\begin{aligned} M \le \frac{L'}{\gamma ^{3/2}} = \frac{\sqrt{3}}{18} \cdot \frac{\sum _{i=1}^{m} \left\| a_i\right\| ^3}{ m\gamma ^{3/2}}. \end{aligned}$$