Anderson Acceleration as a Krylov Method with Application to Convergence Analysis

Abstract

Anderson acceleration (AA) is widely used for accelerating the convergence of nonlinear fixed-point methods, but little is known about how to quantify the asymptotic convergence acceleration provided by AA. As a roadway towards gaining more understanding of convergence acceleration by AA, we study AA(m), i.e., Anderson acceleration with finite window size m, applied to the case of linear fixed-point iterations. We write AA(m) as a Krylov method with polynomial residual update formulas, and derive $(m+2)$-term recurrence relations for the AA(m) polynomials. We derive several results based on these polynomial residual update formulas, including orthogonality relations, acceleration coefficient bounds, nonlinear recursions, and residual convergence bounds. We apply these results to study AA(1) residual convergence patterns and the influence of the initial guess on the asymptotic convergence factor.

Appendices

Appendices

Some Results for AA(m) with General Initial Guess

In this appendix we present results that lead to the proof of Proposition 2 in Sect. 2. We first derive a result, following from expression (16), on writing the residual of the more general AA iteration (9) with general initial guess $\{x_0,x_1,\ldots , x_m\}$ as a sum of $m+1$ vectors which are in $m+1$ Krylov spaces, $\{ \mathcal {K}_s(M,r_j)\}_{j=0}^m$ generated by the $m+1$ initial residuals $r_0,r_1,\ldots , r_m$. We also derive recurrence relations for the polynomials that arise in this expression. Proposition 15 can then easily be specialized to Proposition 2.

Proposition 15

AA(m) iteration (9) with general initial guess $\{x_j\}_{j=0}^{m}$ applied to linear iteration (5) is a multi-Krylov method. That is, the residual can be expressed as

$$\begin{aligned} r_{k+1} = \sum _{j=0}^m p_{k-m+1,j}(M)\,r_j,\quad k\ge m, \end{aligned}$$

(55)

where the $p_{k-m+1,j}(\lambda )$ are polynomials of degree at most $k-m+1$ satisfying the following relations:

$$\begin{aligned} p_{1,j}(\lambda )&=-\beta ^{(m)}_{m-j}\lambda , \quad j=0,\ldots ,m-1; \qquad p_{1,m}(\lambda )=\Big (1+\sum _{i=1}^{m}\beta _i^{(m)}\Big )\lambda ; \end{aligned}$$

(56)

$$\begin{aligned} p_{k-m+1,j}(\lambda )&=\lambda \left( \Big (1+\sum _{i=1}^{m}\beta _i^{(k)}\Big )p_{k-m,j} - \sum _{i=1}^{m}\beta _i^{(k)} p_{k-m-i,j}\right) , \nonumber \\ {}&\qquad \qquad \qquad \qquad \qquad \qquad k-m+1>1, j=0,\ldots , m; \end{aligned}$$

(57)

where for $i=1,\ldots ,m,$ and $j=0,\ldots , m$,

$$\begin{aligned} p_{1-i,j}(\lambda ) = {\left\{ \begin{array}{ll} 1 &{} \text {if}\,\, i=j+1-m,\\ 0 &{} \text {otherwise}. \end{array}\right. } \end{aligned}$$

Proof

The results of (56) are obvious from (16). For (57), when $k\ge 2m+1$, from (16), $r_{k+1}$ is a linear combination of $\{r_{k+1-i}\}_{i=1}^{m+1}$ where the smallest subscript index in the residual is $k-m\ge m+1$. Thus, every term $r_{k+1-j}$ can be rewritten as a linear combination of $\{r_j\}_{j=0}^m$. Then, (57) can be validated easily. As to $k< 2m+1$, since some terms in $\{r_{k+1-i}\}_{i=1}^{m+1}$ do not contain all $\{r_j\}_{j=0}^m$, we require that $p_{1-i,j}(\lambda )=0$ or 1 for (57) to hold. $\square$

Remark 3

Expression (55) indicates that the residual $r_{k+1}$ with $k\ge m$ of AA(m) can be decomposed as a sum of $m+1$ vectors which are in $m+1$ Krylov spaces, $\{ \mathcal {K}_s(M,r_j)\}_{j=0}^m$ or $\{ \mathcal {K}_s(A,r_j)\}_{j=0}^m$, where $s=k-m+2$. Therefore, we can refer to AA(m) with general initial guess as a multi-Krylov space method. Note that, in the case of the usual AA(m) iteration (1) with one initial guess $x_0$, each $r_j\in \{r_j\}_{j=1}^m$ can, by Proposition 15, be expressed as a polynomial in M applied to $r_0$, so AA(m) is a Krylov method, that is, $r_{k+1}\in \mathcal {K}_s (M,r_0)$, as formalized in Proposition 2 in Sect. 2.

Remark 4

In Proposition 15, we can, if desired, also rewrite the residual in terms of polynomials in the matrix A:

$$\begin{aligned} r_{k+1} = \sum _{j=0}^m\widetilde{ p}_{k-m+1,j}(A)\,r_j,\quad k\ge m, \end{aligned}$$

(58)

where the $\widetilde{p}_{k-m+1,j}(\lambda )$ satisfy the relations:

$$\begin{aligned} \widetilde{p}_{1,j}(\lambda )&=-\sum _{i=1}^{m}\beta _i^{(m)}(1-\lambda ), \,\, j=0,\ldots ,m-1; \\ \widetilde{p}_{1,m}(\lambda )&=\Big (1+\sum _{i=1}^{m}\beta _i^{(m)}\Big )(1-\lambda ); \\ \widetilde{p}_{k-m+1,j}(\lambda )&=(1-\lambda )\left( \Big (1+\sum _{i=1}^{m}\beta _i^{(k)}\Big ) \widetilde{p}_{k-m,j} - \sum _{i=1}^{m}\beta _i^{(k)} \widetilde{p}_{k-m-i,j}\right) , \\ {}&\qquad \qquad \qquad \qquad \qquad \qquad k-m+1>1, j=0,\ldots , m; \end{aligned}$$

where for $i=1,\ldots ,m, j=0,\ldots , m$,

$$\begin{aligned} \widetilde{p}_{1-i,j}(\lambda ) = {\left\{ \begin{array}{ll} 1 &{} \text {if}\,\, i=j+1-m,\\ 0 &{} \text {otherwise}. \end{array}\right. } \end{aligned}$$