Abstract
The conjugate gradient method (CG) for solving linear systems of algebraic equations represents a highly nonlinear finite process. Since the original paper of Hestenes and Stiefel published in 1952, it has been linked with the Gauss-Christoffel quadrature approximation of Riemann-Stieltjes distribution functions determined by the data, i.e., with a simplified form of the Stieltjes moment problem. This link, developed further by Vorobyev, Brezinski, Golub, Meurant and others, indicates that a general description of the CG rate of convergence using an asymptotic convergence factor has principal limitations. Moreover, CG is computationally based on short recurrences. In finite precision arithmetic its behaviour is therefore affected by a possible loss of orthogonality among the computed direction vectors. Consequently, any consideration concerning the CG rate of convergence relevant to practical computations must include analysis of effects of rounding errors. Through the example of composite convergence bounds based on Chebyshev polynomials, this paper argues that the facts mentioned above should become a part of common considerations on the CG rate of convergence. It also explains that the spectrum composed of small number of well separated tight clusters of eigenvalues does not necessarily imply a fast convergence of CG or other Krylov subspace methods.
Similar content being viewed by others
References
Arioli, M., Liesen, J., Miedlar, A., Strakoš, Z.: Interplay between discretization and algebraic computation in adaptive numerical solution of elliptic PDE problems. GAMM Mitt. Ges. Angew. Math. Mech. (2013)
Axelsson, O.: A class of iterative methods for finite element equations. Comput. Methods Appl. Mech. Engrg. 9, 123–127 (1976)
Axelsson, O.: Iterative solution methods. Cambridge University Press, Cambridge (1994)
Axelsson, O.: A generalized conjugate gradient minimum residual method with variable preconditioners In: Advanced mathematics: computations and applications (Novosibirsk, 1995), pp. 14–25. NCC Publ.
Axelsson, O.: Optimal preconditioners based on rate of convergence estimates for the conjugate gradient method. Numer. Funct. Anal. Optim. 22, 277–302 (2001)
Axelsson, O., Kaporin, I.: On the sublinear and superlinear rate of convergence of conjugate gradient methods. Numer. Algorithms 25, 1–22 (2000)
Axelsson, O., Karátson, J.: Equivalent operator preconditioning for elliptic problems. Numer. Algorithms 50, 297–380 (2009)
Axelsson, O., Lindskog, G.: On the eigenvalue distribution of a class of preconditioning methods. Numer. Math. 48, 479–498 (1986)
Beckermann, B., Kuijlaars, A.B.J.: On the sharpness of an asymptotic error estimate for conjugate gradients. BIT 41, 856–867 (2001)
Beckermann, B., Kuijlaars, A.B.J.: Superlinear convergence of conjugate gradients. SIAM J. Numer. Anal. 39, 300–329 (2001)
Beckermann, B., Kuijlaars, A.B.J.: Superlinear CG convergence for special right-hand sides. Electron. Trans. Numer. Anal. 14, 1–19 (2002)
Brezinski, C.: Projection Methods for Systems of Equations. Vol. 7 of Studies in Computational Mathematics. North-Holland, Amsterdam (1997)
Brezinski, C.: Error estimates for the solution of linear systems. SIAM J. Sci. Comput. 21, 764–781 (1999)
Calvetti, D., Morigi, S., Reichel, L., Sgallari, F.: Computable error bounds and estimates for the conjugate gradient method. Numer. Algorithms 25, 75–88 (2000)
Calvetti, D., Morigi, S., Reichel, L., Sgallari, F.: An iterative method with error estimators. J. Comput. Appl. Math. 127, 93–119 (2001)
Dahlquist, G., Björck, Å.: Numerical Methods in Scientific Computing, vol. 1. SIAM, Philadelphia (2008)
Dahlquist, G., Eisenstat, S.C., Golub, G.H.: Bounds for the error of linear systems of equations using the theory of moments. J. Math. Anal. Appl. 37, 151–166 (1972)
Dahlquist, G., Golub, G.H., Nash, S.G.: Bounds for the error in linear systems. In: Semi-infinite programming (Proc. Workshop, Bad Honnef, 1978). Lecture Notes in Control and Information Sci., vol. 15, pp. 154–172. Springer, Berlin (1979)
Daniel, J.W.: The conjugate gradient method for linear and nonlinear operator equations. SIAM J. Numer. Anal. 4, 10–26 (1967)
Deuflhard, P.: Cascadic conjugate gradient methods for elliptic partial differential equations, algorithm and numerical results In: Domain decomposition methods in scientific and engineering computing (University Park, PA , 1993), vol. 180 of Contemp. Math American Mathematical Society, pp 29–42. Providence (1994)
Engeli, M., Ginsburg, T., Rutishauser, H., Stiefel, E.: Refined iterative methods for computation of the solution and the eigenvalues of self-adjoint boundary value problems. Mitt. Inst. Angew. Math. Zürich. 8, 107p. (1959)
Faber, V., Manteuffel, T.A., Parter, S.V.: On the theory of equivalent operators and application to the numerical solution of uniformly elliptic partial differential equations. Adv. Appl. Math. 11, 109–163 (1990)
Flanders, D.A., Shortley, G.: Numerical determination of fundamental modes. J. Appl. Phys. 21, 1326–1332 (1950)
Golub, G.H., Meurant, G.: Matrices, moments and quadrature with applications. Princeton Series in Applied Mathematics. Princeton University Press, Princeton (2010)
Golub, G.H., Strakoš, Z.: Estimates in quadratic formulas. Numer. Algorithms 8, 241–268 (1994)
Gratton, S., Titley-Peloquin, D., Toint, P., Tshimanga, J.: Linearizing the method of conjugate gradients. Technical Report naXys-15-2012, Namur Centre for Complex Systems. FUNDP–University of Namur, Belgium (2012)
Greenbaum, A.: Behaviour of slightly perturbed Lanczos and conjugate-gradient recurrences. Linear Algebra Appl. 113, 7–63 (1989)
Greenbaum, A.: Iterative Methods for Solving Linear Systems, vol. 17 of Frontiers in Applied Mathematics. SIAM, Philadelphia (1997)
Greenbaum, A., Strakoš, Z.: Predicting the behavior of finite precision Lanczos and conjugate gradient computations. SIAM J. Matrix Anal. Appl. 13, 121–137 (1992)
Günnel, A., Herzog, R., Sachs, E.: A note on preconditioners and scalar products for Krylov methods in Hilbert space. Preprint (2013)
Gutknecht, M.H., Strakoš, Z.: Accuracy of two three-term and three two-term recurrences for Krylov space solvers. SIAM J. Matrix Anal. Appl. 22, 213–229 (2000)
Hackbusch, W.: Iterative solution of large sparse systems of equations. Vol. 95 of Applied Mathematical Sciences. Translated and Revised from the 1991 German Original. Springer-Verlag, New York (1994)
Hestenes, M.R., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Research Nat. Bur. Standards 49, 409–436 (1952)
Hiptmair, R.: Operator preconditioning. Comput. Math. Appl. 52, 699–706 (2006)
Jennings, A.: Influence of the eigenvalue spectrum on the convergence rate of the conjugate gradient method. J. Inst. Math. Appl. 20, 61–72 (1977)
Lanczos, C.: Chebyshev polynomials in the solution of large-scale linear systems In: Proceedings of the Association for Computing Machinery, Toronto, 1952 (1953), pp 124–133. Sauls Lithograph Co. (for the Association for Computing Machinery), Washington, D. C.
Liesen, J., Strakoš, Z.: Krylov subspace methods: principles and analysis. Numerical Mathematics and Scientific Computation. Oxford University Press, Oxford (2012)
Mardal, K.-A., Winther, R.: Preconditioning discretizations of systems of partial differential equations. Numer. Linear Algebra Appl. 18, 1–40 (2011)
Markoff, A.: Démonstration de certaines inégalités de M. Tchébychef. Math. Ann. 24, 172–180 (1884)
Meurant, G.: The Lanczos and conjugate gradient algorithms: from theory to finite precision computations. Vol. 19 of Software, Environments, and Tools. Society for Industrial and Applied Mathematics. SIAM, Philadelphia (2006)
Meurant, G., Strakoš, Z.: The Lanczos and conjugate gradient algorithms in finite precision arithmetic. Acta. Numer. 15, 471–542 (2006)
Meurant, G., Tichý, P.: On computing quadrature-based bounds for the A-norm of the error in conjugate gradients. Numer. Algorithms 62, 163–191 (2013)
Naiman, A.E., Babuška, I.M., Elman, H.C.: A note on conjugate gradient convergence. Numer. Math. 76, 209–230 (1997)
Naiman, A.E., Engelberg, S.: A note on conjugate gradient convergence. II, III. Numer. Math. 85, 665–683, 685–696 (2000)
Nevanlinna, O.: Convergence of iterations for linear equations. Lectures in Mathematics ETH Zürich. Birkhäuser Verlag, Basel (1993)
Notay, Y.: On the convergence rate of the conjugate gradients in presence of rounding errors. Numer. Math. 65, 301–317 (1993)
O’Leary, D.P., Strakoš, Z., Tichý, P.: On sensitivity of Gauss-Christoffel quadrature. Numer. Math. 107, 147–174 (2007)
Paige, C.C.: Error analysis of the Lanczos algorithm for tridiagonalizing a symmetric matrix. J. Inst. Math. Appl. 18, 341–349 (1976)
Paige, C.C.: Accuracy and effectiveness of the Lanczos algorithm for the symmetric eigenproblem. Linear Algebra Appl. 34, 235–258 (1980)
Richardson, L.F.: The approximate arithmetical solution by finite differences of physical problems involving differential equations, with an application to the stresses in a masonry dam. Phil. Trans. Roy. Soc. London A 210, 307–357 (1911)
Rivlin, T.J.: Chebyshev Polynomials, 2nd edn. Pure and Applied Mathematics. Wiley, New York (1990)
Silvester, D.J., Simoncini, V.: An optimal iterative solver for symmetric indefinite systems stemming from mixed approximation. ACM Trans. Math. Softw. 37, Art. 42, 22 (2011)
Spielman, D.A., Woo, J.: A note on preconditioning by low-stretch spanning trees. Computing Research Repository (2009)
Strakoš, Z.: On the real convergence rate of the conjugate gradient method. Linear Algebra Appl. 154–156, 535–549 (1991)
Strakoš, Z., Tichý, P.: On error estimation in the conjugate gradient method and why it works in finite precision computations. Electron. Trans. Numer. Anal. 13, 56–80 (2002)
Strakoš, Z., Tichý, P.: Error estimation in preconditioned conjugate gradients. BIT 45, 789–817 (2005)
Tyrtyshnikov, E.E.: A Brief Introduction to Numerical Analysis. Birkhäuser, Boston (1997)
van der Sluis, A., van der Vorst, H.A.: The rate of convergence of conjugate gradients. Numer. Math. 48, 543–560 (1986)
van der Vorst, H.A.: Iterative solution methods for certain sparse linear systems with a nonsymmetric matrix arising from PDE-problems. J. Comput. Phys. 44, 1–19 (1981)
Varga, R.S.: Matrix iterative analysis, expanded, 2nd edn. Vol. 27 of Springer Series in Computational Mathematics. Springer-Verlag, Berlin (2000)
Vorobyev, Y.V.: Methods of moments in applied mathematics. Translated from the Russian by Bernard Seckler. Gordon and Breach Science Publishers, New York (1965)
Winther, R.: Some superlinear convergence results for the conjugate gradient method. SIAM J. Numer. Anal. 17, 14–17 (1980)
Young, D.: On Richardson’s method for solving linear systems with positive definite matrices. J. Math. Phys. 32, 243–255 (1954)
Author information
Authors and Affiliations
Corresponding author
Additional information
The work was supported by the ERC-CZ project LL1202, by the GACR grant 201/09/0917 and by the GAUK grant 695612.
Rights and permissions
About this article
Cite this article
Gergelits, T., Strakoš, Z. Composite convergence bounds based on Chebyshev polynomials and finite precision conjugate gradient computations. Numer Algor 65, 759–782 (2014). https://doi.org/10.1007/s11075-013-9713-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11075-013-9713-z
Keywords
- Conjugate gradient method
- Stieltjes moment problem
- Chebyshev semi-iterative method
- Composite polynomial convergence bounds
- Finite precision computations
- Clusters of eigenvalues