Skip to main content
SpringerLink
Log in

On the numerical stability of algorithmic differentiation

  • Published:
Computing Aims and scope Submit manuscript

Abstract

In contrast to integration, the differentiation of a function is an ill-conditioned process, if only an oracle is available for its pointwise evaluation. That is, unrelated small variations in the value of the composite function are allowed at nearly identical arguments. In contrast, we show here that, if the function is defined by an evaluation procedure as a composition of arithmetic operations and elementary functions, then automatic, or algorithmic differentiation is backward stable in the sense of Wilkinson. More specifically, the derivative values obtained are exact for a perturbation of the elementary components at the level of the machine precision. We also provide a forward error analysis for both the forward and reverse mode. The theoretical analysis is confirmed by numerical experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Berz M, Bischof C, Corliss G, Griewank A (eds) (1996) Computational differentiation: techniques, applications, and tools. Proceedings series. SIAM

  2. Bücker M, Corliss G, Hovland P, Naumann U, Norris B (eds) (2005) Automatic differentiation: applications, theory, and tools. In: Lecture notes in computational science and engineering, vol 50. Springer, Berlin

  3. Cacuci D, Weber C, Oblow E, Marable J (1980) Sensitivity theory for general systems of nonlinear equations. Nucl Sci Eng 88: 88–110

    Google Scholar 

  4. Christianson B (1991) Reverse accumulation and accurate rounding error estimate for Taylor series coefficients. Optim Method Softw 1: 81–94

    Article  Google Scholar 

  5. Corliss, G, Faure, C, Griewank, A, Hascoet, L, Naumann, U (eds) (2002) Automatic differentiation of algorithms—from simulation to optimization. Springer, Berlin

    MATH  Google Scholar 

  6. Corliss, G, Griewank, A (eds) (1991) Automatic differentiation: theory, implementation, and application. Proceedings series. SIAM, Philadelphia

    Google Scholar 

  7. Griewank A, Utke J, Walther A (2000) Evaluating higher derivative tensors by forward propagation of univariate Taylor series. Math Comput 69(231): 1117–1130

    Article  MathSciNet  MATH  Google Scholar 

  8. Griewank A, Walther A (2008) Principles and techniques of algorithmic differentiation, 2nd edn. SIAM, Philadelphia

    MATH  Google Scholar 

  9. Higham NJ (2002) Accuracy and stability of numerical algorithms, 2nd edn. SIAM, Philadelphia

    Book  MATH  Google Scholar 

  10. Iri M (1991) History of automatic differentiation and rounding error estimation. In: Griewank A, Corliss GF (eds) Automatic differentiation of algorithms: theory, implementation, and application. SIAM, Philadelphia, pp 1–16

    Google Scholar 

  11. Iri M, Tsuchiya T, Hoshi M (1988) Automatic computation of partial derivatives and rounding error estimates with applications to large-scale systems of nonlinear equations. Comput Appl Math 24:365–392. Original Japanese version appeared in J Inf Process 26:1411–1420 (1985)

    Google Scholar 

  12. Lefèvre V, Muller JM, Tisserand A (1998) Toward correctly rounded transcendentals. IEEE Trans Comput 47(11): 1235–1243

    Article  Google Scholar 

  13. Li XS, Demmel JW, Bailey DH, Henry G, Hida Y, Iskandar J, Kahan W, Kapur A, Martin MC, Tung T, Yoo DJ (2002) Design, implementation and testing of extended and mixed precision blas. ACM Trans Math Softw 28:152–205. http://crd.lbl.gov/~xiaoye/XBLAS

    Google Scholar 

  14. Linnainmaa S (1976) Taylor expansion of the accumulated rounding error. BIT (Nordisk Tidskrift for Informationsbehandling) 16(1): 146–160

    MathSciNet  MATH  Google Scholar 

  15. MathWorks T. Matlab. http://www.mathworks.com/

  16. Michelsen M (1982) The isothermal flash problem. Part I: stability. Fluid Phase Equilibria 9(1): 1–19

    Article  Google Scholar 

  17. Miller W, Wrathall C (1980) Software for Roundoff analysis of matrix algorithms. Academic Press, New York

    MATH  Google Scholar 

  18. Muller JM, Brisebarre N, de Dinechin F, Jeannerod CP, Lefèvre V, Melquiond G, Revol N, Stehlé D, Torres S (2010) Handbook of floating-point arithmetic. Birkhäuser, Boston. ACM G.1.0; G.1.2; G.4; B.2.0; B.2.4; F.2.1. ISBN:978-0-8176-4704-9

  19. Ogita T, Rump S, Oishi S (2005) Accurate sum and dot product. SIAM J Sci Comput 26(6): 1955–1988

    Article  MathSciNet  MATH  Google Scholar 

  20. Ring M (2001) MAPM, a portable arbitrary precision math library in C. C/C++ Users J. http://www.tc.umn.edu/~ringx004/mapm-main.html

  21. Rump S (2001) Rigorous and portable standard functions. BIT 41(3): 540–562

    Article  MathSciNet  MATH  Google Scholar 

  22. Rump S, Böhm H (1983) Least significant bit evaluation for arithmetic expressions. Computing 30: 189–199

    Article  MathSciNet  MATH  Google Scholar 

  23. Society IC (2008) IEEE standard for floating-point arithmetic. Tech rep, Microprocessor Standards Committee of the IEEE Computer Society

  24. Speelpenning B (1980) Compiling fast partial derivatives of functions given by algorithms. PhD thesis, University of Illinois at Urbana, Champaign

  25. Wilkinson J (1971) Modern error analysis. SIAM Rev 13: 548–568

    Article  MathSciNet  MATH  Google Scholar 

  26. Ziv A (1991) Fast evaluation of elementary mathematical functions with correctly rounded last bit. ACM Trans Math Softw 17: 410–423

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institut für Mathematik, Humboldt Universität zu Berlin, Berlin, Germany

    Andreas Griewank

  2. Institut für Mathematik, Universität Paderborn, Paderborn, Germany

    Kshitij Kulshreshtha & Andrea Walther

Authors
  1. Andreas Griewank
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kshitij Kulshreshtha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Walther
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kshitij Kulshreshtha.

Additional information

The authors have presented some of the results of this paper during the SCAN 2010 conference in Lyon, September 2010.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Griewank, A., Kulshreshtha, K. & Walther, A. On the numerical stability of algorithmic differentiation. Computing 94, 125–149 (2012). https://doi.org/10.1007/s00607-011-0162-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-011-0162-z

Keywords

Mathematics Subject Classification (2000)

Search

Navigation