skip to main content
research-article

Algorithmic Differentiation of Numerical Methods: Tangent and Adjoint Solvers for Parameterized Systems of Nonlinear Equations

Published: 12 October 2015 Publication History

Abstract

We discuss software tool support for the algorithmic differentiation (AD), also known as automatic differentiation, of numerical simulation programs that contain calls to solvers for parameterized systems of n nonlinear equations. The local computational overhead and the additional memory requirement for the computation of directional derivatives or adjoints of the solution of the nonlinear system with respect to the parameters can quickly become prohibitive for large values of n. Both are reduced drastically by analytical (and symbolic) approaches to differentiation of the underlying numerical methods. Following the discussion of the proposed terminology, we develop the algorithmic formalism building on prior work by other colleagues and present an implementation based on the AD software dco/c++. A representative case study supports the theoretically obtained computational complexity results with practical runtime measurements.

References

[1]
C. Bischof, M. Bücker, P. Hovland, U. Naumann, and J. Utke (Eds.). 2008. Advances in Automatic Differentiation. Lecture Notes in Computational Science and Engineering, Vol. 64. Springer, Berlin.
[2]
C. Broyden. 1970. The convergence of a class of double-rank minimization algorithms. Journal of the Institute of Mathematics and Its Applications 6, 76--90.
[3]
M. Bücker, G. Corliss, P. Hovland, U. Naumann, and B. Norris (Eds.). 2006. Automatic Differentiation: Applications, Theory, and Tools. Lecture Notes in Computational Science and Engineering, Vol. 50. Springer, Berlin.
[4]
L. Capriotti. 2011. Fast Greeks by algorithmic differentiation. Journal of Computational Finance 14, 3.
[5]
B. Christianson. 1994. Reverse accumulation and attractive fixed points. Optimization Methods and Software 3, 4, 311--326.
[6]
A. J. Davies, D. B. Christianson, L. C. W. Dixon, R. Roy, and P. van der Zee. 1997. Reverse differentiation and the inverse diffusion problem. Advances in Engineering Software 28, 4, 217--221.
[7]
N. Dunford and J. Schwartz. 1988. Linear Operators, Part 1, General Theory. Wiley.
[8]
S. Forth, P. Hovland, E. Phipps, J. Utke, and A. Walther (Eds.). 2012. Recent Advances in Algorithmic Differentiation. Lecture Notes in Computational Science and Engineering, Vol. 87. Springer, Berlin.
[9]
A. Gebremedhin, F. Manne, and A. Pothen. 2005. What color is your Jacobian? Graph coloring for computing derivatives. SIAM Review 47, 4, 629--705.
[10]
R. Giering and T. Kaminski. 1998. Recipes for adjoint code construction. ACM Transactions on Mathematical Software 24, 437--474.
[11]
J. C. Gilbert. 1992. Automatic differentiation and iterative processes. Optimization Methods and Software 1, 13--21.
[12]
M. Giles. 2008. Collected matrix derivative results for forward and reverse mode algorithmic differentiation. In Advances in Automatic Differentiation. Lecture Notes in Computational Science and Engineering, Vol. 64. Springer, 35--44.
[13]
M. Giles and P. Glasserman. 2006. Smoking adjoints: Fast Monte Carlo Greeks. Risk 19, 88--92.
[14]
A. Griewank. 1992. Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation. Optimization Methods and Software 1, 35--54.
[15]
A. Griewank and C. Faure. 2002. Reduced functions, gradients and Hessians from fixed-point iterations for state equations. Numerical Algorithms 30, 2, 113--139.
[16]
A. Griewank, D. Juedes, and J. Utke. 1996. ADOL-C, a package for the automatic differentiation of algorithms written in C/C++. ACM Transactions on Mathematical Software 22, 131--167.
[17]
A. Griewank and A. Walther. 2000. Algorithm 799: Revolve: An implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software 26, 1, 19--45.
[18]
A. Griewank and A. Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (2nd ed.). Other Titles in Applied Mathematics, No. 105. SIAM, Philadelphia, PA.
[19]
L. Hascoët, U. Naumann, and V. Pascual. 2005. To-be-recorded analysis in reverse mode automatic differentiation. Future Generation Computer Systems 21, 1401--1417.
[20]
L. Hascoet and V. Pascual. 2013. The tapenade automatic differentiation tool: Principles, model, and specification. ACM Transactions on Mathematical Software 39, 3, 20.
[21]
M. Henrard. 2014. Adjoint algorithmic differentiation: Calibration and implicit function theorem. Journal of Computational Finance 17, 4.
[22]
U. Lehmann and A. Walther. 2002. The implementation and testing of time-minimal and resource-optimal parallel reversal schedules. In Computational Science—ICCS 2002. Lecture Notes in Computer Science, Vol. 2330. Springer, 1049--1058.
[23]
U. Naumann. 2008. Call tree reversal is NP-complete. In Advances in Automatic Differentiation. Lecture Notes in Computational Science and Engineering, Vol. 64. Springer, 13--22.
[24]
U. Naumann. 2009. DAG reversal is NP-complete. Journal of Discrete Algorithms 7, 402--410.
[25]
U. Naumann. 2012. The Art of Differentiating Computer Programs. An Introduction to Algorithmic Differentiation. Software, Environments, and Tools, No. 24. SIAM, Philadelphia, PA.
[26]
U. Naumann and J. du Toit. 2014. Adjoint Algorithmic Differentiation Tool Support for Typical Numerical Patterns in Computational Finance. Technical Report AIB2014-13. RWTH Aachen University, Aachen, Germany. http://sunsite.informatik.rwth-aachen.de/Publications/AIB/2014/2014-13.pdf.
[27]
U. Naumann, K. Leppkes, and J. Lotz. 2014. dco/c++ User Guide. Technical Report AIB-2014-03. RWTH Aachen University, Aachen, Germany. http://aib.informatik.rwth-aachen.de/2014/2014-03.pdf.gz.
[28]
A. Nemili, E. Özkaya, N. Gauger, F. Kramer, A. Carnarius, and F. Thiele. 2013. Discrete adjoint based sensitivity analysis for optimal flow control of a 3D high-lift configuration. In Proceedings of the 21st AIAA Computational Fluid Dynamics Conference. 1--14.
[29]
J. Nocedal and S. J. Wright. 2006. Numerical Optimization (2nd ed.). Springer-Verlag, New York, NY.
[30]
F. Rauser, J. Riehme, K. Leppkes, P. Korn, and U. Naumann. 2010. On the use of discrete adjoints in goal error estimation for shallow water equations. Procedia Computer Science 1, 1, 107--115. 10.1016/j.procs.2010.04.013
[31]
N. Safiran, J. Lotz, and U. Naumann. 2014. Algorithmic Differentiation of Numerical Methods: Second-Order Tangent and Adjoint Solvers for Systems of Nonlinear Equations. Technical Report AIB-2014-07. RWTH Aachen University, Aachen, Germany. http://aib.informatik.rwth-aachen.de/2014/2014-07.pdf.gz.
[32]
M. Sagebaum, N. Gauger, J. Lotz, and U. Naumann. 2013. Algorithmic differentiation of a large simulation code including libraries. Procedia Computer Science 18, 208--217.
[33]
P. Stumm and A. Walther. 2010. New algorithms for optimal online checkpointing. SIAM Journal on Scientific Computing 32, 2, 836--854.
[34]
M. Towara and U. Naumann. 2013. Toward discrete adjoint OpenFOAM. Procedia Computer Science 18, 429--438.
[35]
J. Ungermann, J. Blank, J. Lotz, K. Leppkes, L. Hoffmann, T. Guggenmoser, M. Kaufmann, P. Preusse, U. Naumann, and M. Riese. 2011. A 3-D tomographic retrieval approach with advection compensation for the air-borne limb-imager GLORIA. Atmospheric Measurement Techniques 4, 11, 2509--2529.

Cited By

View all
  • (2025)Non-Classical States of Light for Device-Independent Quantum Key DistributionNon-classical States of Light10.1007/978-3-031-73769-5_5(219-262)Online publication date: 23-Jan-2025
  • (2024)A Matrix-Free Exact Newton MethodSIAM Journal on Scientific Computing10.1137/23M157017X46:3(A1423-A1440)Online publication date: 2-May-2024
  • (2023)Efficient GPU Implementation of Automatic Differentiation for Computational Fluid Dynamics2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)10.1109/HiPC58850.2023.00055(377-386)Online publication date: 18-Dec-2023
  • Show More Cited By

Index Terms

  1. Algorithmic Differentiation of Numerical Methods: Tangent and Adjoint Solvers for Parameterized Systems of Nonlinear Equations

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Transactions on Mathematical Software
        ACM Transactions on Mathematical Software  Volume 41, Issue 4
        October 2015
        160 pages
        ISSN:0098-3500
        EISSN:1557-7295
        DOI:10.1145/2835205
        Issue’s Table of Contents
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 12 October 2015
        Accepted: 01 December 2014
        Revised: 01 December 2014
        Received: 01 September 2012
        Published in TOMS Volume 41, Issue 4

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Tangent/adjoint nonlinear solver
        2. algorithmic differentiation

        Qualifiers

        • Research-article
        • Research
        • Refereed

        Funding Sources

        • DFG

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)24
        • Downloads (Last 6 weeks)2
        Reflects downloads up to 20 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2025)Non-Classical States of Light for Device-Independent Quantum Key DistributionNon-classical States of Light10.1007/978-3-031-73769-5_5(219-262)Online publication date: 23-Jan-2025
        • (2024)A Matrix-Free Exact Newton MethodSIAM Journal on Scientific Computing10.1137/23M157017X46:3(A1423-A1440)Online publication date: 2-May-2024
        • (2023)Efficient GPU Implementation of Automatic Differentiation for Computational Fluid Dynamics2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)10.1109/HiPC58850.2023.00055(377-386)Online publication date: 18-Dec-2023
        • (2022)Numerical Stability of Tangents and Adjoints of Implicit FunctionsComputational Science – ICCS 202210.1007/978-3-031-08757-8_17(181-187)Online publication date: 21-Jun-2022
        • (2020)Towards compiler-aided correctness checking of adjoint MPI applications2020 IEEE/ACM 4th International Workshop on Software Correctness for HPC Applications (Correctness)10.1109/Correctness51934.2020.00010(40-48)Online publication date: Nov-2020
        • (2020)Dual Numbers and Automatic Differentiation to Efficiently Compute Velocities and AccelerationsActa Applicandae Mathematicae: an international survey journal on applying mathematics and mathematical applications10.1007/s10440-020-00351-9170:1(649-659)Online publication date: 1-Dec-2020
        • (2020)Patient-Specific Cardiac Parametrization from Eikonal SimulationsComputational Science – ICCS 202010.1007/978-3-030-50371-0_21(290-303)Online publication date: 3-Jun-2020
        • (2019)Adjoint Code Design PatternsACM Transactions on Mathematical Software10.1145/332616245:3(1-32)Online publication date: 30-Jul-2019
        • (2019)Convergence and error analysis of an automatically differentiated finite volume based heat conduction codeInternational Journal of Numerical Methods for Heat & Fluid Flow10.1108/HFF-09-2018-0489ahead-of-print:ahead-of-printOnline publication date: 25-Jul-2019
        • (2019)A two-level computational graph method for the adjoint of a finite volume based compressible unsteady flow solverParallel Computing10.1016/j.parco.2018.12.00181(68-84)Online publication date: Jan-2019
        • Show More Cited By

        View Options

        Login options

        Full Access

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media