Skip to main content
SpringerLink
Log in

Optimal accumulation of Jacobian matrices by elimination methods on the dual computational graph

  • Published:
Mathematical Programming Submit manuscript

Abstract.

The accumulation of the Jacobian matrix F’ of a vector function can be regarded as a transformation of its linearized computational graph into a subgraph of the directed complete bipartite graph K n,m . This transformation can be performed by applying different elimination techniques that may lead to varying costs for computing F’. This paper introduces face elimination as the basic technique for accumulating Jacobian matrices by using a minimal number of arithmetic operations. Its superiority over both edge and vertex elimination methods is shown. The intention is to establish the conceptual basis for the ongoing development of algorithms for optimizing the computation of Jacobian matrices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aho, A., Sethi, R., Ullman, J.: Compilers, Principles, Techniques and Tools. Addison-Wesley, Reading MA, 1986

  2. Albrecht, A., Gottschling, P., Naumann, U.: Markowitz-type heuristics for computing Jacobian matrices efficiently. In: Proceedings of International Conference on Computational Science, Springer, LNCS. 2658, 575–584 (2003)

    Google Scholar 

  3. Ausiello, G., Crescenzi, P., Gambosi, G., Kann, V., Marchetti-Spaccamela, A., Protasi, M.: Complexity and Approximation. Springer, Berlin, 1999

  4. Berz, M., Bischof, C., Corliss, G., Griewank, A.: Computational Differentiation: Techniques, Applications, and Tools. Proceedings Series, Philadelphia, 1996, SIAM

  5. Bischof, C., Haghighat, M.: Hierarchical approaches to Automatic Differentiation. In: [4] pp. 82–94, 1996

  6. Bischof, C., Khademi, P., Boucharicha, A., Carle, A.: Efficient Computation of Gradients and Jacobians by Dynamic Exploitation of Sparsity in Automatic Differentiation. Optim. Meth. Softw. 7, 1–39 (1997)

    MATH  Google Scholar 

  7. Corliss, G., Faure, C., Griewank, A., Hascoet, L., Naumann, U., editors: Automatic Differentiation of Algorithms. From Simulation to Optimization, Springer, New York, 2002

  8. Corliss, G., Griewank, A. editors: Automatic Differentiation: Theory Implementation and Application. Proceedings Series, Philadelphia, 1991, SIAM

  9. Curtis, A., Powell, M., Reid, J.: On the Estimation of Sparse Jacobian Matrices. J. Inst. Math. Appl. 13, 117–119 (1974)

    MATH  Google Scholar 

  10. Van der Wijngaart, R., Saphir, W.: On the Efficacy of source code optimizations for cache-based processors. Technical report, NAS, June 2000

  11. Garey, M., Johnson, D.: Computers and Intractability – A Guide to the Theory of NP-completeness. W. H. Freeman and Company, San Francisco, 1979

  12. Gilbert, J.: A note on the NP-completeness of vertex elimination on directed graphs. J. Alg. Disc. Meth. 1(3), 292–294 (SIAM), September, 1980

    MATH  Google Scholar 

  13. Griewank, A.: Evaluating Derivatives. Principles and Techniques of Algorithmic Differentiation. Number 19 in Frontiers in Applied Mathematics. (SIAM) Philadelphia, 2000

  14. Griewank, A., Naumann, U.: Accumulating Jacobians as chained sparse matrix products. Math. Program. 3(95), 555–571 2003 (Springer)

    MATH  Google Scholar 

  15. Griewank, A., Reese, S.: On the calculation of Jacobian matrices by the Markovitz rule. In: [8], 1991, pp. 126–135

  16. Herley, K.: A Note on the NP-completeness of optimum Jacobian accumulation by vertex elimination. Presentation at: Theory Institute on Combinatorial Challenges in Computational Differentiation. 1993

  17. Horowitz, E., Sahni, S.: Fundamentals of Computer Algorithms. Computer Science Press, Rockville, 1978

  18. Iri, M.: History of Automatic Differentiation and rounding error estimation. In: [8], 1991, pp. 3–17

  19. Jessani, R., Putrino, M.: Comparison of single- and dual-pass multiply-add fused floating-point units. IEEE Trans. Comp. 47(9), 927–936 (1998)

    Article  Google Scholar 

  20. NAG Numerical Libraries. http://www.nag.co.uk/numeric/numerical_libraries.asp, NAG Ltd., Oxford, UK

  21. Marshall, J., Hill, C., Perelman, L., Adcroft, A.: Hydrostatic, quasi-hydrostatic and nonhydrostatic ocean modeling. J. Geophys. Res. 102 C3(5), 733–5, 752 (1997)

    Google Scholar 

  22. Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–92 (1953)

    Google Scholar 

  23. Miller, W., Wrathall, C.: Software for Roundoff Analysis of Matrix Algorithms. Academic Press, New York, 1980

  24. Naumann, U.: Efficient Calculation of Jacobian Matrices by Optimized Application of the Chain Rule to Computational Graphs. PhD thesis, Technical University Dresden, Feb. 1999

  25. Naumann, U.: An Enhanced Markowitz rule for accumulating Jacobians efficiently. In: K. Mikula, (ed.) ALGORITHMY’2000 Conference on Scientific Computing, Slovak University of Technology, Bratislava, Slovakia, September 2000, pp. 320–329

  26. Naumann, U.: Cheaper Jacobians by Simulated Annealing. SIAM J. Opt. 13(3), 660–674, March 2002

    Article  MATH  Google Scholar 

  27. Naumann, U.: Elimination techniques for cheap jacobians. In: [7] 2002, pp. 247–253

  28. Naumann, U.: Optimal Pivoting in Tangent-Linear and Adjoint Systems of Nonlinear Equations. Preprint ANL-MCS/P944-0402, Argonne National Laboratory, 2002

  29. Naumann, U., Albrecht, A.: Combinatorial optimization methods for fast derivative code. Case for Support EPSRC Grant GR/R38101/01, University of Hertfordshire, Hatfield, UK, 2001

  30. Naumann, U., Gottschling, P.: Prospects for Simulated Annealing in Automatic Differentiation. In: K. Steinhø øfel, (ed.), SAGA 2002 Stochastic Algorithms, Foundations and Applications, volume 2264 of LNCS. Springer, Berlin, 2001

  31. Newsam, G., Ramsdell, J.: Estimation of sparse jacobian Matrices. SIAM J. Alg. Dis. Meth. 4, 404–417 (1983)

    MathSciNet  MATH  Google Scholar 

  32. Rose, D., Tarjan, R.: Algorithmic aspects of vertex elimination on directed graphs. J. Appl. Math. 34(1), 176–197 (SIAM) January 1978

    MATH  Google Scholar 

  33. Basic Linear Algebra Subprograms. http://www.netlib.org/blas/

  34. Tadjouddine, M., Forth, S., Pryce, J.: AD Tools and Prospects for Optimal AD in CFD Flux Calculations. In: [7], 2002, pp. 255–261

  35. Tadjouddine, M., Forth, S., Pryce, J., Reid, J.: performance issues for vertex elimination methods in computing Jacobians using Automatic Differentiation. In: Proceedings of the ICCS 2000 Conference, Volume 2330 of Springer LNCS, 2002, pp. 1077–1086

  36. Wengert, R.: A Simple automatic derivative evaluation program. Comm. ACM 7, 463–464 (1964)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mathematic and Computer Science Division, Argonne National Laboratory, Argonne, IL 60439

    Uwe Naumann

Authors
  1. Uwe Naumann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Uwe Naumann.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Naumann, U. Optimal accumulation of Jacobian matrices by elimination methods on the dual computational graph. Math. Program., Ser. A 99, 399–421 (2004). https://doi.org/10.1007/s10107-003-0456-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-003-0456-9

Key words.

Search

Navigation