skip to main content
research-article

Algorithm 1005: Fortran Subroutines for Reverse Mode Algorithmic Differentiation of BLAS Matrix Operations

Published: 20 March 2020 Publication History

Abstract

A set of Fortran subroutines for reverse mode algorithmic (or automatic) differentiation of the basic linear algebra subprograms (BLAS) is presented. This is preceded by a description of the mathematical tools used to obtain the formulae of these derivatives, with emphasis on special matrices supported by the BLAS: triangular, symmetric, and band. All single and double precision BLAS derivatives have been implemented, together with the Cholesky factorization from Linear Algebra Package (LAPACK). The subroutines are written in Fortran 2003 with a Fortran 77 interface to allow use from C and C++, as well as dynamic languages such as R, Python, Matlab, and Octave. The subroutines are all implemented by calling BLAS, thereby attaining fast runtime. Timing results show derivative runtimes that are about twice those of the corresponding BLAS, in line with theory. The emphasis is on reverse mode because it is more important for the main application that we have in mind, numerical optimization. Two examples are presented, one dealing with the least squares modeling of groundwater, and the other dealing with the maximum likelihood estimation of the parameters of a vector autoregressive time series. The article contains comprehensive tables of formulae for the BLAS derivatives as well as for several non-BLAS matrix operations commonly used in optimization.

Supplementary Material

ZIP File (1005.zip)
Software for Fortran Subroutines for Reverse Mode Algorithmic Differentiation of BLAS Matrix Operations

References

[1]
2017. Intel Math Kernel Library. Retrieved October 13, 2017 from https://software.intel.com/en-us/mkl.
[2]
2017. OpenBLAS: An optimized BLAS library. Retrieved December 4, 2017 from http://www.openblas.net/.
[3]
Ai Barraud. 1977. A numerical algorithm to solve AT XA − X = Q. IEEE Transactions on Automatic Control 22, 5 (1977), 883--885.
[4]
Thomas F. Coleman and Arun Verma. 1998. ADMAT: An automatic differentiation toolbox for MATLAB. In Proceedings of the SIAM Workshop on Object Oriented Methods for Inter-Operable Scientific and Engineering Computing, SIAM, Philadelphia, PA, Vol. 2.
[5]
Thomas F. Coleman and Wei Xu. 2016. Automatic Differentiation in MATLAB Using ADMAT with Applications. SIAM.
[6]
Jack J. Dongarra, Jeremy Du Croz, Sven Hammarling, and Iain S. Duff. 1990. A set of level 3 basic linear algebra subprograms. ACM Transactions on Mathematical Software (TOMS) 16, 1 (1990), 1--17.
[7]
Jack J. Dongarra and David W. Walker. 1995. Software libraries for linear algebra computations on high performance computers. SIAM Review 37, 2 (1995), 151--180.
[8]
Paul S. Dwyer and M. S. MacPhail. 1948. Symbolic matrix derivatives. The Annals of Mathematical Statistics 19, 4 (1948), 517--534.
[9]
Shaun A. Forth. 2006. An efficient overloaded implementation of forward mode automatic differentiation in MATLAB. ACM Transactions on Mathematical Software (TOMS) 32, 2 (2006), 195--222.
[10]
Ralf Giering and Thomas Kaminski. 1998. Recipes for adjoint code construction. ACM Transactions on Mathematical Software (TOMS) 24, 4 (1998), 437--474.
[11]
M. B. Giles. 2008. An Extended Collection of Matrix Derivative Results for Forward and Reverse Mode Automatic Differentiation. Technical Report. Oxford University Computing Laboratory.
[12]
Mike B. Giles. 2008. Collected matrix derivative results for forward and reverse mode algorithmic differentiation. In Advances in Automatic Differentiation, Christian H. Bischof, H. Martin Bücker, Paul D. Hovland, Uwe Naumann, and J. Utke (Eds.). Springer, Berlin, 35--44.
[13]
Kazushige Goto and Robert van de Geijn. 2008. High-performance implementation of the level-3 BLAS. ACM Transactions on Mathematical Software (TOMS) 35, 1 (2008), 4.
[14]
Andreas Griewank. 1989. On automatic differentiation. In Mathematical Programming, Masao Iri and K. Tanabe (Eds.). Kluwer Academic Publishers, Dordrecht, 83--108.
[15]
Andreas Griewank. 2003. A mathematical view of automatic differentiation. In Acta Numerica. Vol. 12. Cambridge University Press, Cambridge, 321--398.
[16]
Andreas Griewank. 2012. Who invented the reverse mode of differentiation? Documenta Mathematica, Extra Volume ISMP (2012), 389--400.
[17]
Andreas Griewank. 2015. Automatic differentiation, Mark R. Dennis, Paul Glendinning, Paul A. Martin, Fadil Santosa, and Jared Tanner (Eds.). Princeton University Press, 749--752.
[18]
Andreas Griewank, David Juedes, and Jean Utke. 1996. Algorithm 755: ADOL-C: A package for the automatic differentiation of algorithms written in C/C++. ACM Transactions on Mathematical Software 22, 2 (1996), 131--167.
[19]
Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (2nd ed.). Number 105 in Other Titles in Applied Mathematics. SIAM, Philadelphia, PA.
[20]
Masao Iri. 1991. History of automatic differentiation and rounding error estimation. In Automatic Differentiation of Algorithms: Theory, Implementation, and Application, Andreas Griewank and George F. Corliss (Eds.). SIAM, Philadelphia, 1--16.
[21]
Kristjan Jonasson. 2008. Algorithm 878: Exact VARMA likelihood and its gradient for complete and incomplete data with Matlab. ACM Transactions on Mathematical Software (TOMS) 35, 1 (2008), 6:1--6:11.
[22]
Kristjan Jonasson. 2009. Algorithm 892: DISPMODULE, a Fortran 95 module for pretty-printing matrices. ACM Transactions on Mathematical Software (TOMS) 36, 1 (2009), 6:1--6:7.
[23]
Gershon Kedem. 1980. Automatic differentiation of computer programs. ACM Trans. Math. Software 6, 2 (June 1980), 150--165.
[24]
Chuck L. Lawson, Richard J. Hanson, David R. Kincaid, and Fred T. Krogh. 1979. Basic linear algebra subprograms for Fortran usage. ACM Transactions on Mathematical Software (TOMS) 5, 3 (1979), 308--323.
[25]
Uwe Naumann. 2008. Optimal Jacobian accumulation is NP-complete. Mathematical Programming 112, 2 (2008), 427--441.
[26]
Uwe Naumann. 2012. The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation. Siam, Philadelphia, PA.
[27]
S. P. Smith. 1995. Differentiation of the Cholesky algorithm. Journal of Computational and Graphical Statistics 4, 2 (1995), 134--147.
[28]
Arun Verma. 1998. Structured Automatic Differentiation. Ph.D. Dissertation. Department of Computer Science, Cornell University, Ithaca, NY.
[29]
Arun Verma. 1999. ADMAT: Automatic differentiation in MATLAB using object oriented methods. In Object Oriented Methods for Interoperable Scientific and Engineering Computing: Proceedings of the 1998 SIAM Workshop, M. E. Henderson, C. R. Anderson, and S. L. Lyons (Eds.). SIAM, Philadelphia, 174--183.
[30]
Sebastian F. Walter and Lutz Lehmann. 2013. Algorithmic differentiation in Python with AlgoPy. Journal of Computational Science 4, 5 (2013), 334--344.
[31]
Qian Wang, Xianyi Zhang, Yunquan Zhang, and Qing Yi. 2013. AUGEM: Automatically generate high performance dense linear algebra kernels on x86 CPUs. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. ACM, 25:1--25:12.
[32]
Matthew J. Weinstein and Anil V. Rao. 2017. Algorithm 984: ADiGator, a toolbox for the algorithmic differentiation of mathematical functions in matlab using source transformation via operator overloading. ACM Transactions on Mathematical Software (TOMS) 44, 2 (2017), 21:1--21:25.
[33]
R. Clint Whaley, Antoine Petitet, and Jack J. Dongarra. 2001. Automated empirical optimizations of software and the ATLAS project. Parallel Computing 27, 1 (2001), 3--35.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Mathematical Software
ACM Transactions on Mathematical Software  Volume 46, Issue 1
March 2020
214 pages
ISSN:0098-3500
EISSN:1557-7295
DOI:10.1145/3387915
Issue’s Table of Contents
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 March 2020
Accepted: 01 August 2019
Revised: 01 August 2019
Received: 01 December 2017
Published in TOMS Volume 46, Issue 1

Check for updates

Author Tags

  1. AD
  2. Numerical linear algebra
  3. RMAD
  4. algorithmic differentiation
  5. automatic differentiation
  6. matrix operations
  7. reverse mode

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 412
    Total Downloads
  • Downloads (Last 12 months)13
  • Downloads (Last 6 weeks)1
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media