Algorithm 1005: Fortran Subroutines for Reverse Mode Algorithmic Differentiation of BLAS Matrix Operations
Authors: 
Kristjan Jonasson, Sven Sigurdsson, Hordur Freyr Yngvason, Petur Orri Ragnarsson, Pall MelstedAuthors Info & Claims
Kristjan Jonasson, Sven Sigurdsson, Hordur Freyr Yngvason, Petur Orri Ragnarsson, Pall MelstedAuthors Info & ClaimsArticle No.: 9, Pages 1 - 20
Abstract
A set of Fortran subroutines for reverse mode algorithmic (or automatic) differentiation of the basic linear algebra subprograms (BLAS) is presented. This is preceded by a description of the mathematical tools used to obtain the formulae of these derivatives, with emphasis on special matrices supported by the BLAS: triangular, symmetric, and band. All single and double precision BLAS derivatives have been implemented, together with the Cholesky factorization from Linear Algebra Package (LAPACK). The subroutines are written in Fortran 2003 with a Fortran 77 interface to allow use from C and C++, as well as dynamic languages such as R, Python, Matlab, and Octave. The subroutines are all implemented by calling BLAS, thereby attaining fast runtime. Timing results show derivative runtimes that are about twice those of the corresponding BLAS, in line with theory. The emphasis is on reverse mode because it is more important for the main application that we have in mind, numerical optimization. Two examples are presented, one dealing with the least squares modeling of groundwater, and the other dealing with the maximum likelihood estimation of the parameters of a vector autoregressive time series. The article contains comprehensive tables of formulae for the BLAS derivatives as well as for several non-BLAS matrix operations commonly used in optimization.
Supplementary Material
Software for Fortran Subroutines for Reverse Mode Algorithmic Differentiation of BLAS Matrix Operations
- Download
- 1.29 MB
References
[1]
2017. Intel Math Kernel Library. Retrieved October 13, 2017 from https://software.intel.com/en-us/mkl.
[2]
2017. OpenBLAS: An optimized BLAS library. Retrieved December 4, 2017 from http://www.openblas.net/.
[3]
Ai Barraud. 1977. A numerical algorithm to solve AT XA − X = Q. IEEE Transactions on Automatic Control 22, 5 (1977), 883--885.
[4]
Thomas F. Coleman and Arun Verma. 1998. ADMAT: An automatic differentiation toolbox for MATLAB. In Proceedings of the SIAM Workshop on Object Oriented Methods for Inter-Operable Scientific and Engineering Computing, SIAM, Philadelphia, PA, Vol. 2.
[5]
Thomas F. Coleman and Wei Xu. 2016. Automatic Differentiation in MATLAB Using ADMAT with Applications. SIAM.
[6]
Jack J. Dongarra, Jeremy Du Croz, Sven Hammarling, and Iain S. Duff. 1990. A set of level 3 basic linear algebra subprograms. ACM Transactions on Mathematical Software (TOMS) 16, 1 (1990), 1--17.
[7]
Jack J. Dongarra and David W. Walker. 1995. Software libraries for linear algebra computations on high performance computers. SIAM Review 37, 2 (1995), 151--180.
[8]
Paul S. Dwyer and M. S. MacPhail. 1948. Symbolic matrix derivatives. The Annals of Mathematical Statistics 19, 4 (1948), 517--534.
[9]
Shaun A. Forth. 2006. An efficient overloaded implementation of forward mode automatic differentiation in MATLAB. ACM Transactions on Mathematical Software (TOMS) 32, 2 (2006), 195--222.
[10]
Ralf Giering and Thomas Kaminski. 1998. Recipes for adjoint code construction. ACM Transactions on Mathematical Software (TOMS) 24, 4 (1998), 437--474.
[11]
M. B. Giles. 2008. An Extended Collection of Matrix Derivative Results for Forward and Reverse Mode Automatic Differentiation. Technical Report. Oxford University Computing Laboratory.
[12]
Mike B. Giles. 2008. Collected matrix derivative results for forward and reverse mode algorithmic differentiation. In Advances in Automatic Differentiation, Christian H. Bischof, H. Martin Bücker, Paul D. Hovland, Uwe Naumann, and J. Utke (Eds.). Springer, Berlin, 35--44.
[13]
Kazushige Goto and Robert van de Geijn. 2008. High-performance implementation of the level-3 BLAS. ACM Transactions on Mathematical Software (TOMS) 35, 1 (2008), 4.
[14]
Andreas Griewank. 1989. On automatic differentiation. In Mathematical Programming, Masao Iri and K. Tanabe (Eds.). Kluwer Academic Publishers, Dordrecht, 83--108.
[15]
Andreas Griewank. 2003. A mathematical view of automatic differentiation. In Acta Numerica. Vol. 12. Cambridge University Press, Cambridge, 321--398.
[16]
Andreas Griewank. 2012. Who invented the reverse mode of differentiation? Documenta Mathematica, Extra Volume ISMP (2012), 389--400.
[17]
Andreas Griewank. 2015. Automatic differentiation, Mark R. Dennis, Paul Glendinning, Paul A. Martin, Fadil Santosa, and Jared Tanner (Eds.). Princeton University Press, 749--752.
[18]
Andreas Griewank, David Juedes, and Jean Utke. 1996. Algorithm 755: ADOL-C: A package for the automatic differentiation of algorithms written in C/C++. ACM Transactions on Mathematical Software 22, 2 (1996), 131--167.
[19]
Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (2nd ed.). Number 105 in Other Titles in Applied Mathematics. SIAM, Philadelphia, PA.
[20]
Masao Iri. 1991. History of automatic differentiation and rounding error estimation. In Automatic Differentiation of Algorithms: Theory, Implementation, and Application, Andreas Griewank and George F. Corliss (Eds.). SIAM, Philadelphia, 1--16.
[21]
Kristjan Jonasson. 2008. Algorithm 878: Exact VARMA likelihood and its gradient for complete and incomplete data with Matlab. ACM Transactions on Mathematical Software (TOMS) 35, 1 (2008), 6:1--6:11.
[22]
Kristjan Jonasson. 2009. Algorithm 892: DISPMODULE, a Fortran 95 module for pretty-printing matrices. ACM Transactions on Mathematical Software (TOMS) 36, 1 (2009), 6:1--6:7.
[23]
Gershon Kedem. 1980. Automatic differentiation of computer programs. ACM Trans. Math. Software 6, 2 (June 1980), 150--165.
[24]
Chuck L. Lawson, Richard J. Hanson, David R. Kincaid, and Fred T. Krogh. 1979. Basic linear algebra subprograms for Fortran usage. ACM Transactions on Mathematical Software (TOMS) 5, 3 (1979), 308--323.
[25]
Uwe Naumann. 2008. Optimal Jacobian accumulation is NP-complete. Mathematical Programming 112, 2 (2008), 427--441.
[26]
Uwe Naumann. 2012. The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation. Siam, Philadelphia, PA.
[27]
S. P. Smith. 1995. Differentiation of the Cholesky algorithm. Journal of Computational and Graphical Statistics 4, 2 (1995), 134--147.
[28]
Arun Verma. 1998. Structured Automatic Differentiation. Ph.D. Dissertation. Department of Computer Science, Cornell University, Ithaca, NY.
[29]
Arun Verma. 1999. ADMAT: Automatic differentiation in MATLAB using object oriented methods. In Object Oriented Methods for Interoperable Scientific and Engineering Computing: Proceedings of the 1998 SIAM Workshop, M. E. Henderson, C. R. Anderson, and S. L. Lyons (Eds.). SIAM, Philadelphia, 174--183.
[30]
Sebastian F. Walter and Lutz Lehmann. 2013. Algorithmic differentiation in Python with AlgoPy. Journal of Computational Science 4, 5 (2013), 334--344.
[31]
Qian Wang, Xianyi Zhang, Yunquan Zhang, and Qing Yi. 2013. AUGEM: Automatically generate high performance dense linear algebra kernels on x86 CPUs. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. ACM, 25:1--25:12.
[32]
Matthew J. Weinstein and Anil V. Rao. 2017. Algorithm 984: ADiGator, a toolbox for the algorithmic differentiation of mathematical functions in matlab using source transformation via operator overloading. ACM Transactions on Mathematical Software (TOMS) 44, 2 (2017), 21:1--21:25.
[33]
R. Clint Whaley, Antoine Petitet, and Jack J. Dongarra. 2001. Automated empirical optimizations of software and the ATLAS project. Parallel Computing 27, 1 (2001), 3--35.
Index Terms
- Algorithm 1005: Fortran Subroutines for Reverse Mode Algorithmic Differentiation of BLAS Matrix Operations
Recommendations
Practical Use of Some Krylov Subspace Methods for Solving Indefinite and Nonsymmetric Linear Systems
The main purpose of this paper is to develop stable versions of some Krylov subspace methods for solving linear systems of equations $Ax = b$. As in the case of Paige and Saunders's SYMMLQ [SIAM J. Numer. Anal., 12 (1975), pp. 617–624], our algorithms ...
Row Modifications of a Sparse Cholesky Factorization
Given a sparse, symmetric positive definite matrix C and an associated sparse Cholesky factorization LDL$\tr$, we develop sparse techniques for updating the factorization after a symmetric modification of a row and column of C. We show how the ...
Solving unsymmetric sparse systems of linear equations with PARDISO
Special issue: Selected numerical algorithmsSupernode partitioning for unsymmetric matrices together with complete block diagonal supernode pivoting and asynchronous computation can achieve high gigaflop rates for parallel sparse LU factorization on shared memory parallel computers. The progress ...
Comments
Information & Contributors
Information
Published In
March 2020
214 pages
ISSN:0098-3500
EISSN:1557-7295
DOI:10.1145/3387915
- Editors:
- Zhaojun Bai,
- Wolfgang Bangerth
Copyright © 2020 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 March 2020
Accepted: 01 August 2019
Revised: 01 August 2019
Received: 01 December 2017
Published in TOMS Volume 46, Issue 1
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 412Total Downloads
- Downloads (Last 12 months)13
- Downloads (Last 6 weeks)1
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format