A communication-avoiding 3D sparse triangular solver

Sao, Piyush; Kannan, Ramakrishnan {ramki}; Li, Xiaoye Sherry; Vuduc, Richard

doi:10.1145/3330345.3330357

Title: A communication-avoiding 3D sparse triangular solver

Conference · Sat Jun 01 00:00:00 EDT 2019

DOI:https://doi.org/10.1145/3330345.3330357· OSTI ID:1558528

^[1];

^[1]; Li, Xiaoye Sherry ^[2]; Vuduc, Richard ^[3]

ORNL
Lawrence Berkeley National Laboratory (LBNL)
Georgia Institute of Technology, Atlanta

We present a novel distributed memory algorithm to improve the strong scalability of the solution of a sparse triangular system. This operation appears in the solve phase of direct methods for solving general sparse linear systems, Ax = b. Our 3D sparse triangular solver employs several techniques, including a 3D MPI process grid, elimination tree parallelism, and data replication, all of which reduce the per-process communication when combined. We present analytical models to understand the communication cost of our algorithm and show that our 3D sparse triangular solver can reduce the per-process communication volume asymptotically by a factor of O(n1/4) and O(n1/6) for problems arising from the finite element discretizations of 2D "planar" and 3D "non-planar" PDEs, respectively. We implement our algorithm for use in SuperLU_DIST3D, using a hybrid MPI+OpenMP programming model. Our 3D triangular solve algorithm, when run on 12k cores of Cray XC30, outperforms the current state-of-the-art 2D algorithm by 7.2x for planar and 2.7x for the non-planar sparse matrices, respectively.

View Conference

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1558528

Resource Relation:: Conference: International Conference on Supercomputing (ICS 2019) - Phoenix, Arizona, United States of America - 6/26/2019 8:00:00 AM-6/28/2019 8:00:00 AM

Country of Publication:: United States

Language:: English

References (16)

Nested Dissection of a Regular Finite Element Mesh George, Alan SIAM Journal on Numerical Analysis, Vol. 10, Issue 2 https://doi.org/10.1137/0710032	journal	April 1973
Communication-Avoiding Parallel Sparse-Dense Matrix-Matrix Multiplication Koanantakool, Penporn; Azad, Ariful; Buluc, Aydin 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.117	conference	May 2016
On asynchronous iterations Frommer, Andreas; Szyld, Daniel B. Journal of Computational and Applied Mathematics, Vol. 123, Issue 1-2 https://doi.org/10.1016/S0377-0427(00)00409-X	journal	November 2000
Communication results for parallel sparse Cholesky factorization on a hypercube George, Alan; Liu, Joseph W. H.; Ng, Esmond Parallel Computing, Vol. 10, Issue 3 https://doi.org/10.1016/0167-8191(89)90101-4	journal	May 1989
Structure-adaptive parallel solution of sparse triangular linear systems Totoni, Ehsan; Heath, Michael T.; Kale, Laxmikant V. Parallel Computing, Vol. 40, Issue 9 https://doi.org/10.1016/j.parco.2014.06.006	journal	October 2014
Convergence Models and Surprising Results for the Asynchronous Jacobi Method Chow, Edmond; Chow, Edmond 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2018.00103	conference	May 2018
Highly scalable parallel algorithms for sparse matrix factorization Gupta, A.; Karypis, G.; Kumar, V. IEEE Transactions on Parallel and Distributed Systems, Vol. 8, Issue 5 https://doi.org/10.1109/71.598277	journal	May 1997
A Communication-Avoiding 3D LU Factorization Algorithm for Sparse Matrices Sao, Piyush; Li, Xiaoye Sherry; Vuduc, Richard 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2018.00100	conference	May 2018
Integrated Model, Batch, and Domain Parallelism in Training Neural Networks Gholami, Amir; Azad, Ariful; Jin, Peter Proceedings of the 30th on Symposium on Parallelism in Algorithms and Architectures https://doi.org/10.1145/3210377.3210394	conference	July 2018
Communication-Avoiding Parallel Algorithms for Solving Triangular Systems of Linear Equations Wicky, Tobias; Solomonik, Edgar; Hoefler, Torsten 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2017.104	conference	May 2017
Exploiting Multiple Levels of Parallelism in Sparse Matrix-Matrix Multiplication Azad, Ariful; Ballard, Grey; Buluç, Aydin SIAM Journal on Scientific Computing, Vol. 38, Issue 6 https://doi.org/10.1137/15M104253X	journal	January 2016
Efficient Parallel Sparse Triangular Solution Using Selective Inversion Raghavan, Padma Parallel Processing Letters, Vol. 08, Issue 01 https://doi.org/10.1142/S0129626498000067	journal	March 1998
Trading Replication for Communication in Parallel Distributed-Memory Dense Solvers Irony, Dror; Toledo, Sivan Parallel Processing Letters, Vol. 12, Issue 01 https://doi.org/10.1142/S0129626402000847	journal	March 2002
A New Data-Mapping Scheme for Latency-Tolerant Distributed Sparse Triangular Solution Teranishi, K.; Raghavan, P. ACM/IEEE SC 2002 Conference (SC'02) https://doi.org/10.1109/SC.2002.10020	conference	January 2002
Parallel Algorithms for Sparse Linear Systems Heath, Michael T.; Ng, Esmond; Peyton, Barry W. SIAM Review, Vol. 33, Issue 3 https://doi.org/10.1137/1033099	journal	September 1991
Avoiding communication in sparse matrix computations Demmel, James; Hoemmen, Mark; Mohiyuddin, Marghoob Distributed Processing Symposium (IPDPS), 2008 IEEE International Symposium on Parallel and Distributed Processing https://doi.org/10.1109/IPDPS.2008.4536305	conference	April 2008

Similar Records

A communication-avoiding 3D algorithm for sparse LU factorization on heterogeneous systems

Journal Article · Mon Aug 19 00:00:00 EDT 2019 · Journal of Parallel and Distributed Computing · OSTI ID:1558528

Sao, Piyush; Li, Xiaoye Sherry; Vuduc, Richard

A Communication-Avoiding 3D LU Factorization Algorithm for Sparse Matrices

Conference · Tue May 01 00:00:00 EDT 2018 · OSTI ID:1558528

Sao, Piyush; Li, Xiaoye Sherry; Vuduc, Richard

Highly scalable distributed-memory sparse triangular solution algorithms.

Conference · Mon Jan 01 00:00:00 EST 2018 · OSTI ID:1558528

Liu, Yang; Jacquelin, Mathias; Ghysels, Pieter; +1 more

Title: A communication-avoiding 3D sparse triangular solver

Citation Formats

References (16)

Similar Records

Related Subjects