Skip to main content
SpringerLink
Log in

Towards efficient tile low-rank GEMM computation on sunway many-core processors

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Tile low-rank general matrix multiplication (TLR GEMM) is a novel method of matrix multiplication on large data-sparse matrices, which can significantly reduce storage footprint and arithmetic complexity under given accuracy. To implement high-performance TLR GEMM on Sunway many-core processor, the following challenges remain to be addressed: 1) design an efficient parallel scheme; 2) provide an efficient kernel library of math functions commonly used in TLR GEMM. This paper proposes swTLR GEMM, an efficient implementation of TLR GEMM. We assign LR GEMM computation to a single computing processing element (CPE) and use grouped task queue to process different data tiles of the TLR matrix. Moreover, we implement an efficient kernel library (swLR Kernels) for low-rank matrix operations. To scale to massive (CGs), we organize the CGs into the CG grid and partition the matrices into blocks accordingly. We also apply Cannon’s algorithm to enable efficient communication when processing the matrix blocks across CGs simultaneously. The experiment results show that the DGEMM kernel in swLR Kernels achieves 102\(\times\) speedup on average. In terms of overall performance, swTLR GEMM-LLD and swTLR GEMM-LLL achieve 91\(\times\) and 20.1\(\times\) speedup on average, respectively. In addition, our implementation of swTLR GEMM exhibits good scalability when running on 1,024 CGs of Sunway processors (66,560 cores in total).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Wolfgang Hackbusch (1999) A sparse matrix arithmetic based on \(\cal{H}\)-matrices. part i: Introduction to \({\cal{H}}\)-matrices. Computing 62(2):89–108

    Article  MathSciNet  Google Scholar 

  2. Grasedyck L, Hackbusch Wolfgang (2003) Construction and arithmetics of \({\cal{H}}\)-matrices. Computing 70(4):295–334

    Article  MathSciNet  Google Scholar 

  3. Akbudak K, Ltaief H, Mikhalev A, and Keyes D 2017) Tile low rank cholesky factorization for climate/weather modeling applications on manycore architectures. In: International Supercomputing Conference, pp 22–40. Springer

  4. Charara A, Keyes D, and Ltaief H (2018) Tile low-rank gemm using batched operations on gpus. In: European Conference on Parallel Processing, pp 811–825. Springer

  5. Susan BL, Antoine P, Roldan P, Karin R, Clint WR, James D, Jack D, Iain D, Sven H, Greg Henry et al (2002) An updated set of basic linear algebra subprograms (blas). ACM Trans Math Softw 28(2):135–151

    Article  MathSciNet  Google Scholar 

  6. Kriemann Ronald (2005) Parallel \({\cal{H}}\)-matrix arithmetics on shared memory systems. Computing 74(3):273–297

    Article  MathSciNet  Google Scholar 

  7. Halim BW, George T, Hatem L, Keyes David E (2018) Batched qr and svd algorithms on gpus with applications in hierarchical matrix compression. Parallel Comput 74:19–33

    Article  MathSciNet  Google Scholar 

  8. Nvidia CUDA (2008) Cublas library. NVIDIA Corporation, Santa Clara, CaliforniaSanta Clara, CaliforniaSanta Clara, CaliforniaSanta Clara, California, p 31

    Google Scholar 

  9. Augonnet C, Thibault S, Namyst R, Wacrenier Pierre-André (2011) Starpu: a unified platform for task scheduling on heterogeneous multicore architectures. Concurr Comput: Pract Exp 23(2):187–198

    Article  Google Scholar 

  10. Dongarra J (2016) Report on the sunway taihulight system. PDF). www. netlib. org. Retrieved June, 20,

  11. Haohuan F, Liao J, Yang J, Wang L, Song Z, Huang X, Yang C, Xue W, Liu F, Qiao Fangli et al (2016) The sunway taihulight supercomputer: system and applications. Sci China Inf Sci 59(7):072001

    Article  Google Scholar 

  12. Jiang L, Yang C, Ao Y, Yin W, Ma W, Sun Q, Liu F, Lin R, and Zhang P (2017) Towards highly efficient dgemm on the emerging sw26010 many-core processor. In: 2017 46th International Conference on Parallel Processing (ICPP), pp 422–431. IEEE

  13. Fang J, Fu H, Zhao W, Chen B, Zheng W, and Yang G (2017) swdnn: a library for accelerating deep learning applications on sunway taihulight. In: 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp 615–624. IEEE

  14. de Dinechin BD Ayrignac R, Beaucamps PE, Couvert P, Ganne B, de Massas PG Jacquet F, Jones S, Chaisemartin NM, Riss F et al (2013) A clustered manycore processor architecture for embedded and accelerated applications. In: 2013 IEEE High Performance Extreme Computing Conference (HPEC), pp 1–6. IEEE

  15. Çatalyürek Ümit V, Feo J, Gebremedhin AH, Halappanavar M, Pothen A (2012) Graph coloring algorithms for multi-core and massively multithreaded architectures. Parallel Comput 38(10–11):576–594

    Article  MathSciNet  Google Scholar 

  16. Williams S, Shalf J , Oliker L, Kamil S, Husbands P, and Yelick K (2006) The potential of the cell processor for scientific computing. In: Proceedings of the 3rd Conference on Computing Frontiers, pp 9–20

  17. Hackbusch W, Khoromskij B, Sauter SA (2000) On \({\cal{H}}^2\)-matrices. Lectures on applied mathematics. Springer, Berlin, pp 9–29

    MATH  Google Scholar 

  18. Rouet FH, Li XS, Ghysels P, Napov A (2016) A distributed-memory package for dense hierarchically semi-separable matrix computations using randomization. ACM Trans Math Softw (TOMS) 42(4):27

    Article  MathSciNet  Google Scholar 

  19. Ambikasaran S, Darve E (2013) An \({\cal{O}}(n \log n)\) fast direct solver for partial hierarchically semi-separable matrices. J Sci Comput 57(3):477–501

    Article  MathSciNet  Google Scholar 

  20. Amestoy P, Ashcraft C, Boiteau O, Buttari A, L’Excellent JY, Weisbecker Clément (2015) Improving multifrontal methods by means of block low-rank representations. SIAM J Sci Comput 37(3):A1451–A1474

    Article  MathSciNet  Google Scholar 

  21. Kriemann Ronald (2013) \({\cal{H}}\)-lu factorization on many-core systems. Comput Visualiz Sci 16(3):105–117

    Article  MathSciNet  Google Scholar 

  22. Noha Al-Harthi, Rabab Alomairy, Kadir Akbudak, Rui Chen, Hatem Ltaief, Hakan Bagci, and David E. Keyes. Solving acoustic boundary integral equations using high performance tile low-rank LU factorization. In: 2020 International Conference on High Performance Computing (ISC), pp 209–229. Springer

  23. Cao Q, Pei Y, Akbudak K, Mikhalev A, Bosilca G, Ltaief H, Keyes D, and Dongarra J (2020) Extreme-scale task-based cholesky factorization toward climate and weather prediction applications. In: Proceedings of the Platform for Advanced Scientific Computing Conference, pp 1–11

  24. Duan X, Gao P, Zhang T, Zhang M, Liu W, Zhang W , Xue W, Fu H, Gan L, Chen D et al (2018) Redesigning lammps for peta-scale and hundred-billion-atom simulation on sunway taihulight. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, p 12. IEEE Press

  25. Chen B, Fu H, Wei Y, He C, Zhang W, Li Y, Wan W, Zhang W, Gan L, Zhang W et al (2018) Simulating the wenchuan earthquake with accurate surface topography on sunway taihulight. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, p 40. IEEE Press

  26. Lin H, Zhu X, Yu B, Tang X, Xue W, Chen W, Zhang L , Hoefler T, Ma X, Liu X et al (2018) hentu: processing multi-trillion edge graphs on millions of cores in seconds. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, pp 56. IEEE Press

  27. Yongmin H, Yang H, Luan Z, Gan L, Yang G, Qian Depei (2019) Massively scaling seismic processing on sunway taihulight supercomputer. IEEE Trans Parallel Distrib Syst 31(5):1194–1208

    Google Scholar 

  28. Fu H, Liao J, Ding N, Duan X, Gan L, Liang Y, Wang X, Yang J, Zheng Y, Liu W et al (2017) Redesigning cam-se for peta-scale climate modeling performance and ultra-high resolution on sunway taihulight. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, p 1. ACM

  29. Liu C, Yang H, Sun R, Luan Z, and Qian D (2019) swtvm: Exploring the automated compilation for deep learning on sunway architecture. arXiv preprint arXiv:1904.07404,

  30. Li L, Fang J, Fu H, Jiang J, Zhao W, He C, You X, and Yang G (2018) swcaffe: a parallel framework for accelerating deep learning applications on sunway taihulight. In: 2018 IEEE International Conference on Cluster Computing (CLUSTER), pp 413–422. IEEE

  31. Zhong X, Li M, Yang H, Liu Y, Qian D (2018) swMR: a framework for accelerating mapreduce applications on sunway taihulight. IEEE Trans Emerg Topics Comput. https://doi.org/10.1109/TETC.2018.2881265

    Article  Google Scholar 

  32. Liu C, Xie B, Liu X, Xue W, Yang H, and Liu X (2018) Towards efficient spmv on sunway manycore architectures. In: Proceedings of the 2018 International Conference on Supercomputing, pp 363–373. ACM

  33. Li M, Liu Y, Yang H, Luan Z, and Qian D (2018) Multi-role sptrsv on sunway many-core architecture. In: 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pp 594–601. IEEE

  34. Wang X, Liu W, Xue W , and Wu L (2018) swsptrsv: a fast sparse triangular solve with sparse level tile layout on sunway architectures. In: Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pp 338–353. ACM

  35. Ayguadé E, Copty N, Duran A, Hoeflinger J, Lin Y, Massaioli F, Teruel X, Unnikrishnan P, Zhang G (2008) The design of openmp tasks. IEEE Trans Parallel Distrib Syst 20(3):404–418

    Article  Google Scholar 

  36. Alejandro D, Eduard A, Badia Rosa M, Jesús L, Luis M, Xavier M, Judit P (2011) Ompss: a proposal for programming heterogeneous multi-core architectures. Parallel process lett 21(02):173–193

    Article  MathSciNet  Google Scholar 

  37. Kishore Kumar N, Schneider J (2017) Literature survey on low rank approximation of matrices. Linear Multilinear Algebra 65(11):2212–2244

    Article  MathSciNet  Google Scholar 

  38. Bebendorf M (2011) Adaptive cross approximation of multivariate functions. Construct Approx 34(2):149–179

    Article  MathSciNet  Google Scholar 

  39. Chan TF (1987) Rank revealing qr factorizations. Linear algebra Appl 88:67–82

    MathSciNet  MATH  Google Scholar 

  40. Halko N, Martinsson PG, Tropp JA (2011) Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev 53(2):217–288

    Article  MathSciNet  Google Scholar 

  41. Murphy KP (2012) Machine learning: a probabilistic perspective. MIT press, Cambridge

    MATH  Google Scholar 

  42. Skillicorn David (2007) Understanding complex datasets: data mining with matrix decompositions. CRC Press, Boca Raton

    Book  Google Scholar 

  43. Li X, Shen B, Liu BD, Zhang YJ (2016) A locality sensitive low-rank model for image tag completion. IEEE Trans Multimed 18(3):474–483

    Article  Google Scholar 

  44. Park H and Elden L (2003) Matrix rank reduction for data analysis and feature extraction. Technical report, Tr 03-015, University of Minnesota

  45. Li M, Liu Y, Yang H, Luan Z, Gan L, Yang G, Qian D (2019) Accelerating sparse cholesky factorization on sunway manycore architecture. IEEE Trans Parallel Distrib Syst 31(7):1636–1650

    Article  Google Scholar 

  46. Van Zee Field G, Van De Geijn RA (2015) Blis: a framework for rapidly instantiating blas functionality. ACM Trans Math Softw 41(3):1–33

    MathSciNet  MATH  Google Scholar 

  47. Anderson E, Bai Z, Bischof C, Blackford S, Dongarra J, Du Croz J, Greenbaum A, Hammarling S, McKenney A, Sorensen D (1999) LAPACK users’ guide, vol 9. Society for industrial and applied mathematics

  48. Gander Walter (1980) Algorithms for the qr decomposition. Res. Rep 80(02):1251–1268

    Google Scholar 

  49. Golub HG, Van Loan Charles F (1996) Matrix computations. Johns hopkins university Press, London

    MATH  Google Scholar 

  50. Wilkinson JH, Bauer FL, Reinsch C (2013) Linear algebra, vol 2. Springer, Berlin

    Google Scholar 

  51. Cannon LE (1969) A cellular computer to implement the Kalman filter algorithm. PhD thesis, Montana State University-Bozeman, College of Engineering

  52. Strassen V (1969) Gaussian elimination is not optimal. Numer Mathem 13(4):354–356

    Article  MathSciNet  Google Scholar 

  53. Van De Geijn RA, Watts J (1997) Summa: scalable universal matrix multiplication algorithm. Concurr: Pract Exp 9(4):255–274

    Article  Google Scholar 

  54. Solomonik E and Demmel J (2011) Communication-optimal parallel 2.5 d matrix multiplication and lu factorization algorithms. In: European Conference on Parallel Processing, pp 90–109. Springer

  55. Demmel J, Eliahu D, Fox A, Kamil S, Lipshitz B, Schwartz O, and Spillinger O (2013) Communication-optimal parallel recursive rectangular matrix multiplication. In: 2013 IEEE 27th International Symposium on Parallel and Distributed Processing, pp 261–272. IEEE

  56. Kwasniewski G, Kabić M, Besta M, VandeVondele J , Solcà R, and Hoefler T (2019) Red-blue pebbling revisited: near optimal parallel matrix-matrix multiplication. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp 1–22

  57. Yi-Han X, Yang CC, Hua M, Zhou Wen (2020) Deep deterministic policy gradient (ddpg)-based resource allocation scheme for noma vehicular communications. IEEE Access 8:18797–18807

    Article  Google Scholar 

  58. Yi-Han X, Xie JW, Zhang YG, Hua M, Zhou Wen (2020) Reinforcement learning (rl)-based energy efficient resource allocation for energy harvesting-powered wireless body area network. Sensors 20(1):44

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank all anonymous reviewers for their insightful comments and suggestions. This work is supported by National Key R&D Program of China (Grant No. 2020YFB150001), National Natural Science Foundation of China (Grant No. 62072018), the Open Project Program of the State Key Laboratory of Mathematical Engineering and Advanced Computing (Grant No. 2019A12) and Center for High Performance Computing and System Simulation, Pilot National Laboratory for Marine Science and Technology (Qingdao). Hailong Yang is the corresponding author.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Beihang University, Beijing, 100191, China

    Qingchang Han, Hailong Yang, Zhongzhi Luan & Depei Qian

  2. State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, China

    Hailong Yang

  3. School of Cyber Science and Technology, Beihang University, Beijing, 100191, China

    Ming Dun

  4. Department of Computer Science and Technology, Tsinghua University, Beijing, 100084, China

    Lin Gan & Guangwen Yang

Authors
  1. Qingchang Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hailong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ming Dun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhongzhi Luan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lin Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guangwen Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Depei Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hailong Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, Q., Yang, H., Dun, M. et al. Towards efficient tile low-rank GEMM computation on sunway many-core processors. J Supercomput 77, 4533–4564 (2021). https://doi.org/10.1007/s11227-020-03444-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-020-03444-2

Keywords

Search

Navigation