Skip to main content
Log in

High performance computing for first-principles Kohn-Sham density functional theory towards exascale supercomputers

  • Regular Paper
  • Published:
CCF Transactions on High Performance Computing Aims and scope Submit manuscript

Abstract

High performance computing (HPC) plays an essential role in enabling first-principles calculations based on the Kohn–Sham density functional theory (KS-DFT) for investigating quantum structural and electronic properties of large-scale molecules and solids in condensed matter physics, quantum chemistry and materials science. This review focuses on recent advances for HPC software development in large-scale KS-DFT calculations containing tens of thousands of atoms on modern heterogeneous supercomputers, especially for the HPC software with independent intellectual property rights supported on the Chinese domestic exascale supercomputers. We first introduce three various types of DFT software developed on modern heterogeneous supercomputers, involving PWDFT (Plane-Wave Density Functional Theory), HONPAS (Hefei Order-N Packages for Ab initio Simulations) and DGDFT (Discontinuous Galerkin Density Functional Theory), respectively based on three different types of basis sets (plane waves, numerical atomic orbitals and adaptive local basis functions). Then, we describe the theoretical algorithms and parallel implementation of these three software on modern heterogeneous supercomputers in detail. Finally, we conclude this review and propose several promising research fields for future large-scale KS-DFT calculations towards exascale supercomputers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Adamo, C., Barone, V.: Toward reliable density functional methods without adjustable parameters: the PBE0 model. J. Chem. Phys. 110(13), 6158–6170 (1999)

    Article  Google Scholar 

  • Apra, E., Bylaska, E.J., De Jong, W.A., Govind, N., Kowalski, K., Straatsma, T.P., Valiev, M., van Dam, H.J., Alexeev, Y., Anchell, J., et al.: Nwchem: Past, present, and future. J. Chem. Phys. 152(18), 184102 (2020)

    Article  Google Scholar 

  • Arita, M., Arapan, S., Bowler, D.R., Miyazaki, T.: Large-scale dft simulations with a linear-scaling dft code conquest on k-computer. J. Adv. Simulat. Sci. Eng. 1(1), 87–97 (2014)

    Article  Google Scholar 

  • Asadchev, A., Gordon, M.S.: Mixed-precision evaluation of two-electron integrals by rys quadrature. Comput. Phys. Commun. 183(8), 1563–1567 (2012)

    Article  Google Scholar 

  • Banerjee, A.S., Lin, L., Hu, W., et al.: Chebyshev polynomial filtered subspace iteration in the discontinuous galerkin method for large-scale electronic structure calculations. J. Chem. Phys. 145, 154101 (2016)

    Article  Google Scholar 

  • Banerjee, A.S., Lin, L., Suryanarayana, P., Yang, C., Pask, J.E.: Two-level chebyshev filter based complementary subspace method: Pushing the envelope of large-scale electronic structure calculations. J. Chem. Theory Comput. 14(6), 2930–2946 (2018)

    Article  Google Scholar 

  • Becke, A.D.: Density-functional exchange-energy approximation with correct asymptotic behavior. Phys. Rev. A 38(6), 3098 (1988)

    Article  Google Scholar 

  • Blum, V., Gehrke, R., Hanke, F., Havu, P., Havu, V., Ren, X., Reuter, K., Scheffler, M.: Ab initio molecular simulations with numeric atom-centered orbitals. Comput. Phys. Commun. 180(11), 2175–2196 (2009)

    Article  MATH  Google Scholar 

  • Bottin, F., Leroux, S., Knyazev, A., Zérah, G.: Large-scale ab initio calculations based on three levels of parallelization. Comp. Mater. Sci. 42(2), 329–336 (2008)

    Article  Google Scholar 

  • Bowler, D., Choudhury, R., Gillan, M., Miyazaki, T.: Recent progress with large-scale ab initio calculations: the conquest code. Phys. Status. Solidi. B. 243(5), 989–1000 (2006)

    Article  Google Scholar 

  • Bowler, D.R., Miyazaki, T.: Calculations for millions of atoms with density functional theory: linear scaling shows its potential. J. Phys. 22(7), 074207 (2010)

    Google Scholar 

  • Bowler, D.R., Miyazaki, T.: \(\cal{O}\)(n) methods in electronic structure calculations. Rep. Prog. Phys 75(3), 036503 (2012)

    Article  Google Scholar 

  • Buluç, A., Gilbert, J.R.: Parallel sparse matrix-matrix multiplication and indexing: Implementation and experiments. SIAM. J. Sci. Comput. 34(4), 170–191 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  • Cao, Y., Romero, J., Olson, J.P., Degroote, M., Johnson, P.D., Kieferova, M., Kivlichan, I.D., Menke, T., Peropadre, B., Sawaya, N.P.D., Sim, S., Veis, L., Aspuru-Guzik, A.: Quantum chemistry in the age of quantum computing. Chem. Rev. 119, 10856–10915 (2019)

    Article  Google Scholar 

  • Ceperley, D.M., Alder, B.J.: Ground state of the electron gas by a stochastic method. Phys. Rev. Lett. 45(7), 566 (1980)

    Article  Google Scholar 

  • Das, S., Motamarri, P., Gavini, V., Turcksin, B., Li, Y.W., Leback, B.: Fast, scalable and accurate finite-element based ab initio calculations using mixed precision computing: 46 pflops simulation of a metallic dislocation system. Association for Computing Machinery, New York, NY, USA (2019)

    Book  Google Scholar 

  • Davidson, E.R.: The iterative calculation of a few of the lowest eigenvalues and corresponding eigenvectors of large real-symmetric matrices. J. Comput. Phys. 17, 87–94 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  • Ehrenreich, H., Cohen, M.H.: Self-consistent field approach to the many-electron problem. Phys. Rev. 115(4), 786 (1959)

    Article  MathSciNet  MATH  Google Scholar 

  • Ernzerhof, M., Scuseria, G.E.: Assessment of the perdew–burke–ernzerhof exchange-correlation functional. J. Chem. Phys. 110(11), 5029–5036 (1999)

    Article  Google Scholar 

  • Fu, H., Liao, J., Yang, J., Wang, L., Song, Z., Huang, X., Yang, C., Xue, W., Liu, F., Qiao, F., et al.: The sunway taihulight supercomputer: system and applications. Sci. China Inf. Sci. 59(7), 1–16 (2016)

    Article  Google Scholar 

  • Giannozzi, P., Andreussi, O., Brumme, T., Bunau, O., Nardelli, M.B., Calandra, M., Car, R., Cavazzoni, C., Ceresoli, D., Cococcioni, M., et al.: Advanced capabilities for materials modelling with quantum espresso. J. Phys. 29(46), 465901 (2017)

    Google Scholar 

  • Giannozzi, P., Baroni, S., Bonini, N., et al.: QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. J. Phys. 21(39), 395502 (2009)

    Google Scholar 

  • Goedecker, S.: Linear scaling electronic structure methods. Rev. Mod. Phys. 71, 1085–1123 (1999)

    Article  Google Scholar 

  • Gonze, X., Amadon, B., Anglade, P.-M., Beuken, J.-M., Bottin, F., Boulanger, P., Bruneval, F., Caliste, D., Caracas, R., Côté, M., Deutsch, T., Genovese, L., Ghosez, P., Giantomassi, M., Goedecker, S., Hamann, D.R., Hermet, P., Jollet, F., Jomard, G., Leroux, S., Mancini, M., Mazevet, S., Oliveira, M.J.T., Onida, G., Pouillon, Y., Rangel, T., Rignanese, G.-M., Sangalli, D., Shaltaf, R., Torrent, M., Verstraete, M.J., Zerah, G., Zwanziger, J.W.: Abinit: First-principles approach to material and nanosystem properties. Comput. Phys. Commun. 180(12), 2582–2615 (2009)

    Article  Google Scholar 

  • Gygi, F., Draeger, E.W., Schulz, M., de Supinski, B.R., Gunnels, J.A., Austel, V., Sexton, J.C., Franchetti, F., Kral, S., Ueberhuber, C.W., Lorenz, J.: Large-scale electronic structure calculations of high-z metals on the BlueGene/L platform. Association for Computing Machinery, New York, NY, USA (2006)

    Google Scholar 

  • Hasegawa, Y., Iwata, J.-I., Tsuji, M., Takahashi, D., Oshiyama, A., Minami, K., Boku, T., Shoji, F., Uno, A., Kurokawa, M., Inoue, H., Miyoshi, I., Yokokawa, M.: First-principles calculations of electron states of a silicon nanowire with 100,000 atoms on the k computer. Association for Computing Machinery, New York, NY, USA (2011)

    Google Scholar 

  • Heyd, J., Scuseria, G.E., Ernzerhof, M.: Hybrid functionals based on a screened coulomb potential. J. Chem. Phys. 118(18), 8207–8215 (2003)

    Article  Google Scholar 

  • Hohenberg, P., Kohn, W.: Inhomogeneous electron gas. Phys. Rev. 136, 864–871 (1964)

    Article  MathSciNet  Google Scholar 

  • Hu, W., Lin, L., Banerjee, A.S., et al.: Adaptively compressed exchange operator for large-scale hybrid density functional calculations with applications to the adsorption of water on silicene. J. Chem. Theory Comput. 13(3), 1188–1198 (2017)

    Article  Google Scholar 

  • Hu, W., Lin, L., Yang, C.: DGDFT: A massively parallel method for large scale density functional theory calculations. J. Chem. Phys. 143(12), 124110 (2015)

    Article  Google Scholar 

  • Hu, W., Lin, L., Yang, C.: Edge reconstruction in armchair phosphorene nanoribbons revealed by discontinuous galerkin density functional theory. Phys. Chem. Chem. Phys. 17(47), 31397–31404 (2015)

    Article  Google Scholar 

  • Hu, W., Lin, L., Yang, C.: Interpolative separable density fitting decomposition for accelerating hybrid density functional calculations with applications to defects in silicon. J. Chem. Theory Comput. 13(11), 5420–5431 (2017)

    Article  Google Scholar 

  • Hu, W., Lin, L., Yang, C.: Projected commutator diis method for accelerating hybrid functional electronic structure calculations. J. Chem. Theory Comput. 13, 5458–5467 (2017)

    Article  Google Scholar 

  • Hu, W., Qin, X., Jiang, Q., Chen, J., An, H., Jia, W., Li, F., Liu, X., Chen, D., Liu, F., et al.: High performance computing of DGDFT for tens of thousands of atoms using millions of cores on Sunway TaihuLight. Sci. Bull. 66(2), 111–119 (2021)

    Article  Google Scholar 

  • Igram, D., Bhattarai, B., Biswas, P., Drabold, D.A.: Large and realistic models of amorphous silicon. J. Non. Cryst. Solids. 492, 27–32 (2018)

    Article  Google Scholar 

  • Jia, W., Cao, Z., Wang, L., Fu, J., Chi, X., Gao, W., Wang, L.-W.: The analysis of a plane wave pseudopotential density functional theory code on a gpu machine. Comput. Phys. Commun. 184(1), 9–18 (2013)

    Article  Google Scholar 

  • Knyazev, A.V.: Toward the optimal preconditioned eigensolver: locally optimal block preconditioned conjugate gradient method. SIAM J Sci. Comput. 23(2), 517–541 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  • Kohn, W.: Density functional and density matrix method scaling linearly with the number of atoms. Phys. Rev. Lett. 76, 3168–3171 (1996)

    Article  Google Scholar 

  • Kohn, W., Sham, L.J.: Self-consistent equations including exchange and correlation effects. Phys. Rev. 140(4A), 1133 (1965)

    Article  MathSciNet  Google Scholar 

  • Kresse, G., Hafner, J.: Ab initio molecular dynamics for liquid metals. Phys Rev B 47, 558 (1993)

    Article  Google Scholar 

  • Kühne, T.D., Iannuzzi, M., Del Ben, M., Rybkin, V.V., Seewald, P., Stein, F., Laino, T., Khaliullin, R.Z., Schütt, O., Schiffmann, F., et al.: CP2K: An electronic structure and molecular dynamics software package-quickstep: Efficient and accurate electronic structure calculations. J. Chem. Phys. 152(19), 194103 (2020)

    Article  Google Scholar 

  • Lanczos, C.: Applied Analysis. Dover, New York (1988)

    MATH  Google Scholar 

  • Li, X.S., Demmel, J.W.: Superlu_dist: a scalable distributed-memory sparse direct solver for unsymmetric linear systems. ACM Trans. Math.AD Softw. (TOMS) 29(2), 110–140 (2003)

    Article  MATH  Google Scholar 

  • Li, P., Liu, X., Chen, M., Lin, P., Ren, X., Lin, L., Yang, C., He, L.: Large-scale ab initio simulations based on systematically improvable atomic basis. Comput. Mater. Sci. 112, 503–517 (2016)

    Article  Google Scholar 

  • Lin, L.: Adaptively compressed exchange operator. J. Chem. Theory Comput. 12(5), 2242–2249 (2016)

    Article  Google Scholar 

  • Lin, L., Chen, M., Yang, C., He, L.: Accelerating atomic orbital-based electronic structure calculation via pole expansion and selected inversion. J. Phys. 25(29), 295501 (2013)

    Google Scholar 

  • Lin, L., García, A., Huhs, G., et al.: SIESTA-PEXSI: Massively parallel method for efficient and accurate ab initio materials simulation without matrix diagonalization. J. Phys.A 26, 305503 (2014)

    Google Scholar 

  • Lin, L., García, A., Huhs, G., Yang, C.: SIESTA-PEXSI: massively parallel method for efficient and accurateab initiomaterials simulation without matrix diagonalization. J. Phys. 26(30), 305503 (2014)

    Google Scholar 

  • Lin, L., Lu, J., Ying, L.E.W.: Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework I: Total energy calculation. J. Comput. Phys 231(4), 2140–2154 (2012)

    Article  MATH  Google Scholar 

  • Lin, L., Lu, J., Ying, L., Car, R.E.W.: Fast algorithm for extracting the diagonal of the inverse matrix with application to the electronic structure analysis of metallic systems. Comm. Math. Sci 7(3), 755–777 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • Lin, P., Ren, X., He, L.: Efficient hybrid density functional calculations for large periodic systems using numerical atomic orbitals. J. Chem. Theory Comput. 17(1), 222–239 (2021)

    Article  Google Scholar 

  • Lucignano, P., Alfè, D., Cataudella, V., Ninno, D., Cantele, G.: Crucial role of atomic corrugation on the flat bands and energy gaps of twisted bilayer graphene at the magic angle \(\theta \sim 1.{08}^{\circ }\). Phys. Rev. B 99(19), 195419 (2019)

    Article  Google Scholar 

  • Ma, H., Wang, L., Wan, L., Li, J., Qin, X., Liu, J., Hu, W., Lin, L., Yang, C., Yang, J.: Realizing effective cubic-scaling coulomb hole plus screened exchange approximation in periodic systems via interpolative separable density fitting with a plane-wave basis set. J. Phys. Chem. A 125(34), 7545–7557 (2021)

    Article  Google Scholar 

  • Paier, J., Ren, X., Rinke, P., Scuseria, G.E., Grüneis, A., Kresse, G., Scheffler, M.: Assessment of correlation energies based on the random-phase approximation. New J. Phys. 14(4), 043002 (2012)

    Article  Google Scholar 

  • Perdew, J.P., Burke, K., Ernzerhof, M.: Generalized gradient approximation made simple. Phys. Rev. Lett. 77(18), 3865 (1996)

    Article  Google Scholar 

  • Perdew, J.P., Schmidt, K.: Jacob’s ladder of density functional approximations for the exchange-correlation energy. AIP Conf. Proc. 577(1), 1–20 (2001)

    Google Scholar 

  • Qin, X., Shang, H., Xiang, H., Li, Z., Yang, J.: HONPAS: A linear scaling open-source solution for large system simulations. Int. J. Quantum. Chem. 115(10), 647–655 (2015)

    Article  Google Scholar 

  • Ratcliff, L.E., Dawson, W., Fisicaro, G., Caliste, D., Mohr, S., Degomme, A., Videau, B., Cristiglio, V., Stella, M., D’Alessandro, M., et al.: Flexibilities of wavelets as a computational basis set for large-scale electronic structure calculations. J. Chem. Phys. 152(19), 194110 (2020)

    Article  Google Scholar 

  • Shang, H., Li, Z., Yang, J.: Implementation of screened hybrid density functional for periodic systems with numerical atomic orbitals: Basis function fitting and integral screening. J. Chem. Phys. 135(3), 034110 (2011)

    Article  Google Scholar 

  • Shang, H., Li, F., Zhang, Y., Zhang, L., Fu, Y., Gao, Y., Wu, Y., Duan, X., Lin, R., Liu,X., Liu, Y., Chen, D.: Extreme-scale ab initio quantum raman spectra simulations on the leadership HPC system in China. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021.

  • Shang, H., Xu, L., Wu, B., Qin, X., Zhang, Y., Yang, J.: The dynamic parallel distribution algorithm for hybrid density-functional calculations in HONPAS package. Comput. Phys. Commun. 254, 107204 (2020)

    Article  MathSciNet  Google Scholar 

  • Soler, J.M., Artacho, E., Gale, J.D., García, A., Junquera, J., Ordejón, P., Sánchez-Portal, D.: The SIESTA method for ab initio order-N materials simulation. J. Phys. Condens. Matter. 14(11), 2745 (2002)

    Article  Google Scholar 

  • Stephens, P.J., Devlin, F.J., Chabalowski, C.F., Frisch, M.J.: Ab initio calculation of vibrational absorption and circular dichroism spectra using density functional force fields. J. Phys. Chem. 98(45), 11623–11627 (1994)

    Article  Google Scholar 

  • Strohmaier, E., Dongarra, J., Simon, H., Meuer, M., Meuer, H.: Top500 list. https://www.top500.org (2021)

  • Sun, J., Ruzsinszky, A., Perdew, J.P.: Strongly constrained and appropriately normed semilocal density functional. Phys. Rev. Lett. 115(3), 036402 (2015)

    Article  Google Scholar 

  • Tao, J., Perdew, J.P., Staroverov, V.N., Scuseria, G.E.: Climbing the density functional ladder: Nonempirical meta-generalized gradient approximation designed for molecules and solids. Phys. Rev. Lett. 91(14), 146401 (2003)

    Article  Google Scholar 

  • Tian, Y., Suo, B., Ma, Y., Jin, Z.: Optimizing two-electron repulsion integral calculations with McMurchie-Davidson method on graphic processing unit. J. Chem. Phys. 155(3), 034112 (2021)

    Article  Google Scholar 

  • Tsuchida, E., Choe, Y.-K.: Iterative diagonalization of symmetric matrices in mixed precision and its application to electronic structure calculations. Comput. Phys. Commun. 183(4), 980–985 (2012)

    Article  MATH  Google Scholar 

  • Valeev, E.F.: Libint: A library for the evaluation of molecular integrals of many-body operators over Gaussian functions. http://libint.valeyev.net/. version 2.7.1 (2021)

  • VandeVondele, J., Borštnik, U., Hutter, J.: Linear scaling self-consistent field calculations with millions of atoms in the condensed phase. J. Chem. Theory Comput. 8(10), 3565–3573 (2012)

    Article  Google Scholar 

  • Vecharynski, E., Yang, C., Pask, J.E.: A projected preconditioned conjugate gradient algorithm for computing many extreme eigenpairs of a Hermitian matrix. J. Comput. Phys. 290(1), 73–89 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  • Vinson, J.: Faster exact exchange in periodic systems using single-precision arithmetic. J. Chem. Phys. 153(20), 204106 (2020)

    Article  Google Scholar 

  • Wan, L., Liu, X., Liu, J., Qin, X., Hu, W., Yang, J.: Hybrid mpi and openmp parallel implementation of large-scale linear-response time-dependent density functional theory with plane-wave basis set. Electron. Struct. 3, 024004 (2021)

    Article  Google Scholar 

  • Wang, L.: Divide-and-conquer quantum mechanical material simulations with exascale supercomputers. Natl. Sci. Rev. 1(4), 604–617 (2014)

    Article  Google Scholar 

  • Wang, L.-W., Lee, B., Shan, H., Zhao, Z., Meza, J., Strohmaier, E., Bailey, D.H.: Linearly scaling 3D fragment method for large-scale electronic structure calculations. IEEE Press, New York, NY, USA (2008)

    Book  Google Scholar 

  • Yang, W.: Direct calculation of electron density in density-functional theory. Phys. Rev. Lett. 66, 1438–1441 (1991)

    Article  Google Scholar 

  • Zhang, G., Lin, L., Hu, W., et al.: Adaptive local basis set for Kohn-Sham density functional theory in a discontinuous Galerkin framework II: Force, vibration, and molecular dynamics calculations. J. Comput. Phys. 335, 426–443 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  • Zhou, Y., Saad, Y., Tiago, M.L., et al.: Self-consistent-field calculations using chebyshev-filtered subspace iteration. J. Comput. Phys. 219, 172–184 (2006)

    Article  MATH  Google Scholar 

Download references

Acknowledgements

This work is partly supported by the National Natural Science Foundation of China (21688102, 21803066, 22003061, 22173093), by the Hefei National Laboratory for Physical Sciences at the Microscale (KF2020003), the Chinese Academy of Sciences Pioneer Hundred Talents Program (KJ2340000031), the Anhui Initiative in Quantum Information Technologies (AHY090400), the CAS Project for Young Scientists in Basic Research (YSBR-005), the Strategic Priority Research Program of Chinese Academy of Sciences (XDC01040100), the Fundamental Research Funds for the Central Universities (WK2340000091, WK2060000018), the Hefei National Laboratory for Physical Sciences at the Microscale (SK2340002001), the Research Start-Up Grants (KY2340000094) and the Academic Leading Talents Training Program (KY2340000103) from University of Science and Technology of China. The authors thank the Hefei Advanced Computing Center, the Supercomputing Center of Chinese Academy of Sciences (SunRising-1), the Supercomputing Center of USTC, the National Supercomputing Center in Wuxi, Tianjin, Shanghai, and Guangzhou for the computational resources.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Wei Hu.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, X., Chen, J., Luo, Z. et al. High performance computing for first-principles Kohn-Sham density functional theory towards exascale supercomputers. CCF Trans. HPC 5, 26–42 (2023). https://doi.org/10.1007/s42514-022-00120-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42514-022-00120-0

Keywords

Navigation