Skip to main content
SpringerLink
Log in

Implementation of scalable bidomain-based 3D cardiac simulations on a graphics processing unit cluster

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Computational models of the human cardiac cells provide detailed properties of human ventricular cells. The execution time for a realistic 3D heart simulation based on these models is a major barrier for physicians to study and understand the heart diseases, and evaluate hypotheses rapidly toward developing treatments. Graphics processing unit (GPU)-based parallelization efforts to this date have been shown to be more effective than parallelization efforts on the CPU-based clusters in terms of addressing the 3D cardiac simulation time challenge. In this paper, we review all GPU-based studies and investigate both the cardiac cell models and cardiac tissue models in 3D space. We propose algorithmic optimizations based on red black successive over-relaxation method for reducing the number of simulation iterations and convergence method for dependence elimination between neighboring cells of the heart tissue. We investigate data transfer reduction and 2D mesh partitioning strategies, evaluate their impact on thread utilization, and propose a strongly scalable cardiac simulation. Our implementation results with reducing the execution time by a factor of five compared to the state-of-the-art baseline implementation. More importantly, our implementation is an important step toward achieving real-time cardiac simulations as it achieves the strongest scalability among all other cluster-based implementations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Desai AS, Stevenson LW (2012) Rehospitalization for heart failure: predict or prevent? Circulation 126(4):501–506

    Article  Google Scholar 

  2. Cheng A, Dalal D, Butcher B, Norgard S, Zhang Y, Dickfeld T, Eldadah ZA, Ellenbogen KA, Guallar E, Tomaselli GF (2013) Prospective observational study of implantable cardioverter-defibrillators in primary prevention of sudden cardiac death: study design and cohort description. J Am Heart Assoc 2(1):e000083

    Article  Google Scholar 

  3. Nimmagadda VK, Akoglu A, Hariri S, Moukabary T (2012) Cardiac simulation on multi-GPU platform. J Supercomput 59(3):1360–1378

    Article  Google Scholar 

  4. Biffard R, Leon LJ (2003) Cardiac tissue simulation using graphics hardware. In: Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2003. IEEE, vol 3, pp 2838–2840

  5. Rocha BM, Campos FO, Amorim RM, Plank G, dos Santos RW, Liebmann M, Haase G (2011) Accelerating cardiac excitation spread simulations using graphics processing units. Concurr Comput Pract Exp 23(7):708–720

    Article  Google Scholar 

  6. Vigmond EJ, Boyle PM, Leon LJ, Plank G (2009) Near-real-time simulations of biolelectric activity in small mammalian hearts using graphical processing units. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2009. EMBC 2009. IEEE, pp 3290–3293

  7. Amorim R, Haase G, Liebmann M, Dos Santos RW (2009) Comparing CUDA and OpenGL implementations for a Jacobi iteration. In: HPCS’09. International Conference on High Performance Computing & Simulation, 2009. IEEE, pp 22–32

  8. Amorim RM, Rocha BM, Campos FO, dos Santos RW (2010) Automatic code generation for solvers of cardiac cellular membrane dynamics in GPUs. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp 2666–2669

  9. Bartocci E, Cherry EM, Glimm J, Grosu R, Smolka SA, Fenton FH (2011) Toward real-time simulation of cardiac dynamics. In: Proceedings of the 9th International Conference on Computational Methods in Systems Biology. ACM, pp 103–112

  10. Garcia VM, Liberos A, Climent AM, Vidal A, Millet J, Gonzalez A (2011) An adaptive step size GPU ODE solver for simulating the electric cardiac activity. In: Computing in Cardiology, 2011. IEEE, pp 233–236

  11. García-Molla VM, Liberos A, Vidal A, Guillem M, Millet J, Gonzalez A, Martínez-Zaldívar FJ, Climent AM (2014) Adaptive step ODE algorithms for the 3D simulation of electric heart activity with graphics processing units. Comput Biol Med 44:15–26

    Article  Google Scholar 

  12. Jararweh Y, Jarrah M, Hariri S (2012) Exploiting GPUs for compute-intensive medical applications. In: 2012 International Conference on Multimedia Computing and Systems (ICMCS). IEEE, pp 29–34

  13. Esmaili E, Akoglu A, Ditzler G, Hariri S, Moukabary T, Szep J (2017) Autonomic management of 3D cardiac simulations. In: 2017 International Conference on Cloud and Autonomic Computing (ICCAC). IEEE, pp 1–9

  14. Yu D, Du D, Yang H, Tu Y (2014) Parallel computing simulation of electrical excitation and conduction in the 3D human heart. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp 4315–4319

  15. Chai J, Wen M, Wu N, Huang D, Yang J, Cai X, Zhang C, Yang Q (2013) Simulating cardiac electrophysiology in the era of GPU-cluster computing. IEICE Trans Inf Syst 96(12):2587–2595

    Article  Google Scholar 

  16. Neic A, Liebmann M, Hoetzl E, Mitchell L, Vigmond EJ, Haase G, Plank G (2012) Accelerating cardiac bidomain simulations using graphics processing units. IEEE Trans Biomed Eng 59(8):2281–2290

    Article  Google Scholar 

  17. Higham J, Aslanidi O, Zhang H (2011) Large speed increase using novel GPU based algorithms to simulate cardiac excitation waves in 3D rabbit ventricles. In: Computing in Cardiology, 2011. IEEE, pp 9–12

  18. Zhang L, Wang K, Zuo W, Gai C (2014) G-Heart: a GPU-based system for electrophysiological simulation and multi-modality cardiac visualization. J Comput 9(2):360–368

    Google Scholar 

  19. Xia Y, Wang K, Zhang H (2015) Parallel optimization of 3D cardiac electrophysiological model using GPU. Comput Math Methods Med 2015:1–10

    Article  MATH  Google Scholar 

  20. Mirams GR, Arthurs CJ, Bernabeu MO, Bordas R, Cooper J, Corrias A, Davit Y, Dunn SJ, Fletcher AG, Harvey DG et al (2013) Chaste: an open source C++ library for computational physiology and biology. PLoS Comput Biol 9(3):e1002970

    Article  MathSciNet  Google Scholar 

  21. Yang J, Chai J, Wen M, Wu N, Zhang C (2013) Solving the cardiac model using multi-core CPU and many integrated cores (MIC). In: 2013 IEEE 10th International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing (HPCC\_EUC). IEEE, pp 1009–1015

  22. Langguth J, Lan Q, Gaur N, Cai X, Wen M, Zhang CY (2016) Enabling tissue-scale cardiac simulations using heterogeneous computing on Tianhe-2. In: 2016 IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS). IEEE, pp 843–852

  23. Arevalo HJ, Vadakkumpadan F, Guallar E, Jebb A, Malamas P, Wu KC, Trayanova NA (2016) Arrhythmia risk stratification of patients after myocardial infarction using personalized heart models. Nat Commun 7:11437

    Article  Google Scholar 

  24. Karma A (1994) Electrical alternans and spiral wave breakup in cardiac tissue. Chaos: an Interdisciplinary. J Nonlinear Sci 4(3):461–472

    Google Scholar 

  25. Iyer V, Mazhari R, Winslow RL (2004) A computational model of the human left-ventricular epicardial myocyte. Biophys J 87(3):1507–1525

    Article  Google Scholar 

  26. Ten Tusscher K, Noble D, Noble PJ, Panfilov AV (2004) A model for human ventricular tissue. Am J Physiol-Heart Circ Physiol 286(4):H1573–H1589

    Article  Google Scholar 

  27. Ten Tusscher KH, Panfilov AV (2006) Alternans and spiral breakup in a human ventricular tissue model. Am J Physiol-Heart Circ Physiol 291(3):H1088–H1100

    Article  Google Scholar 

  28. Luo Ch, Rudy Y (1991) A model of the ventricular cardiac action potential. Depolarization, repolarization, and their interaction. Circ Res 68(6):1501–1526

    Article  Google Scholar 

  29. Mahajan A, Shiferaw Y, Sato D, Baher A, Olcese R, Xie LH, Yang MJ, Chen PS, Restrepo JG, Karma A et al (2008) A rabbit ventricular action potential model replicating cardiac dynamics at rapid heart rates. Biophys J 94(2):392–410

    Article  Google Scholar 

  30. Bondarenko VE, Szigeti GP, Bett GC, Kim SJ, Rasmusson RL (2004) Computer model of action potential of mouse ventricular myocytes. Am J Physiol-Heart Circ Physiol 287(3):H1378–H1403

    Article  Google Scholar 

  31. Shannon TR, Wang F, Puglisi J, Weber C, Bers DM (2004) A mathematical treatment of integrated Ca dynamics within the ventricular myocyte. Biophys J 87(5):3351–3371

    Article  Google Scholar 

  32. Nickerson DP, Hunter PJ (2010) Cardiac cellular electrophysiological modeling. Cardiac electrophysiology methods and models. Springer, Boston, pp 135–158

    Chapter  Google Scholar 

  33. Majumder R, Nayak AR, Pandit R (2011) Scroll-wave dynamics in human cardiac tissue: lessons from a mathematical model with inhomogeneities and fiber architecture. PLOS ONE 6(4):e18052

    Article  Google Scholar 

  34. Majumder R, Nayak AR, Pandit R (2012) Nonequilibrium arrhythmic states and transitions in a mathematical model for diffuse fibrosis in human cardiac tissue. PLoS ONE 7(10):e45040

    Article  Google Scholar 

  35. Nayak AR, Shajahan T, Panfilov A, Pandit R (2013) Spiral-wave dynamics in a mathematical model of human ventricular tissue with myocytes and fibroblasts. PloS ONE 8(9):e72950

    Article  Google Scholar 

  36. Smaill BH, Hunter PJ (2010) Computer modeling of electrical activation: from cellular dynamics to the whole heart. Cardiac electrophysiology methods and models. Springer, Boston, pp 159–185

    Chapter  Google Scholar 

  37. Morton KW, Mayers DF (2005) Numerical solution of partial differential equations: an introduction. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  38. Cuellar AA, Lloyd CM, Nielsen PF, Bullivant DP, Nickerson DP, Hunter PJ (2003) An overview of CellML 1.1, a biological model description language. Simulation 79(12):740–747

    Article  Google Scholar 

  39. Nvidia Corporation (2017) Nvidia CUDA C programming guide, version 8.0. https://docs.nvidia.com/cuda/cuda-c-programming-guide/. Accessed June 2017

  40. Eager DL, Zahorjan J, Lazowska ED (1989) Speedup versus efficiency in parallel systems. IEEE Trans Comput 38(3):408–423

    Article  Google Scholar 

  41. Arioli M (2004) A stopping criterion for the conjugate gradient algorithm in a finite element method framework. Numer Math 97(1):1–24

    Article  MathSciNet  MATH  Google Scholar 

  42. Zhang C, Lan H, Ye Y, Estrade BD (2005) Parallel SOR iterative algorithms and performance evaluation on a Linux cluster. Technical report, Naval Research Laboratory Stennis Space Center MS Oceanography Division

  43. Hadjidimos A (2000) Successive overrelaxation (SOR) and related methods. J Comput Appl Math 123(1–2):177–199

    Article  MathSciNet  MATH  Google Scholar 

  44. Hackbusch W (1994) Iterative solution of large sparse systems of equations, vol 95. Springer, New York

    Book  MATH  Google Scholar 

  45. Marsh M (2012) An assessment of numerical methods for cardiac simulation. Ph.D. thesis, University of Saskatchewan

Download references

Acknowledgements

This material is based upon the work supported by the National Science Foundation under Grant No. CNS 1624668 I/UCRC: Industry/University Cooperative Research Center for Cloud and Autonomic Computing.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ, 85721, USA

    Ehsan Esmaili, Ali Akoglu & Salim Hariri

  2. Carondelet Heart and Vascular Institute - Cardiology West, 445 N. Silverbell Rd., #201, Tucson, AZ, 85745, USA

    Talal Moukabary

Authors
  1. Ehsan Esmaili
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Akoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Salim Hariri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Talal Moukabary
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ehsan Esmaili.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Esmaili, E., Akoglu, A., Hariri, S. et al. Implementation of scalable bidomain-based 3D cardiac simulations on a graphics processing unit cluster. J Supercomput 75, 5475–5506 (2019). https://doi.org/10.1007/s11227-019-02796-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-019-02796-8

Keywords

Search

Navigation