Article

Blue matter: approaching the limits of concurrency for classical molecular dynamics

Authors:

Blake G. Fitch,

Aleksandr Rayshubskiy,

Maria Eleftheriou,

T. J. Christopher Ward,

Michael C. Pitman,

Robert S. GermainAuthors Info & Claims

SC '06: Proceedings of the 2006 ACM/IEEE conference on Supercomputing

Pages 87 - es

https://doi.org/10.1145/1188455.1188547

Published: 11 November 2006 Publication History

Abstract

This paper describes a novel spatial-force decomposition for N-body simulations for which we observe O(sqrt(p)) communication scaling. This has enabled Blue Matter to approach the effective limits of concurrency for molecular dynamics using particle-mesh (FFT-based) methods for handling electrostatic interactions. Using this decomposition, Blue Matter running on Blue Gene/L has achieved simulation rates in excess of 1000 time steps per second and demonstrated significant speed-ups to O(1) atom per node. Blue Matter employs a Communicating Sequential Process (CSP) style model with application communication state machines compiled to hardware interfaces. The scalability achieved has enabled methodologically rigorous biomolecular simulations on biologically interesting systems, such as membrane-bound proteins, whose time scales dwarf previous work on those systems, Major scaling improvements will require exploration of alternative algorithms for treating the long range electrostatics.

References

[1]

Allen, F., et al., 2001. Blue Gene: a vision for protein science using a petaflop supercomputer. IBM Systems Journal 40, 2, 310--327.

Digital Library

[2]

Deserno, M., and Holm, C. 1998. How to mesh up Ewald sums. i. a theoretical and numerical comparison of various particle mesh routines. J. Chem. Phys. 109, 18, 7678--7693.

[3]

Eleftheriou, M., Fitch, B., Rayshubskiy, A., Ward, T., and Germain, R. 2005. Performance measurements of the 3d FFT on the Blue Gene/L supercomputer. In Euro-Par 2005 Parallel Processing: 11th International Euro-Par Conference, Lisbon, Portugal, August 30-September 2, 2005. Springer-Verlag, J. Cunha and P. Medeiros, Eds., vol. 3648 of Lecture Notes in Computer Science, 795--803.

Digital Library

[4]

Eleftheriou, M., Germain, R., Royyuru, A., and Zhou, R. 2006. Thermal denaturing of mutant lysozyme with both oplsaa and charmm force fields. submitted to J. Am. Chem. Soc.

[5]

Eleftheriou, M., Rayshubskiy, A., Pitera, J. W., Fitch, B. G., Zhou, R., and Germain, R. S. 2006. Parallel implementation of the replica exchange molecular dynamics algorithm on Blue Gene/L. In Fifth IEEE International Workshop on High Performance Computational Biology.

[6]

Fitch, B., Germain, R., Mendell, M., Pitera, J., Pitman, M., Rayshubskiy, A., Sham, Y., Suits, F., Swope, W., Ward, T., Zhestkov, Y., and Zhou, R. 2003. Blue Matter, an application framework for molecular simulation on Blue Gene. Journal of Parallel and Distributed Computing 63, 759--773.

Digital Library

[7]

Fitch, B. G., Rayshubskiy, A., Eleftheriou, M., Ward, T. C., Giampapa, M., Zhestkov, Y., Pitman, M. C., Suits, F., Grossfield, A., Pitera, J., Swope, W., Zhou, R., Germain, R. S., and Feller, S. 2005. Blue matter: Strong scaling of molecular dynamics on Blue Gene/L. Research Report RC23688, IBM Research Division, August.

[8]

Fitch, B. G., Rayshubskiy, A., Eleftheriou, M., Ward, T. C., Giampapa, M., Zhestkov, Y., Pitman, M. C., Suits, F., Grossfield, A., Pitera, J., Swope, W., Zhou, R., Feller, S., and Germain, R. S. 2006. Blue Matter: Strong scaling of molecular dynamics on Blue Gene/L. In International Conference on Computational Science (ICCS 2006), Springer-Verlag, V. Alexandrov, D. van Albada, P. Sloot, and J. Dongarra, Eds., vol. 3992 of LNCS, 846--854.

Digital Library

[9]

Frenkel, D., and Smit, B. 1996. Understanding Molecular Simulation. Academic Press, San Diego, CA.

Digital Library

[10]

Gara, A., ET AL. 2005. Overview of the Blue Gene/L system architecture. IBM Journal of Research and Development 49, 2/3, 195--212.

Digital Library

[11]

Germain, R. S., Fitch. B., Rayshubskiy, A., Eleftheriou, M., Pitman, M. C., Suits. F., Giampapa, M., And T. J. Christopher Ward, T. C. 2005. Blue Matter on Blue Gene/L: massively parallel computation for biomolecular simulation. In CODES+ISSS '05: Proceedings of the 3rd IEEE/ACM/IFIP international conference on Hardware/software codesign and system synthesis, ACM Press, New York, NY, USA, 207--212.

Digital Library

[12]

Germain, R., Zhestkov, Y., Eleftheriou, M., Rayshubskiy, A., Suits, F., Ward, T., and Fitch, B. 2005. Early performance data on the Blue Matter molecular simulation framework. IBM Journal of Research and Development 49, 2/3, 447--456.

Digital Library

[13]

Giampapa, M., Bellofatto, R., Blumrich, M. A., Chen, D., Dombrowa, M. B., Gara, A., Haring, R. A., Heidelberger, P., Hoenicke, D., Kopcsay, G. V., Nathanson, B. J., Steinmacher-Burow, B. D., Ohmacht, M., Salapura, V., and Vranas, P. 2005, Blue Gene/L advanced diagnostics environment. IBM Journal of Research and Development 49, 2/3, 319--332.

Digital Library

[14]

Grossfield, A., Feller, S. E., and Pitman, M. C. 2006. A role for direct interactions in the modulation of rhodopsin by omega-3 polyunsaturated lipids. PNAS 103, 13, 4888--4893.

[15]

Hendrickson, B., and Plimpton, S. 1995. Parallel many body simulations without all to all communication. Journal of Parallel and Distributed Computing 27, 1, 1525.

Digital Library

[16]

Hoare, C. 1985. Communicating Sequential Processes. Prentice-Hall.

Digital Library

[17]

Kumar, S., Huang, C., Almasi, G., and Kale, L. V. 2006. Achieving strong scaling with NAMD on Blue Gene/l. 20th IEEE International Parallel and Distributed Processing Symposium, IEEE. http://charm.cs.uiuc.edu/papers/NAMDIDPDSO6.pdf.

Digital Library

[18]

Phillips, James C. and Zheng, G., Kumar, S., and Kale, L. V. 2002. NAMD: biomolecular simulation on thousands of processors. In Supercomputing '02: Proceedings of the 2002 ACM/IEEE conference on Supercomputing, IEEE Computer Society Press, Los Alamitos, CA, USA, 1--18.

Digital Library

[19]

Pitman, M. C., Suits, F., MacKerell, Alexander D., J., and Feller, S. E. 2004. Molecular-level organization of saturated and polyunsaturated fatty acids in a phosphatidylcholine bilayer containing cholesterol. Biochemistry 43, 49, 15318--15328.

[20]

Pitman, M. C., Grossfield, A., Suits, F., and Feller, S. E. 2005. Role of cholesterol and polyunsaturated chains in lipid-protein interactions: Molecular dynamics simulation of rhodopsin in a realistic membrane environment. Journal of the American Chemical Society 127, 13, 4576--4577.

[21]

Pitman, M. C., Suits, F., Gawrisch, K., and Feller, S. E. 2005. Molecular dynamics investigation of dynamical properties of phosphatidylethanolamine lipid bilayers. Journal of Chemical Physics 122, 24, 244715.

[22]

Plimpton, S., and Hendrickson, B. 1996. A new parallel method for molecular dynamics simulation of macromolecular systems. Journal of Computational Chemistry 17, 3, 326--337.

[23]

Shaw, D. 2004. An asymptotic improvement in the parallel evaluation of pairwise particle interactions. Presented at Philadelpha American Chemical Society meeting, September.

[24]

Shaw, D. E. 2005. A fast, scalable method for the parallel evaluation of distance-limited pairwise particle interactions. Journal of Computational Chemistry 26, 13, 1318--1328.

[25]

Snir, M. 2004. A note on n-body computations with cutoffs. Theory of Computing Systems 37, 295--318.

[26]

Sugita, Y., and Okamoto, Y. 1999. Replica-exchange molecular dynamics method for protein folding. Chem. Phys. Lett. 314, 141--151.

[27]

Suits, F., Pitman, M. C., and Feller, S. E. 2005. Molecular dynamics investigation of the structural properties of phosphatidylethanolamine lipid bilayers. Journal of Chemical Physics 122, 24.

[28]

Swope, W., Andersen, H., Berens, P., and Wilson, K. 1982. A computer simulation method for the calculation of equilibrium constants for the formation of physical clusters of molecules: Application to small water clusters. Journal of Chemical Physics 76, 637--649.

[29]

Swope, W. C., Pitera, J. W., Suits, F., Pitman, M., Eleftheriou, M., Fitch, B. G., Germain, R. S., Rayshubskiy, A., Ward, T. J. C., Zhestkov, Y., and Zhou, R. 2004. Describing protein folding kinetics by molecular dynamics simulations. 2. example applications to alanine dipeptide and a β-hairpin peptide. J. Phys. Chem. B 108, 21, 6582--6594.

[30]

Taylor, V., Stevens, R., and Arnold, K. 1997. Parallel molecular dynamics: implications for massively parallel machines. Journal of Parallel and Distributed Computing 45, 2 (September), 166--175.

Digital Library

Cited By

Schwetman HPatel ARobinson LZheng XWood AKrishnamoorthy A(2016)An Energy-Efficient Optical Interconnect Architecture for Bandwidth-Balanced SystemsJournal of Lightwave Technology10.1109/JLT.2016.254323034:12(2905-2919)Online publication date: 15-Jun-2016
https://doi.org/10.1109/JLT.2016.2543230
Kumar SSun YKale L(2013)Acceleration of an Asynchronous Message Driven Programming Paradigm on IBM Blue Gene/QProceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing10.1109/IPDPS.2013.83(689-699)Online publication date: 20-May-2013
https://dl.acm.org/doi/10.1109/IPDPS.2013.83
Sun YZheng GMei CBohm EPhillips JKalé LJones THollingsworth J(2012)Optimizing fine-grained communication in a biomolecular simulation application on Cray XK6Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/2388996.2389071(1-11)Online publication date: 10-Nov-2012
https://dl.acm.org/doi/10.5555/2388996.2389071
Show More Cited By

Index Terms

Blue matter: approaching the limits of concurrency for classical molecular dynamics

Recommendations

GPGPU-based parallel computation: application to molecular dynamics problems
COMPUTE '11: Proceedings of the Fourth Annual ACM Bangalore Conference

Parallel and distributed computing techniques have been applied to solve problems that have long running times. Molecular dynamics (MD) is used extensively in physics, chemistry, material sciences and biology to study statistical mechanical properties ...
Homology modeling, molecular docking and MD simulation studies to investigate role of cysteine protease from Xanthomonas campestris in degradation of Aβ peptide

Cysteine protease is known to degrade amyloid beta peptide which is a causative agent of Alzheimer's disease. This cleavage mechanism has not been studied in detail at the atomic level. Hence, a three-dimensional structure of cysteine protease from ...
Understanding the lid movements of LolA in Escherichia coli using molecular dynamics simulation and in silico point mutation

Molecular dynamic analysis of open to close conformation for two LolA derivatives, LolA(R43L) and MsL43R.In silico point mutation of Leu43 to Arg43 induced conformational changes leading complete closed conformation of protein.Conformational changes ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '06: Proceedings of the 2006 ACM/IEEE conference on Supercomputing

November 2006

746 pages

ISBN:0769527000

DOI:10.1145/1188455

Conference Chair:
Barbara Horner-Miller
Arctic Region Supercomputing Center

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SC '06

Sponsor:

SIGARCH
IEEE-CS

SC '06: International Conference for High Performance Computing, Networking, Storage and Analysis

November 11 - 17, 2006

Florida, Tampa

Acceptance Rates

SC '06 Paper Acceptance Rate 54 of 239 submissions, 23%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
265
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Schwetman HPatel ARobinson LZheng XWood AKrishnamoorthy A(2016)An Energy-Efficient Optical Interconnect Architecture for Bandwidth-Balanced SystemsJournal of Lightwave Technology10.1109/JLT.2016.254323034:12(2905-2919)Online publication date: 15-Jun-2016
https://doi.org/10.1109/JLT.2016.2543230
Kumar SSun YKale L(2013)Acceleration of an Asynchronous Message Driven Programming Paradigm on IBM Blue Gene/QProceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing10.1109/IPDPS.2013.83(689-699)Online publication date: 20-May-2013
https://dl.acm.org/doi/10.1109/IPDPS.2013.83
Sun YZheng GMei CBohm EPhillips JKalé LJones THollingsworth J(2012)Optimizing fine-grained communication in a biomolecular simulation application on Cray XK6Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/2388996.2389071(1-11)Online publication date: 10-Nov-2012
https://dl.acm.org/doi/10.5555/2388996.2389071
Christodoulopoulos KRuffini MO'Mahony DKatrinis K(2012)Topology configuration in hybrid EPS/OCS interconnectsProceedings of the 18th international conference on Parallel Processing10.1007/978-3-642-32820-6_70(701-715)Online publication date: 27-Aug-2012
https://dl.acm.org/doi/10.1007/978-3-642-32820-6_70
Bhatele AJain NGropp WKale LLathrop SCosta JKramer W(2011)Avoiding hot-spots on two-level direct networksProceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/2063384.2063486(1-11)Online publication date: 12-Nov-2011
https://dl.acm.org/doi/10.1145/2063384.2063486
Mei CSun YZheng GBohm EKale LPhillips JHarrison CLathrop SCosta JKramer W(2011)Enabling and scaling biomolecular simulations of 100 million atoms on petascale machines with a multicore-optimized message-driven runtimeProceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/2063384.2063466(1-11)Online publication date: 12-Nov-2011
https://dl.acm.org/doi/10.1145/2063384.2063466
Chiu MHerbordt M(2010)Molecular Dynamics Simulations on High-Performance Reconfigurable Computing SystemsACM Transactions on Reconfigurable Technology and Systems10.1145/1862648.18626533:4(1-37)Online publication date: 1-Nov-2010
https://dl.acm.org/doi/10.1145/1862648.1862653
Doi JNegishi YHess B(2010)Overlapping Methods of All-to-All Communication and FFT Algorithms for Torus-Connected Massively Parallel SupercomputersProceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC.2010.38(1-9)Online publication date: 13-Nov-2010
https://dl.acm.org/doi/10.1109/SC.2010.38
Dror RGrossman JMackenzie KTowles BChow ESalmon JYoung CBank JBatson BDeneroff MKuskin JLarson RMoraes MShaw DHess B(2010)Exploiting 162-Nanosecond End-to-End Communication Latency on AntonProceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC.2010.23(1-12)Online publication date: 13-Nov-2010
https://dl.acm.org/doi/10.1109/SC.2010.23
Shaw DDror RSalmon JGrossman JMackenzie KBank JYoung CDeneroff MBatson BBowers KChow EEastwood MIerardi DKlepeis JKuskin JLarson RLindorff-Larsen KMaragakis PMoraes MPiana SShan YTowles BPinfold W(2009)Millisecond-scale molecular dynamics simulations on AntonProceedings of the Conference on High Performance Computing Networking, Storage and Analysis10.1145/1654059.1654126(1-11)Online publication date: 14-Nov-2009
https://dl.acm.org/doi/10.1145/1654059.1654126
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten