The Blue Gene/L system at the Department of Energy Lawrence Livermore National Laboratory in Livermore, California is the world’s most powerful supercomputer. It has achieved groundbreaking performance in both standard benchmarks as well as real scientific applications. In that process, it has enabled new science that simply could not be done before. Blue Gene/L was developed by a relatively small team of dedicated scientists and engineers. This article is both a description of the Blue Gene/L supercomputer as well as an account of how that system was designed, developed, and delivered. It reports on the technical characteristics of the system that made it possible to build such a powerful supercomputer. It also reports on how teams across the world worked around the clock to accomplish this milestone of high-performance computing.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
G. Almasi, S. Chatterjee, A. Gara, J. Gunnels, M. Gupta, A. Henning, J. E. Moreira and B. Walkup, Unlocking the Performance of the BlueGene/L Supercomputer, In Proc IEEE/ACM SC04, Pittsburgh, PA (November 2004).
ASC/Alliances Center for Astrophysical Thermonuclear Flashes, University of Chicago; see http://flash.uchicago.edu/website/home/.
A. Bright, M. Ellavsky, A. Gara, R. Haring, G. Kopcsay, R. Lembach, J. Marcella, M. Ohmacht, and V. Salapura. Creating the BlueGene/L Supercomputer from Low Power SoC ASICs, In Proc ISSCC 2005 – IEEE International Solid-State Circuits Conference (February 2005).
V. Bulatov, W. Cai, J. Fier, M. Hiratani, G. Hommes, T. Pierce, M. Tang, M. Rhee, R. K. Yates, and T. Arsenlis, Scalable Line Dynamics in ParaDiS, In Proc IEEE/ACM SC04, Pittsburgh, PA (November 2004).
A. W. Cook, W. H. Cabot, M. L. Welcome, P. L. Williams, B. J. Miller, B. R. de Supinski, and R. K. Yates, Tera-Scalable Algorithms for Variable-Density Elliptic Hydrodynamics with Spectral Accuracy, In Proc IEEE/ACM SC05, Seattle, WA (November 2005).
K. Davis, A. Hoisie, G. Johnson, D. J. Kerbyson, M. Lang, S. Pakin, and F. Petrini, A Performance and Scalability Analysis of the BlueGene/L Architecture, In Proc IEEE/ACM SC04, Pittsburgh, PA (November 2004).
B. G. Fitch, A. Rayshubskiy, M. Eleftheriou, T. J. Christopher Ward, M. E. Giampapa M. C. Pitman, and Robert S. Germain. Blue Matter: Approaching the Limits of Concurrency for Classical Molecular Dynamics, In Proc IEEE/ACM SC06, Tampa, FL (November 2006).
Fryxell B., Olson K., Ricker P., Timmes F.X., Zingale M., Lamb D.Q., MacNeice P., Rosner R., Tufo H. (2000). FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes. Astrophysical J. Supplement 131:273
Gara A., Blumrich M.A., Chen D., Chiu G.L.-T., Coteus P., Giampapa M.E., Haring R.A., Heidelberger P., Hoenicke D., Kopcsay G.V., Liebsch T.A., Ohmacht M., Steinmacher-Burow B.D., Takken T., Vranas P., (March/May 2005). Overview of the Blue Gene/L System Architecture. IBM J. Res. Dev. 49(2/3):195–212
Germain R.S., Zhestkov Y., Eleftheriou M., Rayshubskiy A., Suits F., Ward T.J.C., Fitch B.G., (March/May 2005). Early Performance Data on the Blue Matter Molecular Simulation Framework. IBM J. Res. Dev. 49(2/3):447–456
D. S. Greenberg, R. Brightwell, L. A. Fisk, A. B. Maccabe, and R. E. Riesen, A System Software Architecture for High-end Computing, IEEE/ACM SC97, San Jose, CA, pp. 1–15 (November 1997).
F. Gygi, E. W. Draeger, M. Schulz, B. R. De Supinski, J. A. Gunnels, V. Austel, J. C. Sexton, F. Franchetti, S. Kral, C. Ueberhuber, and J. Lorenz, Large-Scale Electronic Structure Calculations of High-Z Metals on the BlueGene/L Platform, In Proc IEEE/ACM SC06 Tampa, FL (November 2006).
F. Gygi, E. W. Draeger, B. R. de Supinski, R. K. Yates, F. Franchetti, S. Kral, J. Lorenz, C. W. Ueberhuber, J. A. Gunnels, and J. C. Sexton. Large-scale First-Principles Molecular Dynamics Simulations on the BlueGene/L Platform using the Qbox code, IEEE/ACM SC05, Seattle, WA (November 2005).
A. Henning. BlueGene/L: Improving Application Memory Performance on a Massively Parallel Machine. M.E. Thesis. Cornell University (2005).
IBM Blue Gene Team, Blue Gene: A Vision for Protein Science using a Petaflop Supercomputer, IBM Syst. J., 40(2): (2001).
IBM Corporation, Blue Gene/L: Application development. 2006. http://www.redbooks. ibm.com/abstracts/sg247179.html?Open
E. Ipek, B. R. de Supinski, M. Schulz, and S. A. McKee, An Approach to Performance Prediction for Parallel Applications, In Proc 2005 Euro-Par, Lisbon, Portugal (August 2005).
S. Louis, B. R. de Supinski, BlueGene/L: Early application scaling results. BlueGene System Software Workshop February 23–24, 2005, Salt Lake City, Utah. http://www-unix.mcs.anl.gov/~beckman/bluegene/SSW-Utah-2005/BGL-SSW22-LLNL-Apps.pdf
J. E. Moreira, G. Almási, C. Archer, R. Bellofatto, P. Bergner, J. R. Brunheroto, M. Brutman, J. G. Castaños, P. G. Crumley, M. Gupta, T. Inglett, D. Lieber, D. Limpert, P. McCarthy, M. Megerian, M. Mendell, M. Mundy, D. Reed, R. K. Sahoo, A. Sanomiya, R. Shok, B. Smith, and G. G. Stewart, Blue Gene/L Programming and Operating environment, IBM J. Res. Dev., 49(2/3): (March/May 2005).
F. Petrini, D. Kerbyson, and S. Pakin, The case of the missing supercomputer performance: achieving optimal performance on the 8,192 processors of ASCI Q. IEEE/ACM SC03, Phoenix, AZ, November 2003.
R. Ross, J. E. Moreira, K. Cupps, W. Pfeiffer. Parallel I/O on the IBM Blue Gene/L system. Blue Gene/L Consortium Quarterly Newsletter. Argonne National Laboratory. 1st quarter 2006. http://www-fp.mcs.anl.gov/bgconsortium/file%20system%20newsletter2.pdf.
V. Salapura, R. Bickford, M. Blumrich, A. Bright, D. Chen, P. Coteus, A. Gara, M. Giampapa, M. Gschwind, M. Gupta, S. Hall, R. A. Haring, P. Heidelberger, D. Hoenicke, G. V. Kopcsay, M. Ohmacht, R. A. Rand, T. Takken, and P. Vranas, Power and Performance Optimization at the System Level, In Proc ACM Computing Frontiers 2005, Ischia, Italy (May 2005).
V. Salapura, R. Walkup, and A. Gara, Exploiting Workload Parallelism for Performance and Power Optimization in Blue Gene, IEEE Micro (September/October 2006).
V. Salapura, J. R. Brunheroto, F. Redìgolo, D. Hoenicke, and A. Gara, Exploiting eDRAM Bandwidth with Data Prefetching: Simulation and Measurements, IBM internal document.
K. van der Schaaf, Blue Gene in the Heart of a Wide Area Sensor Network, In Proc QCDOC and Blue Gene: Next Generation of HPC Architecture Workshop. Edinburgh, UK (October 2005).
M. Seager, The BlueGene/L Computing Environment. Lawrence Livermore National Laboratory. October 2003. http://www.llnl.gov/asci/platforms/bluegene/papers/16seager.pdf.
K. Singh, E. Ipek, S. A. McKee, B. R. de Supinski, M. Schulz, and R. Caruana, Predicting Parallel Application Performance via Machine Learning Approaches, In Proc Concurrency and Computation: Practice and Experience, 2006 (To appear).
F. H. Streitz, J. N. Glosli, M. V. Patel, B. Chan, R. K. Yates, B. R. de Supinski, J. Sexton, J. A. Gunnels, 100+ TFlop solidification simulations on BlueGene/L, In Proc IEEE/ACM SC05, Seattle, WA (November 2005).
University of Mannheim, University of Tennessee, and NERSC/LBNL. TOP500 Supercomputer sites. http://www.top500.org/.
University of Tennessee. HPC Challenge Benchmark. http://icl.cs.utk.edu/hpcc/.
P. Vranas, G. Bhanot, M. Blumrich, D. Chen, A. Gara, P. Heidelberger, V. Salapura, and J. Sexton, The BlueGene/L Supercomputer and Quantum ChromoDynamics. 2006 Gordon Bell Prize. In Proc IEEE/ACM SC06, Tampa, FL (November 2006).
S. R. Wheat, A. B. Maccabe, R. Riesen, D. W. van Dresser, and T. M. Stallcup, PUMA: An Operating System for Massively Parallel Systems, In Proc Proceedings of the 27th Hawaii International Conference on System Sciences, pp. 56–65 (1994).
Wheat S.R., Maccabe A.B., Riesen R., van Dresser D.W., Stallcup T.M., (1994). PUMA: An Operating System for Massively Parallel Systems. Sci. Programming 3:275–288
A. Yoo, E. Chow, K. Henderson, W. McLendon, B. Hendrickson, U. Catalyurek. A scalable distributed parallel breadth-first search algorithm on BlueGene/L, In Proc IEEE/ACM SC05, Seattle, WA (November 2005).
Author information
Authors and Affiliations
Corresponding author
Additional information
This paper is an expanded version of two other papers: (1) Designing a Highly-Scalable Operating System: The Blue Gene/L Story, in Proceedings of SC’06, and (2) Delivering Teraflops: An Account of how Blue Gene was Brought to Life, in Proceedings of JVA 2006.
Rights and permissions
About this article
Cite this article
Moreira, J.E., Salapura, V., Almasi, G. et al. The Blue Gene/L Supercomputer: A Hardware and Software Story. Int J Parallel Prog 35, 181–206 (2007). https://doi.org/10.1007/s10766-007-0037-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10766-007-0037-2