research-article

Physics-based seismic hazard analysis on petascale heterogeneous supercomputers

Authors:

P. J. Maechling,

T. H. JordanAuthors Info & Claims

SC '13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Article No.: 70, Pages 1 - 12

https://doi.org/10.1145/2503210.2503300

Published: 17 November 2013 Publication History

Abstract

We have developed a highly scalable and efficient GPU-based finite-difference code (AWP) for earthquake simulation that implements high throughput, memory locality, communication reduction and communication/computation overlap and achieves linear scalability on Cray XK7 Titan at ORNL and NCSA's Blue Waters system. We simulate realistic 0-10 Hz earthquake ground motions relevant to building engineering design using high-performance AWP. Moreover, we show that AWP provides a speedup by a factor of 110 in key strain tensor calculations critical to probabilistic seismic hazard analysis (PSHA). These performance improvements to critical scientific application software, coupled with improved co-scheduling capabilities of our workflow-managed systems, make a statewide hazard model a goal reachable with existing supercomputers. The performance improvements of GPU-based AWP are expected to save millions of core-hours over the next few years as physics-based seismic hazard analysis is developed using heterogeneous petascale supercomputers.

References

[1]

Bielak, J., Graves, R., Olsen, K. B., Taborda, R., Ramirez-Guzman, L., Day, S., Ely, G., Roten, D., Jordan, T., Maechling, P., Urbanic, J., Cui, Y. and Juve, G. 2010. The ShakeOut earthquake scenario: Verification of three simulation sets. Geophysical Journal International., 180, 1 (Jan. 2010), 375--404.

[2]

Blanch, J. O., Robertsson, J. O. and Symes, W. W. 1995. Modeling of a constant Q: methodology and algorithm for an efficient and optimally inexpensive viscoelastic technique. Geophysics, 60, 1, 176--184.

[3]

Callaghan, S., Deelman, E., Gunter, D., Juve, G., Maechling, P., Brooks, C., Vahi, K., Milner, K., Graves, R., Field, E., Okaya, D. and Jordan, T. 2010. Scaling up workflow-based applications. Journal of Computer and System Sciences, 76, 6 (Sep. 2010), 428--446.

Digital Library

[4]

Cerjan, C., Kosloff, D., Kosloff, R. and Reshef, M. 1985. A nonreflecting boundary condition for discrete acoustic and elastic wave equations. Geophysics, 50, 4, 705--708.

[5]

Chen, P., Jordan, T. H. and Zhao, L. 2007. Full three-dimensional tomography: a comparison between the scattering- integral and adjoint-wavefield methods. Geophysical Journal International., 170, 1, 175--181.

[6]

Chen, P., Zhao, L. and Jordan, T. H. 2007, Full 3D tomography for the crustal structure of the Los Angeles region. Bulletin of the Seismological Society of America, 97, 4, 1094--1120.

[7]

Cui, Y., Olsen, K. B., Jordan, T. H., Lee, K., Zhou, J., Small, P., Roten, D., Ely, G., Panda, D. K., Chourasia, A., Levesque, J., Day, S. M. and Maechling, P. 2010. Scalable earthquake simulation on petascale supercomputers. Proc. of Int'l. Conf. for High Performance Computing, Networking, Storage and Analysis (SC'10, New Orleans, November 2010), 1--20.

Digital Library

[8]

Dalguer, L. A. and Day, S. M. 2007. Staggered-grid split-node method for spontaneous rupture simulation. Journal of Geophysical Research: Solid Earth, 112, B02302.

[9]

Day, S. M. 1998. Efficient simulation of constant Q using coarse-grained memory variables. Bulletin of the Seismological Society of America, 88, 4, 1051--1062.

[10]

Day, S. M. and Bradley, C. R. 2001. Memory-efficient simulation of anelastic wave propagation. Bulletin of the Seismological Society of America, 91, 3, 520--531.

[11]

Field, E. H., Dawson, T. E., Felzer, K. R., Frankel, A. D., Gupta, V., Jordan, T. H., Parsons, T., Petersen, M. D., Stein, R. S., Weldon II, R. J. and Wills, C. J. 2009. Uniform california earthquake rupture forecast, version 2 (UCERF 2). Bulletin of the Seismological Society of America, vol. 99, no. 4 (Aug. 2009), 2053--2107.

[12]

Field, E. H., Jordan, T. H. and Cornell, C. A. 2003. OpenSHA: A developing community-modeling environment for seismic hazard analysis. Seismological Research Letters, 74, 4, 406--419.

[13]

Georgia Tech 2013. Keeneland User Guide. {Online}. https://www.xsede.org/gatech-keeneland.

[14]

Graves, R. W. 1996. Simulating seismic wave propagation in 3D elastic media using staggered-grid finite differences. Bulletin of the Seismological Society of America, 86, 4, 1091--1106.

[15]

Graves, R. W. and Pitarka, A. 2010. Broadband ground-motion simulation using a hybrid approach. Bulletin of the Seismological Society of America, 100, 5A, 2095--2123.

[16]

Graves, R., Aagaard, B., Hudnut, K., Star, L., Stewart, J. and Jordan, T. H. 2008. Broadband simulations for Mw 7.8 southern San Andreas earthquakes: ground motion sensitivity to rupture speed. Geophysical Research Letters, 35, L22302 (Nov. 2008). 1--5.

[17]

Graves, R., Jordan, T. H., Callaghan, S. Deelman, E., Field, E., Juve, G., Kesselman, C., Maechling, P. Mehta, G., Milner, K., Okaya, D., Small, P. and Vahi, K. 2011. CyberShake: A physics-based seismic hazard model for southern california. Pure and Applied Geophysics, vol. 168, no. 3 (Mar. 2011), 367--381.

[18]

Mai, P. M., Imperatori, W. and Olsen, K. B. 2010. Hybrid broadband ground motion simulations: combining long-period deterministic synthetics with high frequency multiple S-to-S back-scattering. Bulletin of the Seismological Society of America, 100, 5A, 2124--2142.

[19]

National Research Council 2011. National earthquake resilience: Research, implementation, and outreach. National Academies Press, 198.

[20]

Oak Ridge Leadership Computing Facility 2013. Titan User Guide. {Online}. https://www.olcf.ornl.gov/support/system-user-guides/titan-user-guide.

[21]

Olsen, K. B. 1994. Simulation of three-dimensional wave propagation in the Salt Lake basin. University of Utah, Doctoral dissertation.

[22]

Olsen, K. B., Day, S. M., Minster, J. B., Cui, Y., Chourasia, A., Faerman, M., Moore, R., Maechling, P. and Jordan, T. H. 2006. Strong shaking in Los Angeles expected from southern San Andreas earthquake, Geophysical Research Letters, 33, 7 (Apr. 2006).

[23]

Olsen, K. B., Day, S. M., Minster, J. B., Cui, Y., Chourasia, Okaya, D., Maechling, P. and Jordan, T. H. 2008. TeraShake2: spontaneous rupture simulations of mw 7.7 earthquakes on the southern San Andreas fault. Bulletin of the Seismological Society of America, vol. 98, no. 3, (Jun. 2008), 1162--1185.

[24]

Porter, K., Hudnut, K., Perry, S., Reichle, M., Scawthorn, C. and Wein, A. 2011. Forward to the special issue on ShakeOut. Earthquake Spectra, 27, 2, 235--237.

[25]

Schmedes, J., Archuleta, R. J. and Lavallée, D. 2010. Correlation of earthquake source parameters inferred from dynamic rupture simulations. Journal of Geophysical Research: Solid Earth, 115, B3.

[26]

Shi, Z. and Day, S. M. 2013. Rupture dynamics and ground motion from 3-D rough-fault simulations. Journal of Geophysical research, 118, 1--20.

[27]

Southern California Earthquake Center. 2013. ShakeOut. {Online}. http://shakeout.org.

[28]

Taborda, R. and Bielak, J. 2013. Ground-motion simulation and validation of the 2008 Chino Hills. Bulletin of the Seismological Society of America, 103, 131--156.

[29]

Tape, C., Liu, Q., Maggi, A. and Tromp, J. 2010. Seismic tomography of the southern California crust based on spectral-element and adjoint methods. Geophysical Journal International., 180, 1, 433--462.

[30]

University of Illinois NCSA 2013. Blue Waters System Overview. {Online}. https://bluewaters.ncsa.illinois.edu/user-guide.

[31]

Wald, D. J. and Graves R. W. 2001. Resolution analysis of finite fault source inversion using one-and three-dimensional Green's functions: 2. Combining seismic and geodetic data. Journal of Geophysical Research, 106.

[32]

Zhao, L., Chen, P. and. Jordan, T. H. 2006. Strain Green's tensors, reciprocity, and their applications to seismic source and structure studies. Bulletin of the Seismological Society of America, 96, 5, 1753--1763.

[33]

Zhou, J., Unat, D., Choi, D., Guest, C. an Cui, Y. 2012. Hands-on performance tuning of 3D finite difference earthquake simulation on GPU fermi chipset. In Proc. of Int'l. Conf. on Computational Science (ICCS'12, Omaha, Nebraska, June 4--6, 2012), 9, 976--985.

[34]

Zhou, J., Cui, Y., Poyraz, E., Choi, D. and Guest, C. 2013. Multi-GPU implementation of a 3D finite difference time domain earthquake code on heterogeneous supercomputers. In Proc. of Int'l. Conf. on Computational Science (ICCS'13, Barcelona, June 5--7, 2013), 18, 1255--1264.

Cited By

Xu KOlsen KYeh T(2025)Three-dimensional 0–10 Hz physics-based simulations of the 2020 Magna, Utah, earthquake sequenceEarthquake Spectra10.1177/87552930241307797Online publication date: 4-Jan-2025
https://doi.org/10.1177/87552930241307797
Sharma VAuthor HGade MDhanya J(2025)ANN-Based Ground Motion and Physics-Based Broadband Models for Vertical SpectraPure and Applied Geophysics10.1007/s00024-025-03660-yOnline publication date: 1-Feb-2025
https://doi.org/10.1007/s00024-025-03660-y
Callaghan SMaechling PSilva FSu MMilner KGraves ROlsen KCui YVahi KKottke AGoulet CDeelman EJordan TBen-Zion Y(2024)Using open-science workflow tools to produce SCEC CyberShake physics-based probabilistic seismic hazard modelsFrontiers in High Performance Computing10.3389/fhpcp.2024.13607202Online publication date: 1-May-2024
https://doi.org/10.3389/fhpcp.2024.1360720
Show More Cited By

Physics-based seismic hazard analysis on petascale heterogeneous supercomputers
1. Applied computing
  1. Physical sciences and engineering

Recommendations

Adaptive Optimization for Petascale Heterogeneous CPU/GPU Computing
CLUSTER '10: Proceedings of the 2010 IEEE International Conference on Cluster Computing

In this paper, we describe our experiment developing an implementation of the Linpack benchmark for TianHe-1, a petascale CPU/GPU supercomputer system, the largest GPU-accelerated system ever attempted before. An adaptive optimization framework is ...
High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers
SC '16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

The omission of nonlinear effects in large-scale 3D ground motion estimation, which are particularly challenging due to memory and scalability issues, can result in costly misguidance for structural design in earthquake-prone regions. We have ...
Optimizing linpack benchmark on GPU-accelerated petascale supercomputer
Special issue on Community Analysis and Information Recommendation

In this paper we present the programming of the Linpack benchmark on TianHe-1 system, the first petascale supercomputer system of China, and the largest GPU-accelerated heterogeneous system ever attempted before. A hybrid programming model consisting of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

November 2013

1123 pages

ISBN:9781450323789

DOI:10.1145/2503210

General Chair:
William Gropp
University of Illinois at Urbana-Champaign, Urbana, Illinois
,
Program Chair:
Satoshi Matsuoka
Tokyo Institute of Technology, Tokyo, Japan

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SC13

Sponsor:

SIGHPC
SIGARCH
IEEE-CS

SC13: International Conference for High Performance Computing, Networking, Storage and Analysis

November 17 - 21, 2013

Colorado, Denver

Acceptance Rates

SC '13 Paper Acceptance Rate 91 of 449 submissions, 20%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

58
Total Citations
View Citations
302
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)3

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xu KOlsen KYeh T(2025)Three-dimensional 0–10 Hz physics-based simulations of the 2020 Magna, Utah, earthquake sequenceEarthquake Spectra10.1177/87552930241307797Online publication date: 4-Jan-2025
https://doi.org/10.1177/87552930241307797
Sharma VAuthor HGade MDhanya J(2025)ANN-Based Ground Motion and Physics-Based Broadband Models for Vertical SpectraPure and Applied Geophysics10.1007/s00024-025-03660-yOnline publication date: 1-Feb-2025
https://doi.org/10.1007/s00024-025-03660-y
Callaghan SMaechling PSilva FSu MMilner KGraves ROlsen KCui YVahi KKottke AGoulet CDeelman EJordan TBen-Zion Y(2024)Using open-science workflow tools to produce SCEC CyberShake physics-based probabilistic seismic hazard modelsFrontiers in High Performance Computing10.3389/fhpcp.2024.13607202Online publication date: 1-May-2024
https://doi.org/10.3389/fhpcp.2024.1360720
Rojas OMonterrubio-Velasco MRodríguez JCallaghan SAbril CHalldorsson BKowsari MBayat FOlsen KGabriel Ade la Puente J(2024)Earthquake Fault Rupture Modeling and Ground-Motion Simulations for the Southwest Iceland Transform Zone Using CyberShakeBulletin of the Seismological Society of America10.1785/0120240064Online publication date: 10-Dec-2024
https://doi.org/10.1785/0120240064
Roten DYeh TOlsen KDay SCui Y(2023)Implementation of Iwan-Type Nonlinear Rheology in a 3D High-Order Staggered-Grid Finite-Difference MethodBulletin of the Seismological Society of America10.1785/0120230011113:6(2275-2291)Online publication date: 8-Sep-2023
https://doi.org/10.1785/0120230011
Yeh TOlsen K(2023)Fault Damage Zone Effects on Ground Motions during the 2019 Mw 7.1 Ridgecrest, California, EarthquakeBulletin of the Seismological Society of America10.1785/0120220249113:4(1724-1738)Online publication date: 26-Apr-2023
https://doi.org/10.1785/0120220249
Simpson BZhu MSeki AScott M(2023)Challenges in GPU-Accelerated Nonlinear Dynamic Analysis for Structural SystemsJournal of Structural Engineering10.1061/JSENDH.STENG-11311149:3Online publication date: Mar-2023
https://doi.org/10.1061/JSENDH.STENG-11311
Rekoske JGabriel AMay D(2023)Instantaneous Physics‐Based Ground Motion Maps Using Reduced‐Order ModelingJournal of Geophysical Research: Solid Earth10.1029/2023JB026975128:8Online publication date: 11-Aug-2023
https://doi.org/10.1029/2023JB026975
Ichimura TFujita KKoyama KKusakabe RKikuchi YHori THori MMaddegedara LOhi NNishiki TInoue HMinami KNishizawa STsuji MUeda N(2022)152K-computer-node parallel scalable implicit solver for dynamic nonlinear earthquake simulationInternational Conference on High Performance Computing in Asia-Pacific Region10.1145/3492805.3492814(18-29)Online publication date: 7-Jan-2022
https://dl.acm.org/doi/10.1145/3492805.3492814
Hu ZOlsen KDay S(2022)0–5 Hz deterministic 3-D ground motion simulations for the 2014 La Habra, California, EarthquakeGeophysical Journal International10.1093/gji/ggac174230:3(2162-2182)Online publication date: 10-May-2022
https://doi.org/10.1093/gji/ggac174
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten