Wubing Wan
ABSTRACT
A high-scalable and fully optimized earthquake model is presented based on the latest Sunway supercomputer. Contributions include: 1) the curvilinear grid finite-difference method (CGFDM) and flexible model applying perfectly matched layer (PML) and enabling more accurate and realistic terrain descriptions; 2) a hybrid and non-uniform domain decomposition scheme that efficiently maps the model across different levels of the computing system; and 3) sophisticated optimizations that largely alleviate or even eliminate bottlenecks in memory, communication, etc., obtaining a speedup of over 140×. Combining all innovations, the design fully exploits the hardware potential of all aspects and enables us to perform the largest CGFDM-based earthquake simulation ever reported (69.7 PFlops using over 39 million cores). Based on our design, the Turkey earthquakes (February 6, 2023), and the Ridgecrest earthquake (July 4, 2019), are successfully simulated with a maximum resolution of 12-m. Precise hazard evaluations for the hazardous reduction of earthquake-stricken areas are also conducted.
- J Bielak. 1998. Ground motion modeling using 3D finite element methods. Proc. 2nd Intl. Sympo. Effects of Surface Geology on Seismic Motion, 1998 1 (1998), 121--133.Google Scholar
- Jacobo Bielak, Omar Ghattas, and EJ Kim. 2005. Parallel octree-based finite element method for large-scale earthquake ground motion simulation. Computer Modeling in Engineering and Sciences 10, 2 (2005), 99.Google Scholar
- Alexander Breuer, Alexander Heinecke, and Yifeng Cui. 2017. EDGE: Extreme Scale Fused Seismic Simulations with the Discontinuous Galerkin Method. In International Supercomputing Conference. Springer, 41--60.Google Scholar
- Bingwei Chen, Haohuan Fu, Yanwen Wei, Conghui He, Wenqiang Zhang, Yuxuan Li, Wubin Wan, Wei Zhang, Lin Gan, Zhenguo Zhang, et al. 2018. Simulating the Wenchuan earthquake with accurate surface topography on Sunway TaihuLight. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 517--528.Google ScholarDigital Library
- Josep de la Puente, Martin Käser, Michael Dumbser, and Heiner Igel. 2007. An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes-IV. Anisotropy. Geophysical Journal International 169, 3 (2007), 1210--1228.Google ScholarCross Ref
- Michael Dumbser and Martin Käser. 2006. An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes---II. The three-dimensional isotropic case. Geophysical Journal International 167, 1 (2006), 319--336.Google ScholarCross Ref
- Haohuan Fu, Conghui He, Bingwei Chen, Zekun Yin, Zhenguo Zhang, Wenqiang Zhang, Tingjian Zhang, Wei Xue, Weiguo Liu, Wanwang Yin, et al. 2017. 18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 2.Google ScholarDigital Library
- Haohuan Fu, Junfeng Liao, Jinzhe Yang, Lanning Wang, Zhenya Song, Xiaomeng Huang, Chao Yang, Wei Xue, Fangfang Liu, Fangli Qiao, et al. 2016. The Sunway TaihuLight supercomputer: system and applications. Science China Information Sciences 59, 7 (2016), 1--16.Google ScholarCross Ref
- Sezim Ezgi Güvercin, Hayrullah Karabulut, A Özgün Konca, Uğur Doğan, and Semih Ergintav. 2022. Active seismotectonics of the East Anatolian Fault. Geophysical Journal International 230, 1 (2022), 50--69.Google ScholarCross Ref
- Ufuk Hancılar, Karin Şeşetyan, Eser Çaktı, Erdal Şafak, Nesrin Yenihayat, Fatma S. Malcıoģlu, Kökcan Dönmez, Tuģce Tetik, Hakan Süleyman Gómez Capera, and Augusto Antonio. 2023. Kahramanmaraş-Gaziantep Türkiye M7. 7 Earthquake, 6 February 2023 (04: 17 GMT+ 03: 00): Strong Ground Motion and Building Damage Estimations Preliminary Report (v6). (2023).Google Scholar
- Stephen H Hartzell, David L Carver, and Kenneth W King. 1994. Initial investigation of site and topographic effects at Robinwood Ridge, California. Bulletin of the Seismological Society of America 84, 5 (1994), 1336--1349.Google ScholarCross Ref
- Alexander Heinecke, Alexander Breuer, Sebastian Rettenberger, Michael Bader, Alice-Agnes Gabriel, Christian Pelties, Arndt Bode, William Barth, Xiang-Ke Liao, Karthikeyan Vaidyanathan, et al. 2014. Petascale high order dynamic rupture earthquake simulations on heterogeneous supercomputers. In High Performance Computing, Networking, Storage and Analysis, SC14: International Conference for. IEEE, 3--14.Google Scholar
- Ray Hixon. 1997. On increasing the accuracy of MacCormack schemes for aeroacoustic applications. In 3rd AIAA/CEAS Aeroacoustics Conference. 1586.Google ScholarCross Ref
- Ray Hixon. 1998. Evaluation of a high-accuracy MacCormack-type scheme using benchmark problems. Journal of Computational Acoustics 6, 03 (1998), 291--305.Google ScholarCross Ref
- Tsuyoshi Ichimura, Kohei Fujita, Kentaro Koyama, Yuma Kikuchi, Ryota Kusakabe, Kazuo Minami, Hikaru Inoue, Seiya Nishizawa, Miwako Tsuji, Tatsuo Nishiki, et al. 2021. Fast scalable implicit solver with convergence of equation-based modeling and data-driven learning: earthquake city simulation on low-order unstructured finite element. In Proceedings of the Platform for Advanced Scientific Computing Conference. 1--12.Google ScholarDigital Library
- Tsuyoshi Ichimura, Kohei Fujita, Kentaro Koyama, Ryota Kusakabe, Yuma Kikuchi, Takane Hori, Muneo Hori, Lalith Maddegedara, Noriyuki Ohi, Tatsuo Nishiki, et al. 2022. 152K-computer-node parallel scalable implicit solver for dynamic nonlinear earthquake simulation. In International Conference on High Performance Computing in Asia-Pacific Region. 18--29.Google ScholarDigital Library
- Tsuyoshi Ichimura, Kohei Fujita, Ryota Kusakabe, Kentaro Koyama, Sota Murakami, Yuma Kikuchi, Takane Hori, Muneo Hori, Hikaru Inoue, Takafumi Nose, et al. 2022. Extreme scale earthquake simulation with uncertainty quantification. In 2022 SC22: International Conference for High Performance Computing, Networking, Storage and Analysis (SC). IEEE Computer Society, 37--47.Google ScholarCross Ref
- Tsuyoshi Ichimura, Kohei Fujita, Pher Errol Balde Quinay, Lalith Maddegedara, Muneo Hori, Seizo Tanaka, Yoshihisa Shizawa, Hiroshi Kobayashi, and Kazuo Minami. 2015. Implicit nonlinear wave simulation with 1.08 T DOF and 0.270 T unstructured finite elements to enhance comprehensive earthquake simulation. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 4.Google ScholarDigital Library
- Tsuyoshi Ichimura, Kohei Fujita, Takuma Yamaguchi, Akira Naruse, Jack C Wells, Thomas C Schulthess, Tjerk P Straatsma, Christopher J Zimmer, Maxime Martinasso, Kengo Nakajima, et al. 2018. A fast scalable implicit solver for nonlinear time-evolution earthquake city problem on low-ordered unstructured finite elements with artificial intelligence and transprecision computing. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 627--637.Google ScholarDigital Library
- Saad Khan, Mark van der Meijde, Harald van der Werff, and Muhammad Shafique. 2020. The impact of topography on seismic amplification during the 2005 Kashmir earthquake. Natural Hazards and Earth System Sciences 20, 2 (2020), 399--411.Google ScholarCross Ref
- Dimitri Komatitsch and Jeroen Tromp. 1999. Introduction to the spectral element method for three-dimensional seismic wave propagation. Geophysical Journal International 139, 3 (1999), 806--822.Google ScholarCross Ref
- Dimitri Komatitsch, Seiji Tsuboi, Chen Ji, and Jeroen Tromp. 2003. A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator. In Supercomputing, 2003 ACM/IEEE Conference. IEEE, 4--4.Google ScholarDigital Library
- Dimitri Komatitsch and Jean-Pierre Vilotte. 1998. The spectral element method: an efficient tool to simulate the seismic response of 2D and 3D geological structures. Bulletin of the Seismological Society of America 88, 2 (1998), 368--392.Google ScholarCross Ref
- Diego Melgar, Tuncay Taymaz, Athanassios Ganas, Brendan Crowell, Taylan Öcalan, Metin Kahraman, Varvara Tsironi, Seda Yolsal-Çevikbilen, Sotiris Valkaniotis, Tahir Serkan Irmak, et al. 2023. Sub-and super-shear ruptures during the 2023 Mw 7.8 and Mw 7.6 earthquake doublet in SE Türkiye. Seismica 2, 3 (2023).Google Scholar
- NG Reitman, RW Briggs, WD Barnhart, JA Thompson, CB DuRoss, AE Hatem, RD Gold, and JD Mejstrik. 2023. Preliminary fault rupture mapping of the 2023 M7. 8 and M7. 5 Türkiye Earthquakes.Google Scholar
- Hannes Isaak Reuter, Andy Nelson, and Andrew Jarvis. 2007. An evaluation of void-filling interpolation methods for SRTM data. International Journal of Geographical Information Science 21, 9 (2007), 983--1008.Google ScholarDigital Library
- Max Rietmann, Peter Messmer, Tarje Nissen-Meyer, Daniel Peter, Piero Basini, Dimitri Komatitsch, Olaf Schenk, Jeroen Tromp, Lapo Boschi, and Domenico Giardini. 2012. Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures. In SC'12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. IEEE, 1--11.Google ScholarDigital Library
- Zachary E Ross, Benjamín Idini, Zhe Jia, Oliver L Stephenson, Minyan Zhong, Xin Wang, Zhongwen Zhan, Mark Simons, Eric J Fielding, Sang-Ho Yun, et al. 2019. Hierarchical interlocked orthogonal faulting in the 2019 Ridgecrest earthquake sequence. Science 366, 6463 (2019), 346--351.Google Scholar
- Daniel Roten, Yifeng Cui, Kim B Olsen, Steven M Day, Kyle Withers, William H Savran, Peng Wang, and Dawei Mu. 2016. High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers. In SC'16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 957--968.Google ScholarDigital Library
- USGS M7.6 Shakemap. 2023. M 7.5 - Elbistan earthquake, Kahramanmaras earthquake sequence. https://earthquake.usgs.gov/earthquakes/eventpage/us6000jlqa/shakemapGoogle Scholar
- USGS M7.8 Shakemap. 2023. M7.8 - Pazarcik earthquake, Kahramanmaras earthquake sequence. https://earthquake.usgs.gov/earthquakes/eventpage/us6000jllz/shakemapGoogle Scholar
- Patrick Small, David Gill, Philip J Maechling, Ricardo Taborda, Scott Callaghan, Thomas H Jordan, Kim B Olsen, Geoffrey P Ely, and Christine Goulet. 2017. The SCEC unified community velocity model software framework. Seismological Research Letters 88, 6 (2017), 1539--1552.Google ScholarCross Ref
- Paul Spudich, Margaret Hellweg, and WHK Lee. 1996. Directional topographic site response at Tarzana observed in aftershocks of the 1994 Northridge, California, earthquake: implications for mainshock motions. Bulletin of the Seismological Society of America 86, 1B (1996), S193--S208.Google ScholarCross Ref
- Jean Virieux. 1984. SH-wave propagation in heterogeneous media: Velocity-stress finite-difference method. Geophysics 49, 11 (1984), 1933--1942.Google ScholarCross Ref
- Jean Virieux. 1986. P-SV wave propagation in heterogeneous media: Velocity-stress finite-difference method. Geophysics 51, 4 (1986), 889--901.Google ScholarCross Ref
- Xiaodong Yu, Sheng Di, Kai Zhao, Jiannan Tian, Dingwen Tao, Xin Liang, and Franck Cappello. 2022. Ultrafast Error-bounded Lossy Compression for Scientific Datasets. In Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing. 159--171.Google ScholarDigital Library
- Wei Zhang and Yang Shen. 2010. Unsplit complex frequency-shifted PML implementation using auxiliary differential equations for seismic wave modeling. Geophysics 75, 4 (2010), T141--T154.Google ScholarCross Ref
- Wei Zhang, Zhenguo Zhang, and Xiaofei Chen. 2012. Three-dimensional elastic wave numerical modelling in the presence of surface topography by a collocated-grid finite-difference method on curvilinear grids. Geophysical Journal International 190, 1 (2012), 358--378.Google ScholarCross Ref
- Zhenguo Zhang, Wenqiang Zhang, Danhua Xin, Kejie Chen, and Xiaofei Chen. 2021. A dynamic-rupture model of the 2019 M w 7.1 Ridgecrest earthquake being compatible with the observations. Seismological Research Letters 92, 2A (2021), 870--876.Google ScholarCross Ref
- Gengshang Zhu, Zhenguo Zhang, Jian Wen, Wei Zhang, and Xiaofei Chen. 2013. Preliminary results of strong ground motion simulation for the Lushan earthquake of 20 April 2013, China. Earthquake Science 26 (2013), 191--197.Google ScholarCross Ref
Recommendations
18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios
SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and AnalysisThis paper reports our large-scale nonlinear earthquake simulation software on Sunway TaihuLight. Our innovations include: (1) a customized parallelization scheme that employs the 10 million cores efficiently at both the process and the thread levels; (...
Extreme scale multi-physics simulations of the tsunamigenic 2004 sumatra megathrust earthquake
SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and AnalysisWe present a high-resolution simulation of the 2004 Sumatra-Andaman earthquake, including non-linear frictional failure on a megathrustsplay fault system. Our method exploits unstructured meshes capturing the complicated geometries in subduction zones ...
Towards Efficient SpMV on Sunway Manycore Architectures
ICS '18: Proceedings of the 2018 International Conference on SupercomputingSparse Matrix-Vector Multiplication (SpMV) is an essential computation kernel for many data-analytic workloads running in both supercomputers and data centers. The intrinsic irregularity in SpMV is challenging to achieve high performance, especially ...
Comments