Abstract
TOUGH2 is a widely used reservoir simulator for solving subsurface flow related problems such as nuclear waste geologic isolation, environmental remediation of soil and groundwater contamination, and geothermal reservoir engineering. It solves a set of coupled mass and energy balance equations using a finite volume method. This contribution presents the design and analysis of a parallel version of TOUGH2. The parallel implementation first partitions the unstructured computational domain. For each time step, a set of coupled non-linear equations is solved with Newton iteration. In each Newton step, a Jacobian matrix is calculated and an ill-conditioned non-symmetric linear system is solved using a preconditioned iterative solver. Communication is required for convergence tests and data exchange across partitioning borders. Parallel performance results on Cray T3E-900 are presented for two real application problems arising in the Yucca Mountain nuclear waste site study. The execution time is reduced from 7504 seconds on two processors to 126 seconds on 128 processors for a 2D problem involving 52,752 equations. For a larger 3D problem with 293,928 equations the time decreases from 10,055 seconds on 16 processors to 329 seconds on 512 processors.
Similar content being viewed by others
References
E. Anderson, J. Brooks, C. Grassl, and S. Scott. Performance of the CRAY T3E multiprocessor. In Proceedings of Supercomputing '97, 1997.
R. Barrett, M. Berry, T. F. Chan, J. Demmel, J. Donato, J. Dongarra, V. Eijkhout, R. Pozo, C. Romine, and H. V. der Vorst. Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, 2nd ed. SIAM, Philadelphia, 1994.
G. Bodvarsson, T. Bandurraga, and Y. Wu. The site-scale unsaturated zone model of Yucca Mountain, Nevada, for the viability assessment. Yucca Mountain Characterization Project Report LBNL-40376, UC-814, Earth Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, Calif., 1997.
E. Elmroth. On grid partitioning for a high performance groundwater simulation software. In B. Engquist et. al., ed. Simulation and Visualization on the Grid. Lecture Notes in Computational Science and Engineering, Vol. 13, pp. 221-234. Berlin, 2000.
E. Elmroth, C. Ding, Y.-S. Wu, and K. Pruess. A parallel implementation of the TOUGH2 software package for large scale multiphase fluid and heat flow simulations. In Proceedings of Supercomputing '99, 1999.
MPI Forum. A message-passing interface standard. International Journal of Supercomputing Applications and High Performance Computing 8(3–4), 1994.
S. Hutchinson, L. Prevost, J. Shadid, C. Tong, and R. Tuminaro. Aztec users' guide, version 2.0. Technical report, Massively Parallel Computing Research Center, Sandia National Laboratories, Albuquerque, NM, 1998.
G. Karypsis and V. Kumar. METIS. A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices, version 4.0. Technical report, Department of Computer Science, University of Minnesota.
K. Pruess. TOUGH users' guide. Technical report LBNL-29400, UC-251, Earth Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, Calif., 1987.
K. Pruess. TOUGH2—a general-purpose numerical simulator for multiphase fluid and heat flow. Technical report LBNL-29400, UC-251, Earth Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, Calif., 1991.
K. Pruess, ed. Proceedings of the TOUGH workshop '98. Technical report LBNL-41995, Conf-980559, Earth Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, Calif., 1998.
Y. Saad. ILUT: a dual threshold incomplete ILU preconditioner. Numerical Linear Algebra with Applications, 1:387-402, 1994.
B. Smith, P. Bjørstad, and W. Gropp. Domain Decomposition. Parallel Multilevel Methods for Elliptic Partial Differential Equations. Cambridge University Press, 1996.
H. Van der Vorst. BICGSTAB: a fast and smoothly converging variant of the BI-CG for the solution of nonsymmetric linear systems. SIAM J. Sci. Stat. Comp., 13:631-644, 1992.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Elmroth, E., Ding, C. & Wu, YS. High Performance Computations for Large Scale Simulations of Subsurface Multiphase Fluid and Heat Flow. The Journal of Supercomputing 18, 235–258 (2001). https://doi.org/10.1023/A:1008117130225
Issue Date:
DOI: https://doi.org/10.1023/A:1008117130225