ABSTRACT
A highly scalable simulation code for turbulent flows which solves the fully compressible Navier-Stokes equations is presented. The code, which supports one, two and three dimensional domain decompositions is shown to scale well on up to 262,144 cores. Introducing multiple levels of parallelism based on distributed message passing and shared-memory paradigms results in a reduction of up to 33% of communication time at large core counts. The code has been used to generate a large database of homogeneous isotropic turbulence in a stationary state created by forcing the largest scales in the flow. The scaling of spectra of velocity and density fluctuations are presented. While the former follow classical theories strictly valid for incompressible flows, the latter presents a more complicated behavior. Fluctuations in velocity gradients and derived quantities exhibit extreme though rare fluctuations, a phenomenon known as intermittency. The simulations presented provide data to disentangle Reynolds and Mach number effects.
- B. J. Bayly, C. D. Levermore, and T. Passot. Density variations in weakly compressible flows. Phys. Fluids, 4:945--954, 1992.Google ScholarCross Ref
- G. A. Blaisdell, E. T. Spyropoulos, and J. H. Qin. The effect of the formulation of nonlinear terms on aliasing errors in spectral methods. Appl. Num. Math., 21:207--219, 1996. Google ScholarDigital Library
- C. Canuto, M. Y. Hussaini, A. Quarteroni, and T. A. Zang. Spectral Methods in Fluid Dynamics. Springer-Verlag, 1988.Google ScholarCross Ref
- A. W. Cook, W. H. Cabot, P. L. Williams, B. J. Miller, B. R. d. Supinski, R. K. Yates, and M. L. Welcome. Tera-scalable algorithms for variable-density elliptic hydrodynamics with spectral accuracy. In Proc. 2005 ACM/IEEE Conf. Supercomp., 2005. Google ScholarDigital Library
- S. Dastgeer and G. P. Zank. Turbulence in nearly incompressible fluids: density spectrum, flows, correlations and implication to the interstellar medium. Nonl. Proc. Geophys., 12(1):139--148, 2005.Google ScholarCross Ref
- D. A. Donzis and K. R. Sreenivasan. The bottleneck effect and the Kolmogorov constant in isotropic turbulence. J. Fluid Mech., 657:171--188, 2010.Google ScholarCross Ref
- D. A. Donzis, P. K. Yeung, and D. Pekurovsky. Turbulence simulations on O(104) processors. TeraGrid 2008 Conference, 2008.Google Scholar
- D. A. Donzis, P. K. Yeung, and K. R. Sreenivasan. Dissipation and enstrophy in isotropic turbulence: scaling and resolution effects in direct numerical simulations. Phys. Fluids, 20:045108, 2008.Google ScholarCross Ref
- F. Ducros, F. Laporte, T. Souleres, V. Guinot, P. Moinat, and B. Caruelle. High-order fluxes for conservative skew-symmetric-like schemes in structured meshes: Application to compressible flows. J. Comp. Phys., 161:114--139, 2000. Google ScholarDigital Library
- U. Frisch. Turbulence. Cambridge University Press, 1995.Google Scholar
- G. H. Golub and C. F. Van Loan. Matrix Computation. The Johns Hopkins University Press, 1996.Google Scholar
- T. Hoefler, P. Gottschling, and A. Lumsdain. Leveraging non-blocking collective communication in high-performance Applications. In SPAA'08: Proceedings of the twentieth annual symposium on parallelism in algorithms and architectures, pages 113--115, 2008. Google ScholarDigital Library
- T. Ishihara, T. Gotoh, and Y. Kaneda. Study of high-Reynolds number isotropic turbulence by direct numerical simulation. Annu. Rev. Fluid Mech., 41:165--180, 2009.Google ScholarCross Ref
- J. Jiménez. Computing high-Reynolds-number turbulence: will simulations ever replace experiments? J. Turbulence, 4:022, 2003.Google ScholarCross Ref
- K. Kandalla, H. Subramoni, K. Tomko, D. Pekurovsky, S. Sur, and D. K. Panda. High-performance and scalable non-blocking all-to-all with collective offload on InfiniBand clusters: a study with parallel 3D FFT. Comput. Sci., 26:237--246, 2011. Google ScholarDigital Library
- A. N. Kolmogorov. Local structure of turbulence in an incompressible fluid for very large Reynolds numbers. Dokl. Akad. Nauk. SSSR, 30:299--303, 1941.Google Scholar
- S. Lee, S. K. Lele, and P. Moin. Eddy shocklets in decaying compressible turbulence. Phys. Fluids, 3:657--664, 1991.Google ScholarCross Ref
- S. Lee, S. K. Lele, and P. Moin. Direct numerical simulation of isotropic turbulence interacting with a weak shock wave. J. Fluid Mech., 251:533--562, 1993.Google ScholarCross Ref
- S. Lele. Compact finite difference schemes with spectral-like resolution. J. Comp. Phys., 103:16--42, 1992.Google ScholarCross Ref
- S. K. Lele. Compressibility effects on turbulence. Annu. Rev. Fluid Mech., 26:211--254, 1994.Google ScholarCross Ref
- X.-G. Lv and J. Le. A note on solving nearly penta-diagonal linear systems. Appl. Math. Comput., 204:707--712, 2008.Google Scholar
- P. D. Mininni, D. Rosenberg, R. Reddy, and A. Pouquet. A hybrid MPI-OpenMP scheme for scalable parallel pseudospectral computations for fluid turbulence. Parall. Comp., 37(6-7):316--326, 2011.Google ScholarCross Ref
- P. Moin and K. Mahesh. Direct numerical simulation: A tool in turbulence research. Annu. Rev. Fluid Mech., 30:539--578, 1998.Google ScholarCross Ref
- A. S. Monin and A. M. Yaglom. Statistical Fluid Mechanics, volume 2. MIT Press, 1975.Google Scholar
- M. R. Petersen and D. Livescu. Forcing for statistically stationary compressible isotropic turbulence. Phys. Fluids, 22:116101, 2010.Google ScholarCross Ref
- S. Pirozzoli and F. Grasso. Direct numerical simulations of isotropic compressible turbulence: Influence of compressibility on dynamics and structures. Phys. Fluids, 16:4386--4407, 2004.Google ScholarCross Ref
- P. Sagaut and C. Cambon. Homogeneous Turbulence Dynamics. Cambridge University Press, Cambridge, 2008.Google ScholarCross Ref
- R. Samtaney, D. I. Pullin, and B. Kosovic. Direct numerical simulation of decaying compressible turbulence and shocklet statistics. Phys. Fluids, 13:1415, 2001.Google ScholarCross Ref
- K. R. Sreenivasan. On the universality of the kolmogorov constant. Phys. Fluids, 7:2778--2784, 1995.Google ScholarCross Ref
- K. R. Sreenivasan and R. A. Antonia. The phenomenology of small-scale turbulence. Annu. Rev. Fluid Mech., 29:435--472, 1997.Google ScholarCross Ref
- J. H. Williamson. Low-storage Runge-Kutta schemes. J. Comp. Phys., 35(1):48--56, 1980.Google ScholarCross Ref
- V. Yakhot and K. R. Sreenivasan. Anomalous scaling of structure functions and dynamic constraints on turbulence simulations. J. Stat. Phys., 121:823--841, 2005.Google ScholarCross Ref
- E. Yilmaz, R. U. Payli, H. U. Akay, and A. Ecer. Hybrid parallelism for CFD simulations: Combining MPI with OpenMP. In Parallel Computational Fluid Dynamics 2007, volume 67, pages 401--408. Springer Berlin Heidelberg, 2009.Google Scholar
Index Terms
Massively parallel direct numerical simulations of forced compressible turbulence: a hybrid MPI/OpenMP approach
Recommendations
Assessment of high-resolution methods for numerical simulations of compressible turbulence with shock waves
Flows in which shock waves and turbulence are present and interact dynamically occur in a wide range of applications, including inertial confinement fusion, supernovae explosion, and scramjet propulsion. Accurate simulations of such problems are ...
Numerical simulations of the flow and sound fields of a heated axisymmetric pulsating jet
The flow and sound fields of a heated axisymmetric pulsating jet have been investigated by direct numerical solution of the compressible Navier-Stokes equations in cylindrical coordinates using highly accurate numerical methods. Effects of pulsating ...
Higher entropy conservation and numerical stability of compressible turbulence simulations
We present a numerical formulation for the treatment of nonlinear instabilities in shock-free compressible turbulence simulations. The formulation is high order and contains no artificial dissipation. Numerical stability is enhanced through semi-...
Comments