skip to main content
10.1145/2335755.2335819acmotherconferencesArticle/Chapter ViewAbstractPublication PagesxsedeConference Proceedingsconference-collections
research-article

Massively parallel direct numerical simulations of forced compressible turbulence: a hybrid MPI/OpenMP approach

Published:16 July 2012Publication History

ABSTRACT

A highly scalable simulation code for turbulent flows which solves the fully compressible Navier-Stokes equations is presented. The code, which supports one, two and three dimensional domain decompositions is shown to scale well on up to 262,144 cores. Introducing multiple levels of parallelism based on distributed message passing and shared-memory paradigms results in a reduction of up to 33% of communication time at large core counts. The code has been used to generate a large database of homogeneous isotropic turbulence in a stationary state created by forcing the largest scales in the flow. The scaling of spectra of velocity and density fluctuations are presented. While the former follow classical theories strictly valid for incompressible flows, the latter presents a more complicated behavior. Fluctuations in velocity gradients and derived quantities exhibit extreme though rare fluctuations, a phenomenon known as intermittency. The simulations presented provide data to disentangle Reynolds and Mach number effects.

References

  1. B. J. Bayly, C. D. Levermore, and T. Passot. Density variations in weakly compressible flows. Phys. Fluids, 4:945--954, 1992.Google ScholarGoogle ScholarCross RefCross Ref
  2. G. A. Blaisdell, E. T. Spyropoulos, and J. H. Qin. The effect of the formulation of nonlinear terms on aliasing errors in spectral methods. Appl. Num. Math., 21:207--219, 1996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. C. Canuto, M. Y. Hussaini, A. Quarteroni, and T. A. Zang. Spectral Methods in Fluid Dynamics. Springer-Verlag, 1988.Google ScholarGoogle ScholarCross RefCross Ref
  4. A. W. Cook, W. H. Cabot, P. L. Williams, B. J. Miller, B. R. d. Supinski, R. K. Yates, and M. L. Welcome. Tera-scalable algorithms for variable-density elliptic hydrodynamics with spectral accuracy. In Proc. 2005 ACM/IEEE Conf. Supercomp., 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. S. Dastgeer and G. P. Zank. Turbulence in nearly incompressible fluids: density spectrum, flows, correlations and implication to the interstellar medium. Nonl. Proc. Geophys., 12(1):139--148, 2005.Google ScholarGoogle ScholarCross RefCross Ref
  6. D. A. Donzis and K. R. Sreenivasan. The bottleneck effect and the Kolmogorov constant in isotropic turbulence. J. Fluid Mech., 657:171--188, 2010.Google ScholarGoogle ScholarCross RefCross Ref
  7. D. A. Donzis, P. K. Yeung, and D. Pekurovsky. Turbulence simulations on O(104) processors. TeraGrid 2008 Conference, 2008.Google ScholarGoogle Scholar
  8. D. A. Donzis, P. K. Yeung, and K. R. Sreenivasan. Dissipation and enstrophy in isotropic turbulence: scaling and resolution effects in direct numerical simulations. Phys. Fluids, 20:045108, 2008.Google ScholarGoogle ScholarCross RefCross Ref
  9. F. Ducros, F. Laporte, T. Souleres, V. Guinot, P. Moinat, and B. Caruelle. High-order fluxes for conservative skew-symmetric-like schemes in structured meshes: Application to compressible flows. J. Comp. Phys., 161:114--139, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. U. Frisch. Turbulence. Cambridge University Press, 1995.Google ScholarGoogle Scholar
  11. G. H. Golub and C. F. Van Loan. Matrix Computation. The Johns Hopkins University Press, 1996.Google ScholarGoogle Scholar
  12. T. Hoefler, P. Gottschling, and A. Lumsdain. Leveraging non-blocking collective communication in high-performance Applications. In SPAA'08: Proceedings of the twentieth annual symposium on parallelism in algorithms and architectures, pages 113--115, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. T. Ishihara, T. Gotoh, and Y. Kaneda. Study of high-Reynolds number isotropic turbulence by direct numerical simulation. Annu. Rev. Fluid Mech., 41:165--180, 2009.Google ScholarGoogle ScholarCross RefCross Ref
  14. J. Jiménez. Computing high-Reynolds-number turbulence: will simulations ever replace experiments? J. Turbulence, 4:022, 2003.Google ScholarGoogle ScholarCross RefCross Ref
  15. K. Kandalla, H. Subramoni, K. Tomko, D. Pekurovsky, S. Sur, and D. K. Panda. High-performance and scalable non-blocking all-to-all with collective offload on InfiniBand clusters: a study with parallel 3D FFT. Comput. Sci., 26:237--246, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. A. N. Kolmogorov. Local structure of turbulence in an incompressible fluid for very large Reynolds numbers. Dokl. Akad. Nauk. SSSR, 30:299--303, 1941.Google ScholarGoogle Scholar
  17. S. Lee, S. K. Lele, and P. Moin. Eddy shocklets in decaying compressible turbulence. Phys. Fluids, 3:657--664, 1991.Google ScholarGoogle ScholarCross RefCross Ref
  18. S. Lee, S. K. Lele, and P. Moin. Direct numerical simulation of isotropic turbulence interacting with a weak shock wave. J. Fluid Mech., 251:533--562, 1993.Google ScholarGoogle ScholarCross RefCross Ref
  19. S. Lele. Compact finite difference schemes with spectral-like resolution. J. Comp. Phys., 103:16--42, 1992.Google ScholarGoogle ScholarCross RefCross Ref
  20. S. K. Lele. Compressibility effects on turbulence. Annu. Rev. Fluid Mech., 26:211--254, 1994.Google ScholarGoogle ScholarCross RefCross Ref
  21. X.-G. Lv and J. Le. A note on solving nearly penta-diagonal linear systems. Appl. Math. Comput., 204:707--712, 2008.Google ScholarGoogle Scholar
  22. P. D. Mininni, D. Rosenberg, R. Reddy, and A. Pouquet. A hybrid MPI-OpenMP scheme for scalable parallel pseudospectral computations for fluid turbulence. Parall. Comp., 37(6-7):316--326, 2011.Google ScholarGoogle ScholarCross RefCross Ref
  23. P. Moin and K. Mahesh. Direct numerical simulation: A tool in turbulence research. Annu. Rev. Fluid Mech., 30:539--578, 1998.Google ScholarGoogle ScholarCross RefCross Ref
  24. A. S. Monin and A. M. Yaglom. Statistical Fluid Mechanics, volume 2. MIT Press, 1975.Google ScholarGoogle Scholar
  25. M. R. Petersen and D. Livescu. Forcing for statistically stationary compressible isotropic turbulence. Phys. Fluids, 22:116101, 2010.Google ScholarGoogle ScholarCross RefCross Ref
  26. S. Pirozzoli and F. Grasso. Direct numerical simulations of isotropic compressible turbulence: Influence of compressibility on dynamics and structures. Phys. Fluids, 16:4386--4407, 2004.Google ScholarGoogle ScholarCross RefCross Ref
  27. P. Sagaut and C. Cambon. Homogeneous Turbulence Dynamics. Cambridge University Press, Cambridge, 2008.Google ScholarGoogle ScholarCross RefCross Ref
  28. R. Samtaney, D. I. Pullin, and B. Kosovic. Direct numerical simulation of decaying compressible turbulence and shocklet statistics. Phys. Fluids, 13:1415, 2001.Google ScholarGoogle ScholarCross RefCross Ref
  29. K. R. Sreenivasan. On the universality of the kolmogorov constant. Phys. Fluids, 7:2778--2784, 1995.Google ScholarGoogle ScholarCross RefCross Ref
  30. K. R. Sreenivasan and R. A. Antonia. The phenomenology of small-scale turbulence. Annu. Rev. Fluid Mech., 29:435--472, 1997.Google ScholarGoogle ScholarCross RefCross Ref
  31. J. H. Williamson. Low-storage Runge-Kutta schemes. J. Comp. Phys., 35(1):48--56, 1980.Google ScholarGoogle ScholarCross RefCross Ref
  32. V. Yakhot and K. R. Sreenivasan. Anomalous scaling of structure functions and dynamic constraints on turbulence simulations. J. Stat. Phys., 121:823--841, 2005.Google ScholarGoogle ScholarCross RefCross Ref
  33. E. Yilmaz, R. U. Payli, H. U. Akay, and A. Ecer. Hybrid parallelism for CFD simulations: Combining MPI with OpenMP. In Parallel Computational Fluid Dynamics 2007, volume 67, pages 401--408. Springer Berlin Heidelberg, 2009.Google ScholarGoogle Scholar

Index Terms

  1. Massively parallel direct numerical simulations of forced compressible turbulence: a hybrid MPI/OpenMP approach

                Recommendations

                Comments

                Login options

                Check if you have access through your login credentials or your institution to get full access on this article.

                Sign in
                • Published in

                  cover image ACM Other conferences
                  XSEDE '12: Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the eXtreme to the campus and beyond
                  July 2012
                  423 pages
                  ISBN:9781450316026
                  DOI:10.1145/2335755

                  Copyright © 2012 ACM

                  Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

                  Publisher

                  Association for Computing Machinery

                  New York, NY, United States

                  Publication History

                  • Published: 16 July 2012

                  Permissions

                  Request permissions about this article.

                  Request Permissions

                  Check for updates

                  Qualifiers

                  • research-article

                  Acceptance Rates

                  Overall Acceptance Rate129of190submissions,68%

                PDF Format

                View or Download as a PDF file.

                PDF

                eReader

                View online with eReader.

                eReader