skip to main content
10.1145/2488551.2488574acmotherconferencesArticle/Chapter ViewAbstractPublication PageseurompiConference Proceedingsconference-collections
research-article

Run-time creation of the turbulent channel flow database by an HPC simulation using MPI-DB

Published: 15 September 2013 Publication History

Abstract

We demonstrate a method based on MPI client-server implementation for storing the output of computations directly into the database. Our method automates the previously used inefficient ingestion process which required development of special tools for each simulation. In large-scale channel flow simulation experiments we were able to ingest the output data sets in real time, without delaying the simulation process. This was accomplished by building a Fortran interface to the MPI-DB software library and using it within the simulation code. The channel flow simulation data set will be exposed for analysis to researchers using the JHU Public Turbulence Database [7].

References

[1]
P. Baumann, A. Dehmel, P. Furtado, R. Ritsch, and N. Widmann. The multidimensional database system rasdaman. In SIGMOD, 1998.
[2]
P. G. Brown. Overview of scidb: large scale array storage, processing and analysis. In SIGMOD, 2010.
[3]
F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A distributed storage system for structured data. ACM Trans. Comput. Syst., 26(2):4:1--4:26, June 2008.
[4]
G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: amazon's highly available key-value store. In SOSP, 2007.
[5]
E. Givelberg, A. Szalay, K. Kanov, and R. Burns. An architecture for a data-intensive computer. In Proceedings of the first international workshop on Network-aware data management, NDM '11, pages 57--64, New York, NY, USA, 2011. ACM.
[6]
Y. Gu and R. L. Grossman. Udt: Udp-based data transfer for high-speed wide area networks. In Computer Networks, volume 51. Elsevier, May 2007. Issue 7.
[7]
JHU Turbulence Database Group. JHU Turbulence Database, Oct. 2012. http://turbulence.pha.jhu.edu.
[8]
J. Kim, P. Moin, and R. Moser. Turbulence statistics in fully developed channel flow at low reynolds number. J. Fluid Mech., 177:133--166, 1987.
[9]
S. K. Lele. Compact finite differences schemes with spectral-like resolution. J. Comput. Phys., 174, 1992.
[10]
Y. Li, E. Perlman, M. Wan, Y. Yang, C. Meneveau, R. Burns, S. Chen, A. Szalay, and G. Eyink. A public turbulence database cluster and applications to study lagrangian evolution of velocity increments in turbulence. J. Turbulence, 9(31), 2008.
[11]
L. Libkin, R. Machlin, and L. Wong. A query language for multidimensional arrays: design, implementation, and optimization techniques. In SIGMOD, 1996.
[12]
A. P. Marathe and K. Salem. Query processing techniques for arrays. The VLDB Journal, 11(1):68--91, 2002.
[13]
Message Passing Interface Forum. MPI: A Message-Passing Interface Standard (Version 2.2). Technical report, University of Tennessee, Knoxville, Tennessee, 2009.
[14]
E. Perlman, R. Burns, Y. Li, and C. Meneveau. Data exploration of turbulence simulations using a database cluster. In Proceedings of the 2007 ACM/IEEE conference on Supercomputing, SC '07, pages 23:1--23:11, New York, NY, USA, 2007. ACM.
[15]
R. Rew and G. Davis. Netcdf: an interface for scientific data access. Computer Graphics and Applications, IEEE, 10(4):76--82, 1990.
[16]
The HDF Group. Hierarchical data format version 5, 2000--2010. http://www.hdfgroup.org/HDF5.

Cited By

View all
  • (2014)ADR visualization: A generalized framework for ranking large-scale scientific data using Analysis-Driven Refinement2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV)10.1109/LDAV.2014.7013203(43-50)Online publication date: Nov-2014

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EuroMPI '13: Proceedings of the 20th European MPI Users' Group Meeting
September 2013
289 pages
ISBN:9781450319034
DOI:10.1145/2488551
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

  • ARCOS: Computer Architecture and Technology Area, Universidad Carlos III de Madrid

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 September 2013

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. databases
  2. high performance computing
  3. scientific simulations
  4. turbulence

Qualifiers

  • Research-article

Funding Sources

Conference

EuroMPI '13
Sponsor:
  • ARCOS
EuroMPI '13: 20th European MPI Users's Group Meeting
September 15 - 18, 2013
Madrid, Spain

Acceptance Rates

EuroMPI '13 Paper Acceptance Rate 22 of 47 submissions, 47%;
Overall Acceptance Rate 66 of 139 submissions, 47%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 12 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2014)ADR visualization: A generalized framework for ranking large-scale scientific data using Analysis-Driven Refinement2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV)10.1109/LDAV.2014.7013203(43-50)Online publication date: Nov-2014

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media