research-article

Optimizing fastquery performance on lustre file system

Authors:

Kesheng WuAuthors Info & Claims

SSDBM '13: Proceedings of the 25th International Conference on Scientific and Statistical Database Management

Article No.: 29, Pages 1 - 12

https://doi.org/10.1145/2484838.2484853

Published: 29 July 2013 Publication History

Abstract

FastQuery is a parallel indexing and querying system we developed for accelerating analysis and visualization of scientific data. We have applied it to a wide variety of HPC applications and demonstrated its capability and scalability using a petascale trillion-particle simulation in our previous work. Yet, through our experience, we found that performance of reading and writing data with FastQuery, like many other HPC applications, could be significantly affected by various tunable parameters throughout the parallel I/O stack. In this paper, we describe our success in tuning the performance of FastQuery on a Lustre parallel file system. We study and analyze the impact of parameters and tunable settings at file system, MPI-IO library, and HDF5 library levels of the I/O stack. We demonstrate that a combined optimization strategy is able to improve performance and I/O bandwidth of FastQuery significantly. In our tests with a trillion-particle dataset, the time to index the dataset reduced by more than one half.

References

[1]

IPCC Fifth Assessment Report. http://en.wikipedia.org/wiki/IPCC_Fifth_Assessment_Report.

[2]

B. Behzad, J. Huchette, H. Luu, R. Aydt, S. Byna, Y. Yao, Q. Koziol, and Prabhat. A framework for auto-tuning hdf5 applications. In HPDC, 2013. https://sdm.lbl.gov/~sbyna/research/papers/hpdc2013.pdf.

Digital Library

[3]

K. J. Bowers, B. J. Albright, L. Yin, B. Bergen, and T. J. T. Kwan. Ultrahigh performance three-dimensional electromagnetic relativistic kinetic plasma simulation. Physics of Plasmas, 15(5):7, 2008.

[4]

S. Byna, J. Chou, O. Rübel, Prabhat, H. Karimabadi, W. S. Daughton, V. Roytershteyn, E. W. Bethel, M. Howison, K.-J. Hsu, K.-W. Lin, A. Shoshani, A. Uselton, and K. Wu. Parallel i/o, analysis, and visualization of a trillion particle simulation. In SC, pages 59:1--59:12, 2012.

Digital Library

[5]

Y. Chen, M. Winslett, Y. Cho, and S. Kuo. Automatic parallel I/O performance optimization using genetic algorithms. In HPDC, pages 155--162, jul 1998.

Digital Library

[6]

Y. Chen, M. Winslett, Y. Cho, S. Kuo, and C. Y. Chen. Automatic Parallel I/O Performance Optimization in Panda. In In Proceedings of the 10th Annual ACM Symposium on Parallel Algorithms and Architectures, pages 108--118, 1998.

Digital Library

[7]

Y. Chen, M. Winslett, S.-w. Kuo, Y. Cho, M. Subramaniam, and K. Seamons. Performance modeling for the panda array I/O library. In SC, 1996.

Digital Library

[8]

J. Chou, K. Wu, and Prabhat. FastQuery: A general indexing and querying system for scientific data. In SSDBM, pages 573--574, 2011.

Digital Library

[9]

J. Chou, K. Wu, and Prabhat. FastQuery: A parallel indexing system for scientific data. In Proceedings of Workshop on Interfaces and Abstractions for Scientific Data Storage, 2011.

Digital Library

[10]

J. Chou, K. Wu, O. Rübel, M. Howison, J. Qiang, Prabhat, B. Austin, E. W. Bethel, R. D. Ryne, and A. Shoshani. Parallel index and query for large scale data analysis. In SC, pages 30:1--30:11, 2011.

Digital Library

[11]

D. Comer. The ubiquitous b-tree. ACM Comput. Surv., 11(2):121--137, June 1979.

Digital Library

[12]

Cray. Getting Started on MPI I/O, Dec. 2009. CrayDoc S-2490-40.

[13]

C. M. Herb Wartens, Jim Garlick. LMT - The Lustre Monitoring Tool. https://github.com/chaos/lmt/wiki.

[14]

T. Hey, S. Tansley, and K. Tolle, editors. The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft, Oct. 2009.

[15]

M. Howison, Q. Koziol, D. Knaak, J. Mainzer, and J. Shalf. Tuning HDF5 for Lustre File Systems. In Proceedings of Workshop on Interfaces and Abstractions for Scientific Data Storage, Heraklion, Crete, Greece, Sept. 2010. LBNL-4803E.

[16]

M. Howison, Q. Koziol, D. Knaak, J. Mainzer, and J. Shalf. Tuning HDF5 for Lustre File Systems. In Proceedings of Workshop on Interfaces and Abstractions for Scientific Data Storage, Heraklion, Crete, Greece, Sept. 2010. LBNL-4803E.

[17]

J. Kim, H. Abbasi, L. Chacón, C. Docan, S. Klasky, Q. Liu, N. Podhorszki, A. Shoshani, and K. Wu. Parallel in situ indexing for data-intensive computing. In LDAV, pages 65--72. IEEE, 2011.

[18]

J. Li, W. keng Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale. Parallel netCDF: A high-performance scientific I/O interface. In SC, page 39, 2003.

Digital Library

[19]

W.-k. Liao and A. Choudhary. Dynamically adapting file domain partitioning methods for collective i/o based on underlying parallel file system locking protocols. In SC, pages 3:1--3:12, 2008.

Digital Library

[20]

J. F. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, and C. Jin. Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS). In CLADE, pages 15--24, 2008.

Digital Library

[21]

J. Mache, V. Lo, and S. Garg. The impact of spatial layout of jobs on I/O hotspots in mesh networks. JPDC, 65(10):1190--1203, Oct. 2005.

Digital Library

[22]

P. O'Neil and E. O'Neil. Database: principles, programming, and performance. Morgan Kaugmann, 2nd edition, 2000.

Digital Library

[23]

P. E. O'Neil. Model 204 architecture and performance. In Proceedings of the 2nd International Workshop on High Performance Transaction Systems, pages 40--59, London, UK, UK, 1989. Springer-Verlag.

Digital Library

[24]

A. Shoshani and D. e. Rotem. Scientific Data Management: Challenges, Technology, and Deployment. Chapman & Hall/CRC Press, 2009.

Digital Library

[25]

K. Stockinger, J. Shalf, W. Bethel, and K. Wu. Query-driven visualization of large data sets. In IEEE Visualization, pages 167--174, Oct. 2005.

[26]

The HDF Group. HDF5 user guide. http://hdf.ncsa.uiuc.edu/HDF5/doc/H5.user.html.

[27]

Unidata. The NetCDF users' guide. http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/.

[28]

VisIt Visualization Tool. https://wci.llnl.gov/codes/visit/.

[29]

K. Wu. FastBit: an efficient indexing technology for accelerating data-intensive science. Journal of Physics: Conference Series, 16:556--560, 2005.

[30]

K. Wu, E. Otoo, and A. Shoshani. Optimizing bitmap indices with efficient compression. ACM Transactions on Database Systems, 31:1--38, 2006.

Digital Library

[31]

W. Yu, J. Vetter, and H. Oral. Performance characterization and optimization of parallel I/O on the Cray XT. In IPDPS, pages 1--11, Apr. 2008.

Cited By

Williams JMedeiros DCostea STskhakaya DPoeschel FWidera RHuebl AKlasky SPodhorszki NKos LPodolnik AHromadka JNarwal TSteiniger KBussmann MLaure EMarkidis S(2024)Enabling High- Throughput Parallel I/O in Particle-in-Cell Monte Carlo Simulations with openPMD and Darshan I/O Monitoring2024 IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops)10.1109/CLUSTERWorkshops61563.2024.00022(86-95)Online publication date: 24-Sep-2024
https://doi.org/10.1109/CLUSTERWorkshops61563.2024.00022
Hu JChe XKan BShao Y(2024)LS-HTC: an HTC system for large-scale jobsCCF Transactions on High Performance Computing10.1007/s42514-024-00183-16:3(301-318)Online publication date: 11-Mar-2024
https://doi.org/10.1007/s42514-024-00183-1
Khoshlessan MParaskevakos IFox GJha SBeckstein O(2020)Parallel performance of molecular dynamics trajectory analysisConcurrency and Computation: Practice and Experience10.1002/cpe.578932:19Online publication date: 27-Apr-2020
https://doi.org/10.1002/cpe.5789
Show More Cited By

Index Terms

Optimizing fastquery performance on lustre file system

Recommendations

File system performance and transaction support
File System Performance and Transaction Support
Evaluation of a Performance Model of Lustre File System
CHINAGRID '10: Proceedings of the The Fifth Annual ChinaGrid Conference

As a large-scale global parallel file system, Lustre file system plays a key role in High Performance Computing (HPC) system, and the potential performance of such systems can be difficult to predict because the potential impact to application ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SSDBM '13: Proceedings of the 25th International Conference on Scientific and Statistical Database Management

July 2013

401 pages

ISBN:9781450319218

DOI:10.1145/2484838

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 July 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

U.S. Department of Energy

Conference

SSDBM '13

SSDBM '13: Conference on Scientific and Statistical Database Management

July 29 - 31, 2013

Maryland, Baltimore, USA

Acceptance Rates

Overall Acceptance Rate 56 of 146 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
142
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Williams JMedeiros DCostea STskhakaya DPoeschel FWidera RHuebl AKlasky SPodhorszki NKos LPodolnik AHromadka JNarwal TSteiniger KBussmann MLaure EMarkidis S(2024)Enabling High- Throughput Parallel I/O in Particle-in-Cell Monte Carlo Simulations with openPMD and Darshan I/O Monitoring2024 IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops)10.1109/CLUSTERWorkshops61563.2024.00022(86-95)Online publication date: 24-Sep-2024
https://doi.org/10.1109/CLUSTERWorkshops61563.2024.00022
Hu JChe XKan BShao Y(2024)LS-HTC: an HTC system for large-scale jobsCCF Transactions on High Performance Computing10.1007/s42514-024-00183-16:3(301-318)Online publication date: 11-Mar-2024
https://doi.org/10.1007/s42514-024-00183-1
Khoshlessan MParaskevakos IFox GJha SBeckstein O(2020)Parallel performance of molecular dynamics trajectory analysisConcurrency and Computation: Practice and Experience10.1002/cpe.578932:19Online publication date: 27-Apr-2020
https://doi.org/10.1002/cpe.5789
Jackson AWeiland MParsons MHomölle B(2019)An Architecture for High Performance Computing and Data Systems Using Byte-Addressable Persistent MemoryHigh Performance Computing10.1007/978-3-030-34356-9_21(258-274)Online publication date: 16-Jun-2019
https://dl.acm.org/doi/10.1007/978-3-030-34356-9_21
Yildiz BWu KByna SShoshani A(2019)Parallel membership queries on very large scientific data sets using bitmap indexesConcurrency and Computation: Practice and Experience10.1002/cpe.515731:15Online publication date: 28-Jan-2019
https://doi.org/10.1002/cpe.5157

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten