research-article

Performance and scalability evaluation of the Ceph parallel file system

Authors:

Bradley W. Settlemyer,

Blake Caldwell,

Jason HillAuthors Info & Claims

PDSW '13: Proceedings of the 8th Parallel Data Storage Workshop

Pages 14 - 19

https://doi.org/10.1145/2538542.2538562

Published: 17 November 2013 Publication History

Get Access

Abstract

Ceph is an emerging open-source parallel distributed file and storage system. By design, Ceph leverages unreliable commodity storage and network hardware, and provides reliability and fault-tolerance via controlled object placement and data replication. This paper presents our file and block I/O performance and scalability evaluation of Ceph for scientific high-performance computing (HPC) environments. Our work makes two unique contributions. First, our evaluation is performed under a realistic setup for a large-scale capability HPC environment using a commercial high-end storage system. Second, our path of investigation, tuning efforts, and findings made direct contributions to Ceph's development and improved code quality, scalability, and performance. These changes should benefit both Ceph and the HPC community at large.

References

[1]

IOR HPC benchmark. https://github.com/chaos/ior.

Google Scholar

[2]

D. Karger, E. Lehman, T. Leighton, R. Panigrahy, M. Levine, and D. Lewin. Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web. In Proceedings of the twenty-ninth annual ACM symposium on Theory of computing, pages 654--663. ACM, 1997.

Digital Library

Google Scholar

[3]

J. Schutt. Understanding delays due to throttling under very heavy write load. http://marc.info/?l=ceph-devel&m=133009796706284&w=2, 2012.

Google Scholar

[4]

F. Wang, M. Nelson, S. Oral, D. Fuller, S. Atchley, B. Caldwell, B. Settlemyer, J. Hill, and S. Weil. Ceph parallel file system evaluation report. Technical Report ORNL/TM-2013/151, Oak Ridge National Laboratory, 2013.

Crossref

Google Scholar

[5]

S. A. Weil, S. A. Brandt, E. L. Miller, D. D. E. Long, and C. Maltzahn. Ceph: a scalable, high-performance distributed file system. In Proceedings of the 7th symposium on Operating systems design and implementation, OSDI '06, pages 307--320, Berkeley, CA, USA, 2006. USENIX Association.

Digital Library

Google Scholar

[6]

S. A. Weil, S. A. Brandt, E. L. Miller, and C. Maltzahn. Crush: controlled, scalable, decentralized placement of replicated data. In Proceedings of the 2006 ACM/IEEE conference on Supercomputing, SC '06, New York, NY, USA, 2006. ACM.

Digital Library

Google Scholar

[7]

S. A. Weil, K. T. Pollack, S. A. Brandt, and E. L. Miller. Dynamic metadata management for petabyte-scale file systems. In Proceedings of the 2004 ACM/IEEE conference on Supercomputing, page 4. IEEE Computer Society, 2004.

Digital Library

Google Scholar

Cited By

View all

Chum SPark HChoi J(2021)Supporting SLA via Adaptive Mapping and Heterogeneous Storage Devices in CephElectronics10.3390/electronics1007084710:7(847)Online publication date: 2-Apr-2021
https://doi.org/10.3390/electronics10070847
Wu ZWei JZhang FGuo WXie G(2020)MDLB: a metadata dynamic load balancing mechanism based on reinforcement learningFrontiers of Information Technology & Electronic Engineering10.1631/FITEE.190012121:7(1034-1046)Online publication date: 29-Jul-2020
https://doi.org/10.1631/FITEE.1900121
Chum SChoi JLi JPark H(2020)SLA-Aware Adaptive Mapping Scheme in Bigdata Distributed Storage SystemsThe 9th International Conference on Smart Media and Applications10.1145/3426020.3426053(135-140)Online publication date: 17-Sep-2020
https://dl.acm.org/doi/10.1145/3426020.3426053
Show More Cited By

Recommendations

File system performance and transaction support
File System Performance and Transaction Support
Scalable performance of the Panasas parallel file system
FAST'08: Proceedings of the 6th USENIX Conference on File and Storage Technologies

The Panasas file system uses parallel and redundant access to object storage devices (OSDs), per-file RAID, distributed metadata management, consistent client caching, file locking services, and internal cluster management to provide a scalable, fault ...

Comments

Information & Contributors

Information

Published In

PDSW '13: Proceedings of the 8th Parallel Data Storage Workshop

November 2013

55 pages

ISBN:9781450325059

DOI:10.1145/2538542

Conference Chairs:
Dean Hildebrand
IBM
,
Karsten Schwan
Georgia Tech

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SC13

Sponsor:

SC13: International Conference for High Performance Computing, Networking, Storage and Analysis

November 17 - 21, 2013

Colorado, Denver

Acceptance Rates

PDSW '13 Paper Acceptance Rate 8 of 16 submissions, 50%;

Overall Acceptance Rate 17 of 41 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
893
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)2

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Chum SPark HChoi J(2021)Supporting SLA via Adaptive Mapping and Heterogeneous Storage Devices in CephElectronics10.3390/electronics1007084710:7(847)Online publication date: 2-Apr-2021
https://doi.org/10.3390/electronics10070847
Wu ZWei JZhang FGuo WXie G(2020)MDLB: a metadata dynamic load balancing mechanism based on reinforcement learningFrontiers of Information Technology & Electronic Engineering10.1631/FITEE.190012121:7(1034-1046)Online publication date: 29-Jul-2020
https://doi.org/10.1631/FITEE.1900121
Chum SChoi JLi JPark H(2020)SLA-Aware Adaptive Mapping Scheme in Bigdata Distributed Storage SystemsThe 9th International Conference on Smart Media and Applications10.1145/3426020.3426053(135-140)Online publication date: 17-Sep-2020
https://dl.acm.org/doi/10.1145/3426020.3426053
Kolosov OYadgar GLiram MTamo IBarg A(2020)On Fault Tolerance, Locality, and Optimality in Locally Repairable CodesACM Transactions on Storage10.1145/338183216:2(1-32)Online publication date: 22-May-2020
https://dl.acm.org/doi/10.1145/3381832
Khan AHamandawana PKim Y(2020)A Content Fingerprint-Based Cluster-Wide Inline Deduplication for Shared-Nothing Storage SystemsIEEE Access10.1109/ACCESS.2020.30390568(209163-209180)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3039056
Daoud HDagenais M(2020)Performance analysis of distributed storage clusters based on kernel and userspace tracesSoftware: Practice and Experience10.1002/spe.288951:1(5-24)Online publication date: 7-Sep-2020
https://doi.org/10.1002/spe.2889
Zhang XWang YWang QZhao X(2019)A New Approach to Double I/O Performance for Ceph Distributed File System in Cloud Computing2019 2nd International Conference on Data Intelligence and Security (ICDIS)10.1109/ICDIS.2019.00018(68-75)Online publication date: Jun-2019
https://doi.org/10.1109/ICDIS.2019.00018
Noel RMehra RLama P(2019)Towards Self-Managing Cloud Storage with Reinforcement Learning2019 IEEE International Conference on Cloud Engineering (IC2E)10.1109/IC2E.2019.000-9(34-44)Online publication date: Jun-2019
https://doi.org/10.1109/IC2E.2019.000-9
Song UJeong BPark SLee K(2019)Optimizing communication performance in scale-out storage systemCluster Computing10.1007/s10586-018-2831-622:2(335-346)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1007/s10586-018-2831-6
Sevilla MJimenez IWatkins NLeFevre JAlvaro PFinkelstein SDonnelly PMaltzahn C(2018)Cudele: An API and Framework for Programmable Consistency and Durability in a Global Namespace2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS.2018.00105(960-969)Online publication date: May-2018
https://doi.org/10.1109/IPDPS.2018.00105
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Cited By

Recommendations

File system performance and transaction support

File System Performance and Transaction Support

Scalable performance of the Panasas parallel file system

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Recommendations

File system performance and transaction support

File System Performance and Transaction Support

Scalable performance of the Panasas parallel file system

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations