Skip to main content
SpringerLink
Log in

DMetabench—a metadata benchmark for distributed file systems

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

The performance of metadata processing in large distributed file systems currently presents larger challenges than scaling of data throughput. The paper presents a novel, distributed benchmark called DMetabench for measuring the performance of metadata operations. DMetabench runs in environments with potentially thousands of nodes and allows an assessment of the scalability of metadata operations. Additionally, precise run-time performance data is preserved which allows for a better understanding of performance artifacts. Example results from production file systems are provided and discussed. Possible applications of knowledge about metadata performance scaling include the choice of an optimal parallelization strategy for metadata-intensive workloads in a specific runtime environment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Callaghan B, Pawlowski B, Staubach P (1995) NFS version 3 protocol specification. http://www.ietf.org/rfc/rfc1813.txt

  2. Campbell R (1998) Managing AFS: the Andrew File System. Prentice-Hall, New York

    Google Scholar 

  3. Shepard L, Eppe E (2006) SGI InfiniteStorage shares filesystem CXFS: a high-performance, multi-OS filesystem from SGI. Technical report, Silicon Graphics

  4. Cluster File Systems, Inc. (2007) Lustre 1.6 operations manual

  5. The Open Group (2004) The single UNIX specification, version 3. Technical report

  6. Eisler M, Corbett P, Kazar M, Nydick DS, Wagner JC (2007) Data ONTAP GX: a scalable storage cluster. In: Proceedings of FAST’ 07

  7. Hitz D, Lau J, Malcolm M File system design for an NFS file server appliance. Technical report, Network Appliance (TR 3002)

  8. Sweeney A, Doucette D, Hu W, Anderson C, Nishimoto M, Peck G (1996) Scalability in the XFS File System. In: Proceedings of the USENIX 1996 Technical Conference, San Diego, CA, USA (22–26 1996) 1–14

  9. Norcott WD, Capps D (2006) Iozone filesystem benchmark. http://www.iozone.org/

  10. Intel Corporation (1998) Iometer. http://www.iometer.org/

  11. Rabenseifner R, Koniges AE, Prost JP, Hedges R (2001) The parallel effective I/O bandwidth benchmark: b eff io. Technical report, High-Performance Computing Center (HLRS)

  12. Howard J, Kazar M, Menees S, Nichols D, Satyanarayanan M, Sidebotham R, West M (1988) Scale and performance in a distributed file system. ACM Trans Comput Syst 6:51–81

    Article  Google Scholar 

  13. Katcher J (1997) PostMark: A new file system benchmark. Technical report 3022, Network Appliance

  14. SPEC (2008) SPECsfs 2008 user’s guide. Technical report version 1.0, Standard Performance Evaluation Corporation (SPEC)

  15. McDougall R, Mauro J (2006) Filebench tutorial. http://www.solarisinternals.com/si/tools/filebench

Download references

Author information

Authors and Affiliations

  1. Leibniz-Rechenzentrum (LRZ) der Bayerischen Akademie der Wissenschaften, Boltzmannstr. 1, 85748, Garching, Germany

    Christoph Biardzki

  2. German Climate Computing Centre (DKRZ), Bundesstrasse 55, 20146, Hamburg, Germany

    Thomas Ludwig

Authors
  1. Christoph Biardzki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Ludwig
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christoph Biardzki.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Biardzki, C., Ludwig, T. DMetabench—a metadata benchmark for distributed file systems. J Supercomput 57, 179–188 (2011). https://doi.org/10.1007/s11227-010-0389-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-010-0389-z

Keywords

Search

Navigation