Skip to main content
SpringerLink
Log in

Hint Controlled Distribution with Parallel File Systems

  • Conference paper
Recent Advances in Parallel Virtual Machine and Message Passing Interface (EuroPVM/MPI 2005)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 3666))

  • 810 Accesses

Abstract

The performance of scientific parallel programs with high file-I/O-activity running on top of cluster computers strongly depends on the qualitative and quantitative characteristics of the requested I/O-accesses. It also depends on the corresponding mechanisms and policies being used at the parallel file system level. This paper presents the motivation and design of a set of MPI-IO-hints. These hints are used to select the distribution function with which a parallel file system manipulates an opened file. The implementation of a new physical distribution function called varstrip_dist is also presented in this article. This function is proposed based upon spatial characteristics presented by I/O-access patterns observed at the application level.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Patterson, D.A., Chen, P.M.: Storage Performance - Metrics and Benchmarks (1998), http://citeseer.ist.psu.edu/91919.html

  2. Patterson, D.A., Chen, P.M.: Maximizing Performance in a Striped Disk Array. In: Proc. 17th Annual Symposium on Computer Architecture (17th ISCA 1990), Computer Architecture News, pp. 322–331 (1990)

    Google Scholar 

  3. Hsu, W.W., Smith, A.J.: Characteristics of I/O traffic in personal computer and server workloads. IBM Syst. J. 42, 347–372 (2003)

    Article  Google Scholar 

  4. Hsu, W.W., Smith, A.J.: The performance impact of I/O optimizations and disk improvements. IBM Journal of Research and Development 48, 255–289 (2004)

    Article  Google Scholar 

  5. Sterling, T.: An Overview of Cluster Computing. Beowulf Cluster Computing with Linux., pp. 15–29 (2002)

    Google Scholar 

  6. PVFS2, http://www.pvfs.org/pvfs2/

  7. ROMIO, http://www-unix.mcs.anl.gov/romio/

  8. Ligon, W.B., Ross, R.B.: Implementation and Performance of a Parallel File System for High Performance Distributed Applications. In: Proceedings of the Fifth IEEE International Symposium on High Performance Distributed Computing, pp. 471–480 (1996)

    Google Scholar 

  9. Ross, R.B., Carns, P.H., Ligon III, W.B., Latham, R.: Using the Parallel Virtual File System (2002), http://www.parl.clemson.edu/pvfs/user-guide.html

  10. Madhyastha, T.M.: Automatic Classification of Input/Output Access Patterns. PhD Thesis (1997)

    Google Scholar 

  11. Madhyastha, T.M., Reed, D.A.: Exploiting Global Input/Output Access Pattern Classification. In: Proceedings of SC 1997: High Performance Networking and Computing (1997)

    Google Scholar 

  12. Thakur, R., Gropp, W., Lusk, E.: On implementing MPI-IO portably and with high performance. In: Proceedings of the 6th Workshop on I/O in Parallel and Distributed Systems (IOPADS 1999), pp. 23–32 (1999)

    Google Scholar 

  13. Thakur, R.S., Gropp, W., Lusk, E.: A Case for ung MPI’s derived datatypes to improve I/O Performance. In: Proceedings of Supercomputing 1998, CD-ROM (1998)

    Google Scholar 

  14. Rabenseifner, R., Koniges, A.E., Prost, J.-P., Hedges, R.: The Parallel Effective I/O Bandwidth Benchmark: b_eff_io. Parallel I/O for Cluster Computing, 107–132 (2004)

    Google Scholar 

  15. Miller, E.L., Katz, R.H.: Input/output behavior of supercomputing applications. SC, 567–576 (1991)

    Google Scholar 

  16. MPI-2, http://www.mpi-forum.org

  17. Gropp, W., Lusk, E., Rajeev, T.: Using MPI-2: Advanced Features of the Message-Passing Interface, pp. 15–16 (1999)

    Google Scholar 

  18. Patterson, D., Gibson, G., Randy, K.: A case for redundant arrays of inexpensive disks (RAID). In: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 109–116 (1988)

    Google Scholar 

  19. PVFS Development Team: PVFS 2 Concepts: the new guy’s guide to PVFS. PVFS 2 Documentation (2004)

    Google Scholar 

  20. PVFS Development Team: PVFS 2 Distribution Design Notes. PVFS 2 Documentation (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Parallele und Verteilte Systeme, Institut für Informatik, Ruprecht-Karls-Universität Heidelberg, 69120, Heidelberg, Germany

    Hipolito Vasquez Lucas & Thomas Ludwig

Authors
  1. Hipolito Vasquez Lucas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Ludwig
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Dipartimento di Ingegneria dell’ Informazione, Second University of Naples - Italy, Real Casa dell’Annunziata - via Roma, 29, 81031, Aversa, CE, Italy

    Beniamino Di Martino

  2. GUP, Institute of Graphics and Parallel Processing, Johannes Kepler University, Altenbergerstraße 69, A-4040, Linz, Austria

    Dieter Kranzlmüller

  3. Computer Science Department, University of Tennessee, 37996-3450, Knoxville, TN, USA

    Jack Dongarra

Rights and permissions

Reprints and permissions

Copyright information

© 2005 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lucas, H.V., Ludwig, T. (2005). Hint Controlled Distribution with Parallel File Systems. In: Di Martino, B., Kranzlmüller, D., Dongarra, J. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2005. Lecture Notes in Computer Science, vol 3666. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11557265_18

Download citation

  • DOI: https://doi.org/10.1007/11557265_18

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-29009-4

  • Online ISBN: 978-3-540-31943-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation