Skip to main content
SpringerLink
Log in
Book cover

International Conference on High Performance Computing

ISC High Performance 2017: High Performance Computing pp 712–723Cite as

JULEA: A Flexible Storage Framework for HPC

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10524))

Abstract

JULEA is a flexible storage framework that allows offering arbitrary client interfaces to applications. To be able to rapidly prototype new approaches, it offers data and metadata backends that can either be client-side or server-side; backends for popular storage technologies such as POSIX, LevelDB and MongoDB have already been implemented. Additionally, JULEA allows dynamically adapting the I/O operations’ semantics and can thus be adjusted to different use-cases. It runs completely in user space, which eases development and debugging. Its goal is to provide a solid foundation for storage research and teaching.

This is a preview of subscription content, log in via an institution.

Notes

  1. 1.

    https://github.com/wr-hamburg/julea.

  2. 2.

    The network’s round-trip time is 0.110ms, which results in a maximum of 9,090 ops/s.

  3. 3.

    The existing backends are between 200 and 400 lines of code each.

References

  1. Al-Kiswany, S., Gharaibeh, A., Ripeanu, M.: The case for a versatile storage system. Operating Syst. Rev. 44(1), 10–14 (2010). http://doi.acm.org/10.1145/1740390.1740394

    Article  Google Scholar 

  2. Albadri, N., Watson, R., Dekeyser, S.: TreeTags: bringing tags to the hierarchical file system. In: Proceedings of the Australasian Computer Science Week Multiconference, Canberra, Australia, 2–5 February 2016, p. 21 (2016). http://doi.acm.org/10.1145/2843043.2843868

  3. BigStorage: Storage-Based Convergence Between HPC and Cloud to Handle Big Data (2017). http://bigstorage-project.eu/. Accessed Mar 2017

  4. Carns, P.H., Ligon III, W.B., Ross, R.B., Thakur, R.: PVFS: A parallel file system for linux clusters. In: 4th Annual Linux Showcase and Conference, Atlanta, Georgia, USA, 10–14 October 2000. https://www.usenix.org/conference/als-2000/pvfs-parallel-file-system-linux-clusters

  5. ESiWACE: Centre of Excellence in Simulation of Weather and Climate in Europe (2017). https://www.esiwace.eu/. Accessed Mar 2017

  6. GLib: GLib Reference Manual (2017). https://developer.gnome.org/glib/. Accessed Mar 2017

  7. Gu, P., Wang, J., Ross, R.: Bridging the gap between parallel file systems and local file systems: A case study with PVFS. In: 2008 International Conference on Parallel Processing, ICPP 2008, 8–12 September 2008, Portland, Oregon, USA, pp. 554–561 (2008). http://dx.doi.org/10.1109/ICp.2008.43

  8. Kuhn, M.: A semantics-aware I/O interface for high performance computing. In: Kunkel, J.M., Ludwig, T., Meuer, H.W. (eds.) ISC 2013. LNCS, vol. 7905, pp. 408–421. Springer, Heidelberg (2013). doi:10.1007/978-3-642-38750-0_31

    Chapter  Google Scholar 

  9. Kuhn, M.: Dynamically adaptable I/O semantics for high performance computing. In: Kunkel, J.M., Ludwig, T. (eds.) ISC High Performance 2015. LNCS, vol. 9137, pp. 240–256. Springer, Cham (2015). doi:10.1007/978-3-319-20119-1_18

    Chapter  Google Scholar 

  10. Kuhn, M.: Dynamically adaptable I/O semantics for high performance computing. Ph.D. Thesis, Universität Hamburg (2015). http://ediss.sub.uni-hamburg.de/volltexte/2015/7302/

  11. Kuhn, M., Kunkel, J.M., Ludwig, T.: Dynamic file system semantics to enable metadata optimizations in PVFS. Concurrency Comput. Pract. Experience 21(14), 1775–1788 (2009). http://dx.doi.org/10.1002/cpe.1439

    Article  Google Scholar 

  12. Lofstead, J.F., Jimenez, I., Maltzahn, C., Koziol, Q., Bent, J., Barton, E.: DAOS and friends: a proposal for an exascale storage system. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2016, Salt Lake City, UT, USA, 13–18 November 2016, pp. 585–596 (2016). http://dx.doi.org/10.1109/SC.2016.49

  13. Matri, P., Costan, A., Antoniu, G., Montes, J., Pérez, M.S.: Týr: blob storage meets built-in transactions. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2016, Salt Lake City, UT, USA, 13–18 November 2016, pp. 49:1–49:12 (2016). http://dl.acm.org/citation.cfm?id=3014970

  14. MongoDB, I.: Libbson: A BSON utility library (2017). https://github.com/mongodb/libbson. Accessed Mar 2017

  15. MongoDB, I.: Libmongoc: A high-performance MongoDB driver for C (2017). https://github.com/mongodb/mongo-c-driver. Accessed Mar 2017

  16. Schröder, S.: Design, Implementation, and Evaluation of a Low-Level Extent-Based Object Store. Master’s Thesis, Universität Hamburg (2013)

    Google Scholar 

  17. Seltzer, M.I., Murphy, N.: Hierarchical file systems are dead. In: Proceedings of HotOS 2009: 12th Workshop on Hot Topics in Operating Systems, 18–20 May 2009, Monte Verità, Switzerland (2009). http://www.usenix.org/events/hotos09/tech/full_papers/seltzer/seltzer.pdf

  18. Stender, J., Kolbeck, B., Hupfeld, F., Cesario, E., Focht, E., Hess, M., Malo, J., Martí, J.: Striping without sacrifices: Maintaining POSIX semantics in a parallel file system. In: Proceedings of the First USENIX Workshop on Large-Scale Computing, LASCO 2008, 23 June 2008, Boston, MA, USA (2008). http://www.usenix.org/events/wiov08/tech/full_papers/stender/stender.pdf

  19. Vilayannur, M., Nath, P., Sivasubramaniam, A.: Providing tunable consistency for a parallel file store. In: Proceedings of the FAST 2005 Conference on File and Storage Technologies, 13–16 December 2005, San Francisco, California, USA (2005). http://www.usenix.org/events/fast05/tech/vilayannur.html

  20. Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: A scalable, high-performance distributed file system. In: 7th Symposium on Operating Systems Design and Implementation (OSDI 2006), 6–8 November 2006, Seattle, WA, USA, pp. 307–320 (2006). http://www.usenix.org/events/osdi06/tech/weil.html

  21. Wright, C.P., Spillane, R.P., Sivathanu, G., Zadok, E.: Extending ACID semantics to the file system. TOS 3(2), 4:1–4:42 (2007). http://doi.acm.org/10.1145/1242520.1242521

Download references

Author information

Authors and Affiliations

  1. Universität Hamburg, 20146, Hamburg, Germany

    Michael Kuhn

Authors
  1. Michael Kuhn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Kuhn .

Editor information

Editors and Affiliations

  1. Deutsches Klimarechenzentrum (DKRZ), Hamburg, Hamburg, Germany

    Julian M. Kunkel

  2. TITECH, Tokyo, Japan

    Rio Yokota

  3. Department of Computer Science, University of Delaware, Newark, Delaware, USA

    Michela Taufer

  4. Lawrence Berkeley National Laboratory, Berkeley, California, USA

    John Shalf

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Kuhn, M. (2017). JULEA: A Flexible Storage Framework for HPC. In: Kunkel, J., Yokota, R., Taufer, M., Shalf, J. (eds) High Performance Computing. ISC High Performance 2017. Lecture Notes in Computer Science(), vol 10524. Springer, Cham. https://doi.org/10.1007/978-3-319-67630-2_51

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-67630-2_51

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-67629-6

  • Online ISBN: 978-3-319-67630-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation