Skip to main content
SpringerLink
Log in

Global Access to Legacy Data-Sets in Multi-cloud Applications with Onedata

  • Conference paper
  • First Online:
Parallel Processing and Applied Mathematics (PPAM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13826))

Abstract

Data access and management for multi-cloud applications proves challenging where there is a need for efficient access to large pre-existing, legacy data-sets. To address this problem, we created an indexing subsystem incorporated into Onedata data management system achieving a global multi-cloud integration of legacy storage systems containing large data-sets. The solution is based on metadata management, organization and periodic monitoring of legacy data-sets, what makes possible scheme-less co-existence a legacy storage system and Onedata for managing the same data collections. Thanks to block based data transfer provided by Onedata large files stored on legacy storage systems can be efficiently accessed from the cloud. The approach has been initially evaluated in a multi-cloud deployment scenario, where a large data collection stored on legacy storage system located in a super-computing center is processed on a commercial cloud.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amazon: Amazon s3 developer guide. Technical Report, Amazon (2010). http://aws.amazon.com/documentation/s3/

  2. Haustein, N., Christ, A.: IBM SpectrumScale. automation of storage services. https://www.ibm.com/support/pages/system/files/inline-files/Spectrum_Scale_Automation_v1.6.pdf

  3. Kalavade, A.: AWS storage gateway in 2019. https://aws.amazon.com/blogs/storage/aws-storage-gateway-in-2019/. Accessed 3 Feb 2020

  4. Linthicum, D.S.: Emerging hybrid cloud patterns. IEEE Cloud Comput. 3(1), 88–91 (2016). https://doi.org/10.1109/MCC.2016.22

    Article  Google Scholar 

  5. Orzechowski, M., Baliś, B., Słota, R.G., Kitowski, J.: Reproducibility of computational experiments on Kubernetes-managed container clouds with HyperFlow. In: Krzhizhanovskaya, V.V., et al. (eds.) ICCS 2020. LNCS, vol. 12137, pp. 220–233. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50371-0_16

    Chapter  Google Scholar 

  6. Röblitz, T.: Towards implementing virtual data infrastructures - a case study with iRODS. Comput. Sci. (AGH) 13(4), 21–34 (2012)

    Article  Google Scholar 

  7. Schmuck, F.B., Haskin, R.L.: GPFS: a shared-disk file system for large computing clusters. In: D.D.E. Long (ed.) FAST, pp. 231–244. USENIX (2002)

    Google Scholar 

  8. Selvaganesan, M., Liazudeen, M.A.: An insight about glusterfs and its enforcement techniques. In: ICCCRI, pp. 120–127. IEEE Computer Society (2016)

    Google Scholar 

  9. Shepler, S., et al.: NFS version 4 Protocol. RFC 3010 (Proposed Standard) (2000)

    Google Scholar 

  10. The IEEE and the open group: the open group base specifications issue 6 - IEEE Std 1003.1, 2004 Edition. IEEE, New York, NY, USA (2004)

    Google Scholar 

  11. Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: a scalable, high-performance distributed file system. In: Bershad, B.N., Mogul, J.C. (eds.) OSDI, pp. 307–320. USENIX Association (2006)

    Google Scholar 

  12. Wrzeszcz, M., Łukasz Dutka, Słota, R.G., Kitowski, J.: New approach to global data access in computational infrastructures. Future Gener. Comput. Syst. 125, 575–589 (2021). https://doi.org/10.1016/j.future.2021.06.054

  13. Wrzeszcz, M., Kitowski, J., Słota, R.G.: Towards trasparent data access with context awareness. Comput. Sci. 19(2), 201–221 (2018). https://doi.org/10.7494/csci.2018.19.2.2844

    Article  Google Scholar 

  14. Wrzeszcz, M., et al.: Consistency models for global scalable data access services. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds.) PPAM 2017. LNCS, vol. 10777, pp. 471–480. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-78024-5_41

    Chapter  Google Scholar 

Download references

Acknowledgements

This scientific work is co-financed in part by an international project and by the program of the Minister of Science and Higher Education entitled “PMW” in the years 2020–2023; contract No. 5145/H2020/2020/2; contract No. 5193/H2020/2021/22 and by The National Centre for Research and Development under the program entitled ERA-NET CO-FUND ICT-AGRI-FOOD, contract No. ICTAGRIFOOD/I/FINDR/ 02/2022. JK and RGS are grateful for support from the subvention of Polish Ministry of Education and Science assigned to AGH University of Science and Technology.

Author information

Authors and Affiliations

  1. Academic Computer Centre CYFRONET AGH, Krakow, Poland

    Michał Orzechowski, Michał Wrzeszcz, Bartosz Kryza, Łukasz Dutka & Jacek Kitowski

  2. Institute of Computer Science, AGH University of Science and Technology, Krakow, Poland

    Renata G. Słota & Jacek Kitowski

Authors
  1. Michał Orzechowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michał Wrzeszcz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bartosz Kryza
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Łukasz Dutka
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Renata G. Słota
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jacek Kitowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michał Orzechowski .

Editor information

Editors and Affiliations

  1. Czestochowa University of Technology, Czestochowa, Poland

    Roman Wyrzykowski

  2. University of Tennessee, Knoxville, TN, USA

    Jack Dongarra

  3. University of Southern California, Marina del Rey, CA, USA

    Ewa Deelman

  4. Czestochowa University of Technology, Czestochowa, Poland

    Konrad Karczewski

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Orzechowski, M., Wrzeszcz, M., Kryza, B., Dutka, Ł., Słota, R.G., Kitowski, J. (2023). Global Access to Legacy Data-Sets in Multi-cloud Applications with Onedata. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds) Parallel Processing and Applied Mathematics. PPAM 2022. Lecture Notes in Computer Science, vol 13826. Springer, Cham. https://doi.org/10.1007/978-3-031-30442-2_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-30442-2_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-30441-5

  • Online ISBN: 978-3-031-30442-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation