Greyfish: An Out-of-the-Box, Reusable, Portable Cloud Storage Service
Article No.: 13, Pages 1 - 6
Abstract
A scalable storage system is an integral requirement for supporting large-scale cloud computing jobs. The raw space on storage systems is made usable with the help of a software layer which is typically called a filesystem (e.g., Google's Cloud Filestore). In this paper, we present the design and implementation of an open-source and free cloud-based filesystem named as "Greyfish" that can be installed on the Virtual Machines (VMs) hosted on different cloud computing systems, such as Jetstream and Chameleon. Greyfish helps in: (1) storing files and directories for different user-accounts in a shared space on the cloud, (2) managing file-access permissions, and (3) purging files when needed. It is currently being used in the implementation of the Gateway-In-A-Box (GIB) project. A simplified version of Greyfish, known as Reef, is already in production in the BOINC@TACC project. Science gateway developers will find Greyfish useful for creating local filesystems that can be mounted in containers. By doing so, they can independently do quick installations of self-contained software solutions in development and test environments while mounting the filesystems on large-scale storage platforms in the production environments only.
References
[1]
Cloud Filestore documentation | Cloud Filestore Documentation | Google Cloud. Retrieved on 2019-04-15 from https://cloud.google.com/filestore/docs/.
[2]
Use Volumes | Docker Documentation. Retrieved on 2019-04-15 from https://docs.docker.com/storage/volumes/.
[3]
What is a Container? | Docker. Retrieved on 2019-04-15 from https://www.docker.com/resources/what-container.
[4]
Gateway-In-A-Box. Retrieved on 2019-04-15 from https://github.com/ritua2/gib.
[5]
Ritu Arora, Carlos Redondo, and Gerald Joshua. Scalable Software Infrastructure for Integrating Supercomputing with Volunteer Computing and Cloud Computing. In Majumdar A. and Arora R., editors, Software Challenges to Exascale Computing. SCEC 2018. Communications in Computer and Information Science, volume 964. Springer, Singapore, 2019.
[6]
Using InfluxDB in Grafana | Grafana Documentation. Retrieved on 2019-04-15 from https://docs.grafana.org/features/datasources/influxdb/.
[7]
Greyfish, Portable Cloud Storage. Retrieved on 2019-04-15 from https://github.com/noderod/greyfish.
[8]
Gunicorn - WSGI server -- Gunicorn 19.9.0 documentation. Retrieved on 2019-04-15 from https://docs.gunicorn.org/en/stable/.
[9]
Redis. Retrieved on 2019-04-15 from https://redis.io/documentation.
[10]
InfluxDB 1.7 documentation | InfluxData Documentation. Retrieved on 2019-04-15 from https://docs.influxdata.com/influxdb/v1.7/.
[11]
Overview of Docker Compose | Docker Documentation. Retrieved on 2019-04-15 from https://docs.docker.com/compose/overview/.
[12]
C.A. Stewart, T.M. Cockerill, I. Foster, D. Hancock, N. Merchant, E. Skidmore, D. Stanzione, J. Taylor, S. Tuecke, G. Turner, M. Vaughn, and N.I. Gaffney. Jetstream: a self-provisioned, scalable science and engineering cloud environment. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure, 2015. ACM: 2792774. p. 1--8.
[13]
John Towns, Timothy Cockerill, Maytal Dahan, Ian Foster, Kelly Gaither, Andrew Grimshaw, Victor Hazlewood, Scott Lathrop, Dave Lifka, Gregory D. Peterson, Ralph Roskies, J. Ray Scott, and Nancy Wilkins-Diehr. XSEDE: Accelerating Scientific Discovery. Computing in Science Engineering, 16(5):62--74, 2014.
[14]
What is Amazon S3? - Amazon Simple Storage Service. Retrieved on 2019-04-15 from https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html.
[15]
Dropbox. Retrieved on 2019-04-15 from https://www.dropbox.com.
[16]
Using Google Drive-New Features, Benefits & Advantages of Google Cloud Storage. Retrieved on 2019-04-15 from https://www.google.com/drive/using-drive/.
[17]
Kate Keahey, Pierre Riteau, Dan Stanzione, Tim Cockerill, Joe Mambretti, Paul Rad, and Paul Ruth. Chameleon: a Scalable Production Testbed for Computer Science Research. In Jeffrey Vetter, editor, Contemporary High Performance Computing: From Petascale toward Exascale, volume 3 of Chapman Hall/CRC Computational Science, chapter 5. CRC Press, Boca Raton, FL, 1 edition, 2018.
Recommendations
The design and implementation of an extensible network backup system in realtime
ICUIMC '09: Proceedings of the 3rd International Conference on Ubiquitous Information Management and CommunicationThis paper proposes a backup system based on mirroring filesystem "GMFS." GMFS has been developed to mirror data in realtime on the filesystem layer. The GMFS is a stackable filesystem which flexibly mirrors without changing the existing environment by ...
hashFS: Applying Hashing to Optimize File Systems for Small File Reads
SNAPI '10: Proceedings of the 2010 International Workshop on Storage Network Architecture and Parallel I/OsToday’s file systems typically need multiple disk accesses for a single read operation of a file. In the worst case, when none of the needed data is already in the cache, the metadata for each component of the file path has to be read in. Once the ...
A study of practical deduplication
We collected file system content data from 857 desktop computers at Microsoft over a span of 4 weeks. We analyzed the data to determine the relative efficacy of data deduplication, particularly considering whole-file versus block-level elimination of ...
Comments
Information & Contributors
Information
Published In
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 July 2019
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
PEARC '19
PEARC '19: Practice and Experience in Advanced Research Computing
July 28 - August 1, 2019
IL, Chicago, USA
Acceptance Rates
Overall Acceptance Rate 133 of 202 submissions, 66%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 75Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in