WebGlusterFS: A web-based administration tool for GlusterFS with resource assignment for various storage demands

Highlights

• A web-based administration tools is designed and implemented to ease the management of GlusterFS.

• A framework is setup to automatically assign resource according to various workload demands.

• Two algorithms are integrated into the tools to allocate the resource to build volumes with lowest cost for two types of workloads.

Abstract

Facing the complex tasks involves making decisions about assignment of workloads to storage backends as well as dynamic and timely adjustment according to the storage demands in Cloud and Big-data environment, an administration tool for GlusterFS, WebGlusterFS, is presented in this article to ease the management and help to assign the storage resource. WebGlusterFS is a web-based tool designed to substitute the command line console manager of GlusterFS and provides an interface for auto-assignment module to build volumes from heterogeneous backend devices. A simple demo module is also implemented to show how various storage demands are fulfilled by building the volumes from properly matched storage resource with minimum cost. The characteristics of underlying storage resource are obtained by benchmarking and used to make the assignment decision. WebGlusterFS setups a base framework for workload aware storage platform for large scale computing environments.

Introduction

With the rapid development of information technology, enormou volumes of data or called “big data” is being generated by many enterprises at all time. Scientific applications, weather forecasting, researches, hospitals, gene information processing and military services are few such major contributors. The need to provide efficient, easy to use solutions has become one of the main issues for these types of computations. The prosperous solution to this issue is the use of Distributed File Systems (DFS) or cloud storage.

There are a number of opensource solutions like GlusterFS, Ceph [3], HDFS [4], Swift [5] and vendor specific solutions like EMC ViPR [6], NetApp ONTAP [7], IBM Virtual Storage [8], etc. All these DFSs are qualified to the basic requirements of cloud storage. The most typical feature of cloud storage involves SAAS (storage as a service) [2], which makes the providing of storage as a service to the users. So, the system need to provide administrators and users with convenient operating and maintaining environment. Besides the easy to manage, there is a need to auto-configure and reconfigure the storage resource easily to provide the storage serve to meet the demand of on-going applications.

Storage administration and data management are challenging and expensive tasks [16], particularly in cloud and Big Data environments where resources are shared among multiple workloads and accessed by various patterns [9,10]. One major and complex portion of these tasks involves making decisions about assignment of workloads to storage backend as well as dynamic and timely adjustment according to changing demands in cloud environments [11]. The assignment decision can be made based on the detailed description of workload characteristics and the performance factors of the underlying storage resource. There are all beyond the scope of these DFSs. So, we need an enhancement to cooperate the workload classification, backend characterization and the resource assignment of the exist DFSs to handle the variety of storage demands from Big-data applications.

In this article, a web-based management tool for GlusterFS is designed and implemented to help managing the storage resource and provide the possibility to make the resource decision by considering the workload requirements and backend characterization. GlusterFS is a scalable open source parallel file system that offers a global namespace, distributed front end, and capable of scaling to hundreds of petabytes without difficulty. It also offers extraordinary cost advantages benefits that are unmatched in the industry. The advantages that make it an ideal cloud storage are as follows [1]: 1) High scalable. Cloud storage can be expanded according to the demand of applications. 2) High reliable and available. Cloud storage will automatically back up the data to software/hardware failures and capable of disaster recovery. 3) Resource controllable. It is capable of controlling the access permissions of resources. 4) High utilization. Cloud storage can consolidate all the storage resources and provide a unified access interface to users. 5) Cost effective. The use of cloud storage can dramatically reduce the cost to running a data center for enterprises, and reduce the need for removable storage devices and efficiently lower the cost of individual users and businesses. GlusterFS has indeed a large user base both in HPC computing farms, and in several Cloud computing facilities. It supports access to storage both in terms of POSIX file-system and via a REST gateway for object storage support. It is worth simplifying the management and enable the workload awareness.

The rest of the paper is organized as follows. Section 2 presents the background of the key challenges and the system architecture of GlusterFS. Section 3 gives the details of designing and implementing the web-based management tool of GlusterFS to ease the management. A scheme is figured out to provide an interface to create volumes based on the workload classification and the performance parameters. Section 4 details two algorithms to assign resource with lowest cost. Section 5 shows some the prototype of this system and the experiment results. We conclude in Section 6.