Is cloud storage ready? Performance comparison of representative IP-based storage systems

Highlights

• We conduct a study on three storage systems with realistic conditions.

• We make several interesting observations through a set of experiments.

• We make several recommendations to practitioners for optimizations.

Abstract

Network based storage systems have traditionally been dominated by Network Attached Storage (NAS) and Storage Area Network (SAN). Cloud based storage systems, including object storage, have gained growing popularity among both private and enterprise users in recent years. Certain enterprises have even considered replacing traditional storage systems with cloud-based systems. Nevertheless, there still lacks a systematic comparative study on the performance of the aforementioned systems to assist such a transition. To fill in this gap, in this paper, we conduct a comprehensive study on the three major network storage systems with realistic network conditions and application behaviours. Specifically, we select one representative from each category for comparison, i.e., Network File System (NFS) from NAS, Internet Small Computer System Interface (iSCSI) from SAN, and OpenStack Swift from cloud storage. As the first study of its kind, we mainly focus on the client-side and take performance as the perspective for comparison. We build a testbed and a suite of micro-benchmarks to study the impact of network complexities and access behaviours on performance. In addition, we employ two widely used macro-benchmarks – PostMark and FileBench – to test the three systems under realistic workloads. Through a set of comprehensive experiments and thorough analysis, we make several key observations. (1) iSCSI excels under good network conditions, e.g., in local area networks (LANs); when network complexities like network delay and packet loss exist, its performance degrades significantly, especially for data-intensive operations. (2) For Internet-like environments, NFS performs poorly, while Swift demonstrates much resilience. (3) Overall, Swift is a viable replacement for NFS in all network scenarios, while it is not ready yet to replace iSCSI for performance-critical environments. (4) System configuration on the client side impacts storage performance significantly and deserves adequate attention. Based on our experimental study, we also make several recommendations to practitioners and pinpoint aspects for system designers to improve each storage system further.

Introduction

Enterprise storage systems have traditionally been dominated by two major technologies, i.e., Network Attached Storage (NAS) (Gibson and Van Meter, 2000) and Storage Area Network (SAN) (Thornburgh and Schoenborn, 2000). Both technologies have been widely deployed in enterprise environment in the past decade (Aiken, Grunwald, Pleszkun, Willeke, 2003, Radkov, Yin, Goyal, Sarkar, Shenoy, 2004), and have proven their performance and reliability by time.

In recent years, cloud computing paradigm has gained significant popularity and starts to replace traditional computing models. As a critical component, the emerging cloud storage (e.g., Amazon S3) provides a highly promising solution to enable a transition from dedicated storage to more platform-independent IP-based storage. By using an Hypertext Transfer Protocol (HTTP) (Fielding, 2000) based Representational State Transfer (RESTful) (Fielding et al., 1999) API, auto scaling, and pay-as-you-go model, cloud storage delivers a set of attractive characteristics for enterprise environment, including elasticity, scalability, cross-platform accessibility, and web integrability. With widely available wireless networks, users may access the same set of data on cloud storage anytime and anywhere conveniently. For this reason, many enterprise IT departments are considering to replace traditional network storage services with private or public cloud-based storage services (Systems, Crump). IETF has also formed a working group recently on Internet-based cloud storage services (Cui et al., 2015b). Nevertheless, without a thorough understanding of such a new storage model in comparison with conventional ones, it is challenging to realize the technology transition efficiently and perform optimizations further. There are two major reasons posing the challenge.

First, in today’s enterprise environment, end users usually rely on wireless networks to gain mobility and flexibility. Unfortunately, such a practice makes it particularly challenging to ensure the quality of storage services - signal strength varies at different locations, and the connection can be on and off intermittently. What is worse, users on mobile often need to access storage services through the unpredictable Internet (e.g., work from home or work on travel). These issues together inevitably introduce significant network complexity, and thus pose strong challenge on the user-perceivable storage performance. Second, although NAS, SAN, and cloud storage are all important in practical environment, there still lacks a systematic study on understanding the three drastically different storage systems in a comparative way. Without a thorough understanding on the benefits, limitations, and implications, it is difficult to make an informed decision. Admittedly, as the three systems are designed with different goals in mind, they are likely to show distinct characteristics under different system environments. We believe none of them is universally perfect that can satisfy all sorts of demands. It is thus important to understand their behaviours and relative strengths and weaknesses.

To address the challenge mentioned above, we conduct an experimental study striving to understand the intrinsic characteristics of NAS, SAN, and cloud storage, and investigate their implications in different scenarios. Admittedly, there are many perspectives that can be compared from and deserve investigation, including performance, scalability, and reliability. Nevertheless, as the first attempt to provide such a study, we primarily focus on the performance perspective in this paper, and leave the other perspectives, e.g., scalability and reliability, for future work. We compare the three systems from the client aspect. Namely, we evaluate the three systems from the end user perceivable performance and explore potential approaches to improve them accordingly.

We selectively choose Network File System (NFS) (Shepler et al., 2003), Internet Small Computer System Interface (iSCSI) (Chadalapaka et al., 2004), and OpenStack Swift (OpenStack Foundation, 2015), as the representative of each technology. In order to provide a fair comparison, we run the experiments on the same hardware setup, integrate the same Ext4 file system for each system, and access them all through standard POSIX APIs. To provide a controlled wireless environment and make the experiments reproducible, we use the Wide Area Network emulator (WANem) (Nambiar et al., 2014) to emulate various network scenarios. We design a set of fine-grained experiments to cover different aspects, including microbenchmarks for file access and metadata operations. To compare the three systems under realistic workloads, we also leverage two widely used macrobenchmark tools, i.e., PostMark and FileBench.

With all the efforts mentioned above, we strive to answer the following questions:

• Is cloud storage a viable replacement in scenarios that are traditionally dominated by NAS and SAN?

• Is cloud storage universally better than NAS and SAN? If not, in what scenarios is cloud storage better?

• How much does network conditions and application behaviours impact the performance of each technology?

Through systematic analysis, we make several important observations that assist in answering the questions:

• We find that under ideal network conditions, SAN is several orders of magnitude better than both NAS and cloud storage, while NAS slightly outperforms cloud storage. Under network conditions similar to the Internet, performance of both NAS and SAN declines fast, while cloud storage remains almost unaffected. Specifically, for data-intensive operations, e.g., large file access, SAN performs as badly as NAS; while for lightweight operations like metadata and small file access, the superiority of SAN from internal bundling schemes makes it still keep the outperformance over both NAS and cloud storage.

• We discover that the performance difference between NAS and cloud storage can largely boil down to implementation details, while their differences from SAN arise from a combination of protocol design and implementation details. Specifically, the intrinsic bundling mechanism from SAN (i.e., protocol design) makes it superior in small file and metadata operations; while the capability of utilizing multiple TCP connections (i.e., implementation details) affects performance significantly, especially under realistic network conditions involving nontrivial network delay and packet loss.

• Moreover, we notice that access behaviours (e.g., Direct vs. Sync I/Os) have a remarkable impact on storage performance, and should be chosen carefully based on the tradeoff between performance and consistency. We also find that host system setup, e.g., I/O schedulers and local file system, plays a significant role on the performance of SAN.

• From performance perspective, we conclude that object-based cloud storage is not suitable to replace SAN in performance critical environment, especially for manipulating small files; nevertheless, it is a potential replacement for NAS in nearly all network conditions, especially in Internet-like environment.

Based on the observations, we discuss important system implications to application designers and practitioners, and present our own ideas on how to further enhance each storage system design. As the first comparative study of its kind, we hope this study can shed light on understanding the intrinsic characteristics and system implications of each solution from the client perspective, and assist in identifying the right position of each technology in today’s storage systems.