SHadoop: Improving MapReduce performance by optimizing job execution mechanism in Hadoop clusters

Highlights

• Analyzed and identified two critical limitations of MapReduce execution mechanism.

• Achieved first optimization by implementing new job setup/cleanup tasks.

• Replaced heartbeat with an instant messaging mechanism to speedup task scheduling.

• Conducted comprehensive benchmarks to evaluate stable performance improvements.

• Passed a production test and integrated our work into Intel Distributed Hadoop.

Abstract

As a widely-used parallel computing framework for big data processing today, the Hadoop MapReduce framework puts more emphasis on high-throughput of data than on low-latency of job execution. However, today more and more big data applications developed with MapReduce require quick response time. As a result, improving the performance of MapReduce jobs, especially for short jobs, is of great significance in practice and has attracted more and more attentions from both academia and industry. A lot of efforts have been made to improve the performance of Hadoop from job scheduling or job parameter optimization level. In this paper, we explore an approach to improve the performance of the Hadoop MapReduce framework by optimizing the job and task execution mechanism. First of all, by analyzing the job and task execution mechanism in MapReduce framework we reveal two critical limitations to job execution performance. Then we propose two major optimizations to the MapReduce job and task execution mechanisms: first, we optimize the setup and cleanup tasks of a MapReduce job to reduce the time cost during the initialization and termination stages of the job; second, instead of adopting the loose heartbeat-based communication mechanism to transmit all messages between the JobTracker and TaskTrackers, we introduce an instant messaging communication mechanism for accelerating performance-sensitive task scheduling and execution. Finally, we implement SHadoop, an optimized and fully compatible version of Hadoop that aims at shortening the execution time cost of MapReduce jobs, especially for short jobs. Experimental results show that compared to the standard Hadoop, SHadoop can achieve stable performance improvement by around 25% on average for comprehensive benchmarks without losing scalability and speedup. Our optimization work has passed a production-level test in Intel and has been integrated into the Intel Distributed Hadoop (IDH). To the best of our knowledge, this work is the first effort that explores on optimizing the execution mechanism inside map/reduce tasks of a job. The advantage is that it can complement job scheduling optimizations to further improve the job execution performance.

Introduction

The MapReduce parallel computing framework  [7], proposed by Google in 2004, has become an effective and attractive solution for big data processing problems. Through simple programming interfaces with two functions, map and reduce, MapReduce significantly simplifies the design and implementation of many data-intensive applications in the real world. Moreover, MapReduce offers other benefits, including load balancing, elastic scalability, and fault tolerance, which makes it a widely adopted parallel computing framework. Hadoop  [1], an open-source implementation of MapReduce, has been widely used in industry and researched in academia.

Both the MapReduce frameworks of Google and Hadoop have been widely recognized for their high-throughput, elastic scalability, and fault tolerance. They focus more on these features than job execution efficiency. This results in relatively poor performance when using Hadoop MapReduce to execute jobs, especially short jobs. The term ‘short job’ has already been used in some related work  [30], [31]. There is no quantitative definition for short jobs now. Usually they refer to MapReduce jobs with execution time ranging from seconds to a few minutes as opposed to long MapReduce jobs that take hours. Facebook names this type jobs as ‘small job’ in its recently released optimized version of Hadoop, Corona  [24].

Some studies show that short jobs compose a large portion of MapReduce jobs  [7], [5]. For example, the average execution time of MapReduce jobs at Google in September 2007 is 395 s  [7]. Response time is most important for short jobs in scenarios where users need the answer quickly, such as query or analysis on log data for debugging, monitoring and business intelligence  [30]. In a pay-by-the-time environment like EC2, improving MapReduce performance means saving monetary costs. Optimizing MapReduce’s execution time can also prevent jobs from occupying system resources too long, which is good for a cluster’s health  [31].

Today there are a number of high-level query and data-analysis systems that provide services on top of MapReduce, such as Google’s Sawzall  [17], Facebook’s Hive  [22] and Yahoo!’s Pig  [16]. These systems execute users’ requests by converting SQL-like queries to a series of MapReduce jobs that are usually short. These high-level declarative languages can greatly simplify the task of developing applications in MapReduce without hand-coded MapReduce programs  [21]. Thus, in practice, these systems play more important roles than hand-coded MapReduce programs. For example, more than 95% Hadoop jobs in Facebook are not hand-coded but generated by Hive and more than 90% MapReduce jobs in Yahoo! are generated by Pig  [12]. In fact, these systems are very sensitive to the execution time of underlying short MapReduce jobs. Therefore, reducing the execution time of MapReduce jobs is very important to these widely-used systems.

For the above reasons, in this paper we concentrate on improving the execution performance of short MapReduce jobs. Having studied the Hadoop MapReduce framework in great detail, we focus on the internal execution mechanisms of an individual job and the tasks inside a job. Through in-depth analysis, we reveal that there are two critical issues that limit the performance of MapReduce jobs. To address these issues, we design and implement SHadoop, an optimized version of Hadoop that is fully compatible with the standard Hadoop. Different from improving the performance on job scheduling or job parameter optimization level, we optimize the underlying execution mechanism of each of tasks inside a job. In implementation, first, we optimize the setup and cleanup tasks, two special tasks when executing a MapReduce job, to reduce the time cost during the initialization and termination stages of the job; second, we add an instant messaging communication mechanism into the standard Hadoop for the fast delivery of the performance-sensitive task scheduling and execution messages between the JobTracker and TaskTrackers. This way the tasks of a job can be scheduled and executed instantly without heartbeat delay. As a consequence, the job execution process becomes more compact and utilization of the slots on the TaskTrackers can be much improved. Experimental results show that SHadoop outperforms the standard Hadoop and can achieve stable performance improvements of around 25% on average for comprehensive benchmarks. Our optimization work has passed a production-level test in Intel and been integrated into the Intel Distributed Hadoop  [11]. To the best of our knowledge, this work is the first effort that explores on optimizing the execution mechanism inside map/reduce tasks. The advantage is that it can complement job scheduling optimization work to further improve the job execution performance.

The rest of this paper is organized as follows: Section  2 introduces the related works on MapReduce performance optimization and comparisons them with SHadoop. Section  3 focuses on analyzing the job/tasks execution mechanism in standard Hadoop MapReduce. Based on this, Section  4 describes our optimization methods for improving job execution efficiency in the standard Hadoop MapReduce. Section  5 discusses experiments and performance evaluations of our optimization work. Finally, we conclude this paper in Section  6.