Balaji Palanisamy
Ling Liu
ABSTRACT
We present Purlieus, a MapReduce resource allocation system aimed at enhancing the performance of MapReduce jobs in the cloud. Purlieus provisions virtual MapReduce clusters in a locality-aware manner enabling MapReduce virtual machines (VMs) access to input data and importantly, intermediate data from local or close-by physical machines. We demonstrate how this locality-awareness during both map and reduce phases of the job not only improves runtime performance of individual jobs but also has an additional advantage of reducing network traffic generated in the cloud data center. This is accomplished using a novel coupling of, otherwise independent, data and VM placement steps. We conduct a detailed evaluation of Purlieus and demonstrate significant savings in network traffic and almost 50% reduction in job execution times for a variety of workloads.
- B. Igou "User Survey Analysis: Cloud-Computing Budgets Are Growing and Shifting; Traditional IT Services Providers Must Prepare or Perish". Gartner Report, 2010Google Scholar
- http://en.wikipedia.org/wiki/Loop_deviceGoogle Scholar
- J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In OSDI, 2004. Google ScholarDigital Library
- G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha and E. Harris. Reining in the Outliers inMap-Reduce Clusters using Mantri. In OSDI, 2010. Google ScholarDigital Library
- http://en.wikipedia.org/wiki/Big-dataGoogle Scholar
- S. Babu. Towards Automatic Optimization of MapReduce Programs. In SOCC, 2010. Google ScholarDigital Library
- http://en.wikipedia.org/wiki/ClickstreamGoogle Scholar
- K. Kambatla, A. Pathak and H. Pucha. Towards Optimizing Hadoop Provisioning in the Cloud. In HotCloud, 2009. Google ScholarDigital Library
- Cloudera. http://www.cloudera.com/blog/2010/08/hadoop-for-fraud-detection-and-prevention/Google Scholar
- K. Morton, A. Friesen, M. Balazinska, D. Grossman. Estimating the Progress of MapReduce Pipelines. In ICDE, 2010.Google ScholarCross Ref
- Hadoop DFS User Guide. http://hadoop.apache.org/.Google Scholar
- T. Wood, P. Shenoy, A. Venkataramani and M. Yousif. Black-box and Gray-box Strategies for Virtual Machine Migration. In NSDI, 2007. Google ScholarDigital Library
- Y. Chen, R. Griffith, J. Liu, R. H. Katz and A. D. Joseph. Understanding TCP Incast Throughput Collapse in Datacenter Networks. In WREN, 2009. Google ScholarDigital Library
- Amazon Elastic MapReduce. http://aws.amazon.com/elasticmapreduce/Google Scholar
- Amazon Elastic Compute Cloud. http://aws.amazon.com/ec2/Google Scholar
- Amazon Simple Storage Service. http://aws.amazon.com/s3/Google Scholar
- T. Gunarathne, T. Wu, J. Qiu, G. Fox MapReduce in the Clouds for Science. In CloudCom, 2010. Google ScholarDigital Library
- M. Cardosa, P. Narang, A. Chandra, H. Pucha and A. Singh. STEAMEngine: Optimizing MapReduce provisioning in the cloud. Dept. of CSE, Univ. of Minnesota, 2010.Google Scholar
- M. Al-Fares, A. Loukissas and A. Vahdat. A scalable, commodity data center network architecture. In SIGCOMM, 2008. Google ScholarDigital Library
- C. Guo, H. Wu, K. Tan, L. Shiy, Y. Zhang, S. Luz. DCell: A Scalable and Fault-Tolerant Network Structure for Data Centers. In SIGCOMM, 2008. Google ScholarDigital Library
- A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. A. Maltz, P. Patel, S. Sengupta. VL2: A Scalable and Flexible Data Center Network. In SIGCOMM, 2009. Google ScholarDigital Library
- M. Al-Fares, S. Radhakrishnan, B. Raghavan, N. Huang, A. Vahdat. Hedera: Dynamic Flow Scheduling for Data Center Networks. In NSDI, 2010. Google ScholarDigital Library
- Hadoop. http://hadoop.apache.org.Google Scholar
- M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, I. Stoica. Improving MapReduce Performance in Heterogeneous Environments. In OSDI, 2008. Google ScholarDigital Library
- M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: fair scheduling for distributed computing clusters. In SOSP, 2009. Google ScholarDigital Library
- G. Wang, A. Butt, P. Pandey, K. Gupta. A Simulation Approach to Evaluating Design Decisions in MapReduce Setups. MASCOTS, 2009.Google Scholar
- R. J. Mokken. Cliques, clubs and clans. In Quality and Quantity, 1973.Google Scholar
- M. R. Garey, D. S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman. ISBN 0-7167-1045-5. Google ScholarDigital Library
- M. A. Kozuch, M. P. Ryan, R. Gass et al. Tashi: Location-aware Cluster Management. In ACDC, 2009. Google ScholarDigital Library
- K. Kambatla, A. Pathak, and H. Pucha. Towards optimizing hadoop provisioning in the cloud. In HotCloud, 2009. Google ScholarDigital Library
- H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, S. Babu Starfish: A Selftuning System for Big Data Analytics. In CIDR, 2011.Google Scholar
- G. Khanna, K. Beaty, G. Kar, and A. Kochut. Application performance management in virtualized server environments. In NOMS, 2006.Google Scholar
- T. Sandholm and K. Lai. Mapreduce optimization using dynamic regulated prioritization. In ACM SIGMETRICS/Performance, 2009. Google ScholarDigital Library
- Scheduling in hadoop. http://www.cloudera.com/blog/tag/scheduling/.Google Scholar
- A. Singh, M. Korupolu, and D. Mohapatra. Server-storage virtualization: Integration and load balancing in data centers. In IEEE/ACM Supercomputing, 2008. Google ScholarDigital Library
- A. Verma, P. Ahuja, and A. Neogi. pMapper: Power and Migration Cost Aware Placement of Applications in Virtualized Systems. In ACM Middleware, 2008. Google ScholarDigital Library
- A. Phanishayee, H. Shah, E. Krevat, D. Andersen, G. Ganger, G. Gibson, B. Mueller, V. Vasudevan. Safe and Effective Fine-grained TCP Retransmissions for Datacenter Communication. In SIGCOMM 2009. Google ScholarDigital Library
- G. Lee, N. Tolia, P. Ranganathan, R. Katz. Topology-Aware Resource Allocation for Data-intensive workloads. In APSys, 2010. Google ScholarDigital Library
Recommendations
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16: Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution EnvironmentsSingle-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
Comments