short-paper

BAShuffler: Maximizing Network Bandwidth Utilization in the Shuffle of YARN

Authors:

Feng Liang,

Francis C.M. LauAuthors Info & Claims

HPDC '16: Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing

Pages 281 - 284

https://doi.org/10.1145/2907294.2907296

Published: 31 May 2016 Publication History

Get Access

Abstract

YARN is a popular cluster resource management platform. It does not, however, manage the network bandwidth resources which can significantly affect the execution performance of those tasks having large volumes of data to transfer within the cluster. The shuffle phase of MapReduce jobs features many such tasks. The impact of under utilization of the network bandwidth in shuffle tasks is more pronounced if the network bandwidth capacities of the nodes in the cluster are varied.

We present BAShuffler, a bandwidth-aware shuffle scheduler, that can maximize the overall network bandwidth utilization by scheduling the source nodes of the fetch flows at the application level. BAShuffler can fully utilize the network bandwidth capacity in a max-min fair network. The experimental results for a variety of realistic benchmarks show that BAShuffler can substantially improve the cluster's shuffle throughput and reduce the execution time of shuffle tasks as compared to the original YARN, especially in heterogeneous network bandwidth environments.

References

[1]

F. Ahmad, S. Lee, M. Thottethodi, and T. Vijaykumar. Puma: Purdue mapreduce benchmarks suite. 2012.

Google Scholar

[2]

M. Alizadeh, T. Edsall, S. Dharmapurikar, R. Vaidyanathan, K. Chu, A. Fingerhut, F. Matus, R. Pan, N. Yadav, G. Varghese, et al. Conga: Distributed congestion-aware load balancing for datacenters. In SIGCOMM, 2014.

Digital Library

Google Scholar

[3]

M. Alizadeh, S. Yang, M. Sharif, S. Katti, N. McKeown, B. Prabhakar, and S. Shenker. pfabric: Minimal near-optimal datacenter transport. SIGCOMM, 2013.

Digital Library

Google Scholar

[4]

D.-M. Chiu and R. Jain. Analysis of the increase and decrease algorithms for congestion avoidance in computer networks. Computer Networks and ISDN systems, 17(1):1--14, 1989.

Digital Library

Google Scholar

[5]

M. Chowdhury, M. Zaharia, J. Ma, M. I. Jordan, and I. Stoica. Managing data transfers in computer clusters with orchestra. In SIGCOMM, 2011.

Digital Library

Google Scholar

[6]

M. Chowdhury, Y. Zhong, and I. Stoica. Efficient coflow scheduling with varys. In SIGCOMM, 2014.

Digital Library

Google Scholar

[7]

J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. Communications of the ACM, 2008.

Digital Library

Google Scholar

[8]

A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. A. Maltz, P. Patel, and S. Sengupta. Vl2: a scalable and flexible data center network. In SIGCOMM, 2009.

Digital Library

Google Scholar

[9]

V. Jacobson. Congestion avoidance and control. In SIGCOMM, 1988.

Digital Library

Google Scholar

[10]

F. Liang and F. C. M. Lau. Bashuffler: Maximizing network bandwidth utilization in the shuffle of yarn. http://i.cs.hku.hk/%7Efliang/paper/BAShuffler.pdf.

Google Scholar

[11]

R. Niranjan Mysore, A. Pamboris, N. Farrington, N. Huang, P. Miri, S. Radhakrishnan, V. Subramanya, and A. Vahdat. Portland: a scalable fault-tolerant layer 2 data center network fabric. In SIGCOMM, 2009.

Digital Library

Google Scholar

[12]

A. Shieh, S. Kandula, A. Greenberg, C. Kim, and B. Saha. Sharing the data center network. In NSDI, 2011.

Digital Library

Google Scholar

[13]

V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, et al. Apache hadoop yarn: Yet another resource negotiator. In SOCC, 2013.

Digital Library

Google Scholar

[14]

M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI, 2012.

Digital Library

Google Scholar

Cited By

View all

Wu SChen HJin HIbrahim S(2022)Shadow: Exploiting the Power of Choice for Efficient Shuffling in MapReduceIEEE Transactions on Big Data10.1109/TBDATA.2019.29434738:1(253-267)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TBDATA.2019.2943473
Biswas DNeuwirth SPaul AButt A(2021)Bridging Network and Parallel I/O Research for Improving Data-Intensive Distributed Applications2021 IEEE Workshop on Innovating the Network for Data-Intensive Science (INDIS)10.1109/INDIS54524.2021.00011(50-56)Online publication date: Nov-2021
https://doi.org/10.1109/INDIS54524.2021.00011
Xu JWang JQi QLiao JSun HHan ZLi T(2021)Network-aware task selection to reduce multi-application makespan in cloudJournal of Network and Computer Applications10.1016/j.jnca.2020.102889176:COnline publication date: 15-Feb-2021
https://dl.acm.org/doi/10.1016/j.jnca.2020.102889
Show More Cited By

Index Terms

BAShuffler: Maximizing Network Bandwidth Utilization in the Shuffle of YARN
1. Computer systems organization
  1. Architectures
    1. Distributed architectures

Recommendations

Provision of Disk I/O Guarantee for MapReduce Applications
TRUSTCOM-BIGDATASE-ISPA '15: Proceedings of the 2015 IEEE Trustcom/BigDataSE/ISPA - Volume 02

YARN is an emerging resource management in the Hadoop ecosystem, where big data in the scale of petabytes/day are processed with the use of commercial off-the-shelf servers. At present YARN supports only RAM and CPU reservation/control. However, the ...
Provision of Disk I/O Guarantee for MapReduce Applications
TRUSTCOM-BIGDATASE-ISPA '15: Proceedings of the 2015 IEEE Trustcom/BigDataSE/ISPA - Volume 02

YARN is an emerging resource management in the Hadoop ecosystem, where big data in the scale of petabytes/day are processed with the use of commercial off-the-shelf servers. At present YARN supports only RAM and CPU reservation/control. However, the ...
Multi-policy-aware MapReduce resource allocation and scheduling for smart computing cluster

When a user submit a MapReduce job in the smart computing cluster, we first need to allocate cluster resource for the job. It is widely concerned that how to save time and resource costs to provide users with computing capacity and services. Here, we ...

Comments

Information & Contributors

Information

Published In

HPDC '16: Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing

May 2016

302 pages

ISBN:9781450343145

DOI:10.1145/2907294

General Chair:
Hiroshi Nakashima
Kyoto University, Japan
,
Program Chairs:
Kenjiro Taura
The University of Tokyo, Japan
,
Jack Lange
University of Pittsburgh, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Hong Kong Research Grants Council

Conference

HPDC'16

Sponsor:

University of Arizona
SIGARCH

HPDC'16: The 25th International Symposium on High-Performance Parallel and Distributed Computing

May 31 - June 4, 2016

Kyoto, Japan

Acceptance Rates

HPDC '16 Paper Acceptance Rate 20 of 129 submissions, 16%;

Overall Acceptance Rate 166 of 966 submissions, 17%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
153
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wu SChen HJin HIbrahim S(2022)Shadow: Exploiting the Power of Choice for Efficient Shuffling in MapReduceIEEE Transactions on Big Data10.1109/TBDATA.2019.29434738:1(253-267)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TBDATA.2019.2943473
Biswas DNeuwirth SPaul AButt A(2021)Bridging Network and Parallel I/O Research for Improving Data-Intensive Distributed Applications2021 IEEE Workshop on Innovating the Network for Data-Intensive Science (INDIS)10.1109/INDIS54524.2021.00011(50-56)Online publication date: Nov-2021
https://doi.org/10.1109/INDIS54524.2021.00011
Xu JWang JQi QLiao JSun HHan ZLi T(2021)Network-aware task selection to reduce multi-application makespan in cloudJournal of Network and Computer Applications10.1016/j.jnca.2020.102889176:COnline publication date: 15-Feb-2021
https://dl.acm.org/doi/10.1016/j.jnca.2020.102889
Jeyaraj RAnanthanarayana VPaul A(2020)Fine-grained data-locality aware MapReduce job scheduler in a virtualized environmentJournal of Ambient Intelligence and Humanized Computing10.1007/s12652-020-01707-7Online publication date: 22-Jan-2020
https://doi.org/10.1007/s12652-020-01707-7
Jeyaraj RS AHaddad HWainwright RChbeir R(2018)Multi-level per node combiner (MLPNC) to minimize mapreduce job latency on virtualized environmentProceedings of the 33rd Annual ACM Symposium on Applied Computing10.1145/3167132.3167149(167-174)Online publication date: 9-Apr-2018
https://dl.acm.org/doi/10.1145/3167132.3167149
Xu JWang JQi QLiao J(2018)Proportion Scheduler to Improve the Mismatched Locality in YARN2018 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications (ISPA/IUCC/BDCloud/SocialCom/SustainCom)10.1109/BDCloud.2018.00068(399-406)Online publication date: Dec-2018
https://doi.org/10.1109/BDCloud.2018.00068
Wu SChen HLin CJin H(2017)Shadow: Exploiting the Power of Choice for Efficient Shuffling in MapReduce2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS.2017.00078(553-560)Online publication date: Dec-2017
https://doi.org/10.1109/ICPADS.2017.00078

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Provision of Disk I/O Guarantee for MapReduce Applications

Provision of Disk I/O Guarantee for MapReduce Applications

Multi-policy-aware MapReduce resource allocation and scheduling for smart computing cluster

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations