skip to main content
10.1145/3232565.3234504acmotherconferencesArticle/Chapter ViewAbstractPublication PagescommConference Proceedingsconference-collections
research-article

Using the Macroflow Abstraction to Minimize Machine Slot-time Spent on Networking in Hadoop

Published: 01 August 2018 Publication History

Abstract

Machine slot-time spent on data transmission has direct impact on average job completion time (JCT). In this paper, we propose Macroflow, a networking abstraction that can capture the primitive scheduling granularity of machine slot-time. We demonstrate that minimizing machine slot-time is equivalent to minimizing the average macroflow completion time (MCT). We prove that minimizing MCT to be strongly NP-hard and focus on developing effective heuristics. We propose the Smallest-Macroflow-First (SMF) and Smallest-Average-Macroflow-First (SAMF) heuristics that greedily schedule macroflows based on their network footprint. To work with existing commodity switches, priority discretization is performed to classify macroflows into a small number of priority queues.

References

[1]
Mohammad Alizadeh, Albert Greenberg, David A Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. 2011. Data center tcp (dctcp). In in Proc. ACM SIGCOMM 2011, Vol. 41. ACM, 63--74.
[2]
Mohammad Alizadeh, Shuang Yang, Milad Sharif, Sachin Katti, Nick McKeown, Balaji Prabhakar, and Scott Shenker. 2013. pfabric: Minimal near-optimal datacenter transport. In ACM SIGCOMM 2013, Vol. 43. ACM, 435--446.
[3]
Wei Bai, Li Chen, Kai Chen, Dongsu Han, Chen Tian, and Hao Wang. 2015. Information-Agnostic Flow Scheduling for Commodity Data Centers. In NSDI. USENIX.
[4]
Mosharaf Chowdhury and Ion Stoica. 2015. Efficient coflow scheduling without prior knowledge. In ACM SIGCOMM Computer Communication Review, Vol. 45. ACM, 393--406.
[5]
Mosharaf Chowdhury, Matei Zaharia, Justin Ma, Michael I Jordan, and Ion Stoica. 2011. Managing data transfers in computer clusters with orchestra. In ACM SIGCOMM, Vol. 41. ACM, 98--109.
[6]
Mosharaf Chowdhury, Yuan Zhong, and Ion Stoica. 2014. Efficient coflow scheduling with Varys. In ACM SIGCOMM. ACM, 443--454.
[7]
Fahad R Dogar, Thomas Karagiannis, Hitesh Ballani, and Antony Rowstron. 2014. Decentralized task-aware scheduling for data center networks. In ACM SIGCOMM Computer Communication Review, Vol. 44. ACM, 431--442.
[8]
Chi-Yao Hong, Matthew Caesar, and P Godfrey. 2012. Finishing flows quickly with preemptive scheduling. In Proc. ACM SIGCOMM 2012, Vol. 42. ACM, 127--138.
[9]
Joseph Y-T Leung, Haibing Li, and Michael Pinedo. 2007. Scheduling orders for multiple product types to minimize total weighted completion time. Discrete Applied Mathematics 155, 8 (2007), 945--970.
[10]
Ali Munir, Ghufran Baig, Syed M Irteza, Ihsan A Qazi, Alex X Liu, and Fahad R Dogar. 2014. Friends, not foes: synthesizing existing transport strategies for data center networks. In ACM SIGCOMM 2014. ACM, 491--502.
[11]
Ali Munir, Ihsan Ayyub Qazi, Zartash Afzal Uzmi, Aisha Mushtaq, Saad N Ismail, M Safdar Iqbal, and Basma Khan. 2013. Minimizing flow completion times in data centers. In in Proc. IEEE INFOCOM, 2013. IEEE, 2157--2165.
[12]
Thomas A Roemer. 2006. A note on the complexity of the concurrent open shop problem. Journal of scheduling 9, 4 (2006), 389--396.
[13]
David Zats, Tathagata Das, Prashanth Mohan, Dhruba Borthakur, and Randy Katz. 2012. DeTail: reducing the flow completion time tail in datacenter networks. In ACM SIGCOMM, Vol. 42. ACM, 139--150.

Cited By

View all
  • (2022)PushBox: Making Use of Every Bit of Time to Accelerate Completion of Data-Parallel JobsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.318203733:12(4256-4269)Online publication date: 1-Dec-2022
  • (2020)Towards High-Efficiency Data Centers via Job-Aware Network SchedulingProceedings of the 49th International Conference on Parallel Processing10.1145/3404397.3404474(1-10)Online publication date: 17-Aug-2020
  • (2020)Supporting Multi-dimensional and Arbitrary Numbers of Ranks for Software Packet Scheduling2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS)10.1109/IWQoS49365.2020.9212844(1-10)Online publication date: Jun-2020
  • Show More Cited By

Index Terms

  1. Using the Macroflow Abstraction to Minimize Machine Slot-time Spent on Networking in Hadoop

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    APNet '18: Proceedings of the 2nd Asia-Pacific Workshop on Networking
    August 2018
    78 pages
    ISBN:9781450363952
    DOI:10.1145/3232565
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 01 August 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Macroflow
    2. datacenter networking
    3. network scheduling

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    • National Science and Technology Major Project of China
    • National Natural Science Foundation of China
    • Fundamental Research Funds for the Central Universities

    Conference

    APNet '18

    Acceptance Rates

    Overall Acceptance Rate 50 of 118 submissions, 42%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)1
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2022)PushBox: Making Use of Every Bit of Time to Accelerate Completion of Data-Parallel JobsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.318203733:12(4256-4269)Online publication date: 1-Dec-2022
    • (2020)Towards High-Efficiency Data Centers via Job-Aware Network SchedulingProceedings of the 49th International Conference on Parallel Processing10.1145/3404397.3404474(1-10)Online publication date: 17-Aug-2020
    • (2020)Supporting Multi-dimensional and Arbitrary Numbers of Ranks for Software Packet Scheduling2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS)10.1109/IWQoS49365.2020.9212844(1-10)Online publication date: Jun-2020
    • (2019)RDMA Load Balancing via Data Partition2019 28th International Conference on Computer Communication and Networks (ICCCN)10.1109/ICCCN.2019.8847077(1-8)Online publication date: Jul-2019
    • (2019)Error Recovery of RDMA Packets in Data Center Networks2019 28th International Conference on Computer Communication and Networks (ICCCN)10.1109/ICCCN.2019.8846946(1-8)Online publication date: Jul-2019
    • (2019)Metaflow: A Better Traffic Abstraction for Distributed Applications2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS)10.1109/HPCC/SmartCity/DSS.2019.00159(1123-1130)Online publication date: Aug-2019
    • (2019)Application-Oriented Network Scheduling With MetaflowIEEE Access10.1109/ACCESS.2019.29577657(175531-175541)Online publication date: 2019

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media