skip to main content
10.1145/3469393.3469396acmotherconferencesArticle/Chapter ViewAbstractPublication PagescommConference Proceedingsconference-collections
research-article

HierCC: Hierarchical RDMA Congestion Control

Published: 01 February 2022 Publication History

Abstract

RDMA has been increasingly deployed in data centers to decrease latency and CPU utilization. However, existing RDMA congestion control schemes fail to address instantaneous large queue build-up or bandwidth under-utilization associated with frequent traffic bursty. In this paper, we argue that traffic uncertainty is the essential reason that constrains data center congestion control from simultaneously achieving high throughput and deterministic latency. Since aggregated flows within the same rack are relatively long-lived, we propose HierCC, which aggregates flows destined to the same IP in a rack and hierarchically controls the rate of flows. The rate of aggregate flows between racks is controlled by a credit-based congestion control mechanism. Then the bandwidth obtained by an aggregate flow in a rack is allocated to the corresponding individual flows from that rack promptly and accurately. We evaluate HierCC using SystemC and large-scale NS3 simulations. Results indicate that HierCC can significantly mitigate buffer usage and reduce the 99th percentile FCT by up to 20% and 40% compared with HPCC and DCQCN under a realistic workload, respectively.

References

[1]
[n.d.]. Arista 7500R Switch Architecture. https://www.arista.com/assets/data/pdf/Whitepapers/Arista7500RSwitchArchitectureWP.pdf.
[2]
[n.d.]. Cisco Silicon One. https://www.cisco.com/c/en/us/solutions/silicon-one.html.
[3]
[n.d.]. Fibre Channel Industry Association. https://fibrechannel.org/.
[4]
2018. BROADCOM. BCM88690, StrataDNX 10 Tb/s Scalable Switching Device, Product Brief. https://www.arista.com/assets/data/pdf/Whitepapers/Arista7500RSwitchArchitectureWP.pdf.
[5]
Hari Balakrishnan, Hariharan S Rahul, and Srinivasan Seshan. 1999. An integrated congestion management architecture for Internet hosts. In Proceedings of the conference on Applications, technologies, architectures, and protocols for computer communication. 175–187.
[6]
Frank Cangialosi, Akshay Narayan, Prateesh Goyal, Radhika Mittal, Mohammad Alizadeh, and Hari Balakrishnan. 2020. Site-to-Site Internet Traffic Control. arXiv preprint arXiv:2011.01258(2020).
[7]
Inho Cho, Keon Jang, and Dongsu Han. 2017. Credit-scheduled delay-bounded congestion control for datacenters. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication. 239–252.
[8]
Nandita Dukkipati. 2006. Rcp: Congestion control to make flows complete quickly. Ph.D. Dissertation. PhD Thesis, Department of Electrical Engineering, Stanford University.
[9]
Peter X Gao, Akshay Narayan, Sagar Karandikar, Joao Carreira, Sangjin Han, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker. 2016. Network requirements for resource disaggregation. In USENIX Symposium on Operating Systems Design and Implementation (OSDI). 249–264.
[10]
Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn. 2016. RDMA over commodity ethernet at scale. In Proceedings of the 2016 ACM SIGCOMM Conference. 202–215.
[11]
Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu, Andrew W Moore, Gianni Antichi, and Marcin Wójcik. 2017. Re-architecting datacenter networks and stacks for low latency and high performance. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication. 29–42.
[12]
Shuihai Hu, Wei Bai, Gaoxiong Zeng, Zilong Wang, Baochen Qiao, Kai Chen, Kun Tan, and Yi Wang. 2020. Aeolus: A building block for proactive transport in datacenters. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 422–434.
[13]
Joseph Izraelevitz, Jian Yang, Lu Zhang, Juno Kim, Xiao Liu, Amirsaman Memaripour, Yun Joon Soh, Zixuan Wang, Yi Xu, Subramanya R Dulloor, 2019. Basic performance measurements of the intel optane DC persistent memory module. arXiv preprint arXiv:1903.05714(2019).
[14]
Dina Katabi, Mark Handley, and Charlie Rohrs. 2002. Congestion control for high bandwidth-delay product networks. In Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications. 89–102.
[15]
Ana Klimovic, Christos Kozyrakis, Eno Thereska, Binu John, and Sanjeev Kumar. 2016. Flash Storage Disaggregation. In EuroSys.
[16]
Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, and et al.2019. HPCC: High Precision Congestion Control. In ACM SIGCOMM (Beijing, China). https://doi.org/10.1145/3341302.3342085
[17]
Radhika Mittal, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, David Zats, 2015. TIMELY: RTT-based Congestion Control for the Datacenter. In ACM SIGCOMM.
[18]
Behnam Montazeri, Yilong Li, Mohammad Alizadeh, and John Ousterhout. 2018. Homa: A receiver-driven low-latency transport protocol using network priorities. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. 221–235.
[19]
Jonathan Perry, Amy Ousterhout, Hari Balakrishnan, Devavrat Shah, and Hans Fugal. 2014. Fastpass: a centralized” zero-queue” datacenter network. In Proceedings of the 2014 ACM conference on SIGCOMM. 307–318.
[20]
Sundararajan Renganathan, Venkata N Padmanabhan, and Akshay Uttama Nambi. 2018. Rethinking Networking for” Five Computers”. In Proceedings of the 17th ACM Workshop on Hot Topics in Networks. 92–98.
[21]
Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C Snoeren. 2015. Inside the Social Network’s (datacenter) Network. In ACM SIGCOMM. 123–137.
[22]
Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C Snoeren. 2015. Inside the social network’s (datacenter) network. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 123–137.
[23]
Ahmed Saeed, Varun Gupta, Prateesh Goyal, Milad Sharif, Rong Pan, Mostafa Ammar, Ellen Zegura, Keon Jang, Mohammad Alizadeh, Abdul Kabbani, 2020. Annulus: A Dual Congestion Control Loop for Datacenter and WAN Traffic Aggregates. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 735–749.
[24]
Parvin Taheri, Danushka Menikkumbura, Erico Vanini, Sonia Fahmy, Patrick Eugster, and Tom Edsall. 2020. RoCC: robust congestion control for RDMA. In Proceedings of the 16th International Conference on emerging Networking EXperiments and Technologies. 17–30.
[25]
Chonggang Wang, Kazem Sohraby, and Bo Li. 2005. SenTCP: A hop-by-hop congestion control protocol for wireless sensor networks. IEEE INFOCOM (Poster Paper)(2005).
[26]
Jilong Xue, Youshan Miao, Cheng Chen, Ming Wu, Lintao Zhang, and Lidong Zhou. 2019. Fast distributed deep learning over RDMA. In Proceedings of the Fourteenth EuroSys Conference 2019. 1–14.
[27]
David Zats, Anand Padmanabha Iyer, Ganesh Ananthanarayanan, Rachit Agarwal, Randy Katz, Ion Stoica, and Amin Vahdat. 2015. Fastlane: making short flows shorter with agile drop notification. In Proceedings of the Sixth ACM Symposium on Cloud Computing. 84–96.
[28]
Jiao Zhang, Fengyuan Ren, Ran Shu, and Peng Cheng. 2016. TFC: token flow control in data center networks. In Proceedings of the Eleventh European Conference on Computer Systems. 1–14.
[29]
Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015. Congestion Control for Large-Scale RDMA Deployments. SIGCOMM Comput. Commun. Rev.4 (2015). https://doi.org/10.1145/2829988.2787484
[30]
Danyang Zhuo, Qiao Zhang, Vincent Liu, Arvind Krishnamurthy, and Thomas Anderson. 2016. Rack-level congestion control. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks. 148–154.
[31]
Noa Zilberman, Gabi Bracha, and Golan Schzukin. 2019. Stardust: Divide and conquer in the data center network. In 16th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 19). 141–160.

Cited By

View all
  • (2024)GTCC: A Game Theoretic Approach for Efficient Congestion Control in Datacenter NetworksIEEE Transactions on Network Science and Engineering10.1109/TNSE.2024.344309911:6(6328-6344)Online publication date: Nov-2024
  • (2024)PACC: A Proactive CNP Generation Scheme for Datacenter NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2024.336177132:3(2586-2599)Online publication date: Jun-2024
  • (2024)PCNPComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2024.110453247:COnline publication date: 18-Jul-2024
  • Show More Cited By

Index Terms

  1. HierCC: Hierarchical RDMA Congestion Control
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        APNet '21: Proceedings of the 5th Asia-Pacific Workshop on Networking
        June 2021
        79 pages
        ISBN:9781450385879
        DOI:10.1145/3469393
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 01 February 2022

        Permissions

        Request permissions for this article.

        Check for updates

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Funding Sources

        Conference

        APNet 2021
        APNet 2021: 5th Asia-Pacific Workshop on Networking
        June 24 - 25, 2021
        Shenzhen, China, China

        Acceptance Rates

        Overall Acceptance Rate 50 of 118 submissions, 42%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)90
        • Downloads (Last 6 weeks)9
        Reflects downloads up to 17 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)GTCC: A Game Theoretic Approach for Efficient Congestion Control in Datacenter NetworksIEEE Transactions on Network Science and Engineering10.1109/TNSE.2024.344309911:6(6328-6344)Online publication date: Nov-2024
        • (2024)PACC: A Proactive CNP Generation Scheme for Datacenter NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2024.336177132:3(2586-2599)Online publication date: Jun-2024
        • (2024)PCNPComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2024.110453247:COnline publication date: 18-Jul-2024
        • (2023)A Survey of RoCEv2 Congestion ControlThe 7th International Conference on Information Science, Communication and Computing10.1007/978-981-99-7161-9_4(42-56)Online publication date: 3-Nov-2023

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media