skip to main content
10.1145/3482898.3483361acmconferencesArticle/Chapter ViewAbstractPublication PagescommConference Proceedingsconference-collections
research-article
Public Access

Nimble: Scalable TCP-Friendly Programmable In-Network Rate-Limiting

Published: 08 November 2021 Publication History

Abstract

There is an emerging need for scalable high-performance in-networkrate-limiting because rate-limiters can be used to provide performance isolation. However, existing approaches to in-network rate-limiting are not scalable or TCP-friendly.
This paper presents the design of Nimble, a new approach to in-network rate-limiting that is scalable, high performance, and TCP-friendly. Nimble uses meters to scalably provide hardware rate-limiting without any dedicated queuing or buffering resources, and Nimble uses ECN-Shaping for TCP-friendly rate-limit enforcement. Nimble also introduces the first algorithm for configuring in-network rate-limiters to enforce network-wide isolation policies.
Through a P4 implementation and experiments with a 100Gbps Barefoot Tofino switch, we find that Nimble is immediately usable and can operate even with high bandwidth rate-limits without needing to recirculate packets or rely on hardware packet generators to generate token refill packets. This overcomes the scalability limitations of prior approaches. Experiments with Apache and Redis show that Nimble can reduce application-level latency by an order of magnitude when compared to not using in-network rate-limiting, and ns-3 simulations demonstrate that Nimble behaves well in larger clusters. We find that Nimble can scale to 100K rate-limiters perswitch when implemented on a Barefoot Tofino switch, and our new rate allocation algorithm reduces rate-limiter updates by a factor of 10x-24x and improves network utilization by 24%.

References

[1]
Apache---HTTP Server Project. https://httpd.apache.org/.
[2]
BESS: Berkeley Extensible Software Switch. https://github.com/NetSys/bess.
[3]
iperf3: Documentation. http://software.es.net/iperf/.
[4]
P4 Language Consortium. https://p4.org/.
[5]
Redis. https://redis.io/.
[6]
sockperf: Network benchmarking utility. https://github.com/Mellanox/sockperf.
[7]
The ns-3 discrete-event network simulator. http://www.nsnam.org.
[8]
Alizadeh, M., Greenberg, A., Maltz, D. A., Padhye, J., Patel, P., PrabhakaR@, B., Sengupta, S., andSridharan, M. Data Center TCP (DCTCP). In SIGCOMM(2010).
[9]
Alizadeh, M., Kabbani, A., Edsall, T., Prabhakar, B., Vahdat, A., andYasuda, M. Less is more: trading a little bandwidth for ultra-low latency in the data center. In NSDI(2012).
[10]
Ballani, H., Costa, P., Karagiannis, T., andRowstron, A. Towards predictable datacenter networks. In SIGCOMM(2011).
[11]
Barefoot. Barefoot Tofino. https://www.barefootnetworks.com/technology/#tofino, 2017.
[12]
Bosshart, P., Gibb, G., Kim, H.-S., Varghese, G., McKeown, N., Izzard, M., Mujica, F. A., andHorowitz, M. Forwarding metamorphosis: fast programmable match-action processing in hardware for SDN. In SIGCOMM(2013).
[13]
Brown, L., Ananthanarayanan, G., Katz-Bassett, E., Krishnamurthy, A., Ratnasamy, S., Schapira, M., andShenker, S. On the future of congestion control for the public internet. In Proceedings of the ACM Workshop on Hot Topics in Networks (HotNets)(2020), Association for Computing Machinery.
[14]
CaviumCorporation. Cavium CNX880XX. http://www.cavium.com/pdfFiles/CNX880XX_PB_Rev1.pdf.
[15]
Chen, X., Feibish, S. L., Koral, Y., Rexford, J., Rottenstreich, O., Monetti, S. A., andWang, T.-Y. Fine-grained queue measurement in the data plane. In Proceedings of the 15th International Conference on Emerging Networking Experiments And Technologies(2019), p. 1529.
[16]
Chen, Y.-C., andXu, X. An adaptive buffer allocation mechanism for token bucket flow control. vol. 4, pp. 3020--3024 Vol. 4.
[17]
Cronkite-Ratcliff, B., Bergman, A., Vargaftik, S., Ravi, M., McKeown, N., Abraham, I., andKeslassy, I. Virtualized congestion control. In SIGCOMM(2016).
[18]
Dalton, M., Schultz, D., Adriaens, J., Arefin, A., Gupta, A., Fahs, B., Rubinstein, D., Zermeno, e. C., Rubow, e., Docauer, J. A., Alpert, J., Ai, J., Olson, J., DeCabooter, K., deKruijf, M., Hua, N., Lewis, N., Kasinadhuni, N., Crepaldi, R., Krishnan, S., Venkata, S., Richter, Y., Naik, U., andVahdat, A. Andromeda: Performance, isolation, and velocity at scale in cloud network virtualization. In NSDI(2018).
[19]
DataCenterBridgingTaskGroup. http://www.ieee802.org/1/pages/dcbridges.html.
[20]
Dukkipati, N. Rate Control Protocol (Rcp): Congestion Control to Make Flows Complete Quickly.PhD thesis, Stanford, CA, USA, 2008. AAI3292347.
[21]
Firestone, D., Putnam, A., Mundkur, S., Chiou, D., Dabagh, A., Andrewartha, M., Angepat, H., Bhanu, V., Caulfield, A., Chung, e., Chandrappa, H. k., Chaturmohta, S., Humphrey, M., Lavier, J., Lam, N., Liu, F., Ovtcharov, K., Padhye, J., Popuri, G., Raindel, S., Sapre, T., Shaw, M., Silva, G., Sivakumar, M., Srivastava, N., Verma, A., Zuhair, Q., Bansal, D., Burger, D., Vaid, K., Maltz, D. A., andGreenberg, A. Azure accelerated networking: SmartNICs in the public cloud. In NSDI(2018), USENIX Association.
[22]
Flach, T., Papageorge, P., Terzis, A., Pedrosa, L., Cheng, Y., Karim, T., Katz-Bassett, E., andGovindan, R. An internet-wide analysis of traffic policing. In SIGCOMM(2016).
[23]
FLOYD, S., AND Jacobson, V. Random early detection gateways for congestion avoidance. IEEE/ACM Trans. Netw.(1993).
[24]
Gember-Jacobson, A., Viswanathan, R., Prakash, C., Grandl, R., Khalid, J., Das, S., andAkella, A. OpenNF: Enabling innovation in network function control. In SIGCOMM(2014).
[25]
Goyal, P., Shah, P., Sharma, N. k., Alizadeh, M., andAnderson, T. e. Backpressure flow control, 2019.
[26]
Han, S., Jang, k., Panda, A., Palkar, S., Han, D., andRatnasamy, S. Softnic: A software nic to augment hardware. Tech. Rep. UCB/EECS-2015-155, EECS Department, University of California, Berkeley, May 2015.
[27]
He, K., Qin, W., Zhang, Q., Wu, W., Yang, J., Pan, T., Hu, C., Zhang, J., Stephens, B., Akella, A., andZhang, Y. Low latency software rate limiters for cloud networks. In APNet(2017), ACM.
[28]
He, K., Rozner, E., Agarwal, K., Gu, Y. J., Felter, W., Carter, J., andAkella, A. AC/DC TCP: Virtual congestion control enforcement for datacenter networks. In SIGCOMM(2016).
[29]
Heinanen, J., Finland, T., andGuerin, R. Rfc2697: A single rate three color marker, 1999.
[30]
Heinanen, J., andGuerin, R. Rfc2698: A two rate three color marker, 1999.
[31]
Hong, C.-Y., Kandula, S., Mahajan, R., Zhang, M., Gill, V., Nanduri, M., andWattenhofer, R. Achieving high utilization with software-driven WAN.
[32]
Hong, C.-Y., Mandal, S., Al-Fares, M., Zhu, M., Alimi, R., B., K. N., Bhagat, C., Jain, S., Kaimal, J., Liang, S., Mendelev, K., Padgett, S., Rabe, F., Ray, S., Tewari, M., Tierney, M., Zahn, M., Zolla, J., Ong, J., andVahdat, A. B4 and after: Managing hierarchy, partitioning, and asymmetry for availability and scale in google's software-defined WAN. In SIGCOMM(2018).
[33]
Iorgulescu, C., Azimi, R., Kwon, Y., Elnikety, S., Syamala, M., Narasayya, V., Herodotou, h., Tomita, P., Chen, A., Zhang, J., andWang, J. Perfiso: Performance isolation for commercial latency-sensitive services. In USENIX ATC(2018), USENIX Association.
[34]
Jain, S., Kumar, A., Mandal, S., Ong, J., Poutievski, L., Singh, A., Venkata, S., Wanderer, J., Zhou, J., Zhu, M., Zolla, J., Hölzle, U., STUART, S., AND Vahdat, A. B4: Experience with a globally-deployed software defined wan. In SIGCOMM(2013).
[35]
Jang, K., Sherry, J., Ballani, H., andMoncaster, T. Silo: Predictable message latency in the cloud. In SIGCOMM(2015).
[36]
Jeyakumar, V., Alizadeh, M., Mazieres, D., Prabhakar, B., Kim, C., AND GREENBERG, A. EyeQ: Practical network performance isolation at the edge. In NSDI(2013).
[37]
Jose, L., Yan, L., Alizadeh, M., Varghese, G., McKeown, N., andKatti, S. High speed networks need proactive congestion control. In HotNets(2015), ACM.
[38]
Kumar, A., Jain, S., Naik, U., Kasinadhuni, N., Zermeno, E. C., Gunn, C. S., Ai, J., Carlin, B., Amarandei-Stavila, M., Robin, M., Siganporia, A., Stuart, S., andVahdat, A. BwE: Flexible, hierarchical bandwidth allocation for WAN distributed computing. In SIGCOMM(2015).
[39]
KUMAR, G., DUKKIPATI, N., JANG, K., WASSEL, H. M. G., Wu, X., Montazeri, B., Wang, Y., Springborn, K., Alfeld, C., Ryan, M., WETHERALL, D., AND Vahdat, A. Swift: Delay is simple and effective for congestion control in the datacenter. In SIGCOMM(2020), Association for Computing Machinery.
[40]
Liu, Z., Manousis, A., Vorsanger, G., Sekar, V., andBraverman, V. One sketch to rule them all: Rethinking network flow monitoring with UnivMon. In SIGCOMM(2016).
[41]
Lo, D., Cheng, L., Govindaraju, R., Ranganathan, P., andKozyrakis, C. Heracles: Improving resource efficiency at scale. In ISCA(2015).
[42]
Marty, M., deKRUIJF, M., ADRIAENS, J., Alfeld, C., Bauer, S., Contavalli, C., Dalton, M., Dukkipati, N., Evans, W. C., Gribble, S., etal. Snap: a microkernel approach to host networking. In Proceedings of the 27th ACM Symposium on Operating Systems Principles(2019).
[43]
MellanoxTechnologies. ConnectX-5 EN Card. http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-5_EN_Card.pdf.
[44]
Mittal, R., Lam, T., Dukkipati, N., Blem, E., Wassel, h., Ghobadi, M., Vahdat, A., Wang, Y., Wetherall, D., andZats, D. TIMELY: RTT-based congestion control for the datacenter. In SIGCOMM(2015).
[45]
Narayana, S., Sivaraman, A., Nathan, V., Goyal, P., Arun, V., Alizadeh, M., JEYAKUMAR, V., AND KIM, C. Language-directed hardware design for network performance monitoring. In SIGCOMM(2017).
[46]
Popa, L., Kumar, G., Chowdhury, M., Krishnamurthy, A., Ratnasamy, S., andStoica, I. FairCloud: Sharing the network in cloud computing. In SIGCOMM(2012).
[47]
Radhakrishnan, S., Geng, Y., Jeyakumar, V., Kabbani, A., Porter, G., andVahdat, A. SENIC: Scalable NIC for end-host rate limiting. In NSDI(2014).
[48]
Roy, A., Zeng, H., Bagga, J., Porter, G., andSnoeren, A. C. Inside the social network's (datacenter) network. In SIGCOMM(2015).
[49]
Saeed, A., Dukkipati, N., Valancius, V., Lam, T., Contavalli, C., andVahdat, A. Carousel: Scalable traffic shaping at end-hosts. In SIGCOMM(2017).
[50]
Saeed, A., Zhao, Y., Dukkipati, N., Zegura, E., Ammar, M., Harras, K., AND Vahdat, A. Eiffel: Efficient and flexible software packet scheduling. In NSDI(2019), USENIX Association.
[51]
Sharma, N. K., Kaufmann, A., Anderson, T., Krishnamurthy, A., Nelson, J., andPeter, S. Evaluating the power of flexible packet processing for network resource allocation. In 14th { USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 17)(2017), pp. 67--82.
[52]
Sharma, N. K., Liu, M., Atreya, K., andKrishnamurthy, A. Approximating fair queueing on reconfigurable switches. In NSDI(2018), USENIX Association.
[53]
Shieh, A., Kandula, S., Greenberg, A., andKim, C. Sharing the data center network. In NSDI(2011).
[54]
Shrivastav, V. Fast, scalable, and programmable packet scheduler in hardware. In SIGCOMM(2019), ACM.
[55]
Singh, A., Ong, J., Agarwal, A., Anderson, G., Armistead, A., Bannon, R., Boving, S., Desai, G., Felderman, B., Germano, P., Kanagala, A., Provost, J., Simmons, J., Tanda, E., Wanderer, J., Hölzle, U., Stuart, S., andVahdat, A. Jupiter rising: A decade of clos topologies and centralized control in google's datacenter network. In SIGCOMM(2015), pp. 183--197.
[56]
Sivaraman, A., Cheung, A., Budiu, M., Kim, C., Alizadeh, M., Balakrishnan, H., Varghese, G., McKeown, N., andLicking, S. Packet transactions: High-level programming for line-rate switches. In SIGCOMM(2016).
[57]
Sivaraman, A., Kim, C., Krishnamoorthy, R., Dixit, A., andBudiu, M. Dc.p4: Programming the forwarding plane of a data-center switch. In SOSR(2015), ACM.
[58]
Sivaraman, A., Subramanian, S., Alizadeh, M., Chole, S., CHUANG, S.-T., Agrawal, A., Balakrishnan, H., Edsall, T., Katti, S., andMcKeown, N. Programmable packet scheduling at line rate. In SIGCOMM(2016).
[59]
Sonchack, J., Michel, o., Aviv, A. J., Keller, E., andSmith, J. M. Scaling hardware accelerated network monitoring to concurrent and dynamic queries with flow. In USENIX ATC(2018).
[60]
Soni, H., Rifai, M., Kumar, P., Doenges, R., andFoster, N. Composing dataplane programs with μp4. SIGCOMM.
[61]
Stephens, B., Akella, A., andSwift, M. Loom: Flexible and efficient NIC packet scheduling. In NSDI(2017).
[62]
Stephens, B., Cox, A. L., Singla, A., Carter J., Dixon, C., andFelter W. Practical DCB for improved data center networks. In INFOCOM(2014).
[63]
Sundarrajan, A., Feng, M., Kasbekar M., andSitaraman, R. K. Footprint Descriptors: Theory and practice of cache provisioning in a global CDN. In CoNEXT(2017), ACM.
[64]
vanHaalen, R., andMalhotra, R. Improving tcp performance with bufferless token bucket policing: A tcp friendly policer. In 2007 15th IEEE Workshop on Local Metropolitan Area Networks(2007), pp. 72--77.
[65]
Wang, S. Y., Hu, H. W., andLin, Y. B. Design and implementation of tcp-friendly meters in p4 switches. IEEE/ACM Transactions on Networking 28, 4 (2020), 1885--1898.
[66]
Yu, Z., Wu, J., BRAVERMAN, V., Stoica, I., AND JIN, X. Twenty years after: Hierarchical core-stateless fair queueing. In USENIX Symposium on Networked Systems Design and Implementation (NSDI)(2021), USENIX Association.
[67]
Zats, D., Das, T., Mohan, P., Borthakur D., andKatz, R. DeTail: Reducing the flow completion time tail in datacenter networks. In SIGCOMM(2012).
[68]
Zhang, Q., Liu, V., Zeng, H., andKrishnamurthy, A. High-resolution measurement of data center microbursts. In Proceedings of the Internet Measurement Conference(2017), IMC, ACM.
[69]
Zhu, Y., Eran, h., Firestone, D., Guo, C., Lipshteyn, M., Liron, Y., Padhye, J., Raindel, S., Yahia, M. H., andZhang, M. Congestion control for large-scale RDMA deployments. In SIGCOMM(August 2015), ACM.
[70]
Zhu, Y., Ghobadi, M., Misra, V., andPadhye, J. ECN or delay: Lessons learnt from analysis of DCQCN and TIMELY. In CoNEXT(2016), ACM.

Cited By

View all
  • (2024)HSDBA: a hierarchical and scalable dynamic bandwidth allocation for programmable data planesHSDBA: 一种面向可编程数据平面的分层可扩展动态带宽分配方法Frontiers of Information Technology & Electronic Engineering10.1631/FITEE.230059325:10(1337-1352)Online publication date: 5-Nov-2024
  • (2024)Carlo: Cross-Plane Collaboration for Multiple In-network Computing ApplicationsIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621111(2528-2537)Online publication date: 20-May-2024
  • (2023)Scalable Real-Time Bandwidth Fairness in SwitchesIEEE/ACM Transactions on Networking10.1109/TNET.2023.331717232:2(1423-1434)Online publication date: 9-Oct-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SOSR '21: Proceedings of the ACM SIGCOMM Symposium on SDN Research (SOSR)
October 2021
190 pages
ISBN:9781450390842
DOI:10.1145/3482898
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2021

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • NSF

Conference

SOSR '21
Sponsor:
SOSR '21: The ACM SIGCOMM Symposium on SDN Research
October 11 - 12, 2021
Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 7 of 43 submissions, 16%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)214
  • Downloads (Last 6 weeks)34
Reflects downloads up to 10 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)HSDBA: a hierarchical and scalable dynamic bandwidth allocation for programmable data planesHSDBA: 一种面向可编程数据平面的分层可扩展动态带宽分配方法Frontiers of Information Technology & Electronic Engineering10.1631/FITEE.230059325:10(1337-1352)Online publication date: 5-Nov-2024
  • (2024)Carlo: Cross-Plane Collaboration for Multiple In-network Computing ApplicationsIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621111(2528-2537)Online publication date: 20-May-2024
  • (2023)Scalable Real-Time Bandwidth Fairness in SwitchesIEEE/ACM Transactions on Networking10.1109/TNET.2023.331717232:2(1423-1434)Online publication date: 9-Oct-2023
  • (2023)RateSheriff: Multipath Flow-aware and Resource Efficient Rate Limiter Placement for Data Center Networks2023 IEEE/ACM 31st International Symposium on Quality of Service (IWQoS)10.1109/IWQoS57198.2023.10188742(01-10)Online publication date: 19-Jun-2023
  • (2023)Scalable Real-Time Bandwidth Fairness in SwitchesIEEE INFOCOM 2023 - IEEE Conference on Computer Communications10.1109/INFOCOM53939.2023.10228997(1-10)Online publication date: 17-May-2023
  • (2023)Toward Next-Generation Distributed Rate-limiters2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing Workshops (CCGridW)10.1109/CCGridW59191.2023.00083(354-356)Online publication date: May-2023
  • (2022)Consistent and Fine-Grained Rule Update with In-Network Control for Distributed Rate Limiting2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS)10.1109/IWQoS54832.2022.9812884(1-10)Online publication date: 10-Jun-2022
  • (2022)ADA: Arithmetic Operations with Adaptive TCAM Population in Programmable Switches2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS54860.2022.00044(1-11)Online publication date: Jul-2022

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media