Yanfang Le
ABSTRACT
Increasingly, smart Network Interface Cards (sNICs) are being used in data centers to offload networking functions (NFs) from host processors thereby making these processors available for tenant applications. Modern sNICs have fully programmable, energy-efficient multi-core processors on which many packet processing functions, including a full-blown programmable switch, can run. However, having multiple switch instances deployed across the host hypervisor and the attached sNICs makes controlling them difficult and data plane operations more complex.
This paper proposes a generalized SDN-controlled NF offload architecture called UNO. It can transparently offload dynamically selected host processors' packet processing functions to sNICs by using multiple switches in the host while keeping the data centerwide network control and management planes unmodified. UNO exposes a single virtual control plane to the SDN controller and hides dynamic NF offload behind a unified virtual management plane. This enables UNO to make optimal use of host's and sNIC's combined packet processing capabilities with local optimization based on locally observed traffic patterns and resource consumption, and without central controller involvement. Experimental results based on a real UNO prototype in realistic scenarios show promising results: it can save processing worth up to 8 CPU cores, reduce power usage by up to 2x, and reduce the control plane overhead by more than 50%.
- Accolade ANIC. https://accoladetechnology.com/whitepapers/ANIC-Features-Overview.pdf.Google Scholar
- Cavium LiquidIO. http://www.cavium.com/pdfFiles/LiquidIO_Server_Adapters_PB_Rev1.0.pdf.Google Scholar
- Data Center Market Trends. http://www.te.com/content/dam/te-com/documents/broadband-network-solutions/global/data-center/brochures/presentation-data-center-market-trends.pdf.Google Scholar
- Emerging Smart NIC Technology. http://www.csit.qub.ac.uk/News/Events/Belfast-2016-6th-Cyber-Security-Summit/PDFs/Filetoupload,631658,en.pdf.Google Scholar
- Floodlight. http://www.projectfloodlight.org/floodlight/.Google Scholar
- How to Port Open vSwitch to New Software or Hardware. http://openvswitch.org/support/dist-docs-2.5/PORTING.md.html.Google Scholar
- Intel Advanced Encryption Standard (Intel AES) Instructions Set - Rev 3.01. https://software.intel.com/en-us/articles/intel-advanced-encryption-standard-aes-instructions-set.Google Scholar
- Intel Clear Containers: A Breakthrough Combination of Speed and Workload Isolation. https://clearlinux.org/sites/default/files/vmscontainers_wp_v5.pdf.Google Scholar
- Intel Gigabit Server Adapters. http://ark.intel.com/products/family/46829.Google Scholar
- Intel QuickAssist Adapter Family for Servers. http://www.intel.com/content/www/us/en/ethernet-products/gigabit-server-adapters/quickassist-adapter-for-servers.html.Google Scholar
- Max-flow min-cut theorem. https://en.wikipedia.org/wiki/Max-flow_min-cut_theorem.Google Scholar
- Mellanox BlueField. http://www.mellanox.com/related-docs/npu-multicore-processors/PB_Bluefield_SoC.pdf.Google Scholar
- Mellanox ConnectX-4. http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-4_VPI_Card.pdf.Google Scholar
- Mellanox ConnectX-5. http://www.mellanox.com/related-docs/user_manuals/ConnectX-5_VPI_Card.pdf.Google Scholar
- Minimum k-cut. https://en.wikipedia.org/wiki/Minimum_k-cut.Google Scholar
- nDPI. http://www.ntop.org/products/deep-packet-inspection/ndpi/.Google Scholar
- Netronome Agilio vRouter. https://netronome.com/media/redactor_files/SB_Netronome_Juniper_vRouter.pdf.Google Scholar
- OpenDaylight. https://www.opendaylight.org.Google Scholar
- OpenStack. https://www.openstack.org.Google Scholar
- OpenStack Compute API. https://developer.openstack.org/api-ref/compute/.Google Scholar
- OPNFV. https://www.opnfv.org.Google Scholar
- PRADS - Passive Real-time Asset Detection System. https://gamelinux.github.io/prads/.Google Scholar
- Processor Counter Monitor. https://github.com/opcm/pcm.Google Scholar
- Programming Intel QuickAssist Technology Hardware Accelerators for Optimal Performance. https://01.org/sites/default/files/page/332125_002_0.pdf.Google Scholar
- Putting Smart NICs in White Boxes. https://www.sdxcentral.com/articles/analysis/nics-white-boxes/2016/11/.Google Scholar
- SD-WAN. https://en.wikipedia.org/wiki/SD-WAN.Google Scholar
- Setting up the Python API of CPLEX. http://www.ibm.com/support/knowledgecenter/SSSA5P_12.5.1/ilog.odms.cplex.help/CPLEX/GettingStarted/topics/set_up/Python_setup.html.Google Scholar
- TILEncore-Gx36. http://www.mellanox.com/related-docs/prod_multi_core/PB_TILEncore-Gx36.pdf.Google Scholar
- Tilera Rescues CPU Cycles with Network Coprocessors. https://www.enterprisetech.com/2013/10/16/tilera-free-expensive-cpu-cycles-network-coprocessors/.Google Scholar
- VMware. Data Center Micro-Segmentation. http://blogs.vmware.com/networkvirtualization/files/2014/06/VMware-SDDC-Micro-Segmentation-White-Paper.pdf.Google Scholar
- Watts Up Meter. https://www.wattsupmeters.com.Google Scholar
- TILE Processor Architecture Overview for the TILE-Gx Series. Technical report, Mellanox, 2012. Doc. No. UG130.Google Scholar
- OpenFlow Switch Specification 1.5.0. Open Network Foundation, 2014.Google Scholar
- A. Al-Shabibi et al. OpenVirteX: Make Your Virtual SDNs Programmable. In Proc. ACM HotSDN, 2014.Google Scholar
- S. P. Antoine Kaufmann and N. K. Sharma. High Performance Packet Processing with FlexNIC. In Proc. ASPLOS, 2016.Google Scholar
- H. Ballani et al. Enabling End-host Network Functions. In Proc. ACM SIGCOMM, 2015.Google ScholarDigital Library
- A. Belay, G. Prekas, A. Klimovic, S. Grossman, C. Kozyrakis, and E. Bugnion. IX: A Protected Dataplane Operating System for High Throughput and Low Latency. In Proc. USENIX OSDI, 2014.Google ScholarDigital Library
- M. Blott and K. Vissers. Dataflow Architectures for 10Gbps Line-rate Key-value-Stores. In Proc. IEEE Hot Chips 25 Symposium, 2013.Google ScholarCross Ref
- P. Bosshart et al. P4: Programming Protocol-Independent Packet Processors. ACM SIGCOMM Computer Communication Review, 44(3), 2014.Google Scholar
- Z. Bozakov and P. Papadimitriou. AutoSlice: Automated and Scalable Slicing for Software-Defined Networks. In Proc. ACM CoNEXT, 2012.Google ScholarDigital Library
- M. Casado, T. Koponen, S. Shenker, and A. Tootoonchian. Fabric: A Retrospective on Evolving SDN. In Proc. ACM HotSDN, 2012.Google ScholarDigital Library
- H. Chang, S. Mukherjee, L. Wang, T. Lakshman, Y. Le, A. Akella, and M. Swift. UNO: Unifying Host and Smart NIC Offload for Flexible Packet Processing. Technical Report ITD-16-56788B, Nokia, 2016.Google Scholar
- Cisco. Data Center Microsegmentation: Enhance Security for Data Center Traffic. http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-732943.html.Google Scholar
- E. Cuervo et al. MAUI: Making Smartphones Last Longer with Code Offload. In Proc. ACM MobiSys, 2010.Google ScholarDigital Library
- H. T. Dang et al. Network Hardware-Accelerated Consensus. In USI Technical Report Series in Informatics, 2016.Google Scholar
- R. R. David F. Bacon and S. Shukla. FPGA Programming for the Masses. ACM QUEUE, 11(2), 2013.Google Scholar
- W. Dietz, J. Cranmer, N. Dautenhahn, and V. Adve. Slipstream: Automatic Interprocess Communication Optimization. In Proc. USENIX ATC, 2015.Google Scholar
- S. K. Fayazbakhsh, L. Chiang, V. Sekar, M. Yu, and J. C. Mogul. Enforcing Network-Wide Policies in the Presence of Dynamic Middlebox Actions using FlowTags. In Proc. USENIX NSDI, 2014.Google ScholarDigital Library
- D. Firestone. SmartNIC: Accelerating Azure's Network with FPGAs on OCS Servers. Open Compute Project, 2016.Google Scholar
- X. Ge, Y. Liu, D. H. Du, L. Zhang, H. Guan, J. Chen, Y. Zhao, and X. Hu. OpenANFV: Accelerating Network Function Virtualization with a Consolidated Framework in OpenStack. In Proc. ACM SIGCOMM, 2014.Google ScholarDigital Library
- A. Gember, P. Prabhu, Z. Ghadiyali, and A. Akella. Toward Software-defined Middlebox Networking. In Proc. ACM HotNets-XI, 2012.Google ScholarDigital Library
- A. Gember-Jacobson et al. OpenNF: Enabling Innovation in Network Function Control. ACM SIGCOMM Computer Communication Review, 44(4), 2015.Google Scholar
- B. Grot et al. Optimizing Data-Center TCO with Scale-Out Processors. IEEE Micro, 32(5), 2012.Google ScholarDigital Library
- B. Han, V. Gopalakrishnan, L. Ji, and S. Lee. Network Functions Virtualization: Challenges and Opportunities for Innovations. IEEE Communication Magazine, 53(2), 2015.Google Scholar
- S. Han, K. Jang, A. Panda, S. Palkar, D. Han, and S. Ratnasamy. SoftNIC: A Software NIC to Augment Hardware. Technical Report UCB/EECS-2015-155, University of California, Berkeley, 2015.Google Scholar
- A. Holt et al. Cloud Computing Takes Off. https://www.morganstanley.com/views/perspectives/cloud_computing.pdf. Morgan Stanley.Google Scholar
- M. Honda, F. Huici, G. Lettieri, and L. Rizzo. mSwitch: A Highly-Scalable, Modular Software Switch. In Proc. ACM SOSR, 2015.Google Scholar
- J. Hwang, K. K. Ramakrishnan, and T. Wood. NetVM: High Performance and Flexible Networking using Virtualization on Commodity Platforms. In Proc. USENIX NSDI, 2014.Google Scholar
- Z. Istvan, D. Sidler, G. Alonso, and M. Vukolic. Consensus in a Box: Inexpensive Coordination in Hardware. In Proc. USENIX NSDI, 2016.Google Scholar
- E. J. Jackson, M. Walls, A. Panda, J. Pettit, B. Pfaff, J. Rajahalme, T. Koponen, and S. Shenker. SoftFlow: A Middlebox Architecture for Open vSwitch. In Proc. USENIX ATC, 2016.Google Scholar
- M. Kablan, A. Alsudais, E. Keller, and F. Le. Stateless Network Functions: Breaking the Tight Coupling of State and Processing. In Proc. USENIX NSDI, 2017.Google Scholar
- N. Kang, Z. Liu, J. Rexford, and D. Walker. Optimizing the One Big Switch Abstraction in Software-Defined Networks. In Proc. ACM CoNEXT, 2013.Google ScholarDigital Library
- Y. Kanizo, D. Hay, and I. Keslassy. Palette: Distributing Tables in Software-Defined Networks. In Proc. ACM CoNEXT, 2013.Google ScholarCross Ref
- N. P. Katta, J. Rexford, and D. Walker. Incremental Consistent Updates. In Proc. ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking, 2013.Google ScholarDigital Library
- S. Kent. IP Encapsulating Security Payload (ESP). RFC 4303, 2005.Google Scholar
- A. Khrabrov and E. de Lara. Accelerating Complex Data Transfer for Cluster Computing. In Proc. USENIX HotCloud, 2016.Google Scholar
- Kindervag, J. Build Security Into Your Network's DNA: The Zero Trust Network Architecture.Google Scholar
- S. Larsen and B. Lee. Platform IO DMA Transaction Acceleration. In Proc. ACM Workshop on Characterizing Applications for Heterogeneous Exascale Systems, 2011.Google Scholar
- J. Li, E. Michael, N. K. Sharma, A. Szekeres, and D. R. K. Ports. Just say NO to Paxos Overhead: Replacing Consensus with Network Ordering. In Proc. USENIX OSDI, 2016.Google Scholar
- K. Lim et al. Thin Servers with Smart Pipes: Designing SoC Accelerators for Memcached. In Proc. ISCA, 2013.Google ScholarDigital Library
- Y. Luo, E. Murray, and T. L. Ficarra. Accelerated Virtual Switching with Programmable NICs for Scalable Data Center Networking. In Proc. ACM VISA, 2010.Google ScholarDigital Library
- H. Mekky, F. Hao, S. Mukherjee, Z.-L. Zhang, and T. Lakshman. Application-aware Data Plane Processing in SDN. In Proc. ACM HotSDN, 2014.Google ScholarDigital Library
- M. Moshref, M. Yu, A. Sharma, and R. Govindan. vCRIB: Virtualized Rule Management in the Cloud. In Proc. USENIX HotCloud, 2012.Google Scholar
- M. Moshref, M. Yu, A. Sharma, and R. Govindan. Scalable Rule Management for Data Centers. In Proc. USENIX NSDI, 2013.Google Scholar
- J. Nam, M. Jamshed, B. Choi, D. Han, and K. Park. Scaling the Performance of Network Intrusion Detection with Many-core Processors. In Proc. ACM/IEEE ANCS, 2015.Google ScholarCross Ref
- S. Palkar, C. Lan, S. Han, K. Jang, A. Panda, S. Ratnasamy, L. Rizzo, and S. Shenker. E2: A Framework for NFV Applications. In Proc. ACM SOSP, 2015.Google ScholarDigital Library
- Palo Alto Networks. Getting Started With a Zero Trust Approach to Network Security. https://www.paloaltonetworks.com/resources/whitepapers/zero-trust-network-security.html.Google Scholar
- T. Park, Y. Kim, and S. Shin. UNISAFE: A Union of Security Actions for Software Switches. In Proc. SDN-NFV Security, 2016.Google ScholarDigital Library
- S. Peter, J. Li, I. Zhang, D. R. K. Ports, D. Woos, A. Krishnamurthy, T. Anderson, and T. Roscoe. Arrakis: The Operating System is the Control Plane. In Proc. USENIX OSDI, 2014.Google ScholarDigital Library
- J. Pettit. Open vSwitch and the Intelligent Edge. In Proc. OpenStack Summit Atlanta, 2014.Google Scholar
- B. Pfaff et al. The Design and Implementation of Open vSwitch. In Proc. USENIX NSDI, 2015.Google ScholarDigital Library
- Z. A. Qazi, C.-C. Tu, L. Chiang, R. Miao, V. Sekar, and M. Yu. SIMPLE-fying Middlebox Policy Enforcement Using SDN. In Proc. ACM SIGCOMM, 2013.Google ScholarDigital Library
- S. Radhakrishnan, Y. Geng, V. Jeyakumar, A. Kabbani, G. Porter, and A. Vahdat. SENIC: Scalable NIC for End-Host Rate Limiting. In Proc. USENIX NSDI, 2014.Google ScholarDigital Library
- B. Raghavan et al. Software-Defined Internet Architecture: Decoupling Architecture from Infrastructure. In Proc. ACM HotNets-XI, 2012.Google ScholarDigital Library
- K. K. Ram, A. L. Cox, M. Chadha, and S. Rixner. Hyper-switch: A scalable software virtual switching architecture. In Proc. USENIX ATC, 2013.Google ScholarDigital Library
- K. K. Ram et al. sNICh: Efficient Last Hop Networking in the Data Center. In Proc. ACM/IEEE ANCS, 2010.Google Scholar
- L. Rizzo, P. Valente, G. Lettieri, and V. Maffione. PSPAT: software packet scheduling at hardware speed. Preprint, 2016.Google Scholar
- G. Sabin and M. Rashti. Security Offload Using the SmartNIC, A Programmable 10 Gbps Ethernet NIC. In Proc. Aerospace and Electronics Conference, 2015.Google ScholarCross Ref
- V. Sekar, N. Egi, S. Ratnasamy, M. K. Reiter, and G. Shi. Design and Implementation of a Consolidated Middlebox Architecture. In Proc. USENIX NSDI, 2012.Google ScholarDigital Library
- A. Shaikh, J. Rexford, and K. G. Shin. Load-Sensitive Routing of Long-Lived IP Flows. In Proc. ACM SIGCOMM, 1999.Google ScholarDigital Library
- J. Sherry, S. Hasan, C. Scott, A. Krishnamurthy, S. Ratnasamy, and V. Sekar. Making Middleboxes Someone else's Problem: Network Processing As a Cloud Service. In Proc. ACM SIGCOMM, 2012.Google ScholarDigital Library
- R. Sherwood et al. FlowVisor: A Network Virtualization Layer. In OpenFlow Switch Consortium, 2009.Google Scholar
- P. Shinde, A. Kaufmann, T. Roscoe, and S. Kaestle. We need to talk about NICs. In Proc. USENIX HotOS, 2013.Google Scholar
- D. Sturgeon. HW Acceleration of Memcached. In Proc. Flash Memory Summit, 2014.Google Scholar
- A. Tootoonchian and Y. Ganjali. HyperFlow: A Distributed Control Plane for OpenFlow. In Proc. Internet Network Management Conference on Research on Enterprise Networking, 2010.Google Scholar
- A. Wang, Y. Guo, F. Hao, T. V. Lakshman, and S. Chen. UMON: Flexible and Fine Grained Traffic Monitoring in Open vSwitch. In Proc. ACM CoNEXT, 2015.Google ScholarDigital Library
- Z. Wang, K. Liu, Y. Shen, J. Y. B. Lee, M. Chen, and L. Zhang. Intra-host Rate Control with Centralized Approach. In Proc. IEEE International Conference on Cluster Computing, 2016.Google ScholarCross Ref
- Y. Weinsberg, D. Dolev, P. Wyckoff, and T. Anker. Accelerating Distributed Computing Applications Using a Network Offloading Framework. In Proc. IEEE Parallel and Distributed Processing Symposium, 2007.Google ScholarCross Ref
- M. Yu, J. Rexford, M. J. Freedman, and J. Wang. Scalable Flow-Based Networking with DIFANE. In Proc. ACM SIGCOMM, 2010.Google ScholarDigital Library
Index Terms
- UNO: uniflying host and smart NIC offload for flexible packet processing
Recommendations
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16: Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution EnvironmentsSingle-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
Comments