research-article

SR-IOV: Performance Benefits for Virtualized Interconnects

Authors:

Glenn K. Lockwood,

Mahidhar Tatineni,

Rick WagnerAuthors Info & Claims

XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment

Article No.: 47, Pages 1 - 7

https://doi.org/10.1145/2616498.2616537

Published: 13 July 2014 Publication History

Abstract

The demand for virtualization within high-performance computing is rapidly growing as new communities, driven by both new application stacks and new computing modalities, continue to grow and expand. While virtualization has traditionally come with significant penalties in I/O performance that have precluded its use in mainstream large-scale computing environments, new standards such as Single Root I/O Virtualization (SR-IOV) are emerging that promise to diminish the performance gap and make high-performance virtualization possible.

To this end, we have evaluated SR-IOV in the context of both virtualized InfiniBand and virtualized 10 gigabit Ethernet (GbE) using micro-benchmarks and real-world applications. We compare the performance of these interconnects on non-virtualized environments, Amazon's SR-IOV-enabled C3 instances, and our own SR-IOV-enabled InfiniBand cluster and show that SR-IOV significantly reduces the performance losses caused by virtualization. InfiniBand demonstrates less than 2% loss of bandwidth and less than 10% increase in latency when virtualized with SR-IOV. Ethernet also benefits, although less dramatically, when SR-IOV is enabled on Amazon's cloud.

References

[1]

Bollen, J., Fox, G., and Singhal, P.R. How and where the TeraGrid supercomputing infrastructure benefits science. Journal of Informetrics 5, 1 (2011), 114--121.

[2]

Dong, Y., Yang, X., Li, J., Liao, G., Tian, K., and Guan, H. High performance network virtualization with SR-IOV. Journal of Parallel and Distributed Computing 72, 11 (2012), 1471--1480.

Digital Library

[3]

Eldred, C. (Pittsburgh S.C. and Michalakes, J. (National C. for A.R. WRF V3 Parallel Benchmark Page. 2008. http://www.mmm.ucar.edu/wrf/WG2/benchv3/.

[4]

Ghoshal, D., Canon, R.S., and Ramakrishnan, L. I/O performance of virtualized cloud environments. Proceedings of the Second International Workshop on Data Intensive Computing in the Clouds, ACM Press (2011), 71--80.

Digital Library

[5]

Giannozzi, P., Baroni, S., Bonini, N., et al. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. Journal of Physics: Condensed Matter 21, 39 (2009), 395502.

[6]

Hines, M.L. and Carnevale, N.T. Expanding NEURON's Repertoire of Mechanisms with NMODL. Neural Computation 12, 5 (2000), 995--1007.

Digital Library

[7]

Huang, Z., Ma, R., Li, J., Chang, Z., and Guan, H. Adaptive and Scalable Optimizations for High Performance SR-IOV. 2012 IEEE International Conference on Cluster Computing, IEEE (2012), 459--467.

Digital Library

[8]

Jackson, K.R., Ramakrishnan, L., Muriki, K., et al. Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud. 2010 IEEE Second International Conference on Cloud Computing Technology and Science, IEEE (2010), 159--168.

Digital Library

[9]

Jose, J., Li, M., Lu, X., Kandalla, K.C., Arnold, M.D., and Panda, D.K. SR-IOV Support for Virtualization on InfiniBand Clusters: Early Experience. 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, IEEE (2013), 385--392.

[10]

Liu, J. Evaluating standard-based self-virtualizing devices: A performance study on 10 GbE NICs with SR-IOV support. 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), IEEE (2010), 1--12.

[11]

Luszczek, P.R., Bailey, D.H., Dongarra, J.J., et al. The HPC Challenge (HPCC) benchmark suite. Proceedings of the 2006 ACM/IEEE conference on Supercomputing - SC '06, ACM Press (2006), 213.

Digital Library

[12]

Mehrotra, P., Djomehri, J., Heistand, S., et al. Performance evaluation of Amazon EC2 for NASA HPC applications. Proceedings of the 3rd workshop on Scientific Cloud Computing Date - ScienceCloud '12, ACM Press (2012), 41--49.

Digital Library

[13]

Miller, M.A., Pfeiffer, W., and Schwartz, T. Creating the CIPRES Science Gateway for inference of large phylogenetic trees. Gateway Computing Environments Workshop (GCE), 2010, (2010).

[14]

Pekurovsky, D. P3DFFT: A Framework for Parallel Computations of Fourier Transforms in Three Dimensions. SIAM Journal on Scientific Computing 34, 4 (2012), C192--C209.

[15]

Shainer, G., Liu, T., Michalakes, J., et al. Weather Research and Forecast (WRF) Model: Performance Analysis on Advanced Multi-core HPC Clusters. The 10th LCI International Conference on High-Performance Clustered Computing, (2009).

[16]

Sivagnanam, S., Astakhov, V., Yoshimoto, K., et al. A neuroscience gateway: software and implementation. Proceedings of the Conference on Extreme Science and Engineering Discovery Environment Gateway to Discovery-XSEDE '13, ACM Press (2013).

Digital Library

[17]

Skamarock, W.C., Klemp, J.B., Dudhia, J., et al. NCAR Technical Note NCAR/TN--475+STR: A Description of the Advanced Research WRF Version 3. Boulder, CO, 2008.

[18]

QuantumESPRESSO - Deisa. 2008. http://www.deisa.eu/science/benchmarking/codes/quantumespresso.

[19]

Single Root I/O Virtualization and Sharing 1.1 specification. 2010. http://www.pcisig.com/specifications/iov/single_root/.

[20]

HPC Advisory Council Best Practices. 2010. http://www.hpcadvisorycouncil.com/best_practices.php.

[21]

AMD I/O Virtualization Technology (IOMMU) Specification. 2011. http://developer.amd.com/wordpress/media/2012/10/48882.pdf.

[22]

OSU Microbenchmarks. 2011. http://mvapich.cse.ohio-state.edu/benchmarks/.

[23]

Intel Virtualization Technology for Directed I/O. 2013. http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf.

Cited By

Guo ZKoufos KDianati MWoodman R(2025)State-of-the-art virtualisation technologies for the centralised automotive E/E architectureFrontiers in Future Transportation10.3389/ffutr.2025.15193906Online publication date: 17-Feb-2025
https://doi.org/10.3389/ffutr.2025.1519390
Kumar AKatkam RChaudhary PNaik PVutukuru M(2024)AppSteer: Framework for Improving Multicore Scalability of Network Functions via Application-aware Packet Steering2024 IEEE 24th International Symposium on Cluster, Cloud and Internet Computing (CCGrid)10.1109/CCGrid59990.2024.00012(18-27)Online publication date: 6-May-2024
https://doi.org/10.1109/CCGrid59990.2024.00012
de Oliveira Filho AFreitas Edo Carmo PSouto EKelner JSadok D(2024)Analysis of SR-IOV in Docker containers using RTT measurementsComputer Communications10.1016/j.comcom.2024.107961228:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.comcom.2024.107961
Show More Cited By

Index Terms

SR-IOV: Performance Benefits for Virtualized Interconnects
1. Information systems
  1. Information retrieval
  2. Information storage systems
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Communications management

Recommendations

A Hypervisor Approach to Enable Live Migration with Passthrough SR-IOV Network Devices
Special Topics

Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (physical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly ...
MVAPICH2 over openstack with SR-IOV: an efficient approach to build HPC clouds
CCGRID '15: Proceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing

Cloud Computing with Virtualization offers attractive flexibility and elasticity to deliver resources by providing a platform for consolidating complex IT resources in a scalable manner. However, efficiently running HPC applications on Cloud Computing ...
Towards the InfiniBand SR-IOV vSwitch Architecture
CLUSTER '15: Proceedings of the 2015 IEEE International Conference on Cluster Computing

To meet the demands of the Exascale era and facilitate Big Data analytics in the cloud while maintaining flexibility, cloud providers will have to offer efficient virtualized High Performance Computing clusters in a pay-as-you-go model. As a consequence,...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment

July 2014

445 pages

ISBN:9781450328937

DOI:10.1145/2616498

General Chair:
Scott Lathrop
National Center for Supercomputing Applications
,
Program Chair:
Jay Alameda
National Center for Supercomputing Applications

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

NSF: National Science Foundation
Drexel University
Indiana University: Indiana University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

XSEDE '14

XSEDE '14: Annual Conference of the Extreme Science and Engineering Discovery Environment

July 13 - 18, 2014

GA, Atlanta, USA

Acceptance Rates

XSEDE '14 Paper Acceptance Rate 80 of 120 submissions, 67%;

Overall Acceptance Rate 129 of 190 submissions, 68%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
341
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)5

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Guo ZKoufos KDianati MWoodman R(2025)State-of-the-art virtualisation technologies for the centralised automotive E/E architectureFrontiers in Future Transportation10.3389/ffutr.2025.15193906Online publication date: 17-Feb-2025
https://doi.org/10.3389/ffutr.2025.1519390
Kumar AKatkam RChaudhary PNaik PVutukuru M(2024)AppSteer: Framework for Improving Multicore Scalability of Network Functions via Application-aware Packet Steering2024 IEEE 24th International Symposium on Cluster, Cloud and Internet Computing (CCGrid)10.1109/CCGrid59990.2024.00012(18-27)Online publication date: 6-May-2024
https://doi.org/10.1109/CCGrid59990.2024.00012
de Oliveira Filho AFreitas Edo Carmo PSouto EKelner JSadok D(2024)Analysis of SR-IOV in Docker containers using RTT measurementsComputer Communications10.1016/j.comcom.2024.107961228:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.comcom.2024.107961
Wiedner FHelm MDaichendt AAndre JCarle G(2023)Containing Low Tail-Latencies in Packet Processing Using Lightweight Virtualization2023 35th International Teletraffic Congress (ITC-35)10.1109/ITC-3560063.2023.10555759(1-9)Online publication date: 3-Oct-2023
https://doi.org/10.1109/ITC-3560063.2023.10555759
Kang SHwang JKim Y(2023)DAQS: Dynamic and Accurate QoS for SR-IOV2023 14th International Conference on Information and Communication Technology Convergence (ICTC)10.1109/ICTC58733.2023.10392450(1440-1442)Online publication date: 11-Oct-2023
https://doi.org/10.1109/ICTC58733.2023.10392450
de Oliveira Filho AFreitas ECarmo PSadok DKelner J(2023)Measuring the impact of SR-IOV and virtualization on packet round-trip timeComputer Communications10.1016/j.comcom.2023.09.013211:C(193-215)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.comcom.2023.09.013
Wiedner FHelm MGallenmuller SCarle G(2022)HVNet: Hardware-Assisted Virtual Networking on a Single Physical HostIEEE INFOCOM 2022 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)10.1109/INFOCOMWKSHPS54753.2022.9798351(1-6)Online publication date: 2-May-2022
https://doi.org/10.1109/INFOCOMWKSHPS54753.2022.9798351
Ou JYin JChen J(2022)Research on evaluation model and technology of 5G core network User Plane Function2022 International Conference on Information Processing and Network Provisioning (ICIPNP)10.1109/ICIPNP57450.2022.00036(134-136)Online publication date: Sep-2022
https://doi.org/10.1109/ICIPNP57450.2022.00036
Hsu KChou J(2021)Performance benchmarking and auto-tuning for scientific applications on virtual clusterThe Journal of Supercomputing10.1007/s11227-021-04103-w78:5(6174-6206)Online publication date: 11-Oct-2021
https://doi.org/10.1007/s11227-021-04103-w
Zhuang JJacob DLin HLundgren EYantosca RGaya JSulprizio MEastham S(2020)Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry ModelJournal of Advances in Modeling Earth Systems10.1029/2020MS00206412:5Online publication date: 6-May-2020
https://doi.org/10.1029/2020MS002064
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten