Abstract
NVMe-based storage is widely used for various data-intensive applications due to their high bandwidth and low access latency. The NVMe-over-Fabrics (NVMe-oF) protocol specification provides efficient remote access to NVMe-SSDs over storage networking fabrics. NVMe-oF provides the opportunity to make storage disaggregation practical by reducing the cost of remote access. Unfortunately, the performance characteristics of different NVMe-oF networking protocols are not well understood. In this paper, we propose a four dimensional (network protocol, I/O pattern, I/O size, and number of cores) evaluation methodology to understand NVMe-oF performance over various commodity networks. We conduct comprehensive microbenchmark analyses using the user-space Intel SPDK library to compare the TCP, IPoIB, RoCE, and RDMA transports. Our analysis reveals interesting, and often counter-intuitive insights and performance tradeoffs among the different transports. We find that InfiniBand with native RDMA is able to deliver the best performance among all tested networking protocols in most experiments. Contrary to expectation, IPoIB could achieve better CPU utilization and lower tail-latency for large I/O operations in some of our experiments. We believe that our analysis helps gain insight into the deployment implications of NVMe-oF in datacenters.
This work was supported in part by NSF research grant CCF #1822987.
This work was done by the authors A. Kashyap and X. Lu while at Ohio State University.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Chelsio Communications. NVM Express over Fabrics (2014). http://www.chelsio.com/wp-content/uploads/resources/NVM_Express_Over_Fabrics.pdf
Guz, Z., (Huan) Li, H., Shayesteh, A., Balakrishnan, H.: NVMe-over-Fabrics performance characterization and the path to low-overhead flash disaggregation. In: Proceedings of the 10th ACM International Systems and Storage Conference, SYSTOR 2017 (2017)
Guz, Z., (Huan) Li, H., Shayesteh, A., Balakrishnan, V.: Performance characterization of NVMe-over-fabrics storage disaggregation. ACM Trans. Storage 14(4), 1–18 (2018)
Han, D., Nam, B.: Improving access to HDFS using NVMeoF. In: 2019 IEEE International Conference on Cluster Computing (CLUSTER), pp. 1–2 (2019)
IBM Research - Zurich. Crail (2017). http://www.crail.io/
Intel. Intel Optane Memory. https://www.intel.com/OptaneMemory
Intel. SPDK NVMe over Fabrics Target (2017). https://spdk.io/doc/nvmf.html
Intel. SPDK NVMe over Fabrics Target Programming Guide (2017). https://spdk.io/doc/nvmf.html
Jia, Y., Anger, E., Chen, F.: When NVMe over fabrics meets arm: performance and implications. In: 2019 35th Symposium on Mass Storage Systems and Technologies (MSST), pp/ 134–140 (2019)
Joglekar, A., Kounavis, M.E., Berry, F.L.: A scalable and high performance software ISCSI implementation. In: Proceedings of the 4th Conference on USENIX Conference on File and Storage Technologies, FAST 2005, vol. 4, p. 20, USA. USENIX Association (2005)
Khosravi, H.M., Joglekar, A., Iyer, R.: Performance characterization of iSCSI processing in a server platform. In: 2005 24th IEEE International Performance, Computing, and Communications Conference, PCCC 2005, pp. 99–107 (2005)
Kim, J., Fair, D.: How ethernet RDMA protocols iWARP and RoCE support NVMe over fabrics (Ethernet Storage Forum) (2016). https://www.snia.org/sites/default/files/ESF/How_Ethernet_RDMA_ Protocols_Support_NVMe_over_Fabrics_Final.pdf
Klimovic, A., Kozyrakis, C., Thereska, E., John, B., Kumar, S.: Flash storage disaggregation. In Proceedings of the Eleventh European Conference on Computer Systems, EuroSys 2016 (2016)
Klimovic, A., Litz, H., Kozyrakis, C.: ReFlex: remote flash \(\approx \) local flash. In: Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2017, pp. 345–359 (2017)
Klimovic, A., Wang, Y., Stuedi, P., Trivedi, A., Pfefferle, J., Kozyrakis, C.: Pocket: elastic ephemeral storage for serverless analytics. In: Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI 2018, pp. 427–444, USA. USENIX Association (2018)
Mickens, J., et al.: Blizzard: fast, cloud-scale block storage for cloud-oblivious applications. In: 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2014), pp. 257–273 (2014)
Dave M., Metz, J.: Under the Hood with NVMe over fabrics. In: Proceedings of the Ethernet Storage Forum (2015)
NVM Express (2011). https://nvmexpress.org/
NVMe-over-Fabrics Specification (2019). https://nvmexpress.org/developers/nvme-of-specification/
SPDK NVMe perf Benchmark. https://github.com/spdk/spdk/tree/master/examples/nvme/perf
SPDK 20.04 NVMe-oF RDMA Performance Report. https://ci.spdk.io/download/performance-reports/SPDK_rdma_perf_report_2004.pdf
SPDK 20.04 NVMe-oF TCP Performance Report. https://ci.spdk.io/download/performance-reports/SPDK_tcp_perf_report_2004.pdf
Linux sysstat. https://github.com/sysstat/sysstat
Xu, Q., et al.: Performance analysis of containerized applications on local and remote storage. In: International Conference on Massive Storage Systems and Technology (2017)
Yang, Z., et al.: SPDK: a development kit to build high performance storage applications. In: 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 154–161 (2017)
Yang, Z., Wan, Q., Cao, G., Latecki, K.: uNVMe-TCP: a user space approach to optimizing NVMe over fabrics TCP transport. In: Hsu, C.-H., Kallel, S., Lan, K.-C., Zheng, Z. (eds.) IOV 2019. LNCS, vol. 11894, pp. 125–142. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-38651-1_13
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kashyap, A., Gugnani, S., Lu, X. (2021). Impact of Commodity Networks on Storage Disaggregation with NVMe-oF. In: Wolf, F., Gao, W. (eds) Benchmarking, Measuring, and Optimizing. Bench 2020. Lecture Notes in Computer Science(), vol 12614. Springer, Cham. https://doi.org/10.1007/978-3-030-71058-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-71058-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-71057-6
Online ISBN: 978-3-030-71058-3
eBook Packages: Computer ScienceComputer Science (R0)