Abstract
The NVMe-over-Fabrics (NVMeoF) protocol enables high-performance Protocol Data Units (PDUs) exchanges between hosts and remote NVMe controllers. The performance benefits of NVMeoF are mainly derived from the multiple deep queue pairs for parallel PDUs transfers. NVMeoF has significantly facilitated NVMe SSD disaggregation from compute nodes for better resource utilization and scaling independence. However, as the performance of NVMe SSD and network infrastructure increases, the near-perfect performance delivery of NVMeoF is harder to achieve. The primary reason is the increased CPU interrupts and performance interference originated from the I/O requests served by the same NVMeoF queue pair.
In this paper, we investigate how intra-queue requests are mutually affected, and propose PINoF, a Performance Isolated remote storage access mechanism for NVMe-over-Fabrics. PINoF separates CMD and Data PDUs resources in each NVMeoF queue pair to achieve intra-queue I/O isolation, transfers PDUs in batch along with read or write specific I/O path to achieve isolated interrupt-coalescing, and introduces differentiated PDU reordering schemes to achieve isolated scheduling. Our experimental results demonstrate that compared with state-of-the-art NVMeoF implementations, PINoF achieves 23.92% lower latency, increases bandwidth by up to 19.59%, and improves IOPS by 12.41% on average.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ahmad, I., Gulati, A., Mashtizadeh, A.: vIC: interrupt coalescing for virtual machine storage device IO. In: 2011 USENIX Annual Technical Conference, pp. 45–58 (2011)
Bjørling, M., Axboe, J., Nellans, D., Bonnet, P.: Linux block IO: introducing multi-queue SSD access on multi-core systems. In: Proceedings of the 6th international systems and storage conference, pp. 1–10 (2013)
Cheng, L., Wang, C.L.: Network performance isolation for latency-sensitive cloud applications. Future Gener. Comput. Syst. 29(4), 1073–1084 (2013)
Cobb, D., Huffman, A.: Nvm express and the PCI express SSD revolution. In: Intel Developer Forum (2012)
Cohen, D., Talpey, T., Kanevsky, A., Cummings, U., Krause, M.: Remote direct memory access over the converged enhanced ethernet fabric: evaluating the options. In: 2009 17th IEEE Symposium on High Performance Interconnects, pp. 123–130 (2009)
Dragojević, A., Narayanan, D., Castro, M., Hodson, O.: Farm: fast remote memory. In: 11th USENIX Symposium on Networked Systems Design and Implementation, pp. 401–414 (2014)
Gao, P.X., Narayan, A., Karandikar, S., Carreira, J., HAN: network requirements for resource disaggregation. In: 12th USENIX Symposium on Operating Systems Design and Implementation, pp. 249–264 (2016)
Guz, Z., Li, H., Shayesteh, A., Balakrishnan, V.: Performance characterization of NVMe-over-fabrics storage disaggregation. ACM Trans. Storage 14(4), 1–18 (2018)
Hwang, J., Cai, Q., Tang, A., Agarwal, R.: Tcp\(\approx \)rdma: CPU-efficient remote storage access with i10. In: 17th USENIX Symposium on Networked Systems Design and Implementation, pp. 127–140 (2020)
Hwang, J., Vuppalapati, M., Peter, S., Agarwal, R.: Rearchitecting linux storage stack for \(\mu \)s latency and high throughput (2021)
Kaufmann, A., Stamler, T., Peter, S., Sharma, N.: TAS: TCP acceleration as an OS service. In: Proceedings of the Fourteenth EuroSys Conference, pp. 1–16 (2019)
Lee, M., Kang, D.H., Lee, M., Eom, Y.I.: Improving read performance by isolating multiple queues in NVMe SSDs. In: International Conference on Ubiquitous Information Management and Communication, p. 36 (2017)
Li, Y.T., Leith, D., Shorten, R.N.: Experimental evaluation of TCP protocols for high-speed networks. IEEE/ACM Trans. Netw. 15(5), 1109–1122 (2007)
Marinos, I., Watson, R.N., Handley, M.: Network stack specialization for performance. ACM SIGCOMM Comput. Commun. Rev. 44(4), 175–186 (2014)
Minturn, D.: Nvm express over fabrics. In: 11th Annual OpenFabrics International OFS Developers’ Workshop (2015)
Nguyen, et al.: Reducing smartphone application delay through read/write isolation. In: the 13th Annual International Conference (MobiSys 15), pp. 287–300 (2015)
Qiao, X., Xie, X., Xiao, L.: Load-aware transmission mechanism for NVMeoF storage networks. In: International Conference on High Performance Computing and Communication (HPCCE 2021), pp. 105–112 (2022)
Son, Y., Kang, H., Han, H., Yeom, H.Y.: An empirical evaluation of nvm express SSD. In: International Conference on Cloud & Autonomic Computing (ICCAC 15), pp. 275–282 (2015)
Tai, A., Smolyar, I., Wei, M., Tsafrir, D.: Optimizing storage performance with calibrated interrupts. In: Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation, pp. 129–145 (2021)
Zheng, S., Hoseinzadeh, M., Swanson, S.: Ziggurat: a tiered file system for non-volatile main memories and disks. In: 17th USENIX Conference on File and Storage Technologies, pp. 207–219 (2019)
Acknowledgements
We would like to thank the NPC reviewers for their insightful feedback. This work was supported in part by Excellent Youth Foundation of Hunan Province under Grant No.2021JJ10050 and the Science Foundation of NUDT under grant ZK21-03. Dezun Dong and Xuchao Xie are the corresponding authors.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Gu, W., Xie, X., Dong, D. (2022). Alleviating Performance Interference Through Intra-Queue I/O Isolation for NVMe-over-Fabrics. In: Liu, S., Wei, X. (eds) Network and Parallel Computing. NPC 2022. Lecture Notes in Computer Science, vol 13615. Springer, Cham. https://doi.org/10.1007/978-3-031-21395-3_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-21395-3_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21394-6
Online ISBN: 978-3-031-21395-3
eBook Packages: Computer ScienceComputer Science (R0)