Sang-Hoon Kim
Skip Abstract Section
Abstract
There have been drastic changes in the storage device landscape recently. At the center of the diverse storage landscape lies the NVMe interface, which allows high-performance and flexible communication models required by these next-generation device types. However, its hardware-oriented definition and specification are bottlenecking the development and evaluation cycle for new revolutionary storage devices. Furthermore, existing emulators lack the capability to support the advanced storage configurations that are currently in the spotlight.
In this article, we present NVMeVirt, a novel approach to facilitate software-defined NVMe devices. A user can define any NVMe device type with custom features, and NVMeVirt allows it to bridge the gap between the host I/O stack and the virtual NVMe device in software. We demonstrate the advantages and features of NVMeVirt by realizing various storage types and configurations, such as conventional SSDs, low-latency high-bandwidth NVM SSDs, zoned namespace SSDs, and key-value SSDs with the support of PCI peer-to-peer DMA and NVMe-oF target offloading. We also make cases for storage research with NVMeVirt, such as studying the performance characteristics of database engines and extending the NVMe specification for the improved key-value SSD performance.
- [1] . 2011. vIOMMU: Efficient IOMMU emulation. In Proceedings of the 2011 USENIX Annual Technical Conference (ATC’11). USENIX Association.Google Scholar
- [2] . n. d. fio: Flexible I/O Tester. https://github.com/axboe/fioGoogle Scholar
- [3] . 2021. FlashNeuron: SSD-enabled large-batch training of very deep neural networks. In Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST’21). USENIX Association.Google Scholar
- [4] . 2021. ZNS: Avoiding the block interface tax for flash-based SSDs. In Proceedings of the 2021 USENIX Annual Technical Conference (ATC’21). USENIX Association.Google Scholar
- [5] . 2017. LightNVM: The Linux open-channel SSD subsystem. In Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST’17). USENIX Association, 359–375.Google Scholar
- [6] . n. d. ForestDB Benchmark. https://github.com/couchbaselabs/ForestDB-BenchmarkGoogle Scholar
- [7] db ranking. n. d. DB-Engines Ranking. https://db-engines.com/en/rankingGoogle Scholar
- [8] . n. d. ZeRO-Inference: Democratizing massive model inference. https://www.deepspeed.ai/2022/09/09/zero-inference.htmlGoogle Scholar
- [9] . 2020. Cost-effective, energy-efficient, and scalable storage computing for large-scale AI applications. ACM Transactions on Storage 16, 4, Article
21 (2020), 37 pages.Google ScholarDigital Library
- [10] . n. d. Samsung Enterprise SSD. https://www.samsung.com/semiconductor/ssd/enterprise-ssd/Google Scholar
- [11] . n. d. NVMe-oF Specification. https://nvmexpress.org/developers/nvme-of-specification/Google Scholar
- [12] . 2022. Reprogramming 3D TLC flash memory based solid state drives. ACM Transactions on Storage 18, 9 (
May 2022), 133 pages.Google Scholar - [13] . 2018. Amber: Enabling precise full-system simulation with detailed modeling of all SSD resources. In Proceedings of the 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO’18). 469–481.Google ScholarDigital Library
- [14] . 2016. Biscuit: A framework for near-data processing of big data workloads. In Proceedings of the 43rd ACM/IEEE International Symposium on Computer Architecture (ISCA’16). 153–165.Google ScholarDigital Library
- [15] . 2021. ZNS+: Advanced zoned namespace interface for supporting in-storage zone compaction. In Proceedings of the 15nd USENIX Symposium on Operating Systems Design and Implementation (OSDI’21).Google Scholar
- [16] . n. d. MySQLTuner. https://github.com/major/MySQLTuner-perlGoogle Scholar
- [17] . 2018. Achieving defect-free multilevel 3D flash memories with one-shot program design. In Proceedings of the 55th ACM/ESDA/IEEE Annual Design Automation Conference (DAC’18). 6 pages.Google ScholarDigital Library
- [18] . 2020. PinK: High-speed in-storage key-value store with bounded tails. In Proceedings of the 2020 USENIX Annual Technical Conference (ATC’20), 173–187.Google Scholar
- [19] . n. d. Intel I/O Acceleration Technology. https://www.intel.co.kr/content/www/kr/ko/wireless-network/accel-technology.htmlGoogle Scholar
- [20] . n. d. Intel Optane SSD 9 Series. https://www.intel.com/content/www/us/en/products/details/memory-storage/consumer-ssds/optane-ssd-9-series.htmlGoogle Scholar
- [21] . n. d. Basic Performance Measurements of the Intel Optane DC Persistent Memory Module.
arxiv:1903.05714 http://arxiv.org/abs/1903.05714Google Scholar - [22] . 2012. PCI Express Technology. MindShare Technology.Google ScholarDigital Library
- [23] . 2017. KAML: A flexible, high-performance key-value SSD. In Proceedings of the 23rd IEEE Symposium on High Performance Computer Architecture (HPCA’17). IEEE, 373–384.Google ScholarCross Ref
- [24] . 2018. GraFBoost: Using accelerated flash storage for external graph analytics. In Proceedings of the 45rd ACM/IEEE International Symposium on Computer Architecture (ISCA’18). Los Angeles, CA.Google ScholarDigital Library
- [25] . 2020. OpenExpress: Fully hardware automated open research framework for future fast NVMe devices. In Proceedings of the 2020 USENIX Annual Technical Conference (ATC’20).Google Scholar
- [26] . 2017. SimpleSSD: Modeling solid state drives for holistic system simulation. IEEE Computer Architecture Letters 17 (
Sept. 2017), 37–41.Google ScholarDigital Library - [27] . 2019. Towards building a high-performance, scale-in key-value storage system. In Proceedings of the 12th ACM International Conference on Systems and Storage (SYSTOR’19). 144–154.Google ScholarDigital Library
- [28] . 2016. NVMeDirect: A user-space I/O framework for application-specific optimization on NVMe SSDs. In Proceedings of the 8th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage’16).Google ScholarDigital Library
- [29] . 2021. Performance modeling and practical use cases for black-box SSDs. ACM Transactions on Storage 17, 2, Article
14 (June 2021), 38 pages.Google ScholarDigital Library - [30] . 2019. Transaction support using compound commands in key-value SSDs. In Proceedings of the 11th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage’19).Google ScholarDigital Library
- [31] kvceph. n. d. Open memory platform development kit. http://github.com/OpenMPDKGoogle Scholar
- [32] . 2020. Cosmos+ OpenSSD: Rapid prototype for flash storage systems. ACM Transactions on Storage 16, 3, Article
15 (Aug. 2020), 35 pages.Google ScholarDigital Library - [33] . 2022. Hardware/software co-programmable framework for computational SSDs to accelerate deep learning service on large-scale graphs. In Proceedings of the 20th USENIX Conference on File and Storage Technologies (FAST’22). USENIX Association.Google Scholar
- [34] . n. d. The DiskSim Simulation Environment v4.0. https://www.pdl.cmu.edu/DiskSim/index.shtmlGoogle Scholar
- [35] . 2016. ActiveSort: Efficient external sorting using active SSDs in the MapReduce framework. Future Generation Computer Systems 65, C (
Dec. 2016), 76–89.Google ScholarDigital Library - [36] . 2018. The CASE of FEMU: Cheap, accurate, scalable and extensible flash emulator. In Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST’18).Google Scholar
- [37] . 2019. Cognitive SSD: A deep learning engine for in-storage data retrieval. In Proceedings of the 2019 USENIX Annual Technical Conference (ATC’19). USENIX Association.Google Scholar
- [38] . 2013. Extending the lifetime of flash-based storage through reducing write amplification from file systems. In Proceedings of the 11th USENIX Conference on File and Storage Technologies (FAST’13). USENIX Association, 257–270.Google ScholarDigital Library
- [39] . 2016. FlexDrive: A framework to explore NVMe storage solutions. In Proceedings of the 2016 IEEE 18th International Conference on High Performance Computing and Communications. 1115–1122.Google ScholarCross Ref
- [40] . 2017. FlashStorageSim: Performance modeling for SSD architectures. In Proceedings of the 2017 International Conference on Networking, Architecture, and Storage (NAS’17). 1–2.Google ScholarCross Ref
- [41] . n. d. Megatron-DeepSpeed. https://github.com/microsoft/Megatron-DeepSpeedGoogle Scholar
- [42] . n. d. FNV Hash. http://isthe.com/chongo/tech/comp/fnv/Google Scholar
- [43] . n. d. GPUDirect Storage: A Direct Path Between Storage and GPU Memory. https://developer.nvidia.com/blog/gpudirect-storageGoogle Scholar
- [44] . n. d. The NVIDIA Magnum IO GPUDIrect Storage Overview Guide. https://docs.nvidia.com/gpudirect-storage/overview-guide/index.htmlGoogle Scholar
- [45] . 2014. SDF: Software-defined flash for web-scale internet storage systems. In Proceedings of the 19th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’14).Google ScholarDigital Library
- [46] . n. d. PCI SIG. https://pcisig.comGoogle Scholar
- [47] pgtune. n. d. PGTune: configuration for PostgreSQL based on the maximum performance for a given hardware configuration. https://pgtune.leopard.in.ua/Google Scholar
- [48] . n. d. QEMU: A Generic and Open Source Machine Emulator and Virtualizer. https://qemu.orgGoogle Scholar
- [49] . n. d. Metaseq. https://github.com/facebookresearch/metaseqGoogle Scholar
- [50] . 2021. KV-SSD: What is it good for? In Proceedings of the 58th ACM/IEEE Annual Design Automation Conference (DAC’21). 1105–1110.Google ScholarDigital Library
- [51] . n. d. OpenMPDK KVSSD. https://github.com/OpenMPDK/KVSSD/Google Scholar
- [52] sysbench. n. d. Scriptable database and system performance benchmark. https://github.com/akopytov/sysbenchGoogle Scholar
- [53] . 2018. MQSim: A framework for enabling realistic studies of modern multi-queue SSD devices. In Proceedings of the16th USENIX Conference on File and Storage Technologies (FAST’18).Google Scholar
- [54] . n. d. Samsung 970 Pro 512 GB. https://www.techpowerup.com/ssd-specs/samsung-970-pro-512-gb.d54Google Scholar
- [55] . 2019. An early evaluation of Intel’s Optane DC persistent memory module and its impact on high-performance scientific applications. In Proceedings of the 2019 International Conference for High Performance Computing, Networking, Storage and Analysis (SC’19).Google ScholarDigital Library
- [56] . 2021. RecSSD: Near data processing for solid state drive based recommendation inference. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’21).Google ScholarDigital Library
- [57] . 2019. Towards an unwritten contract of Intel Optane SSD. In Proceedings of the 11th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage’19).Google ScholarDigital Library
- [58] . 2017. SPDK: A development kit to build high performance storage applications. In Proceedings of the 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom’17). 154–161.Google ScholarCross Ref
- [59] . 2013. VSSIM: Virtual machine based SSD simulator. In Proceedings of the 29th IEEE Conference on Massive Data Storage (MSST’13).Google ScholarCross Ref
- [60] . 2012. De-indirection for flash-based SSDs with nameless writes. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST’12). USENIX Association.Google ScholarDigital Library
Index Terms
- Empowering Storage Systems Research with NVMeVirt: A Comprehensive NVMe Device Emulator
Recommendations
Leveraging NVMe SSDs for Building a Fast, Cost-effective, LSM-tree-based KV Store
Key-value (KV) stores support many crucial applications and services. They perform fast in-memory processing but are still often limited by I/O performance. The recent emergence of high-speed commodity non-volatile memory express solid-state drives (NVMe ...
An Efficient Memory-Mapped Key-Value Store for Flash Storage
SoCC '18: Proceedings of the ACM Symposium on Cloud ComputingPersistent key-value stores have emerged as a main component in the data access path of modern data processing systems. However, they exhibit high CPU and I/O overhead. Today, due to power limitations it is important to reduce CPU overheads for data ...
Migrating server storage to SSDs: analysis of tradeoffs
EuroSys '09: Proceedings of the 4th ACM European conference on Computer systemsRecently, flash-based solid-state drives (SSDs) have become standard options for laptop and desktop storage, but their impact on enterprise server storage has not been studied. Provisioning server storage is challenging. It requires optimizing for the ...
Comments