Skip to main content

Advertisement

SpringerLink
Log in

Virtualize and share non-volatile memories in user space

  • Regular Paper
  • Published:
CCF Transactions on High Performance Computing Aims and scope Submit manuscript

Abstract

Emerging non-volatile memory (NVM) has attractive characteristics such as DRAM-like low-latency together with the non-volatility of storage devices. Recently, byte-addressable, memory bus-attached NVM has become available. This paper addresses the problem of combining a smaller, faster byte-addressable NVM with a larger, slower storage device, such as SSD, to create the impression of a larger and faster byte-addressable NVM which can be shared across multiple applications concurrently. In this paper, we propose vNVML, a user space library for virtualizing and sharing NVM. vNVML provides for applications transaction-like memory semantics that ensures write ordering, durability, and persistency guarantees across system failures. vNVML exploits DRAM for read caching to improve performance and potentially to reduce the number of writes to NVM, extending the NVM lifetime. vNVML is implemented in C and evaluated with realistic workloads to show that vNVML allows applications to share NVM efficiently, both in a single OS and when docker-like containers are employed. The results from the evaluation show that vNVML incurs less than 10% overhead while providing the benefits of an expanded virtualized NVM space to the applications, and allowing applications to safely share the virtual NVM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. Note: while our approach can be applied with magnetic disks as the backing stores, here we limit ourselves to SSDs.

  2. Note: the “share” here means the same physical NVM pages can be reallocated/reused by multiple applications.

  3. Note: our (vNVML) privatemmap mode is different from the standard POSIX privatemmap mode. In our privatemmap, written data can be reflected back to storage devices, but POSIX privatemmap cannot.

  4. Note: the “share” here means that shared regions can be seen and accessed by multiple processes, exactly like the existing POSIX sharedmmap method for volatile memory.

  5. Note: our platform does not have enough NVM to accommodate all files generated by eight instances inserted 10 K as baseline, respectively. Also, when eight MongoDB instances adopting vNVML are running concurrently, some of instances would crash because of out-of-memory (OOM) error from privatemmap. Therefore, executing at most four instances concurrently is the limitation of our platform.

  6. Note: since all four throughputs from YCSB are almost the same (with usually less than 1% difference).

  7. At (a), the normalized throughput is 0.876 (0.776, respectively) when log buffer is 2 GB (128 MB, respectively) and inserted records are 30,000. At (b), the normalized throughput is 0.638 (0.542, respectively) when log buffer is 2 GB (128 MB, respectively) and inserted records are 30,000.

References

  • Chen, A.: A review of emerging non-volatile memory (NVM) technologies and applications. Solid-State Electron. 125, 25–38 (2016)

    Article  Google Scholar 

  • Coburn, J., Caulfield, A.M,, Akel, A., Grupp, L.M., Gupta, R.K., Jhala, R., Swanson, S.: NV-heaps: making persistent objects fast and safe with next-generation, non-volatile memories. In: ASPLOS XVI Proceedings of the Sixteenth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 105–118. ACM, Newport Beach (2011)

  • Condit, J., Nightingale, E.B., Frost, C., Ipek, E., Lee, B., Burger, D., Coetzee, D.: Better i/o through byte-addressable, persistent memory. In: SOSP ’09 Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles. ACM, Big Sky, Montana (2009)

  • Cooper, B.F., Silberstein, A., ErwinTam, Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with YCSB. In: SoCC ’10 Proceedings of the 1st ACM Symposium on Cloud Computing, pp 143–154. ACM, Indianapolis (2010)

  • Docker. Docker container. https://www.docker.com/ (2013)

  • Docker (2018) Docker container bind mounts. https://docs.docker.com/storage/bind-mounts/

  • Doshi, K., Giles, E.R., Varman, P.: Atomic persistence for scm with a non-intrusive backend controller. In: HPCA ’16 Proceedings of the IEEE International Symposium on High Performance Computer Architecture. IEEE, Barcelona (2016)

  • Dulloor, S.R., Kumar, S., Keshavamurthy, A., Lantz, P., Reddy, D., Sankaran, R., Jackson, J.: System software for persistent memory. In: EuroSys ’14 Proceedings of the Ninth European Conference on Computer Systems, ACM, Amsterdam (2014)

  • Eisner, L.A., Mollov, T., Swanson, S.: Quill: Exploiting fast non-volatile memory by transparently bypassing the file system. In: Technical report, UCSD (2013)

  • Fedorov, V., Kim, J., Qin, M., Gratz, P.V., Reddy, A.L.N.: Speculative paging for future NVM storage. In: MEMSYS ’17 Proceedings of the International Symposium on Memory Systems, Alexandria, Virginia (2017)

  • Giles, E.R., Doshi, K., Varman, P.: Softwrap: A lightweight framework for transactional support of storage class memory. In: MSST ’15 Proceedings of the 31st Symposium on Mass Storage Systems and Technologies, pp. 1–14. IEEE, Santa Clara (2015)

  • Intel.: Persistent memory development kit (2015). https://pmem.io/pmdk/

  • Intel. Storage performance development kit (2017). https://spdk.io/

  • Intel: Intel optane dc persistent memory (2019). https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html

  • Joshi, A., Nagarajan, V., Viglas, S., Cintra, M.: Atom: Atomic durability in non-volatile memory through hardware logging. In: HPCA ’17 Proceedings of the IEEE International Symposium on High Performance Computer Architecture. IEEE, Austin (2017)

  • Jung, J., Won, Y., Kim, E., Shin, H., Jeon, B.: Frash: exploiting storage class memory in hybrid file system for hierarchical storage. ACM Trans. Storage (TOS) 6(1), 1–25 (2010)

    Article  Google Scholar 

  • Kwon, Y., Fingler, H., Hunt, T., Peter, S., Witchel, E., Anderson. T.: Strata: A cross media file system. In: SOSP ’17 Proceedings of the 26th Symposium on Operating Systems Principles, pp. 460–477. ACM, Shangha (2017)

  • Lee, B.C., Zhou, P., Yang, J., Zhang, Y., Zhao, B., Ipek, E., Mutlu, O., Burger, D.: Phase-change technology and the future of main memory. IEEE Micro 30, 131–141 (2010)

    Article  Google Scholar 

  • Liang, L., Chen, R., Chen, H., Xia, Y., Park, K., Zang, B., Guan, H.: A case for virtualizing persistent memory. In: SoCC ’16 Proceedings of the Seventh ACM Symposium on Cloud Computing. ACM, Santa Clara (2016)

  • Memaripour, A., Badam, A., Phanishayee, A., Zhou, Y., Alagappan, R., Strauss, K., Swanson, S.: Atomic in-place updates for non-volatile main memories with kamino-tx. In: EuroSys ’17 Proceedings of the Twelfth European Conference on Computer Systems, pp. 499–512.ACM, Belgrade (2017)

  • Microsoft: Transaction isolation levels (2017). https://docs.microsoft.com/en-us/sql/odbc/reference/develop-app/transaction-isolation-levels?view=sql-server-2017

  • mongoDB (2008a) Mmapv1 storage engine. https://docs.mongodb.com/manual/core/mmapv1/

  • mongoDB (2008b) Mongodb data base. https://github.com/mongodb

  • Narayanan, D., Hodson, O.: Whole-system persistence. In: ASPLOS XVII Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems. ACM, London (2012)

  • Pelley, S., Chen, P.M., Wenisch, T.F.: Memory persistency. ISCA ’14 Proceeding of the 41st annual international symposium on Computer architecuture, pp. 265–276. Minneapolis, Minnesota (2014)

  • Qiu, S., Reddy, A.L.N.: Nvmfs: a hybrid file system for improving random write in NAND-flash SSD. In: MSST ’13 IEEE 29th Symposium on Mass Storage Systems and Technologies. IEEE, Long Beach (2013)

  • Qureshi, M.K., Srinivasan, V., Rivers, J.A.: Scalable high performance main memory system using phase-change memory technology. In: ISCA ’09 Proceedings of the 36th Annual International Symposium on Computer Architecture. ACM, Austin (2009)

  • Samsung (2017) Ultra-low latency with Samsung z-nand ssd. https://www.samsung.com/semiconductor/global.semi.static/Ultra-Low_Latency_with_Samsung_Z-NAND_SSD-0.pdf

  • SNIA: NVM Programming Model. In: Storage Networking Industry Association, rev., vol. 1, no. 2 (2017)

  • Swanson, S.: A vision of persistence (2017). https://www.sigarch.org/a-vision-of-persistence/

  • Swift, M.: Persistent memory ordering (2015). http://materials.dagstuhl.de/files/15/15021/15021.MichaelSwift1.Slides.pdf

  • Venkataraman, S., Tolia, N., Ranganathan, P., Campbell, R.H.: Consistent and durable data structures for non-volatile byte-addressable memory. In: FAST ’11 Proceedings of the 9th USENIX Conference on File and Storage Technologies, USENIX (2011)

  • Volos, H., Tack, A.J., Swift, M.M.: Mnemosyne: Lightweight persistent memory. In: ASPLOS XVI Proceedings of the Sixteenth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 91–104. ACM, Newport Beach, California (2011)

  • Volos, H., Nalli, S., Panneerselvam, S., Varadarajan, V., Saxena, P., Swift, M.M.: Aerie: Flexible file-system interfaces to storage-class memory. In: EuroSys ’14 Proceedings of the Ninth European Conference on Computer Systems. ACM, Amsterdam (2014)

  • Wan, H., Lu, Y., Xu, Y., Shu, J.: Empirical study of redo and undo logging in persistent memory. In: NVMSA ’16 Proceeding of the 5th Non-Volatile Memory Systems and Applications Symposium, pp. 1–6. IEEE, Daegu (2016)

  • Wang, C., Wei, Q., Yang, J., Chen, C., Xue, M.: How to be consistent with persistent memory? an evaluation approach. NAS ’15, pp. 186–194. IEEE, Boston (2015)

  • Watts, D.: Intel optane dc persistent memory product guide (2019). https://lenovopress.com/lp1066-intel-optane-dc-persistent-memory

  • Wu, X., Reddy, A.L.N., (2011) SCMFS: a file system for storage class memory. In: SC ’11 Proceedings of International Conference for High Performance Computing. Networking, Storage and Analysis. ACM, Seattle (2011)

  • Xu, J., Swanson, S.: Nova: A log-structured file system for hybrid volatile/non-volatile main memories. In: FAST ’16 Proceedings of the 14th USENIX Conference on File and Storage Technologies, pp. 323–338. USENIX, Santa Clara (2016)

  • Yang, J., Wei, Q., Chen, C., Wang, C., Yong, K.L., He, B.: NV-Tree: Reducing consistency cost for NVM-based single level systems. In: FAST ’15 Proceedings of the 13th USENIX Conference on File and Storage Technologies. USENIX, Santa Clara (2015)

  • Yu, S., Chen, P.Y.: Emerging memory technologies: recent trends and prospects. IEEE Solid-State Circuits Mag. 8(2), 43–56 (2016)

    Article  MathSciNet  Google Scholar 

  • Zhang, Y., Swanson, S.: A study of application performance with non-volatile main memory. In: MSST ’15 Proceedings of the 31st Symposium on Mass Storage Systems and Technologies. IEEE, Santa Clara (2015)

  • Zhang, Y., Yang, J., Memaripour, A., Swanson, S.: Mojim: a reliable and highly-available non-volatile memory system. In: ASPLOS ’15 Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems. ACM, Istanbul (2015)

  • Zhao, J., Li, S., Yoon, D.H., Xie, Y., Jouppi, N.P.: Kiln: closing the performance gap between systems with and without persistence support. In: MICRO ’13 Proceedings of the 46th Annual IEEE/ACM International Symposium on Microarchitecture. IEEE, Davis (2013)

  • Zhou, P., Zhao, B., Yang, J., Zhang, Y.: A durable and energy efficient main memory using phase change memory technology. In: ISCA ’09 Proceedings of the 36th Annual International Symposium on Computer Architecture. ACM, Austin (2009)

Download references

Acknowledgements

The conference version of this paper was presented in MSST 2019. We thank the anonymous reviewers for their valuable comments and feedback to improve the content and quality of this work. We also thank the National Science Foundation, which supports this work through Grants I/UCRC-1439722 and FoMR-1823403, and generous support from Hewlett Packard Enterprise.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA

    Chih Chieh Chou, A. L. Narasimha Reddy & Paul V. Gratz

  2. Samsung Semiconductor, San Jose, CA, USA

    Jaemin Jung

  3. Boise, ID, USA

    Doug Voigt

Authors
  1. Chih Chieh Chou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jaemin Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. L. Narasimha Reddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Paul V. Gratz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Doug Voigt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chih Chieh Chou.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chou, C.C., Jung, J., Reddy, A.L.N. et al. Virtualize and share non-volatile memories in user space. CCF Trans. HPC 2, 16–35 (2020). https://doi.org/10.1007/s42514-020-00019-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42514-020-00019-8

Keywords

Search

Navigation