skip to main content
research-article

HiNFS: A Persistent Memory File System with Both Buffering and Direct-Access

Published: 04 April 2018 Publication History

Abstract

Persistent memory provides data persistence at main memory with emerging non-volatile main memories (NVMMs). Recent persistent memory file systems aggressively use direct access, which directly copy data between user buffer and the storage layer, to avoid the double-copy overheads through the OS page cache. However, we observe they all suffer from slow writes due to NVMMs’ asymmetric read-write performance and much slower performance than DRAM.
In this article, we propose HiNFS, a high-performance file system for non-volatile main memory, to combine both buffering and direct access for fine-grained file system operations. HiNFS uses an NVMM-aware Write Buffer to buffer the lazy-persistent file writes in DRAM, while performing direct access to NVMM for eager-persistent file writes. It directly reads file data from both DRAM and NVMM, by ensuring read consistency with a combination of the DRAM Block Index and Cacheline Bitmap to track the latest data between DRAM and NVMM. HiNFS also employs a Buffer Benefit Model to identify the eager-persistent file writes before issuing I/Os. Evaluations show that HiNFS significantly improves throughput by up to 184% and reduces execution time by up to 64%comparing with state-of-the-art persistent memory file systems PMFS and EXT4-DAX.

References

[1]
Dbt2 test suite. Retrieved from http://sourceforge.net/apps/mediawiki/osdldbt.
[2]
Filebench 1.4.9.1. Retrieved from http://sourceforge.net/projects/filebench/.
[3]
FIU system call io trace. Retrieved from http://sylab-srv.cs.fiu.edu/dokuwiki/doku.php?id=projects:nbw:start.
[4]
Flexible IO (fio) Tester. Retrieved from http://freecode.com/projects/fio.
[5]
Lasr system call io trace. Retrieved from http://iotta.snia.org/historical_section?tracetype_id=1.
[6]
2014. Support ext4 on NV-DIMMs. Retrieved from http://lwn.net/Articles/588218/.
[7]
2014. Supporting file systems in persistent memory. Retrieved from https://lwn.net/Articles/610174/.
[8]
Geoffrey W. Burr, Matthew J. Breitwisch, Michele Franceschini, Davide Garetto, Kailash Gopalakrishnan, Bryan Jackson, Bülent Kurdi, Chung Lam, Luis A. Lastras, Alvaro Padilla, Bipin Rajendran, Simone Raoux, and Rohit S. Shenoy. 2010. Phase change memory technology. J. Vacuum Sci. Technol. B 28, 2 (2010), 223--262.
[9]
Daniel Campello, Hector Lopez, Ricardo Koller, Raju Rangaswami, and Luis Useche. 2015. Non-blocking writes to files. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST’15). 151--165.
[10]
Shimin Chen, Phillip B. Gibbons, and Suman Nath. 2011. Rethinking database algorithms for phase change memory. In Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR’11). 21--31.
[11]
Joel Coburn, Adrian M. Caulfield, Ameen Akel, Laura M. Grupp, Rajesh K. Gupta, Ranjit Jhala, and Steven Swanson. 2011. NV-heaps: Making persistent objects fast and safe with next-generation, non-volatile memories. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’11). 105--118.
[12]
Edward Grady Coffman and Peter J. Denning. 1973. Operating Systems Theory. Vol. 973.
[13]
Jeremy Condit, Edmund B. Nightingale, Christopher Frost, Engin Ipek, Benjamin Lee, Doug Burger, and Derrick Coetzee. 2009. Better I/O through byte-addressable, persistent memory. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (SOSP’09). 133--146.
[14]
Intel Cooperation. 2015. NVDIMM Namespace Specification. Retrieved from http://pmem.io/documents/NVDIMM_ Namespace_Spec.pdf.
[15]
Intel Cooperation. 2016. Intel Architecture Instruction Set Extensions Programming Reference. Retrieved from https://software.intel.com/sites/default/files/managed/b4/3a/319433-024.pdf.
[16]
Peter J. Denning. 1968. The working set model for program behavior. Commun. ACM 11, 5 (1968), 323--333.
[17]
Edward Doller. 2009. Phase change memory and its impacts on memory hierarchy. Retrieved from http://www.pdl.cmu.edu/SDI/2009/slides/Numonyx.pdf.
[18]
Subramanya R. Dulloor, Sanjay Kumar, Anil Keshavamurthy, Philip Lantz, Dheeraj Reddy, Rajesh Sankaran, and Jeff Jackson. 2014. System software for persistent memory. In Proceedings of the 9th European Conference on Computer Systems (EuroSys’14). 15:1--15:15.
[19]
Tyler Harter, Chris Dragga, Michael Vaughn, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2011. A file is not a file: Understanding the I/O behavior of apple desktop applications. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP’11). 71--83.
[20]
Jian Huang, Karsten Schwan, and Moinuddin K. Qureshi. 2014. NVRAM-aware logging in transaction systems. Proc. VLDB Endow. 8, 4 (Dec. 2014), 389--400.
[21]
Taeho Hwang, Jaemin Jung, and Youjip Won. 2015. Heapo: Heap-based persistent object store. ACM Trans. Storage (TOS) 11, 1 (2015), 3.
[22]
Lei Jiang, Bo Zhao, Youtao Zhang, Jun Yang, and B. R. Childers. 2012. Improving write operations in MLC phase change memory. In Proceedings of the 18th International Symposium on High Performance Computer Architecture (HPCA’12). 1--10.
[23]
Heeseung Jo, Jeong-Uk Kang, Seon-Yeong Park, Jin-Soo Kim, and Joonwon Lee. 2006. FAB: Flash-aware buffer management policy for portable media players. IEEE Trans. Consum. Electron. 52, 2 (May 2006), 485--493.
[24]
Theodore Johnson and Dennis Shasha. 1994. 2Q: A low-overhead high-performance buffer management replacement algorithm. In Proceedings of the 20th International Conference on Very Large Data Bases (VLDB’94). 439--450.
[25]
William K. Josephson, Lars A. Bongo, David Flynn, and Kai Li. 2010. DFS: A file system for virtualized flash storage. In Proceedings of the 8th USENIX Conference on File and Storage Technologies (FAST’10). USENIX, Berkeley, CA.
[26]
Myoungsoo Jung, John Shalf, and Mahmut Kandemir. 2013. Design of a large-scale storage-class RRAM system. In Proceedings of the 27th International ACM Conference on International Conference on Supercomputing (ICS’13). 103--114.
[27]
Sooyong Kang, Sungmin Park, Hoyoung Jung, Hyoki Shim, and Jaehyuk Cha. 2009. Performance trade-offs in using NVRAM write buffer for flash memory-based storage devices. IEEE Trans. Comput. (TC) 58, 6 (June 2009), 744--758.
[28]
Jeffrey Katcher. 1997. Postmark: A New File System Benchmark. Technical Report TR3022, Network Appliance.
[29]
Hyojun Kim and Seongjun Ahn. 2008. BPLRU: A buffer management scheme for improving random writes in flash storage. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST’08). 16:1--16:14.
[30]
ESOS LABORATORY. 2013. Mobibench benchmark tool. Retrieved from http://www.mobibench.co.kr/.
[31]
B. C. Lee, Ping Zhou, Jun Yang, Youtao Zhang, Bo Zhao, E. Ipek, O. Mutlu, and D. Burger. 2010. Phase-change technology and the future of main memory. Micro, IEEE 30, 1 (Jan 2010), 143--143.
[32]
Benjamin C. Lee, Engin Ipek, Onur Mutlu, and Doug Burger. 2009. Architecting phase change memory as a scalable dram alternative. In Proceedings of the 36th Annual International Symposium on Computer Architecture (ISCA’09). 2--13.
[33]
Changman Lee, Dongho Sim, Jooyoung Hwang, and Sangyeun Cho. 2015. F2FS: A new file system for flash storage. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST’15). USENIX, Santa Clara, CA. Retrieved from https://www.usenix.org/conference/fast15/technical-sessions/presentation/lee.
[34]
Eunji Lee and Hyokyung Bahn. 2014. Caching strategies for high-performance storage media. ACM Trans. Storage (TOS) 10, 3 (Aug. 2014), 11:1--11:22.
[35]
Youyou Lu, Jiwu Shu, Youmin Chen, and Tao Li. 2017. Octopus: An RDMA-enabled distributed persistent memory file system. In Proceedings of the 2017 USENIX Annual Technical Conference (USENIX ATC’17). USENIX Association, 773--785.
[36]
Y. Lu, J. Shu, and L. Sun. 2015. Blurred persistence in transactional persistent memory. In Proceedings of the 31st Symposium on Mass Storage Systems and Technologies (MSST’15). 1--13.
[37]
Youyou Lu, Jiwu Shu, and Long Sun. 2016. Blurred persistence: Efficient transactions in persistent memory. ACM Trans. Storage (TOS) 12, 1 (2016), 3.
[38]
Y. Lu, J. Shu, L. Sun, and O. Mutlu. 2014. Loose-ordering consistency for persistent memory. In Proceedings of the 32nd IEEE International Conference on Computer Design (ICCD’14). 216--223.
[39]
Youyou Lu, Jiwu Shu, Long Sun, and Onur Mutlu. 2017. Improving performance and endurance of persistent memory with loose-ordering consistency. IEEE Trans. Parall. Distrib. Syst. PP, 99 (2017), 1--1.
[40]
Youyou Lu, Jiwu Shu, and Wei Wang. 2014. ReconFS: A reconstructable file system on flash storage. In Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST’14). USENIX, Berkeley, CA, 75--88.
[41]
Youyou Lu, Jiwu Shu, and Weimin Zheng. 2013. Extending the lifetime of flash-based storage through reducing write amplification from file systems. In Proceedings of the 11th USENIX Conference on File and Storage Technologies (FAST’13). USENIX, Berkeley, CA.
[42]
Nimrod Megiddo and Dharmendra S. Modha. 2003. ARC: A self-tuning, low overhead replacement cache. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies (FAST’03). 115--130.
[43]
Changwoo Min, Kangnyeon Kim, Hyunjin Cho, Sang-Won Lee, and Young Ik Eom. 2012. SFS: Random write considered harmful in solid state drives. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST’12). San Jose, CA, 139--154.
[44]
Iulian Moraru, David G. Andersen, Michael Kaminsky, Niraj Tolia, Parthasarathy Ranganathan, and Nathan Binkert. 2013. Consistent, durable, and safe memory management for byte-addressable nonvolatile main memory. In Proceedings of the First ACM SIGOPS Conference on Timely Results in Operating Systems (TRIOS’13). 1:1--1:17.
[45]
Edmund B. Nightingale, Kaushik Veeraraghavan, Peter M. Chen, and Jason Flinn. 2006. Rethink the sync. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI’06). 1--14.
[46]
Jiaxin Ou and Jiwu Shu. 2016. Fast and failure-consistent updates of application data in non-volatile main memory file system. In Proceedings of the 32st Symposium on Mass Storage Systems and Technologies (MSST’16).
[47]
Jiaxin Ou, Jiwu Shu, and Youyou Lu. 2016. A high-performance file system for non-volatile main memory. In Proceedings of the 11th European Conference on Computer Systems. ACM, 12.
[48]
Jiaxin Ou, Jiwu Shu, Youyou Lu, Letian Yi, and Wei Wang. 2014. EDM: An endurance-aware data migration scheme for load balancing in SSD storage clusters. In Proceedings of the 28th International Parallel and Distributed Processing Symposium (IPDPS’14). 787--796.
[49]
Steven Pelley, Peter M. Chen, and Thomas F. Wenisch. 2014. Memory persistency. In Proceedings of the 41st Annual International Symposium on Computer Architecuture (ISCA’14). 265--276.
[50]
Moinuddin K. Qureshi, Vijayalakshmi Srinivasan, and Jude A. Rivers. 2009. Scalable high-performance main memory system using phase-change memory technology. In Proceedings of the 36th Annual International Symposium on Computer Architecture (ISCA’09). 24--33.
[51]
Luiz E. Ramos, Eugene Gorbatov, and Ricardo Bianchini. 2011. Page placement in hybrid memory systems. In Proceedings of the International Conference on Supercomputing (ICS’11). 85--95.
[52]
Drew Roselli, Jacob R. Lorch, and Thomas E. Anderson. 2000. A comparison of file system workloads. In Proceedings of the Annual Conference on USENIX Annual Technical Conference (ATEC’00). 41--54.
[53]
Chris Ruemmler and John Wilkes. 1993. UNIX disk access patterns. In Proceedings of the USENIX Winter Conference, Vol. 93. 405--420.
[54]
Long Sun, Youyou Lu, and Jiwu Shu. 2015. DP2: Reducing transaction overhead with differential and dual persistency in persistent memory. In Proceedings of the ACM International Conference on Computing Frontiers (CF’15). ACM.
[55]
Kosuke Suzuki and Steven Swanson. 2015. The Non-Volatile Memory Technology Database (NVMDB). Technical Report CS2015-1011. Department of Computer Science 8 Engineering, University of California, San Diego.
[56]
Shivaram Venkataraman, Niraj Tolia, Parthasarathy Ranganathan, and Roy H. Campbell. 2011. Consistent and durable data structures for non-volatile byte-addressable memory. In Proceedings of the 9th USENIX Conference on File and Storage Technologies (FAST’11). San Jose, CA, 61--75.
[57]
Haris Volos, Guilherme Magalhaes, Ludmila Cherkasova, and Jun Li. 2015. Quartz: A lightweight performance emulator for persistent memory software. In Proceedings of the 16th Annual Middleware Conference. ACM, 37--49.
[58]
Haris Volos, Sanketh Nalli, Sankarlingam Panneerselvam, Venkatanathan Varadarajan, Prashant Saxena, and Michael M. Swift. 2014. Aerie: Flexible file-system interfaces to storage-class memory. In Proceedings of the 9th European Conference on Computer Systems (EuroSys’14). 14:1--14:14.
[59]
Haris Volos, Andres Jaan Tack, and Michael M. Swift. 2011. Mnemosyne: Lightweight persistent memory. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’11). 91--104.
[60]
D. L. Willick, D. L. Eager, and R. B. Bunt. 1993. Disk cache replacement policies for network fileservers. In Proceedings the 13th International Conference on Distributed Computing Systems (ICDCS’93). 2--11.
[61]
Xiaojian Wu and A. L. Narasimha Reddy. 2011. SCMFS: A file system for storage class memory. In Proceedings of the 2011 International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’11). 39:1--39:11.
[62]
Jian Xu and Steven Swanson. 2016. NOVA: A log-structured file system for hybrid volatile/non-volatile main memories. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST’16). 323--338.
[63]
Jun Yang, Qingsong Wei, Cheng Chen, Chundong Wang, Khai Leong Yong, and Bingsheng He. 2015. NV-tree: Reducing consistency cost for NVM-based single level systems. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST’15). 167--181.
[64]
J. Joshua Yang and R. Stanley Williams. 2013. Memristive devices in computing system: Promises and challenges. J. Emerg. Technol. Comput. Syst. (JETC’13) 9, 2 (May 2013), 11:1--11:20.
[65]
Jiacheng Zhang, Jiwu Shu, and Youyou Lu. 2016. ParaFS: A log-structured file system to exploit the internal parallelism of flash devices. In Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC’16).
[66]
Yiying Zhang and Steven Swanson. 2015. A study of application performance with non-volatile main memory. In Proceedings of the 31st Symposium on Mass Storage Systems and Technologies (MSST’15). 1--10.
[67]
Yiying Zhang, Jian Yang, Amirsaman Memaripour, and Steven Swanson. 2015. Mojim: A reliable and highly-available non-volatile memory system. In Proceedings of the 20th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’15). 3--18.
[68]
Jishen Zhao, Sheng Li, Doe Hyun Yoon, Yuan Xie, and Norman P. Jouppi. 2013. Kiln: Closing the performance gap between systems with and without persistence support. In Proceedings of the 46th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO’13). ACM, 421--432.
[69]
Ping Zhou, Bo Zhao, Jun Yang, and Youtao Zhang. 2009. A durable and energy efficient main memory using phase change memory technology. In Proceedings of the 36th Annual International Symposium on Computer Architecture (ISCA’09). 14--23.

Cited By

View all
  • (2024)Exploring the Asynchrony of Slow Memory Filesystem with EasyIOProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3629586(624-640)Online publication date: 22-Apr-2024
  • (2024)Delaying Crash Consistency for Building A High-Performance Persistent Memory File SystemIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2024.337579243:9(2620-2634)Online publication date: Sep-2024
  • (2024)CaitiJournal of Systems Architecture: the EUROMICRO Journal10.1016/j.sysarc.2024.103109150:COnline publication date: 1-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Storage
ACM Transactions on Storage  Volume 14, Issue 1
Special Issue on NVM and Storage
February 2018
237 pages
ISSN:1553-3077
EISSN:1553-3093
DOI:10.1145/3190860
  • Editor:
  • Sam H. Noh
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2018
Accepted: 01 July 2017
Received: 01 February 2017
Published in TOS Volume 14, Issue 1

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Persistent memory
  2. buffering
  3. direct access
  4. file system

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)47
  • Downloads (Last 6 weeks)4
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Exploring the Asynchrony of Slow Memory Filesystem with EasyIOProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3629586(624-640)Online publication date: 22-Apr-2024
  • (2024)Delaying Crash Consistency for Building A High-Performance Persistent Memory File SystemIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2024.337579243:9(2620-2634)Online publication date: Sep-2024
  • (2024)CaitiJournal of Systems Architecture: the EUROMICRO Journal10.1016/j.sysarc.2024.103109150:COnline publication date: 1-May-2024
  • (2023)Progress on storage systems for disaggregated data centersSCIENTIA SINICA Informationis10.1360/SSI-2023-003453:8(1503)Online publication date: 17-Aug-2023
  • (2023)Anchor: A Library for Building Secure Persistent Memory SystemsProceedings of the ACM on Management of Data10.1145/36267181:4(1-31)Online publication date: 12-Dec-2023
  • (2023)A Cost-Efficient Failure-Tolerant Scheme for Distributed DNN Training2023 IEEE 41st International Conference on Computer Design (ICCD)10.1109/ICCD58817.2023.00031(150-157)Online publication date: 6-Nov-2023
  • (2022)Checking robustness to weak persistency modelsProceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation10.1145/3519939.3523723(490-505)Online publication date: 9-Jun-2022
  • (2022)Fast and Low Overhead Metadata Operations for NVM-Based File System Using Slotted PagingIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2022.320243641:11(4481-4491)Online publication date: Nov-2022
  • (2022)XPGraph: XPline-Friendly Persistent Memory Graph Stores for Large-Scale Evolving Graphs2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO)10.1109/MICRO56248.2022.00091(1308-1325)Online publication date: Oct-2022
  • (2022)Methods for Transferring Data from a Compute to a Storage Cloud2022 9th International Conference on Future Internet of Things and Cloud (FiCloud)10.1109/FiCloud57274.2022.00017(67-74)Online publication date: Aug-2022
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media