skip to main content
10.1145/3459898.3463907acmconferencesArticle/Chapter ViewAbstractPublication PagesismmConference Proceedingsconference-collections
research-article

Radiant: efficient page table management for tiered memory systems

Published: 22 June 2021 Publication History

Abstract

Modern enterprise servers are increasingly embracing tiered memory systems with a combination of low latency DRAMs and large capacity but high latency non-volatile main memories (NVMMs) such as Intel’s Optane DC PMM. Prior works have focused on the efficient placement and migration of data on a tiered memory system, but have not studied the optimal placement of page tables.
Explicit and efficient placement of page tables is crucial for large memory footprint applications with high TLB miss rates because they incur dramatically higher page walk latency when page table pages are placed in NVMM. We show that (i) page table pages can end up on NVMM even when enough DRAM memory is available and (ii) page table pages that spill over to NVMM due to DRAM memory pressure are not migrated back later when memory is available in DRAM.
We study the performance impact of page table placement in a tiered memory system and propose Radiant, an efficient and transparent page table management technique that (i) applies different placement policies for data and page table pages,(ii) introduces a differentiating policy for page table pages by placing a small but critical part of the page table in DRAM, and (iii) dynamically and judiciously manages the rest of the page table by transparently migrating the page table pages between DRAM and NVMM. Our implementation on a real system equipped with Intel’s Optane NVMM running Linux reduces the page table walk cycles by 12% and total cycles by 20% on an average. This improves the runtime by 20% on an average for a set of synthetic and real-world large memory footprint applications when compared with various default Linux kernel techniques.

References

[1]
Reto Achermann. 2020. mitosis-project/mitosis-workload-btree: The BTree workload used for evaluation. https://github.com/mitosis-project/mitosis-workload-btree (Accessed on 10/03/2020).
[2]
Reto Achermann. 2020. mitosis-project/mitosis-workload-hashjoin: The HashJoin workload used for evaluation. https://github.com/mitosis-project/mitosis-workload-hashjoin (Accessed on 10/03/2020).
[3]
Reto Achermann, Ashish Panwar, Abhishek Bhattacharjee, Timothy Roscoe, and Jayneel Gandhi. 2020. Mitosis: Transparently Self-Replicating Page-Tables for Large-Memory Machines. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’20). Association for Computing Machinery, New York, NY, USA. 283–300. isbn:9781450371025 https://doi.org/10.1145/3373376.3378468
[4]
Thomas W. Barr, Alan L. Cox, and Scott Rixner. 2010. Translation Caching: Skip, Don’t Walk (the Page Table). In Proceedings of the 37th Annual International Symposium on Computer Architecture (ISCA ’10). Association for Computing Machinery, New York, NY, USA. 48–59. isbn:9781450300537 https://doi.org/10.1145/1815961.1815970
[5]
Arkaprava Basu, Jayneel Gandhi, Jichuan Chang, Mark D. Hill, and Michael M. Swift. 2013. Efficient Virtual Memory for Big Memory Servers. In Proceedings of the 40th Annual International Symposium on Computer Architecture (ISCA ’13). Association for Computing Machinery, New York, NY, USA. 237–248. isbn:9781450320795 https://doi.org/10.1145/2485922.2485943
[6]
Arkaprava Basu, Jayneel Gandhi, Jichuan Chang, Mark D. Hill, and Michael M. Swift. 2013. Efficient Virtual Memory for Big Memory Servers. In Proceedings of the 40th Annual International Symposium on Computer Architecture (ISCA ’13). Association for Computing Machinery, New York, NY, USA. 237–248. isbn:9781450320795 https://doi.org/10.1145/2485922.2485943
[7]
Abhishek Bhattacharjee. 2013. Large-Reach Memory Management Unit Caches. In Proceedings of the 46th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-46). Association for Computing Machinery, New York, NY, USA. 383–394. isbn:9781450326384 https://doi.org/10.1145/2540708.2540741
[8]
Filipe Brandenburger. 2020. numactl/numactl: NUMA support for Linux. https://github.com/numactl/numactl (Accessed on 10/03/2020).
[9]
Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking Cloud Serving Systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing (SoCC ’10). Association for Computing Machinery, New York, NY, USA. 143–154. isbn:9781450300360 https://doi.org/10.1145/1807128.1807152
[10]
Jonathan Corbet. 2012. AutoNUMA: the other approach to NUMA scheduling [LWN.net]. https://lwn.net/Articles/488709/ (Accessed on 10/04/2020).
[11]
Jonathan Corbet. 2013. Split PMD locks [LWN.net]. https://lwn.net/Articles/568076/ (Accessed on 09/30/2020).
[12]
Mohammad Dashti, Alexandra Fedorova, Justin Funston, Fabien Gaud, Renaud Lachaize, Baptiste Lepers, Vivien Quema, and Mark Roth. 2013. Traffic Management: A Holistic Approach to Memory Placement on NUMA Systems. SIGPLAN Not., 48, 4 (2013), March, 381–394. issn:0362-1340 https://doi.org/10.1145/2499368.2451157
[13]
DAXCTL. 2020. DAXCTL Man Pages - NDCTL User Guide. https://docs.pmem.io/ndctl-user-guide/daxctl-man-pages (Accessed on 10/05/2020).
[14]
Linux Kernel documentation. 2020. Split page table lock — The Linux Kernel documentation. https://www.kernel.org/doc/html/latest/vm/split_page_table_lock.html (Accessed on 09/30/2020).
[15]
Fabien Gaud, Baptiste Lepers, Justin Funston, Mohammad Dashti, Alexandra Fedorova, Vivien Quéma, Renaud Lachaize, and Mark Roth. 2015. Challenges of Memory Management on Modern NUMA Systems. Commun. ACM, 58, 12 (2015), Nov., 59–66. issn:0001-0782 https://doi.org/10.1145/2814328
[16]
G. Gill, Roshan Dathathri, Loc Hoang, R. Peri, and K. Pingali. 2020. Single machine graph analytics on massive datasets using Intel optane DC persistent memory. Proceedings of the VLDB Endowment, 13 (2020), 1304 – 1318.
[17]
Dave Hansen. 2019. Allow persistent memory to be used like normal RAM. https://patchwork.kernel.org/cover/10829019/
[18]
Mark Hildebrand, Jawad Khan, Sanjeev Trika, Jason Lowe-Power, and Venkatesh Akella. 2020. AutoTM: Automatic Tensor Movement in Heterogeneous Memory Systems Using Integer Linear Programming. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’20). Association for Computing Machinery, New York, NY, USA. 875–890. isbn:9781450371025 https://doi.org/10.1145/3373376.3378465
[19]
Sungjoo Hong. 2010. Memory technology trend and future challenges. In 2010 International Electron Devices Meeting. https://doi.org/10.1109/IEDM.2010.5703348
[20]
Intel. 2020. Intel Optane DC Persistent Memory. https://www.intel.in/content/dam/www/public/us/en/documents/product-briefs/optane-dc-persistent-memory-brief.pdf
[21]
Sudarsun Kannan, Ada Gavrilovska, Vishal Gupta, and Karsten Schwan. 2017. HeteroOS: OS Design for Heterogeneous Memory Management in Datacenter. SIGARCH Comput. Archit. News, 45, 2 (2017), June, 521–534. issn:0163-5964 https://doi.org/10.1145/3140659.3080245
[22]
Kinam Kim. 2008. Future memory technology: challenges and opportunities. In 2008 International Symposium on VLSI Technology, Systems and Applications (VLSI-TSA). 5–9. https://doi.org/10.1109/VTSA.2008.4530774
[23]
Redis Labs. 2020. Redis. https://redis.io/ (Accessed on 10/03/2020).
[24]
Benjamin C. Lee, Engin Ipek, Onur Mutlu, and Doug Burger. 2009. Architecting Phase Change Memory as a Scalable Dram Alternative. In Proceedings of the 36th Annual International Symposium on Computer Architecture (ISCA ’09). Association for Computing Machinery, New York, NY, USA. 2–13. isbn:9781605585260 https://doi.org/10.1145/1555754.1555758
[25]
Baptiste Lepers, Vivien Quéma, and Alexandra Fedorova. 2015. Thread and Memory Placement on NUMA Systems: Asymmetry Matters. In Proceedings of the 2015 USENIX Conference on Usenix Annual Technical Conference (USENIX ATC ’15). USENIX Association, USA. 277–289. isbn:9781931971225
[26]
2020. move_pages - Linux man page. https://linux.die.net/man/2/move_pages
[27]
L. Liu, S. Yang, L. Peng, and X. Li. 2019. Hierarchical Hybrid Memory Management in OS for Tiered Memory Systems. IEEE Transactions on Parallel and Distributed Systems, 30, 10 (2019), 2223–2236.
[28]
Artemiy Margaritov, Dmitrii Ustiugov, Edouard Bugnion, and Boris Grot. 2019. Prefetched Address Translation. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO ’52). Association for Computing Machinery, New York, NY, USA. 1023–1036. isbn:9781450369381 https://doi.org/10.1145/3352460.3358294
[29]
Memcached. 2020. memcached - a distributed memory object caching system. https://memcached.org/ (Accessed on 10/03/2020).
[30]
N. Unnikrishnan Nair, P.G. Sankaran, and N. Balakrishnan. 2018. Chapter 8 - Multivariate Lifetime Models. In Reliability Modelling and Analysis in Discrete Time, N. Unnikrishnan Nair, P.G. Sankaran, and N. Balakrishnan (Eds.). Academic Press, Boston. 387 – 428. isbn:978-0-12-801913-9 https://doi.org/10.1016/B978-0-12-801913-9.00008-7
[31]
NDCTL. 2020. NDCTL Man Pages - NDCTL User Guide. https://docs.pmem.io/ndctl-user-guide/ndctl-man-pages (Accessed on 10/05/2020).
[32]
Chang Hyun Park, Taekyung Heo, Jungi Jeong, and Jaehyuk Huh. 2017. Hybrid TLB Coalescing: Improving TLB Translation Coverage under Diverse Fragmented Memory Allocations. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA ’17). Association for Computing Machinery, New York, NY, USA. 444–456. isbn:9781450348928 https://doi.org/10.1145/3079856.3080217
[33]
Onkar Patil, Latchesar Ionkov, Jason Lee, Frank Mueller, and Michael Lang. 2019. Performance Characterization of a DRAM-NVM Hybrid Memory Architecture for HPC Applications Using Intel Optane DC Persistent Memory Modules. In Proceedings of the International Symposium on Memory Systems (MEMSYS ’19). Association for Computing Machinery, New York, NY, USA. 288–303. isbn:9781450372060 https://doi.org/10.1145/3357526.3357541
[34]
PMEM.IO. 2020. Volatile use of persistent memory as a hotplugged memory region. https://pmem.io/2020/01/20/memkind-dax-kmem.html
[35]
Georgios Psaropoulos, Ismail Oukid, Thomas Legler, Norman May, and Anastasia Ailamaki. 2019. Bridging the Latency Gap between NVM and DRAM for Latency-Bound Operations. In Proceedings of the 15th International Workshop on Data Management on New Hardware (DaMoN’19). Association for Computing Machinery, New York, NY, USA. Article 13, 8 pages. isbn:9781450368018 https://doi.org/10.1145/3329785.3329917
[36]
Jee Ho Ryoo, Nagendra Gulur, Shuang Song, and Lizy K. John. 2017. Rethinking TLB Designs in Virtualized Environments: A Very Large Part-of-Memory TLB. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA ’17). Association for Computing Machinery, New York, NY, USA. 469–480. isbn:9781450348928 https://doi.org/10.1145/3079856.3080210
[37]
Julian Shun and Guy E. Blelloch. 2013. Ligra: A Lightweight Graph Processing Framework for Shared Memory. SIGPLAN Not., 48, 8 (2013), Feb., 135–146. issn:0362-1340 https://doi.org/10.1145/2517327.2442530
[38]
John R Tramm, Andrew R Siegel, Tanzima Islam, and Martin Schulz. 2014. XSBench - The Development and Verification of a Performance Abstraction for Monte Carlo Reactor Analysis. In PHYSOR 2014 - The Role of Reactor Physics toward a Sustainable Future. Kyoto. "https://www.mcs.anl.gov/papers/P5064-0114.pdf"
[39]
Tao Wang. 2020. Baidu Feed Stream Services Restructures Its In-Memory Database with Intel Optane Technology. https://newsroom.intel.com/wp-content/uploads/sites/11/2019/08/baidu-feed-case-study.pdf
[40]
Fengguang Wu. 2018. x86/pgtable: allocate page table pages from DRAM. https://lkml.org/lkml/2018/12/26/145
[41]
Zi Yan, Daniel Lustig, David Nellans, and Abhishek Bhattacharjee. 2019. Nimble Page Management for Tiered Memory Systems. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’19). Association for Computing Machinery, New York, NY, USA. 331–345. isbn:9781450362405 https://doi.org/10.1145/3297858.3304024
[42]
Jian Yang, Juno Kim, Morteza Hoseinzadeh, Joseph Izraelevitz, and Steve Swanson. 2020. An Empirical Guide to the Behavior and Use of Scalable Persistent Memory. In 18th USENIX Conference on File and Storage Technologies (FAST 20). USENIX Association, Santa Clara, CA. 169–182. isbn:978-1-939133-12-0 https://www.usenix.org/conference/fast20/presentation/yang
[43]
Idan Yaniv and Dan Tsafrir. 2016. Hash, Don’t Cache (the Page Table). In Proceedings of the 2016 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Science (SIGMETRICS ’16). Association for Computing Machinery, New York, NY, USA. 337–350. isbn:9781450342667 https://doi.org/10.1145/2896377.2901456
[44]
Kaiyuan Zhang, Rong Chen, and Haibo Chen. 2015. NUMA-Aware Graph-Structured Analytics. In Proceedings of the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP 2015). Association for Computing Machinery, New York, NY, USA. 183–193. isbn:9781450332057 https://doi.org/10.1145/2688500.2688507

Cited By

View all
  • (2024)TelescopeProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692018(409-424)Online publication date: 10-Jul-2024
  • (2023)MEMTIS: Efficient Memory Tiering with Dynamic Page Classification and Page Size DeterminationProceedings of the 29th Symposium on Operating Systems Principles10.1145/3600006.3613167(17-34)Online publication date: 23-Oct-2023
  • (2023)Accelerating Performance of GPU-based Workloads Using CXLProceedings of the 13th Workshop on AI and Scientific Computing at Scale using Flexible Computing10.1145/3589013.3596678(27-31)Online publication date: 10-Aug-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISMM 2021: Proceedings of the 2021 ACM SIGPLAN International Symposium on Memory Management
June 2021
106 pages
ISBN:9781450384483
DOI:10.1145/3459898
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 June 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Intel Optane DC
  2. NVMM
  3. Page Tables

Qualifiers

  • Research-article

Conference

ISMM '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 72 of 156 submissions, 46%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)100
  • Downloads (Last 6 weeks)8
Reflects downloads up to 28 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)TelescopeProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692018(409-424)Online publication date: 10-Jul-2024
  • (2023)MEMTIS: Efficient Memory Tiering with Dynamic Page Classification and Page Size DeterminationProceedings of the 29th Symposium on Operating Systems Principles10.1145/3600006.3613167(17-34)Online publication date: 23-Oct-2023
  • (2023)Accelerating Performance of GPU-based Workloads Using CXLProceedings of the 13th Workshop on AI and Scientific Computing at Scale using Flexible Computing10.1145/3589013.3596678(27-31)Online publication date: 10-Aug-2023
  • (2022)Performance Characterization of AutoNUMA Memory Tiering on Graph Analytics2022 IEEE International Symposium on Workload Characterization (IISWC)10.1109/IISWC55918.2022.00024(171-184)Online publication date: Dec-2022
  • (2022)Virtual PTE Storage: Repurposing Last-level Cache to Accelerate Address Translation for Big Data Workloads2022 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia)10.1109/ICCE-Asia57006.2022.9954665(1-5)Online publication date: 26-Oct-2022
  • (2022)Pinning Page Structure Entries to Last-Level Cache for Fast Address TranslationIEEE Access10.1109/ACCESS.2022.321748410(114552-114565)Online publication date: 2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media