Skip to main content
SpringerLink
Log in

Precise control of page cache for containers

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Container-based virtualization is becoming increasingly popular in cloud computing due to its efficiency and flexibility. Resource isolation is a fundamental property of containers. Existing works have indicated weak resource isolation could cause significant performance degradation for containerized applications and enhanced resource isolation. However, current studies have almost not discussed the isolation problems of page cache which is a key resource for containers. Containers leverage memory cgroup to control page cache usage. Unfortunately, existing policy introduces two major problems in a container-based environment. First, containers can utilize more memory than limited by their cgroup, effectively breaking memory isolation. Second, the OS kernel has to evict page cache to make space for newly-arrived memory requests, slowing down containerized applications. This paper performs an empirical study of these problems and demonstrates the performance impacts on containerized applications. Then we propose pCache (precise control of page cache) to address the problems by dividing page cache into private and shared and controlling both kinds of page cache separately and precisely. To do so, pCache leverages two new technologies: fair account (f-account) and evict on demand (EoD). F-account splits the shared page cache charging based on per-container share to prevent containers from using memory for free, enhancing memory isolation. And EoD reduces unnecessary page cache evictions to avoid the performance impacts. The evaluation results demonstrate that our system can effectively enhance memory isolation for containers and achieve substantial performance improvement over the original page cache management policy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Merkel D. Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014, 239: 2

    Google Scholar 

  2. Zeng R, Hou X F, Zhang L, Li C, Zheng W L, Guo M Y. Performance optimization for cloud computing systems in the microservice era: state-of-the-art and research opportunities. Frontiers of Computer Science, 2022, 16(6): 166106

    Article  Google Scholar 

  3. Hou X F, Li C, Liu J C, Zhang L, Ren S L, Leng J W, Chen Q, Guo M Y. AlphaR: learning-powered resource management for irregular, dynamic microservice graph. In: Proceeding of IEEE International Parallel and Distributed Processing Symposium. 2021, 797–806

  4. Suo K, Zhao Y, Chen W, Rao J. An analysis and empirical study of container networks. In: Proceedings of IEEE INFOCOM 2018-IEEE Conference on Computer Communications. 2018, 189–197

  5. Zhang Y Q, Goiri I, Chaudhry G I, Fonseca R, Elnikety S, Delimitrou C, Bianchini R. Faster and cheaper serverless computing on harvested resources. In: Proceedings of the 28th ACM SIGOPS Symposium on Operating Systems Principles. 2021, 724–739

  6. Huang H, Rao J, Wu S, Jin H, Suo K, Wu X F. Adaptive resource views for containers. In: Proceedings of International Symposium on HighPerformance Parallel and Distributed Computing. 2019, 243–254

  7. Soltesz S, Pötzl H, Fiuczynski M E, Bavier A, Peterson L. Container-based operating system virtualization: a scalable, high-performance alternative to hypervisors. In: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems. 2007, 275–287

  8. Laadan O, Nieh J. Operating System virtualization: practice and experience. In: Proceedings of the 3rd Annual Haifa Experimental Systems Conference. 2010, 17

  9. Khalid J, Rozner E, Felter W, Xu C, Rajamani K, Ferreira A, Akella A. Iron: Isolating network-based CPU in container environments. In: Proceedings of the 15th USENIX Conference on Networked Systems Design and Implementation. 2018, 313–328

  10. Li Y H Z, Zhang J C, Jiang C F, Wan J, Ren Z J. PINE: Optimizing performance isolation in container environments. IEEE Access, 2019, 7: 30410–30422

    Article  Google Scholar 

  11. Senthil K S. Practical LXC and LXD: Linux Containers for Virtualization and Orchestration. New York: Apress, 2017

    Google Scholar 

  12. Xie X L, Wang P, Wang Q. The performance analysis of Docker and rkt based on Kubernetes. In: Proceedings of the 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery. 2017, 2137–2141

  13. Skarlatos D, Chen Q R, Chen J Y, Xu T Y, Torrellas J. Draco: Architectural and operating system support for system call security. In: Proceedings of the 53rd Annual IEEE/ACM International Symposium on Microarchitecture. 2020, 42–57

  14. Do H D, Hayot-Sasson V, Da Silva R F, Steele C, Casanova H, Glatard T. Modeling the Linux page cache for accurate simulation of dataintensive applications. In: Proceedings of 2021 IEEE International Conference on Cluster Computing. 2021, 398–408

  15. Eklov D, Hagersten E. StatStack: Efficient modeling of LRU caches. In: Proceedings of 2010 IEEE International Symposium on Performance Analysis of Systems & Software. 2010, 55–65

  16. Tarasov V, Zadok E, Shepler S. Filebench: A flexible framework for file system benchmarking. The USENIX Magazine, 2016, 41(1): 6–12

    Google Scholar 

  17. Xiang Y C, Wang X L, Huang Z H, Wang Z Y, Luo Y W, Wang Z L. DCAPS: Dynamic cache allocation with partial sharing. In: Proceedings of the Thirteenth EuroSys Conference. 2018, 1–15

  18. Xu M, Thi L, Phan X, Choi H Y, Lee I. vCAT: Dynamic cache management using CAT virtualization. In: Proceedings of 2017 IEEE Real-Time and Embedded Technology and Applications Symposium. 2017, 211–222

  19. Sohal P, Bechtel M, Mancuso R, Yun H, Krieger O. A closer look at Intel Resource Director Technology (RDT). In: Proceedings of the 30th International Conference on Real-Time Networks and Systems. 2022, 127–139

  20. Chaudhuri M. Zero inclusion victim: Isolating core caches from inclusive last-level cache evictions. In: Proceeding of the 48th ACM/IEEE Annual International Symposium on Computer Architecture. 2021, 71–84

  21. Delimitrou C, Kozyrakis C. Bolt: I know what you did last summer… in the cloud. ACM SIGARCH Computer Architecture News, 2017, 45(1): 599–613

    Article  Google Scholar 

  22. Volckaert S. Randomization-based defenses against data-oriented attacks. In: Proceedings of the 8th ACM Workshop on Moving Target Defense. 2021, 1–2

  23. Love R. Linux Kernel Development. 3rd ed. New York: Pearson Education, 2010

    Google Scholar 

  24. Felter W, Ferreira A, Rajamony R, Rubio J. An updated performance comparison of virtual machines and Linux containers. In: Proceedings of 2015 IEEE International Symposium on Performance Analysis of Systems and Software. 2015, 171–172

  25. Sharma P, Chaufournier L, Shenoy P, Tay Y C. Containers and virtual machines at scale: A comparative study. In: Proceedings of the 17th International Middleware Conference. 2016, 1

  26. Plauth M, Feinbube L, Polze A. A performance survey of lightweight virtualization techniques. In: Proceedings of the 6th European Conference on Service-Oriented and Cloud Computing. 2017, 34–48

  27. Matthews J N, Hu W J, Hapuarachchi M, Deshane T, Dimatos D, Hamilton G, McCabe M, Owens J. Quantifying the performance isolation properties of virtualization systems. In: Proceedings of 2007 Workshop on Experimental Computer Science. 2007, 6–es

  28. Xavier M G, De Oliveira I C, Rossi F D, Dos Passos R D, Matteussi K J, De Rose C A. A performance isolation analysis of disk-intensive workloads on container-based clouds. In: Proceedings of the 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing. 2015, 253–260

  29. Yang N Z, Shen W B, Li J K, Yang Y T, Lu K J, Xiao J T, Zhou T Y, Qin C G, Yu W, Ma J F, Ren K. Demons in the shared kernel: Abstract resource attacks against OS-level virtualization. In: Proceedings of 2021 ACM SIGSAC Conference on Computer and Communications Security. 2021, 764–778

  30. Anjali, Caraza-Harter T, Swift M M. Blending containers and virtual machines: A study of firecracker and gVisor. In: Proceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments. 2020, 101–113

  31. Sartakov V A, Vilanova L, Eyers D, Shinagawa T, Pietzuch P. CAP-VMs: Capability-based isolation and sharing in the cloud. In: Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation. 2022, 597–612

  32. Hua Z C, Yu Y, Gu J Y, Xia Y B, Chen H B, Zang B Y. TZ-container: Protecting container from untrusted OS with ARM TrustZone. Science China Information Sciences, 2021, 64(9): 192101

    Article  Google Scholar 

  33. Sun Y Q, Safford D, Zohar M, Pendarakis D, Gu Z S, Jaeger T. Security namespace: making linux security frameworks available to containers. In: Proceedings of the 27th USENIX Conference on Security Symposium. 2018, 1423–1439

  34. Gao X, Gu Z S, Kayaalp M, Pendarakis D, Wang H N. Containerleaks: Emerging security threats of information leakages in container clouds. In: Proceedings of the 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 2017, 237–248

  35. Gao X, Gu Z S, Li Z F, Jamjoom H, Wang C. Houdini’s escape: Breaking the resource rein of Linux control groups. In: Proceedings of 2019 ACM SIGSAC Conference on Computer and Communications Security. 2019, 1073–1086

  36. Huang H, Rao J, Wu S, Jin H, Jiang S, Che H, Wu X F. Towards exploiting CPU elasticity via efficient thread oversubscription. In: Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing. 2021, 215–226

  37. Wu S, Huang Z, Chen P F, Fan H, Ibrahim S, Jin H. Container-aware I/O stack: Bridging the gap between container storage drivers and solid state devices. In: Proceedings of the 18th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments. 2022, 18–30

  38. Heo T, Schatzberg D, Newell A, Liu S, Dhakshinamurthy S, Narayanan I, Bacik J, Mason C, Tang C Q, Skarlatos D. IOCost: Block IO control for containers in datacenters. In: Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 2022, 595–608

  39. Gu L, Guan J J, Wu S, Jin H, Rao J, Suo K, Zeng D Z. CNTC: A container aware network traffic control framework. In: Proceeding of the 14th International Conference on Green, Pervasive, and Cloud Computing. 2019, 208–222

  40. Randazzo A, Tinnirello I. Kata containers: An emerging architecture for enabling mec services in fast and secure way. In: Proceedings of 2019 Sixth International Conference on Internet of Things: Systems, Management and Security. 2019, 209–214

  41. Manco F, Lupu C, Schmidt F, Mendes J, Kuenzer S, Sati S, Yasukata K, Raiciu C, Huici F. My VM is lighter (and safer) than your container. In: Proceedings of the 26th Symposium on Operating Systems Principles. 2017, 218–233

  42. Mavridis I, Karatza H. Combining containers and virtual machines to enhance isolation and extend functionality on cloud computing. Future Generation Computer Systems, 2019, 94: 674–696

    Article  Google Scholar 

  43. Shen Z M, Sun Z, Sela G E, Bagdasaryan E, Delimitrou C, Renesse R V, Weatherspoon H. X-Containers: Breaking down barriers to improve performance and isolation of cloud-native containers. In: Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems. 2019, 121–135

  44. Tazaki H, Moroo A, Kuga Y, Nakamura R. How to design a library OS for practical containers? In: Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments. 2021, 15–28

  45. Li Z J, Cheng J, Chen Q, Guan E Y, Bian Z Z, Tao Y, Zha B, Wang Q, Han W D, Guo M Y. RunD: A lightweight secure container runtime for high-density deployment and high-concurrency startup in serverless computing. In: Proceeding of 2022 USENIX Annual Technical Conference. 2022, 53–68

  46. Lim J T, Nieh J. Optimizing nested virtualization performance using direct virtual hardware. In: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. 2020, 557–574

  47. Huang J, Qureshi M K, Schwan K. An evolutionary study of Linux memory management for fun and profit. In: Proceedings of the 2016 USENIX Conference on USENIX Annual Technical Conference. 2016, 465–478

  48. Kim J, Shin P, Noh S, Ham D, Hong S. Reducing memory interference latency of safety-critical applications via memory request throttling and Linux Cgroup. In: Proceedings of 2018 31st IEEE International System-on-Chip Conference. 2018, 215–220

  49. Zhuang Z Y, Tran C, Weng J, Ramachandra H, Sridharan B. Taming memory related performance pitfalls in linux Cgroups. In: Proceedings of 2017 International Conference on Computing, Networking and Communications. 2017, 531–535

  50. Oh K, Park J, Eom Y I. Weight-based page cache management scheme for enhancing I/O proportionality of Cgroups. In: Proceedings of 2019 IEEE International Conference on Consumer Electronics. 2019, 1–3

  51. Park J, Eom Y I. Weight-aware cache for application-level proportional I/O sharing. IEEE Transactions on Computers, 2021, 71(10): 2395–2407

    Article  Google Scholar 

  52. Zheng D, Burns R, Szalay A S. Toward millions of file system IOPS on low-cost, commodity hardware. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. 2013, 1–12

  53. Bang J, Kim C, Kim S, Chen Q C, Lee C, Byun E K, Lee J, Eom H. Finer-LRU: A scalable page management scheme for HPC manycore architectures. In: Proceeding of 2021 IEEE International Parallel and Distributed Processing Symposium. 2021, 567–576

Download references

Acknowledgements

We thank the anonymous reviewers for their helpful feedback. This work was supported by the National Key Research and Development Program (2022YFB4500704), and the National Natural Science Foundation of China (Grant Nos. 62032008, 62232012 and 62232011).

Author information

Authors and Affiliations

  1. National Engineering Research Center for Big Data Technology and System, Services Computing Technology and System Lab, Cluster and Grid Computing Lab School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, 430074, China

    Kun Wang, Song Wu, Shengbang Li, Zhuo Huang, Hao Fan, Chen Yu & Hai Jin

Authors
  1. Kun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Song Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengbang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhuo Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hao Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chen Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hai Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Song Wu.

Additional information

Kun Wang received the BS from Huazhong University of Science and Technology (HUST), China in 2015. Currently he is a PhD candidate student in Service Computing Technology and System Lab (SCTS) and Cluster and Grid Lab (CGCL), HUST in China. His current research interests include container virtualization and kernel resource isolation.

Song Wu received the PhD degree from Huazhong University of Science and Technology (HUST), China in 2003. He is a professor of computer science at HUST in China. He currently serves as the vice dean of the School of Computer Science and Technology and the vice head of Service Computing Technology and System Lab (SCTS) and the Cluster and Grid Computing Lab (CGCL) in HUST. His current research interests include cloud resource scheduling and system virtualization.

Shengbang Li received the BS from Shandong University (SDU), China in 2021. Currently he is a MS candidate student in Service Computing Technology and System Lab (SCTS) and Cluster and Grid Lab (CGCL), Huazhong University of Science and Technology (HUST) in China. His current research interest is kernel resource isolation.

Zhuo Huang received the BS from Huazhong Agricultural University (HZAU), China in 2014. Currently he is a PhD candidate student in Service Computing Technology and System Lab (SCTS) and Cluster and Grid Lab (CGCL), Huazhong University of Science and Technology (HUST) in China. His current research interests include container virtualization, serverless computing optimization, and storage system.

Hao Fan received the PhD degree from Huazhong University of Science and Technology (HUST), China in 2021. Currently he is working as a post-doctor in Service Computing Technology and System Lab (SCTS) and Cluster and Grid Lab (CGCL), HUST in China. His current research interests include container technology and storage system.

Chen Yu received the PhD degree in information science from Tohoku University, Japan in 2005. From 2005 to 2006, he was a Japan Science and Technology Agency Postdoctoral Researcher with the Japan Advanced Institute of Science and Technology, Japan. In 2006, he was with Japan Society for the Promotion of Science Postdoctoral Fellow with the Japan Advanced Institute of Science and Technology. Since 2008, he has been with the School of Computer Science and Technology, Huazhong University of Science and Technology, China where he is currently a Professor and Special Research Fellow, working in the areas of wireless sensor networks, ubiquitous computing, edge computing, and edge intelligence.

Hai Jin is a Chair Professor of computer science at Huazhong University of Science and Technology (HUST), China. Jin received his PhD in computer engineering from HUST in 1994. In 1996, he was awarded a German Academic Exchange Service fellowship to visit the Technical University of Chemnitz, Germany. Jin worked at The University of Hong Kong, China between 1998 and 2000. He was awarded Excellent Youth Award from the National Science Foundation of China in 2001. Jin is a Fellow of IEEE, Fellow of CCF, and a life member of the ACM. He has co-authored more than 20 books and published over 900 research papers. His research interests include computer architecture, parallel and distributed computing, big data processing, data storage, and system security.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, K., Wu, S., Li, S. et al. Precise control of page cache for containers. Front. Comput. Sci. 18, 182102 (2024). https://doi.org/10.1007/s11704-022-2455-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-022-2455-0

Keywords

Search

Navigation