research-article

Towards Lightweight and Robust Machine Learning for CDN Caching

Author:

Daniel S. BergerAuthors Info & Claims

HotNets '18: Proceedings of the 17th ACM Workshop on Hot Topics in Networks

Pages 134 - 140

https://doi.org/10.1145/3286062.3286082

Published: 15 November 2018 Publication History

Abstract

Recent advances in the field of reinforcement learning promise a general approach to optimize networking systems. This paper argues against the recent trend for generalization by introducing a case study where domain-specific modeling enables the application of lightweight and robust learning techniques.

We study CDN caching systems, which make a good case for optimization as their performance directly affects operational costs, while currently relying on many hand-tuned parameters. In caching, reinforcement learning has been shown to perform suboptimally when compared to simple heuristics. A key challenge is that rewards (cache hits) manifest with large delays, which prevents timely feedback to the learning algorithm and introduces significant complexity.

This paper shows how to significantly simplify this problem by explicitly modeling optimal caching decisions (OPT). While prior work considered deriving OPT impractical, recent theoretical modeling advances change this assumption. Modeling OPT enables even lightweight decision trees to outperform state-of-the-art CDN caching heuristics.

Supplementary Material

MP4 File (p134-berger.mp4)

Download
477.52 MB

References

[1]

Ravindra K Ahuja, Thomas L Magnanti, and James B Orlin. 1993. Network flows: theory, algorithms, and applications. Prentice hall.

Digital Library

[2]

Martin Arlitt, Ludmila Cherkasova, John Dilley, Rich Friedrich, and Tai Jin. 2000. Evaluating content management techniques for web proxy caches. Performance Evaluation Review 27, 4 (2000), 3--11.

Digital Library

[3]

Mihovil Bartulovic, Junchen Jiang, Sivaraman Balakrishnan, Vyas Sekar, and Bruno Sinopoli. 2017. Biases in Data-Driven Networking, and What to Do About Them. In ACM HotNets. 192--198.

Digital Library

[4]

Nathan Beckmann, Haoxian Chen, and Asaf Cidon. 2018. LHD: Improving Hit Rate by Maximizing Hit Density. In USENIX NSDI. 1--14.

[5]

Daniel S. Berger, Nathan Beckmann, and Mor Harchol-Balter. 2018. Practical Bounds on Optimal Caching with Variable Object Sizes. Proc. ACM Meas. Anal. Comput. Syst. 2, 2, Article 32 (June 2018), 38 pages.

Digital Library

[6]

Daniel S. Berger, Ben Berg, Timothy Zhu, Mor Harchol-Balter, and Sid Sen. 2018. RobinHood: Tail Latency-Aware Caching - Dynamically Reallocating from Cache-Rich to Cache-Poor. In USENIX OSDI.

Digital Library

[7]

Daniel S. Berger, Philipp Gland, Sahil Singla, and Florin Ciucu. 2014. Exact analysis of TTL cache networks. Perform. Eval. 79 (2014), 2--23. Special Issue: Performance 2014.

[8]

Daniel S. Berger, Ramesh Sitaraman, and Mor Harchol-Balter. 2017. AdaptSize: Orchestrating the Hot Object Memory Cache in a CDN. In USENIX NSDI. 483--498.

Digital Library

[9]

Aaron Blankstein, Siddhartha Sen, and Michael J Freedman. 2017. Hyperbolic Caching: Flexible Caching for Web Applications. In USENIX ATC. 499--511.

Digital Library

[10]

Ronen I Brafman and Moshe Tennenholtz. 2002. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research 3, Oct (2002), 213--231.

Digital Library

[11]

Christopher J.C. Burges. 2010. From RankNet to LambdaRank to LambdaMART: An Overview. Technical Report. MSR-TR-2010-82.

[12]

Ludmila Cherkasova. 1998. Improving WWW performance with greedy-dual-size-frequency caching policy. Technical Report. HP Labs.

[13]

Marek Chrobak, Gerhard J Woeginger, Kazuhisa Makino, and Haifeng Xu. 2012. Caching is hard---even in the fault model. Algorithmica 63 (2012), 781--794.

Digital Library

[14]

CISCO. 2017. VNI Global IP Traffic Forecast. Available at hhttps://u.nu/ng-i, accessed 20/10/18.

[15]

Renato Costa and Jose Pazos. 2017. MLCache: A Multi-Armed Bandit Policy for an Operating System Page Cache. Technical Report. UBC.

[16]

Asit Dan and Don Towsley. 1990. An Approximate Analysis of LRU and FIFO Replacement Schemes. In ACM SIGMETRICS. 143--152.

Digital Library

[17]

Jeff Dean. 2018. Is Google Using Reinforcement Learning to Improve Caching? Personal communication on 2018-09-27.

[18]

Philippe Flajolet, Daniele Gardy, and Loÿs Thimonier. 1992. Birthday paradox, coupon collectors, caching algorithms and self-organizing search. Discrete Applied Mathematics 39 (1992), 207--229.

Digital Library

[19]

Syed Hasan, Sergey Gorinsky, Constantine Dovrolis, and Ramesh K Sitaraman. 2014. Trade-offs in optimizing the cache deployments of CDNs. In IEEE INFOCOM. 460--468.

[20]

Ying He, F Richard Yu, Nan Zhao, Victor CM Leung, and Hongxi Yin. 2017. Software-defined networks with mobile edge computing and caching for smart cities: A big data deep reinforcement learning approach. IEEE Communications Magazine 55, 12 (2017), 31--37.

Digital Library

[21]

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep Reinforcement Learning that Matters. In AAAI (Conference on Artificial Intelligence).

[22]

Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. 2018. Rainbow: Combining Improvements in Deep Reinforcement Learning. In AAAI (Conference on Artificial Intelligence).

[23]

Qi Huang, Ken Birman, Robbert van Renesse, Wyatt Lloyd, Sanjeev Kumar, and Harry C Li. 2013. An analysis of Facebook photo caching. In ACM SOSP. 167--181.

Digital Library

[24]

Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. 2017. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR) 50, 2 (2017), 21.

Digital Library

[25]

Alex Irpan. 2016. Faulty Reward Functions in the Wild. OpenAI Blog https://blog.openai.com/faulty-reward-functions/.

[26]

Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. 2017. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. In ACM ICML Reproducibility Workshop.

[27]

Akanksha Jain and Calvin Lin. 2016. Back to the future: leveraging Belady's algorithm for improved cache replacement. In ACM/IEEE ISCA. 78--89.

Digital Library

[28]

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Wei-dong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems. 3146--3154.

Digital Library

[29]

Michael Kearns and Daphne Koller. 1999. Efficient reinforcement learning in factored MDPs. In IJCAI, Vol. 16. 740--747.

Digital Library

[30]

W. Frank King. 1971. Analysis of Demand Paging Algorithms. In IFIP Congress (1). 485--490.

[31]

Péter Kovács. 2015. Minimum-cost flow algorithms: an experimental evaluation. Optimization Methods and Software 30, 1 (2015), 94--127.

Digital Library

[32]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436.

[33]

Mathias Lecuyer, Joshua Lockerman, Lamont Nelson, Siddhartha Sen, Amit Sharma, and Aleksandrs Slivkins. 2017. Harvesting Randomness to Optimize Distributed Systems. In ACM HotNets. 178--184.

Digital Library

[34]

Conglong Li and Alan L Cox. 2015. GD-Wheel: a cost-aware replacement policy for key-value stores. In EUROSYS. 1--15.

Digital Library

[35]

Bruce M Maggs and Ramesh K Sitaraman. 2015. Algorithmic nuggets in content delivery. ACM SIGCOMM CCR 45 (2015), 52--66.

Digital Library

[36]

Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula. 2016. Resource management with deep reinforcement learning. In ACM HotNets. 50--56.

Digital Library

[37]

Matthew K Mukerjee, Ilker Nadi Bozkurt, Bruce Maggs, Srinivasan Seshan, and Hui Zhang. 2016. The impact of brokers on the future of content delivery. In ACM HotNets. 127--133.

Digital Library

[38]

A Neelakantan, L Vilnis, QV Le, I Sutskever, L Kaiser, K Kurach, and J Martens. 2016. Adding Gradient Noise Improves Learning for Very Deep Networks. In ICLR Workshop.

[39]

Andrew Y Ng, Stuart J Russell, et al. 2000. Algorithms for inverse reinforcement learning. In ACM ICML. 663--670.

Digital Library

[40]

E. Nygren, Ramesh K. Sitaraman, and J. Sun. 2010. The Akamai Network: A platform for high-performance Internet applications. ACM SIGOPS Operating Systems Review 44, 3 (2010), 2--19.

Digital Library

[41]

Egerváry Research Group on Combinatorial Optimization. 2015. COIN-OR::LEMON Library. Available at https://u.nu/cqrf, accessed 5/5/18.

[42]

Elizabeth J O'Neil, Patrick E O'Neil, and Gerhard Weikum. 1993. The LRU-K page replacement algorithm for database disk buffering. ACM SIGMOD 22, 2 (1993), 297--306.

Digital Library

[43]

James B Orlin. 1997. A polynomial time primal network simplex algorithm for minimum cost flows. Mathematical Programming 78, 2 (1997), 109--129.

Digital Library

[44]

Stéphane Ross and Drew Bagnell. 2010. Efficient reductions for imitation learning. In AISTATS. 661--668.

[45]

Avik Sengupta, SaiDhiraj Amuru, Ravi Tandon, R Michael Buehrer, and T Charles Clancy. 2014. Learning distributed caching strategies in small cell networks. In IEEE ISWCS. 917--921.

[46]

Ramesh K. Sitaraman, Mangesh Kasbekar, Woody Lichtenstein, and Manish Jain. 2014. Overlay networks: An Akamai perspective. In Advanced Content Delivery. John Wiley & Sons.

[47]

Anirudh Sivaraman, Keith Winstein, Pratiksha Thaker, and Hari Balakrishnan. 2014. An experimental study of the learnability of congestion control. In ACM SIGCOMM, Vol. 44. 479--490.

Digital Library

[48]

Yi Sun, Xiaoqi Yin, Junchen Jiang, Vyas Sekar, Fuyuan Lin, Nanshu Wang, Tao Liu, and Bruno Sinopoli. 2016. CS2P: Improving video bitrate selection and adaptation with data-driven throughput prediction. In ACM SIGCOMM. 272--285.

Digital Library

[49]

Aditya Sundarrajan, Mingdong Feng, Mangesh Kasbekar, and Ramesh K Sitaraman. 2017. Footprint Descriptors: Theory and Practice of Cache Provisioning in a Global CDN. In ACM CoNEXT. 55--67.

Digital Library

[50]

Richard S Sutton. 1990. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine Learning Proceedings. 216--224.

Digital Library

[51]

Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement learning: An introduction (2 ed.). MIT press.

[52]

Jun Zhang, Xiao Chen, Yang Xiang, Wanlei Zhou, and Jie Wu. 2015. Robust network traffic classification. IEEE/ACM TON 23, 4 (2015), 1257--1270.

Digital Library

[53]

Chen Zhong, M Cenk Gursoy, and Senem Velipasalar. 2018. A deep reinforcement learning-based framework for content caching. In IEEE CISS (Annual Conference on Information Sciences and Systems). 1--6.

Cited By

Krishna K(2025)Advancements in cache management: a review of machine learning innovations for enhanced performance and securityFrontiers in Artificial Intelligence10.3389/frai.2025.14412508Online publication date: 25-Feb-2025
https://doi.org/10.3389/frai.2025.1441250
Wong DWu HMolder CGunasekar SLu JKhandkar SSharma ABerger DBeckmann NGanger GMa XWon Y(2024)BaleenProceedings of the 22nd USENIX Conference on File and Storage Technologies10.5555/3650697.3650718(347-372)Online publication date: 27-Feb-2024
https://dl.acm.org/doi/10.5555/3650697.3650718
Lyons SRangaswami R(2024)To Cache or Not to CacheAlgorithms10.3390/a1707030117:7(301)Online publication date: 7-Jul-2024
https://doi.org/10.3390/a17070301
Show More Cited By

Recommendations

Selective Victim Caching: A Method to Improve the Performance of Direct-Mapped Caches

Although direct-mapped caches suffer from higher miss ratios as compared to set-associative caches, they are attractive for today's high-speed pipelined processors that require very low access times. Victim caching was proposed by Jouppi [1] as an ...
A machine learning approach for result caching in web search engines

To the best of our knowledge, our work is therst in literature to apply machine learning techniques to the result caching problem in search engines, for both static, dynamic, and state-of-the-art static-dynamic cache organizations.We evaluate a large ...
Cooperative Caching for GPUs

The rise of general-purpose computing on GPUs has influenced architectural innovation on them. The introduction of an on-chip cache hierarchy is one such innovation. High L1 miss rates on GPUs, however, indicate inefficient cache usage due to myriad ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HotNets '18: Proceedings of the 17th ACM Workshop on Hot Topics in Networks

November 2018

191 pages

ISBN:9781450361200

DOI:10.1145/3286062

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

HotNets '18

Sponsor:

SIGCOMM

HotNets '18: The 17th ACM workshop on Hot Topics in Networks

November 15 - 16, 2018

WA, Redmond, USA

Acceptance Rates

Overall Acceptance Rate 110 of 460 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

63
Total Citations
View Citations
928
Total Downloads

Downloads (Last 12 months)83
Downloads (Last 6 weeks)14

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Krishna K(2025)Advancements in cache management: a review of machine learning innovations for enhanced performance and securityFrontiers in Artificial Intelligence10.3389/frai.2025.14412508Online publication date: 25-Feb-2025
https://doi.org/10.3389/frai.2025.1441250
Wong DWu HMolder CGunasekar SLu JKhandkar SSharma ABerger DBeckmann NGanger GMa XWon Y(2024)BaleenProceedings of the 22nd USENIX Conference on File and Storage Technologies10.5555/3650697.3650718(347-372)Online publication date: 27-Feb-2024
https://dl.acm.org/doi/10.5555/3650697.3650718
Lyons SRangaswami R(2024)To Cache or Not to CacheAlgorithms10.3390/a1707030117:7(301)Online publication date: 7-Jul-2024
https://doi.org/10.3390/a17070301
Suriyakumar YTallent NMarquez AKaravanic KKilic O(2024)MemFriend: Understanding Memory Performance with Spatial-Temporal AffinityProceedings of the International Symposium on Memory Systems10.1145/3695794.3695820(270-284)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3695794.3695820
Vanerio JHügerich LSchmid S(2024)Tero: Offloading CDN Traffic to Massively Distributed DevicesProceedings of the 25th International Conference on Distributed Computing and Networking10.1145/3631461.3631556(186-198)Online publication date: 4-Jan-2024
https://dl.acm.org/doi/10.1145/3631461.3631556
Torabi HKhazaei HLitoiu MBalsamo SKnottenbelt WAbad CShang W(2024)A Learning-Based Caching Mechanism for Edge Content DeliveryProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645037(236-246)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645037
Guo XWang HZhou KJiang HHan YXing G(2024)FLOWS: Balanced MRC Profiling for Heterogeneous Object-Size CacheProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3650078(421-440)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3627703.3650078
Wang PJiang HLiu YZhao ZZhou KHuang Z(2024)Beyond Belady to Attain a Seemingly Unattainable Byte Miss Ratio for Content Delivery NetworksIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.345209635:11(1949-1963)Online publication date: Nov-2024
https://doi.org/10.1109/TPDS.2024.3452096
Zhou YWang FShi ZFeng D(2024)An Efficient Deep Reinforcement Learning-Based Automatic Cache Replacement Policy in Cloud Block Storage SystemsIEEE Transactions on Computers10.1109/TC.2023.332562573:1(164-177)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TC.2023.3325625
Ferreira IOki E(2024)Latency-Aware Cache Mechanism for Resolver Service of Domain Name SystemsNOMS 2024-2024 IEEE Network Operations and Management Symposium10.1109/NOMS59830.2024.10575387(1-4)Online publication date: 6-May-2024
https://doi.org/10.1109/NOMS59830.2024.10575387
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten