Skip to main content
SpringerLink
Log in

Intelligent edge content caching: A deep recurrent reinforcement learning method

  • Published:
Peer-to-Peer Networking and Applications Aims and scope Submit manuscript

Abstract

With the rise of 5G network and the rapid growth of user equipment, there exists a gap between the stringent requirements of emerging applications and the actual functionality of the Internet. In particular, transmitting data over long network links imposes high costs, which can be addressed by the edge caching (EC) method. EC caches the content at the edge server to avoid the extraordinary cost of backhaul link communication. However, in existing EC efforts, it is common to assume either known content popularity or a two-phase caching that is predicted content popularity prior to the caching action, the former being less feasible and the latter increasing the cost of deployment to the real world. A caching strategy is proposed in this paper to cope with this problem that can be feasible end-to-end deployed and has a lower caching cost. Specifically, we first investigate the system cost, including network communication cost, cache over storage cost, and cache replacement cost. And we model the EC problem as a Markov Decision Process (MDP). Then, the Double Deep Recurrent Q Network (DDRQN) algorithm is studied to solve the EC-based MDP problem. Finally, compared with other intelligent caching strategies, the proposed caching strategy can improve the system reward by up to 24% and the cache hit rate by up to 22% under certain conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Cisco (2020) Cisco annual internet report (2018–2023) white paper. [Online]. Available: https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html

  2. Pham Q-V, Fang F, Ha VN, Piran MJ, Le M, Le LB, Hwang W-J, Ding Z (2020) A survey of multi-access edge computing in 5g and beyond: Fundamentals, technology integration, and state-of-the-art. IEEE Access 8:116974–117017

    Article  Google Scholar 

  3. Velte T, Velte A, Elsenpeter R (2009) Cloud computing, a practical approach. McGraw-Hill, Inc.

  4. Zhou Z, Chen X, Li E, Zeng L, Luo K, Zhang J (2019) Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proc IEEE 107(8):1738–1762

    Article  Google Scholar 

  5. Xu Y, Gu B, Hu RQ, Li D, Zhang H (2021) Joint computation offloading and radio resource allocation in mec-based wireless-powered backscatter communication networks. IEEE Trans Veh Technol 70(6):6200–6205

    Article  Google Scholar 

  6. Zhuang W, Ye Q, Lyu F, Cheng N, Ren J (2019) Sdn/nfv-empowered future iov with enhanced communication, computing, and caching. Proc IEEE 108(2):274–291

    Article  Google Scholar 

  7. Xia X, Chen F, He Q, Grundy J, Abdelrazek M, Jin H (2020) Online collaborative data caching in edge computing. IEEE Trans Parallel Distrib Syst 32(2):281–294

    Article  Google Scholar 

  8. Nour B, Khelifi H, Moungla H, Hussain R, Guizani N (2020) A distributed cache placement scheme for large-scale information-centric networking. IEEE Network

  9. Ale L, Zhang N, Wu H, Chen D, Han T (2019) Online proactive caching in mobile edge computing using bidirectional deep recurrent neural network. IEEE Internet Things J 6(3):5520–5530

    Article  Google Scholar 

  10. Wu Z, Lu Z, Zhang W, Wu J, Huang S, Hung PC (2018) A data-driven approach of performance evaluation for cache server groups in content delivery network. J Parallel Distrib Comput 119:162–171

    Article  Google Scholar 

  11. Zeng Y, Xie J, Jiang H, Huang G, Yi S, Xiong N, Li J (2019) Smart caching based on user behavior for mobile edge computing. Inf Sci 503:444–468

    Article  Google Scholar 

  12. Rathore S, Ryu JH, Sharma PK, Park JH (2019) Deepcachnet: a proactive caching framework based on deep learning in cellular networks. IEEE Network 33(3):130–138

    Article  Google Scholar 

  13. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533

  14. Zhong C, Gursoy MC, Velipasalar S (2018) A deep reinforcement learning-based framework for content caching. In: 2018 52nd Annual Conference on Information Sciences and Systems (CISS). IEEE, pp 1–6

  15. He X, Wang K, Xu W (2019) Qoe-driven content-centric caching with deep reinforcement learning in edge-enabled iot. IEEE Comput Intell Mag 14(4):12–20

    Article  Google Scholar 

  16. Qiao G, Leng S, Maharjan S, Zhang Y, Ansari N (2019) Deep reinforcement learning for cooperative content caching in vehicular edge computing and networks. IEEE Internet Things J 7(1):247–257

    Article  Google Scholar 

  17. Hou L, Lei L, Zheng K, Wang X (2018) A q-learning-based proactive caching strategy for non-safety related services in vehicular networks. IEEE Internet Things J 6(3):4512–4520

    Article  Google Scholar 

  18. Li S, Li B, Zhao W (2020) Joint optimization of caching and computation in multi-server noma-mec system via reinforcement learning. IEEE Access 8:112762–112771

    Article  Google Scholar 

  19. Hausknecht M, Stone P (2015) Deep recurrent q-learning for partially observable mdps. arXiv preprint: arXiv:1507.06527

  20. Liu Y, He Q, Zheng D, Xia X, Chen F, Zhang B (2020) Data caching optimization in the edge computing environment. IEEE Trans Serv Comput 1–8

  21. Hu Z, Zheng Z, Wang T, Song L, Li X (2016) Game theoretic approaches for wireless proactive caching. IEEE Commun Mag 54(8):37–43

    Article  Google Scholar 

  22. Yu G, Wu J (2020) Content caching based on mobility prediction and joint user prefetch in mobile edge networks. Peer-to-Peer Netw Appl 13(5):1839–1852

    Article  Google Scholar 

  23. Saputra YM, Hoang DT, Nguyen DN, Dutkiewicz E, Niyato D, Kim DI (2019) Distributed deep learning at the edge: a novel proactive and cooperative caching framework for mobile edge networks. IEEE Wireless Commun Lett 8(4):1220–1223

    Article  Google Scholar 

  24. Liang J, Zhu D, Liu H, Ping H, Li T, Zhang H, Geng L, Liu Y (2020) Multi-head attention based popularity prediction caching in social content-centric networking with mobile edge computing. IEEE Commun Lett 25(2):508–512

    Article  Google Scholar 

  25. Bernardini C, Silverston T, Festor O (2014) Sonetor: a social network traffic generator. In: 2014 IEEE International Conference on Communications (ICC). IEEE, pp 3734–3739

  26. Tang B, Kang L (2021) Eicache: a learning-based intelligent caching strategy in mobile edge computing. Peer-to-Peer Netw Appl 1–16

  27. Zhu H, Cao Y, Wang W, Jiang T, Jin S (2018) Deep reinforcement learning for mobile edge caching: Review, new features, and open issues. IEEE Network 32(6):50–57

    Article  Google Scholar 

  28. Wang X, Han Y, Wang C, Zhao Q, Chen X, Chen M (2019) In-edge AI: Intelligentizing mobile edge computing, caching and communication by federated learning. IEEE Network 33(5):156–165

    Article  Google Scholar 

  29. Zhang N, Zheng K, Tao M (2018) Using grouped linear prediction and accelerated reinforcement learning for online content caching. In: 2018 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, pp 1–6

  30. Tang J, Tang H, Zhang X, Cumanan K, Chen G, Wong K-K, Chambers JA (2019) Energy minimization in d2d-assisted cache-enabled internet of things: a deep reinforcement learning approach. IEEE Trans Industr Inf 16(8):5412–5423

    Article  Google Scholar 

  31. Ye Z, Zhang D, Wu Z-G, Yan H (2021) A3c-based intelligent event-triggering control of networked nonlinear unmanned marine vehicles subject to hybrid attacks. IEEE Trans Intell Transport Syst

  32. Zhao Y, Li R, Wang C, Wang X, Leung VC (2021) Neighboring-aware caching in heterogeneous edge networks by actor-attention-critic learning. In: ICC 2021-IEEE International Conference on Communications. IEEE, pp 1–6

  33. Xu Y, Xie H, Li D, Hu RQ (2022) Energy-efficient beamforming for heterogeneous industrial iot networks with phase and distortion noises. IEEE Trans Ind Inform

  34. Ghemawat S, Gobioff H, Leung S-T (2003) The google file system. In: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, pp 29–43

  35. Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 8(3–4):279–292

    Article  MATH  Google Scholar 

  36. Van Hasselt H, Guez A, Silver D (2015) Deep reinforcement learning with double q-learning. arXiv preprint: arXiv:1509.06461

  37. Zhang D, Ye Z, Chen P, Wang Q-G (2020) Intelligent event-based output feedback control with q-learning for unmanned marine vehicle systems. Control Eng Pract 105:104616

    Article  Google Scholar 

  38. Zhang D, Ye Z, Feng G, Li H (2021) Intelligent event-based fuzzy dynamic positioning control of nonlinear unmanned marine vehicles under dos attack. IEEE Trans Cybern

Download references

Funding

This work is supported by National Science Foundation of China under Grant 61703127, Key R&D Program of Zhejiang Province 2021C01114, and the Graduate Scientific Research Foundation of Hangzhou Dianzi University CXJJ2020087.

Author information

Authors and Affiliations

  1. Hangzhou Dianzi University, Hangzhou, Zhejiang, China

    Haitao Xu, Yuejun Sun & Jingnan Gao

  2. Hangzhou Hexing Electrical Co., Ltd, Hangzhou, China

    Jianbo Guo

Authors
  1. Haitao Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuejun Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jingnan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianbo Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuejun Sun.

Ethics declarations

Conflict of interest

All authors in this work declared that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, H., Sun, Y., Gao, J. et al. Intelligent edge content caching: A deep recurrent reinforcement learning method. Peer-to-Peer Netw. Appl. 15, 2619–2632 (2022). https://doi.org/10.1007/s12083-022-01369-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12083-022-01369-6

Keywords

Search

Navigation