Skip to main content
SpringerLink
Log in

Multi-agent learning algorithms for content placement in cache-enabled small cell networks: 4G and 5G use cases

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Caching popular files at the small base stations (SBSs) has proved to be an effective strategy to reduce the content delivery delay in cellular networks and to alleviate the backhaul congestion. In the optimization of the placement of contents into SBS caches (the so-called content placement problem), several key parameters play an important role, such as content popularity, the mobile users’ (MUs’) channel state information (CSI), as well as the capacity of the backhaul links. These parameters are random in general, and their instantaneous values over time give rise to a stochastic process. In this paper, we propose a mathematical formulation for the distributed optimization of content placement with the objective of minimizing the average content delivery latency. Our formulation is applicable to both conventional 4G small cell networks (SCNs) as well as 5G-compatiable mmWave integrated access and backhaul (IAB) cellular communications. In particular, the placement problem is modeled as a potential game among SBSs in which the objective of each SBS is to minimize the average delay of the MUs within its coverage range. In order to compute the Nash equilibrium (NE) of the game, we adopt the learning-theoretic approach that only relies on incomplete information (or implicit feedback) of the system’s underlying stochastic processes; i.e., the content placement is optimized in run-time by gaining experience and through the immediate noisy feedbacks of the actions actually taken in the operating environment. We propose an algorithm based on multi-agent reinforcement learning (MARL) techniques for potential games. It operates in the independent action space and can learn the optimal strategy profile of the SBSs in larger-scale scenarios, even when the actions of its peers are not observable by each SBS. Simulation experiments are conducted to investigate the convergence of the learning algorithm as well as to compare against some schemes using prior knowledge.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Availability of Data and Material

Data sharing is not applicable—no new data generated.

Notes

  1. The prefix mm in mm-SBS is used on several occasions to emphasize the 5G-compatibility of an SBS. To improve readability, however, we may use SBS to refer to both 4G/5G small base stations. The exact type of SBS in each case should be clear from context.

References

  1. Golrezaei N, Molisch AF, Dimakis AG, Caire G (2013) Femtocaching and device-to-device collaboration: a new architecture for wireless video distribution. IEEE Commun Mag 51(4):142–149. https://doi.org/10.1109/MCOM.2013.6495773

    Article  Google Scholar 

  2. Shanmugam K, Golrezaei N, Dimakis AG, Molisch AF, Caire G (2013) Femtocaching: wireless content delivery through distributed caching helpers. IEEE Trans Inf Theory 59(12):8402–8413. https://doi.org/10.1109/TIT.2013.2281606

    Article  MathSciNet  MATH  Google Scholar 

  3. Ahlehagh H, Dey S (2014) Video-aware scheduling and caching in the radio access network. IEEE/ACM Trans Netw 22(5):1444–1462. https://doi.org/10.1109/TNET.2013.2294111

    Article  Google Scholar 

  4. Wang X, Chen M, Taleb T, Ksentini A, Leung VCM (2014) Cache in the air: exploiting content caching and delivery techniques for 5G systems. IEEE Commun Mag 52(2):131–139. https://doi.org/10.1109/MCOM.2014.6736753

    Article  Google Scholar 

  5. Al-Turjman F (2018) Fog-based caching in software-defined information-centric networks. Comput Electr Eng 69:54–67. https://doi.org/10.1016/j.compeleceng.2018.05.018

    Article  Google Scholar 

  6. Liu D, Chen B, Yang C, Molisch AF (2016) Caching at the wireless edge: design aspects, challenges, and future directions. IEEE Commun Mag 54(9):22–28. https://doi.org/10.1109/MCOM.2016.7565183

    Article  Google Scholar 

  7. Yao J, Han T, Ansari N (2019) On mobile edge caching. IEEE Communications Surveys and Tutorials 21(3):2525–2553. https://doi.org/10.1109/COMST.2019.2908280

    Article  Google Scholar 

  8. Jeong M-W, Ryu JY, Kim SH, Lee W, Ban T-W (2020) A completely distributed transmission algorithm for mobile device-to-device caching networks. Comput Electr Eng 87:106803. https://doi.org/10.1016/j.compeleceng.2020.106803

    Article  Google Scholar 

  9. Yang Z, Tian H, Fan S, Chen G (2017) Distributed cooperative caching in backhaul-limited small cell networks. Electron Lett 53(3):158–160. https://doi.org/10.1049/el.2016.3221

    Article  Google Scholar 

  10. Haw R, Kazmi SMA, Thar K, Alam MGR, Hong CS (2019) Cache aware user association for wireless heterogeneous networks. IEEE Access 7:3472–3485. https://doi.org/10.1109/ACCESS.2018.2885571

    Article  Google Scholar 

  11. Lin X, Tang Y, Lei X, Xia J, Zhou Q, Wu H, Fan L (2019) MARL-based distributed cache placement for wireless networks. IEEE Access 7:62606–62615. https://doi.org/10.1109/ACCESS.2019.2916155

    Article  Google Scholar 

  12. Mishra SK, Pandey P, Arya P, Jain A (2018) Efficient proactive caching in storage constrained 5g small cells. In: Proceedings of the 10th international conference on communication systems and networks, pp 291–296. https://doi.org/10.1109/COMSNETS.2018.8328210

  13. Nie T, Luo J, Gao L, Zheng FC, Yu L (2020) Cooperative edge caching in small cell networks with heterogeneous channel qualities. In: IEEE 91st vehicular technology conference. https://doi.org/10.1109/VTC2020-Spring48590.2020.9128365

  14. Jiang W, Feng G, Qin S, Liu Y (2019) Multi-agent reinforcement learning based cooperative content caching for mobile edge networks. IEEE Access 7:61856–61867. https://doi.org/10.1109/ACCESS.2019.2916314

    Article  Google Scholar 

  15. Xu X, Tao M, Shen C (2020) Collaborative multi-agent multi-armed bandit learning for small-cell caching. IEEE Trans Wireless Commun 19(4):2570–2585. https://doi.org/10.1109/TWC.2020.2966599

    Article  Google Scholar 

  16. Zhu H, Cao Y, Wang W, Jiang T, Jin S (2018) Deep reinforcement learning for mobile edge caching: review, new features, and open issues. IEEE Network 32(6):50–57. https://doi.org/10.1109/MNET.2018.1800109

    Article  Google Scholar 

  17. Liao J, Wong K-K, Khandaker MRA, Zheng Z (2017) Optimizing cache placement for heterogeneous small cell networks. IEEE Commun Lett 21(1):120–123. https://doi.org/10.1109/LCOMM.2016.2612197

    Article  Google Scholar 

  18. Chen M, Hao Y, Hu L, Huang K, Lau VKN (2017) Green and mobility-aware caching in 5G networks. IEEE Trans Wireless Commun 16(12):8347–8361. https://doi.org/10.1109/TWC.2017.2760830

    Article  Google Scholar 

  19. Mohammed L, Jaseemuddin M, Anpalagan A (2019) Fuzzy soft-set based approach for femto-caching in wireless networks. In: Proceedings of the 20th international conference on high performance computing and communications, pp 487–494. https://doi.org/10.1109/HPCC/SmartCity/DSS.2018.00095

  20. Zhou F, Fan L, Jiang M, Chen W (2019) Optimal caching strategy for coordinated small-cells with limited backhaul. IEEE Wireless Commun Lett 8(6):1583–1586. https://doi.org/10.1109/LWC.2019.2929156

    Article  Google Scholar 

  21. Liao J, Wong K-K, Zhang Y, Zheng Z, Yang K (2017) MDS coded cooperative caching for heterogeneous small cell networks. In: Proceedings of the IEEE global communications conference, pp 1–7. https://doi.org/10.1109/GLOCOM.2017.8254854

  22. Liao J, Wong K-K, Zhang Y, Zheng Z, Yang K (2017) Coding, multicast, and cooperation for cache-enabled heterogeneous small cell networks. IEEE Trans Wireless Commun 16(10):6838–6853. https://doi.org/10.1109/TWC.2017.2731967

    Article  Google Scholar 

  23. Keshavarzian I, Zeinalpour-Yazdi Z, Tadaion A (2016) A clustered caching placement in heterogeneous small cell networks with user mobility. In: Proceedings of the IEEE international symposium on signal processing and information technology, pp 421–426. https://doi.org/10.1109/ISSPIT.2015.7394372

  24. Xu X, Tao M (2018) Collaborative multi-agent reinforcement learning of caching optimization in small-cell networks. In: IEEE global communications conference. https://doi.org/10.1109/GLOCOM.2018.8647341

  25. Tekin C, Liu M (2015) Online learning methods for networking. Now Publishers Inc

  26. Astely D, Dahlman E, Fodor G, Parkvall S, Sachs J (2013) LTE release 12 and beyond [Accepted from Open Call]. IEEE Commun Mag 51(7):154–160. https://doi.org/10.1109/MCOM.2013.6553692

    Article  Google Scholar 

  27. 3GPP (2017) Study on integrated access and backhaul for NR,” AT&T, Qualcomm, Samsung—Tdoc RP-171880

  28. Nash J (1951) Non-cooperative games. Ann Math 54(2):286–295. https://doi.org/10.2307/1969529

    Article  MathSciNet  MATH  Google Scholar 

  29. Monderer D, Shapley LS (1996) Potential games. Games Econ Behav 14(1):124–143. https://doi.org/10.1006/game.1996.0044

    Article  MathSciNet  MATH  Google Scholar 

  30. Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. The MIT Press, Cambridge

    MATH  Google Scholar 

  31. Guo F, Zhang H, Li X, Ji H (2018) Content caching in energy harvesting powered small cell network. In: Proceedings of the IEEE international symposium on personal, indoor and mobile radio communications, pp 1–5. https://doi.org/10.1109/PIMRC.2017.8292218

  32. Yang L, Zheng FC, Wen W, Jin S (2020) Analysis and optimization of random caching in mmWave heterogeneous networks. IEEE Trans Veh Technol 69(9):10140–10154. https://doi.org/10.1109/TVT.2020.3001203

    Article  Google Scholar 

  33. Gu Z, Lu H, Zhang M, Sun H, Chen CW (2021) Association and caching in relay-assisted Mmwave networks: from a stochastic geometry perspective. IEEE Trans Wireless Commun. https://doi.org/10.1109/TWC.2021.3091815

    Article  Google Scholar 

  34. Zheng TX, Liu HW, Zhang N, Ding Z, Leung VC (2020) Secure content delivery in two-tier cache-enabled mmWave heterogeneous networks. IEEE Trans Inf Forens Secur 16:1640–1654. https://doi.org/10.1109/TIFS.2020.3040877

    Article  Google Scholar 

  35. Zhang T, Biswas S, Ratnarajah T (2019) An analysis on caching in full-duplex enabled mmWave IAB HetNets. In: Proceedings of the 16th international symposium on wireless communication systems, pp 75–80. https://doi.org/10.1109/ISWCS.2019.8877120

  36. Li X, Wang X, Sheng Z, Zhou H, Leung VCM (2018) Resource allocation for cache-enabled cloud-based small cell networks. Comput Commun 127:20–29. https://doi.org/10.1016/j.comcom.2018.05.007

    Article  Google Scholar 

  37. Khan BS, Jangsher S, Qureshi HK, Mumtaz S (2019) Energy efficient caching in cooperative small cell network. In: IEEE annual consumer communications and networking conference. https://doi.org/10.1109/CCNC.2019.8651764

  38. Muller S, Atan O, Van Der Schaar M, Klein A (2017) Context-aware proactive content caching with service differentiation in wireless networks. IEEE Trans Wireless Commun 16(2):1024–1036. https://doi.org/10.1109/TWC.2016.2636139

    Article  Google Scholar 

  39. Zhang C, Ren P, Du Q (2017) A Contextual multi-armed bandit approach to caching in wireless small cell network. In: Proceedings of the 9th international conference on wireless communications and signal processing, pp 1–6. https://doi.org/10.1109/WCSP.2017.8171043

  40. He D, Jiang J, Yang G, Westphal C (2019) Pushing smart caching to the edge with bay-cache. In: Proceedings of the 16th EAI international conference on mobile and ubiquitous systems: computing, networking and services, pp 90–99. https://doi.org/10.1145/3360774.3360823

  41. Alqasir A, Aldubaikhy K, Kamal AE (2021) Integrated access and backhauling with energy harvesting and dynamic sleeping in HetNets. In: Proceedings of the IEEE international conference on communications, pp 1–6. https://doi.org/10.1109/ICC42927.2021.9500432

  42. Noh S, Ying D, Li Q, Ghozlan H, Papathanassiou A, Wu G (2018) System evaluation for millimeter-wave radio access network. In: Proceedings of the IEEE international conference on communications, pp 1–6. https://doi.org/10.1109/ICC.2018.8423043

  43. International Telecommunication Union (2009) Requirements related to technical performance for IMTadvanced radio interfaces. ITU I.2134

  44. Bai T, Heath RW (2014) Coverage and rate analysis for millimeter-wave cellular networks. IEEE Trans Wireless Commun 14(2):1100–1114. https://doi.org/10.1109/TWC.2014.2364267

    Article  Google Scholar 

  45. Thornburg A, Bai T, Heath RW (2016) Performance analysis of outdoor mmWave ad hoc networks. IEEE Trans Signal Process 64(15):4065–4079. https://doi.org/10.1109/TSP.2016.2551690

    Article  MathSciNet  MATH  Google Scholar 

  46. Liu Y, Fang X, Xiao M (2017) Discrete power control and transmission duration allocation for self-backhauling dense nmWave cellular networks. IEEE Trans Commun 66(1):432–447. https://doi.org/10.1109/TCOMM.2017.2757017

    Article  Google Scholar 

  47. Shokri-Ghadikolaei H, Fischione C, Fodor G, Popovski P, Zorzi M (2015) Millimeter wave cellular networks: a MAC layer perspective. IEEE Trans Commun 63(10):3437–3458. https://doi.org/10.1109/TCOMM.2015.2456093

    Article  Google Scholar 

  48. Liu Q, Tian H, Nie G, Wu H (2018) Context-aware data caching and resource allocation in HetNets with self-backhaul. In: Proceedings of the IEEE/CIC international conference on communications, pp 416–420. https://doi.org/10.1109/ICCChina.2018.8641136

  49. Wang M, Dutta A, Buccapatnam S, Chiang M (2016) Smart exploration in Hetnets: minimizing total regret with mmWave. In: IEEE international conference on sensing, communication and networking, p 33

  50. Chapman C, Leslie DS, Rogers A, Jennings NR (2013) Convergent learning algorithms for unknown reward games. Soc Ind Appl Math J Control Optim 51(4):3154–3180. https://doi.org/10.1137/120893501

    Article  MathSciNet  MATH  Google Scholar 

  51. Wang Y, Pavel L (2014) A modified Q-learning algorithm for potential games. Int Fed Autom Control Proc Vol 47(3):8710–8718. https://doi.org/10.3182/20140824-6-ZA-1003.02646

    Article  Google Scholar 

  52. Fudenberg D, Tirole J (1991) Game theory. MIT Press, Cambridge

    MATH  Google Scholar 

  53. Claus C, Boutilier C (1998) The dynamics of reinforcement learning in cooperative multi-agent systems. In: Proceedings of the 15th AAAI national conference on artificial intelligence, pp 746–752

  54. Fudenberg D, Levine DK (1998) The theory of learning in games. MIT Press, Cambridge

    MATH  Google Scholar 

  55. Leslie DS, Collins EJ (2006) Generalised weakened fictitious play. Games Econom Behav 56(2):285–298. https://doi.org/10.1016/j.geb.2005.08.005

    Article  MathSciNet  MATH  Google Scholar 

  56. Mertikopoulos P, Moustakas AL (2010) The emergence of rational behavior in the presence of stochastic perturbations. Ann Appl Probab 20(4):1359–1388. https://doi.org/10.1214/09-AAP651

    Article  MathSciNet  MATH  Google Scholar 

  57. Hofbauer J, Sandholm WH (2007) Evolution in games with randomly disturbed payoffs. J Econ Theory 132(1):47–69. https://doi.org/10.1016/j.jet.2005.05.011

    Article  MathSciNet  MATH  Google Scholar 

  58. Foster DP, Young HP (2006) Regret testing: learning to play nash equilibrium without knowing you have an opponent. Theor Econ 1(3):341–367

    Google Scholar 

  59. Jaakkola T, Jordan MI, Singh PS (1994) On the convergence of stochastic iterative dynamic programming algorithms. Neural Comput 6(6):1185–1201. https://doi.org/10.1162/neco.1994.6.6.1185

    Article  MATH  Google Scholar 

  60. Young HP (2004) Strategic learning and its limits. Oxford University Press, Oxford

    Book  Google Scholar 

  61. Leslie DS, Collins EJ (2006) Individual Q-learning in normal form games. SIAM J Control Optim 44(2):495–514. https://doi.org/10.1137/S0363012903437976

    Article  MathSciNet  MATH  Google Scholar 

  62. Cominetti R, Melo E, Sorin S (2010) A payoff-based learning procedure and its application to traffic games. Games Econom Behav 70(1):71–83. https://doi.org/10.1016/j.geb.2008.11.012

    Article  MathSciNet  MATH  Google Scholar 

  63. Marden JR, Young HP, Arslan G, Shamma JS (2009) Payoff-based dynamics for multiplayer weakly acyclic games. SIAM J Control Optim 48(1):373–396. https://doi.org/10.1137/070680199

    Article  MathSciNet  MATH  Google Scholar 

  64. Chasparis GC, Shamma JS, Rantzer A (2011) Perturbed learning automata in potential games. In: Proceedings of the IEEE conference on decision and control, pp 2453–2458. https://doi.org/10.1109/CDC.2011.6161294

  65. Wang Y (2014) A modified Q-learning algorithm in games. Master’s thesis. University of Toronto. https://doi.org/10.3182/20140824-6-ZA-1003.02646

  66. Meesa-ard E, Pattaramalai S, Madapatha MDC (2018) Outage probability of mobility incorporated alpha-mu Fading distribution with co-channel interference in heterogeneous networks. In: Proceedings of the IEEE international conference on smart internet of things, pp 76–80. https://doi.org/10.1109/SmartIoT.2018.00023

  67. Meesa-Ard E, Pattaramalai S, Madapatha MDC (2018) Evaluating the impact of mobility over K-Il generalized fading channels in digital communication. In: Proceedings of the 8th international conference on electronics information and emergency communication, pp 35–39. https://doi.org/10.1109/ICEIEC.2018.8473558

  68. Mesodiakaki A, Adelantado F, Alonso L, Di Renzo M, Verikoukis C (2016) Energy-and spectrum-efficient user association in millimeter-wave backhaul small-cell networks. IEEE Trans Veh Technol 66(2):1810–1821. https://doi.org/10.1109/TVT.2016.2565539

    Article  Google Scholar 

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran

    Zahra Rashidi & Sara Rajaee

  2. Center of Excellence in Future Networks, School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran

    Vesal Hakami

  3. Department of Industrial Engineering, Iran University of Science and Technology, Tehran, Iran

    Parmida Geranmayeh

Authors
  1. Zahra Rashidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vesal Hakami
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Parmida Geranmayeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sara Rajaee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vesal Hakami.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rashidi, Z., Hakami, V., Geranmayeh, P. et al. Multi-agent learning algorithms for content placement in cache-enabled small cell networks: 4G and 5G use cases. Neural Comput & Applic 34, 11641–11668 (2022). https://doi.org/10.1007/s00521-022-07051-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07051-5

Keywords

Search

Navigation