Skip to main content

Advertisement

Springer Nature Link
Log in

A novel approach to efficient resource allocation in load-balanced cellular networks using hierarchical DRL

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

5G wireless networks require highly spectral-efficient multiple access techniques, which play an important role in determining the performance of mobile communication systems. Multiple access techniques can be classified into orthogonal and nonorthogonal based on the way the resources are allocated to the users. This paper investigates the joint user association and resource allocation problem in an uplink multicast NOMA system to maximize the power efficiency with guaranteeing the quality-of-experience of all subscribers. We also introduce an adaptive load balancing approach that aspires to obtain “almost optimal” fairness among servers from the quality of service (QoS) perspective in which learning automata (LA) has been used to find the optimal solution for this dynamic problem. This approach contains a sophisticated learning automata which consists of time-separation and the “artificial” ergodic paradigms. Different from conventional model-based resource allocation methods, this paper suggested a hierarchical reinforcement learning based frameworks to solve this non-convex and dynamic power optimization problem, referred to as hierarchical deep learning-based resource allocation framework. The entire resource allocation policies of this framework are adjusted by updating the weights of their neural networks according to feedback of the system. The presented learning automata find the \(\epsilon\)-optimal solution for the problem by resorting to a two-time scale-based SLA paradigm. Numerical results show that the suggested hierarchical resource allocation framework in combination with the load balancing approach, can significantly improve the energy efficiency of the whole NOMA system compared with other approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Abedin SF, Alam MGR, Kazmi SA, Tran NH, Niyato D, Hong CS (2018) Resource allocation for ultra-reliable and enhanced mobile broadband IoT applications in fog network. IEEE Trans Commun 67(1):489–502

    Article  Google Scholar 

  • Baidas MW, Alsusa E, Hamdi KA (2019) Joint relay selection and energy-efficient power allocation strategies in energy-harvesting cooperative NOMA networks. Trans Emerg Telecommun Technol 30(7):e3593

    Google Scholar 

  • Cao X, Ma R, Liu L, Shi H, Cheng Y, Sun C (2018) A machine learning-based algorithm for joint scheduling and power control in wireless networks. IEEE Internet Things J 5(6):4308–4318

    Article  Google Scholar 

  • Celdrán AH, Pérez MG, Clemente FJG, Pérez GM (2019) Towards the autonomous provision of self-protection capabilities in 5G networks. J Ambient Intell Humaniz Comput 10(12):4707–4720

    Article  Google Scholar 

  • Colonius F, Rasmussen M (2021) Quasi-ergodic limits for finite absorbing Markov chains. Linear Algebra Appl 609:253–288

    Article  MathSciNet  Google Scholar 

  • Fu S, Fang F, Zhao L, Ding Z, Jian X (2019) Joint transmission scheduling and power allocation in non-orthogonal multiple access. IEEE Trans Commun 67(11):8137–8150

    Article  Google Scholar 

  • Gui G, Huang H, Song Y, Sari H (2018) Deep learning for an effective nonorthogonal multiple access scheme. IEEE Trans Veh Technol 67(9):8440–8450

    Article  Google Scholar 

  • Huang H, Peng Y, Yang J, Xia W, Gui G (2019) Fast beamforming design via deep learning. IEEE Trans Veh Technol 69(1):1065–1069

    Article  Google Scholar 

  • Khan WU, Liu J, Jameel F, Sharma V, Jantti R, Han Z (2020) Spectral efficiency optimization for next generation NOMA-enabled IoT networks. IEEE Trans Veh Technol 69(12):15284–15297

  • Li J, Bao X, Zhang W, Bao N (2020) QoE probability coverage model of indoor visible light communication network. IEEE Access 8:45390–45399

    Article  Google Scholar 

  • Liu C, Jin Z, Gu J, Qiu C (2017) Short-term load forecasting using a long short-term memory network. In: 2017 IEEE PES innovative smart grid technologies conference Europe (ISGT-Europe), pp 1–6

  • Liu M, Song T, Gui G (2018) Deep cognitive perspective: resource allocation for NOMA-based heterogeneous IoT with imperfect SIC. IEEE Internet Things J 6(2):2885–2894

    Article  Google Scholar 

  • Liu P, Li Y, Cheng W, Zhang W, Gao X (2019a) Energy-efficient power allocation for millimeter wave beamspace MIMO-NOMA systems. IEEE Access 7:114582–114592

    Article  Google Scholar 

  • Liu Y, Yu H, Xie S, Zhang Y (2019b) Deep reinforcement learning for offloading and resource allocation in vehicle edge computing and networks. IEEE Trans Veh Technol 68(11):11158–11168

    Article  Google Scholar 

  • Maimó LF, Celdrán AH, Pérez MG, Clemente FJG, Pérez GM (2019) Dynamic management of a deep learning-based anomaly detection system for 5G networks. J Ambient Intell Humaniz Comput 10(8):3083–3097

    Article  Google Scholar 

  • Pan H, Liew SC, Liang J, Shao Y, Lu L (2018) Network-coded multiple access on unmanned aerial vehicle. IEEE J Sel Areas Commun 36(9):2071–2086

    Article  Google Scholar 

  • Praveenchandar J, Tamilarasi A (2020) Dynamic resource allocation with optimized task scheduling and improved power management in cloud computing. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-020-01794-6

  • Qiu C, Hu Y, Chen Y, Zeng B (2019) Deep deterministic policy gradient (DDPG)-based energy harvesting wireless communications. IEEE Internet Things J 6(5):8577–8588

    Article  Google Scholar 

  • Shu Y, Zhu F (2020) An edge computing offloading mechanism for mobile peer sensing and network load weak balancing in 5G network. J Ambient Intell Humaniz Comput 11(2):503–510

    Article  Google Scholar 

  • Song Z, Ni Q, Sun X (2018) Spectrum and energy efficient resource allocation with QoS requirements for hybrid MC-NOMA 5G systems. IEEE Access 6:37055–37069

    Article  Google Scholar 

  • Teng Z, Zhang B, Fan J (2020) Three-step action search networks with deep q-learning for real-time object tracking. Pattern Recogn 101:107188

    Article  Google Scholar 

  • Verhelst M, Moons B (2017) Embedded deep neural network processing: algorithmic and processor techniques bring deep learning to IOT and edge devices. IEEE Solid State Circuits Mag 9(4):55–65

    Article  Google Scholar 

  • Wei Z, Zhao L, Guo J, Ng DWK, Yuan J (2018) Multi-beam NOMA for hybrid mmWave systems. IEEE Trans Commun 67(2):1705–1719

    Article  Google Scholar 

  • Weisz G, Budzianowski P, Su PH, Gašić M (2018) Sample efficient deep reinforcement learning for dialogue systems with large action spaces. IEEE ACM Trans Audio Speech Lang Process 26(11):2083–2097

    Article  Google Scholar 

  • Wu D, Dong X, Shen J, Hoi SC (2020) Reducing estimation bias via triplet-average deep deterministic policy gradient. IEEE Trans Neural Netw Learn Syst 31(11):4933–4945

    Article  MathSciNet  Google Scholar 

  • Xie D, Zhong X (2020) Semicentralized deep deterministic policy gradient in cooperative StarCraft games. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2020.3042943

  • Xu X, Zuo L, Li X, Qian L, Ren J, Sun Z (2018) A reinforcement learning approach to autonomous decision making of intelligent vehicles on highways. IEEE Trans Syst Man Cybern Syst 50(10):3884–3897

    Google Scholar 

  • Ye H, Li GY (2018) Deep reinforcement learning for resource allocation in V2V communications. In: 2018 IEEE international conference on communications (ICC), pp 1–6

  • Ye H, Li GY, Juang BHF (2019) Deep reinforcement learning based resource allocation for V2V communications. IEEE Trans Veh Technol 68(4):3163–3173

    Article  Google Scholar 

  • Zeng M, Yadav A, Dobre OA, Tsiropoulos GI, Poor HV (2017) Capacity comparison between MIMO-NOMA and MIMO-OMA with multiple users in a cluster. IEEE J Sel Areas Commun 35(10):2413–2424

    Article  Google Scholar 

  • Zeng M, Hao W, Dobre OA, Poor HV (2019) Energy-efficient power allocation in uplink mmWave massive MIMO with NOMA. IEEE Trans Veh Technol 68(3):3000–3004

    Article  Google Scholar 

  • Zhai D, Zhang R, Cai L, Li B, Jiang Y (2018) Energy-efficient user scheduling and power allocation for NOMA-based wireless networks with massive IoT devices. IEEE Internet Things J 5(3):1857–1868

    Article  Google Scholar 

  • Zhang K, Koppel A, Zhu H, Basar T (2020) Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM J Control Optim 58(6):3586–3612

    Article  MathSciNet  Google Scholar 

  • Zhao N, Liang YC, Niyato D, Pei Y, Wu M, Jiang Y (2019) Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks. IEEE Trans Wirel Commun 18(11):5141–5152

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Ardabil Branch, Islamic Azad University, Ardabil, Iran

    A. Mirzaei Rahimi

  2. Computer Networks Lab, Communications and Information Technology Institute, Tehran, Iran

    Amir Ziaeddini

  3. Department of Computer Science, Chengdu University of Technology, Sichuan, 610059, China

    Shu Gonglee

Authors
  1. A. Mirzaei Rahimi
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Amir Ziaeddini
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Shu Gonglee
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Shu Gonglee.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahimi, A., Ziaeddini, A. & Gonglee, S. A novel approach to efficient resource allocation in load-balanced cellular networks using hierarchical DRL. J Ambient Intell Human Comput 13, 2887–2901 (2022). https://doi.org/10.1007/s12652-021-03174-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-021-03174-0

Keywords