Skip to main content
SpringerLink
Log in

Online reliability optimization for URLLC in HetNets: a DQN approach

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Heterogeneous cellular networks (HetNets) have been proven as a promising approach to deal with ever-growing data traffic. Supporting ultra-reliable and low-latency communication (URLLC) is also considered as a new feature of the upcoming wireless networks. Due to the overlapping structure and the mutual interference between cells in HetNets, existing resource allocation approaches cannot be directly applied for real-time applications, especially for URLLC services. As a novel unsupervised algorithm, Deep Q Network (DQN) has already been applied to many online complex optimization models successfully. However, it may perform badly for resource allocation optimization in HetNets, due to the tiny state change and the large-scale action space characteristics. In order to cope with them, we first propose an auto-encoder to disturb the similarity of adjacent states to enhance the features and then divide the whole decision process into two phases. DQN is applied to solve each phase, respectively, and we iterate the whole process to find the joint optimized solution. We implement our algorithm in 6 scenarios with different numbers of user equipment (UE), redundant links, and sub-carriers. Simulations results demonstrate that our algorithm has good convergence for the optimization objective. Moreover, by further optimizing the power allocation, a 1–2 nines of reliability improvement is obtained for bad conditions. Finally, the experiment result shows that our algorithm reaches the reliability of 8-nines in common scenarios. As an online method, the algorithm proposed in this paper takes only 0.32 s on average.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Bacci G, Belmega EV, Mertikopoulos P, Sanguinetti L (2014) Energy-aware competitive power allocation for heterogeneous networks under QOS constraints. IEEE Trans Wirel Commun 14(9):4728–4742

    Article  Google Scholar 

  2. Baldi P (2011) Autoencoders, unsupervised learning and deep architectures. In: International conference on unsupervised and transfer learning workshop

  3. Bennis M, Debbah M, Poor HV (2018) Ultra-reliable and low-latency wireless communication: tail, risk and scale. Proc IEEE 106(10):1834–1853

    Article  Google Scholar 

  4. Cao J, Peng T, Qi Z, Duan R, Yuan Y, Wang W (2018) Interference management in ultradense networks: a user-centric coalition formation game approach. IEEE Trans Veh Technol 67(6):5188–5202

    Article  Google Scholar 

  5. Chen J, Jia J, Liu Y, Wang X, Aghvami A (2019) Optimal resource block assignment and power allocation for D2D-enabled NOMA communication. IEEE Access PP:1–1. https://doi.org/10.1109/ACCESS.2019.2926438

    Article  Google Scholar 

  6. Chen J, Zhang L, Liang Y, Ma S (2020) Optimal resource allocation for multicarrier NOMA in short packet communications. IEEE Trans Veh Technol 69(2):2141–2156

    Article  Google Scholar 

  7. Cui J, Liu Y, Nallanathan A (2020) Multi-agent reinforcement learning-based resource allocation for UAV networks. IEEE Trans Wirel Commun 19(2):729–743

    Article  Google Scholar 

  8. D’Oro S, Marotta MA, Both CB, DaSilva L, Palazzo S (2019) Power-efficient resource allocation in c-RANS with SINR constraints and deadlines. IEEE Trans Veh Technol 68(6):6099–6113

    Article  Google Scholar 

  9. Fadlullah Z, Fouda MM, Kato N, Takeuchi A, Iwasaki N, Nozaki Y (2012) Toward intelligent machine-to-machine communications in smart grid. IEEE Commun Soc 49(4):60–65

    Article  Google Scholar 

  10. Forecast CV (2019) Cisco visual networking index: global mobile data traffic forecast update, 2017–2022 white paper

  11. Frotzscher A, Wetzker U, Bauer M, Rentschler M, Beyer M, Elspass S, Klessig H (2014) Requirements and current solutions of wireless communication in industrial automation. In: IEEE international conference on communications workshops

  12. Ghosh A, Mangalvedhe N, Ratasuk R, Mondal B, Cudak M, Visotsky E, Thomas TA, Andrews JG, Xia P, Jo HS et al (2012) Heterogeneous cellular networks: from theory to practice. IEEE Commun Magn 50(6):54–64

    Article  Google Scholar 

  13. Guo C, Liang L, Li GY (2019) Resource allocation for high-reliability low-latency vehicular communications with packet retransmission. IEEE Trans Veh Technol 68(7):6219–6230

    Article  Google Scholar 

  14. Guo S, Zhou X (2019) Robust resource allocation with imperfect channel estimation in NOMA-based heterogeneous vehicular networks. IEEE Trans Commun 67(3):2321–2332

    Article  Google Scholar 

  15. Hasselt HV, Guez A, Silver D (2015) Deep reinforcement learning with double q-learning. In: Computer science

  16. He C, Abbas R, Peng C, Shirvanimoghaddam M, Vucetic B (2017) Ultra-reliable low latency cellular networks: use cases, challenges and approaches. IEEE Commun Mag

  17. Hessel M, Modayil J, Van Hasselt H, Schaul T, Ostrovski G, Dabney W, Horgan D, Piot B, Azar M, Silver D (2018) Rainbow: combining improvements in deep reinforcement learning. In: AAAI conference on artificial intelligence

  18. Jaderberg M, Czarnecki WM, Dunning I, Marris L, Graepel T (2019) Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science 364(6443):859–865

    Article  MathSciNet  Google Scholar 

  19. Jia J, Deng Y, Chen J, Aghvami AH, Nallanathan A (2017) Achieving high availability in heterogeneous cellular networks via spectrum aggregation. IEEE Trans Veh Technol 66(11):10156–10169

    Article  Google Scholar 

  20. Jia J, Deng Y, Chen J, Aghvami AH, Nallanathan A (2017) Availability analysis and optimization in comp and ca-enabled hetnets. IEEE Trans Commun 65(6):2438–2450

    Article  Google Scholar 

  21. Jie J, Deng Y, Ping S, Aghvami H, Nallanathan A (2016) High availability optimization in heterogeneous cellular networks. In: GLOBECOM 2016 IEEE global communications conference

  22. Kanervisto A, Scheller C, Hautamäki V (2020) Action space shaping in deep reinforcement learning

  23. Lee H, Vahid S, Moessner K (2014) A survey of radio resource management for spectrum aggregation in LTE-advanced. IEEE Commun Surv Tutor 16(2):745–760

    Article  Google Scholar 

  24. Liao X, Shi J, Li Z, Zhang L, Xia B (2020) A model-driven deep reinforcement learning heuristic algorithm for resource allocation in ultra-dense cellular networks. IEEE Trans Veh Technol 69(1):983–997

    Article  Google Scholar 

  25. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529

    Article  Google Scholar 

  26. Nguyen HT, Murakami H, Nguyen K, Ishizu K, Hwang WJ (2019) Joint user association and power allocation for millimeter-wave ultra-dense networks. Mob Netw Appl (2)

  27. Nielsen JJ, Liu R, Popovski P (2018) Ultra-reliable low latency communication using interface diversity. IEEE Trans Commun 66(3):1322–1334

    Article  Google Scholar 

  28. Pukite P, Pukite J (1998) Markov modeling for reliability analysis. Wiley, New York

    Book  Google Scholar 

  29. Qin Z, Yue X, Liu Y, Ding Z, Nallanathan A (2018) User association and resource allocation in unified NOMA enabled heterogeneous ultra dense networks. IEEE Commun Mag 56(6):86–92

    Article  Google Scholar 

  30. Raman RK, Jagannathan K (2018) Downlink resource allocation under time-varying interference: fairness and throughput optimality. IEEE Trans Wirel Commun 17(2):722–735

    Article  Google Scholar 

  31. Schaul T, Quan J, Antonoglou I, Silver D (2015) Prioritized experience replay. Comput Sci

  32. Shirvanimoghaddam M, Mohamadi MS, Abbas R, Minja A, Yue C, Matuz B, Han G, Lin Z, Li Y, Johnson S (2018) Short block-length codes for ultra-reliable low-latency communications. In: IEEE communications magazine, pp 1–8

  33. Silver D, Graves A, Antonoglou I, Riedmiller M, Mnih V, Wierstra D, Kavukcuoglu K (2013) Playing atari with deep reinforcement learning. arXiv: Learning

  34. Skarin P, Tärneberg W, Årzen K, Kihl M (2018) Towards mission-critical control at the edge and over 5g. In: 2018 IEEE international conference on edge computing (EDGE), pp 50–57

  35. Sklar B (1997) Rayleigh fading channels in mobile digital communication systems part I: characterization. IEEE Commun Mag 35(9):136–146

    Article  Google Scholar 

  36. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. IEEE Trans Neural Netw 9(5):1054–1054

    Article  Google Scholar 

  37. Wang H, Raj B (2017) On the origin of deep learning. arXiv preprint arXiv:1702.07800

  38. Wang Z, Schaul T, Hessel M, Van Hasselt H, Lanctot M, De Freitas N (2016) Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd international conference on international conference on machine learning, vol 48, ICML’16, JMLR.org, pp 1995–2003

  39. Whitehead SD (1991) A complexity analysis of cooperative mechanisms in reinforcement learning. In: National conference on artificial intelligence

  40. Wu X, Ma Z, Chen X, Labeau F, Han S (2019) Energy efficiency-aware joint resource allocation and power allocation in multi-user beamforming. IEEE Trans Veh Technol 68(5):4824–4833

    Article  Google Scholar 

  41. Yousefvand M, Hamidouche K, Mandayam NB (2019) Learning-based resource optimization in ultra reliable low latency hetnets. In: GLOBECOM 2019 IEEE global communications conference

  42. Zhang H, Venturino L, Prasad N, Li P, Rangarajan S, Wang X (2011) Weighted sum-rate maximization in multi-cell networks via coordinated scheduling and discrete power control. IEEE J Sel Areas Commun 29(6):1214–1224

    Article  Google Scholar 

  43. Zhang L, Li Z, Wu C (2017) An auction approach to spectrum management in hetnets. arXiv: Networking and internet architecture

  44. Zhang N, Cheng N, Gamage AT, Zhang K, Mark JW, Shen X (2015) Cloud assisted hetnets toward 5g wireless networks. IEEE Commun Mag 53(6):59–65

    Article  Google Scholar 

  45. Zhang X, Wang J, Poor HV (2019) Heterogeneous-qos driven resource allocation over mmwave massive-mimo based 5g mobile wireless networks in the non-asymptotic regime. IEEE J Sel Areas Commun PP:1–1. https://doi.org/10.1109/JSAC.2019.2947941

    Article  Google Scholar 

  46. Zhou ZH, Yu Y, Qian C.: Evolutionary learning: advances in theories and algorithms

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 61772126, and 61972079, in part by the National Key Research and Development Program of China under Grants 2018YFC0830601, in part by the Fundamental Research Funds for the Central Universities under Grants N2016004, N2016002, and N2024005-1, in part by the joint Funds of Ministry of Education with China Mobile under Grant MCM20180203, in part by the Central Government Guided Local Science and Technology Development Fund Project under Grant 2020ZY0003, in part by the Young and Middle-aged Scientific and Technological Innovation Talent Support Program of Shenyang under Grant RC200548, and in part by the LiaoNing Revitalization Talents Program under Grant XLYC1802100.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Northeastern University, Shenyang, 110819, China

    Leyou Yang, Jie Jia, Jian Chen & Xingwei Wang

  2. Engineering Research Center of Security Technology of Complex Network System, Ministry of Education, Shenyang, 110819, China

    Leyou Yang, Jie Jia & Xingwei Wang

  3. Key Laboratory of Intelligent Internet Theory and Application, Liaoning Province, Northeastern University, Shenyang, 110819, China

    Jie Jia

  4. Liaoning Research Center of Safety Engineering Technology in Industrial Control, Neusoft Group Research, Shenyang, 110179, China

    Jian Chen

Authors
  1. Leyou Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jian Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xingwei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Jia.

Ethics declarations

Conflict of interest

We promise that this manuscript is the authors’ original work and has not been published nor has it been submitted simultaneously elsewhere. All authors have checked the manuscript and have agreed to the submission.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, L., Jia, J., Chen, J. et al. Online reliability optimization for URLLC in HetNets: a DQN approach. Neural Comput & Applic 33, 7271–7290 (2021). https://doi.org/10.1007/s00521-020-05492-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05492-4

Keywords

Search

Navigation