Abstract
Heterogeneous cellular networks (HetNets) have been proven as a promising approach to deal with ever-growing data traffic. Supporting ultra-reliable and low-latency communication (URLLC) is also considered as a new feature of the upcoming wireless networks. Due to the overlapping structure and the mutual interference between cells in HetNets, existing resource allocation approaches cannot be directly applied for real-time applications, especially for URLLC services. As a novel unsupervised algorithm, Deep Q Network (DQN) has already been applied to many online complex optimization models successfully. However, it may perform badly for resource allocation optimization in HetNets, due to the tiny state change and the large-scale action space characteristics. In order to cope with them, we first propose an auto-encoder to disturb the similarity of adjacent states to enhance the features and then divide the whole decision process into two phases. DQN is applied to solve each phase, respectively, and we iterate the whole process to find the joint optimized solution. We implement our algorithm in 6 scenarios with different numbers of user equipment (UE), redundant links, and sub-carriers. Simulations results demonstrate that our algorithm has good convergence for the optimization objective. Moreover, by further optimizing the power allocation, a 1–2 nines of reliability improvement is obtained for bad conditions. Finally, the experiment result shows that our algorithm reaches the reliability of 8-nines in common scenarios. As an online method, the algorithm proposed in this paper takes only 0.32 s on average.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig10_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig11_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig12_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig13_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig14_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig15_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig16_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig17_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00521-020-05492-4/MediaObjects/521_2020_5492_Fig18_HTML.png)
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Bacci G, Belmega EV, Mertikopoulos P, Sanguinetti L (2014) Energy-aware competitive power allocation for heterogeneous networks under QOS constraints. IEEE Trans Wirel Commun 14(9):4728–4742
Baldi P (2011) Autoencoders, unsupervised learning and deep architectures. In: International conference on unsupervised and transfer learning workshop
Bennis M, Debbah M, Poor HV (2018) Ultra-reliable and low-latency wireless communication: tail, risk and scale. Proc IEEE 106(10):1834–1853
Cao J, Peng T, Qi Z, Duan R, Yuan Y, Wang W (2018) Interference management in ultradense networks: a user-centric coalition formation game approach. IEEE Trans Veh Technol 67(6):5188–5202
Chen J, Jia J, Liu Y, Wang X, Aghvami A (2019) Optimal resource block assignment and power allocation for D2D-enabled NOMA communication. IEEE Access PP:1–1. https://doi.org/10.1109/ACCESS.2019.2926438
Chen J, Zhang L, Liang Y, Ma S (2020) Optimal resource allocation for multicarrier NOMA in short packet communications. IEEE Trans Veh Technol 69(2):2141–2156
Cui J, Liu Y, Nallanathan A (2020) Multi-agent reinforcement learning-based resource allocation for UAV networks. IEEE Trans Wirel Commun 19(2):729–743
D’Oro S, Marotta MA, Both CB, DaSilva L, Palazzo S (2019) Power-efficient resource allocation in c-RANS with SINR constraints and deadlines. IEEE Trans Veh Technol 68(6):6099–6113
Fadlullah Z, Fouda MM, Kato N, Takeuchi A, Iwasaki N, Nozaki Y (2012) Toward intelligent machine-to-machine communications in smart grid. IEEE Commun Soc 49(4):60–65
Forecast CV (2019) Cisco visual networking index: global mobile data traffic forecast update, 2017–2022 white paper
Frotzscher A, Wetzker U, Bauer M, Rentschler M, Beyer M, Elspass S, Klessig H (2014) Requirements and current solutions of wireless communication in industrial automation. In: IEEE international conference on communications workshops
Ghosh A, Mangalvedhe N, Ratasuk R, Mondal B, Cudak M, Visotsky E, Thomas TA, Andrews JG, Xia P, Jo HS et al (2012) Heterogeneous cellular networks: from theory to practice. IEEE Commun Magn 50(6):54–64
Guo C, Liang L, Li GY (2019) Resource allocation for high-reliability low-latency vehicular communications with packet retransmission. IEEE Trans Veh Technol 68(7):6219–6230
Guo S, Zhou X (2019) Robust resource allocation with imperfect channel estimation in NOMA-based heterogeneous vehicular networks. IEEE Trans Commun 67(3):2321–2332
Hasselt HV, Guez A, Silver D (2015) Deep reinforcement learning with double q-learning. In: Computer science
He C, Abbas R, Peng C, Shirvanimoghaddam M, Vucetic B (2017) Ultra-reliable low latency cellular networks: use cases, challenges and approaches. IEEE Commun Mag
Hessel M, Modayil J, Van Hasselt H, Schaul T, Ostrovski G, Dabney W, Horgan D, Piot B, Azar M, Silver D (2018) Rainbow: combining improvements in deep reinforcement learning. In: AAAI conference on artificial intelligence
Jaderberg M, Czarnecki WM, Dunning I, Marris L, Graepel T (2019) Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science 364(6443):859–865
Jia J, Deng Y, Chen J, Aghvami AH, Nallanathan A (2017) Achieving high availability in heterogeneous cellular networks via spectrum aggregation. IEEE Trans Veh Technol 66(11):10156–10169
Jia J, Deng Y, Chen J, Aghvami AH, Nallanathan A (2017) Availability analysis and optimization in comp and ca-enabled hetnets. IEEE Trans Commun 65(6):2438–2450
Jie J, Deng Y, Ping S, Aghvami H, Nallanathan A (2016) High availability optimization in heterogeneous cellular networks. In: GLOBECOM 2016 IEEE global communications conference
Kanervisto A, Scheller C, Hautamäki V (2020) Action space shaping in deep reinforcement learning
Lee H, Vahid S, Moessner K (2014) A survey of radio resource management for spectrum aggregation in LTE-advanced. IEEE Commun Surv Tutor 16(2):745–760
Liao X, Shi J, Li Z, Zhang L, Xia B (2020) A model-driven deep reinforcement learning heuristic algorithm for resource allocation in ultra-dense cellular networks. IEEE Trans Veh Technol 69(1):983–997
Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529
Nguyen HT, Murakami H, Nguyen K, Ishizu K, Hwang WJ (2019) Joint user association and power allocation for millimeter-wave ultra-dense networks. Mob Netw Appl (2)
Nielsen JJ, Liu R, Popovski P (2018) Ultra-reliable low latency communication using interface diversity. IEEE Trans Commun 66(3):1322–1334
Pukite P, Pukite J (1998) Markov modeling for reliability analysis. Wiley, New York
Qin Z, Yue X, Liu Y, Ding Z, Nallanathan A (2018) User association and resource allocation in unified NOMA enabled heterogeneous ultra dense networks. IEEE Commun Mag 56(6):86–92
Raman RK, Jagannathan K (2018) Downlink resource allocation under time-varying interference: fairness and throughput optimality. IEEE Trans Wirel Commun 17(2):722–735
Schaul T, Quan J, Antonoglou I, Silver D (2015) Prioritized experience replay. Comput Sci
Shirvanimoghaddam M, Mohamadi MS, Abbas R, Minja A, Yue C, Matuz B, Han G, Lin Z, Li Y, Johnson S (2018) Short block-length codes for ultra-reliable low-latency communications. In: IEEE communications magazine, pp 1–8
Silver D, Graves A, Antonoglou I, Riedmiller M, Mnih V, Wierstra D, Kavukcuoglu K (2013) Playing atari with deep reinforcement learning. arXiv: Learning
Skarin P, Tärneberg W, Årzen K, Kihl M (2018) Towards mission-critical control at the edge and over 5g. In: 2018 IEEE international conference on edge computing (EDGE), pp 50–57
Sklar B (1997) Rayleigh fading channels in mobile digital communication systems part I: characterization. IEEE Commun Mag 35(9):136–146
Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. IEEE Trans Neural Netw 9(5):1054–1054
Wang H, Raj B (2017) On the origin of deep learning. arXiv preprint arXiv:1702.07800
Wang Z, Schaul T, Hessel M, Van Hasselt H, Lanctot M, De Freitas N (2016) Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd international conference on international conference on machine learning, vol 48, ICML’16, JMLR.org, pp 1995–2003
Whitehead SD (1991) A complexity analysis of cooperative mechanisms in reinforcement learning. In: National conference on artificial intelligence
Wu X, Ma Z, Chen X, Labeau F, Han S (2019) Energy efficiency-aware joint resource allocation and power allocation in multi-user beamforming. IEEE Trans Veh Technol 68(5):4824–4833
Yousefvand M, Hamidouche K, Mandayam NB (2019) Learning-based resource optimization in ultra reliable low latency hetnets. In: GLOBECOM 2019 IEEE global communications conference
Zhang H, Venturino L, Prasad N, Li P, Rangarajan S, Wang X (2011) Weighted sum-rate maximization in multi-cell networks via coordinated scheduling and discrete power control. IEEE J Sel Areas Commun 29(6):1214–1224
Zhang L, Li Z, Wu C (2017) An auction approach to spectrum management in hetnets. arXiv: Networking and internet architecture
Zhang N, Cheng N, Gamage AT, Zhang K, Mark JW, Shen X (2015) Cloud assisted hetnets toward 5g wireless networks. IEEE Commun Mag 53(6):59–65
Zhang X, Wang J, Poor HV (2019) Heterogeneous-qos driven resource allocation over mmwave massive-mimo based 5g mobile wireless networks in the non-asymptotic regime. IEEE J Sel Areas Commun PP:1–1. https://doi.org/10.1109/JSAC.2019.2947941
Zhou ZH, Yu Y, Qian C.: Evolutionary learning: advances in theories and algorithms
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grants 61772126, and 61972079, in part by the National Key Research and Development Program of China under Grants 2018YFC0830601, in part by the Fundamental Research Funds for the Central Universities under Grants N2016004, N2016002, and N2024005-1, in part by the joint Funds of Ministry of Education with China Mobile under Grant MCM20180203, in part by the Central Government Guided Local Science and Technology Development Fund Project under Grant 2020ZY0003, in part by the Young and Middle-aged Scientific and Technological Innovation Talent Support Program of Shenyang under Grant RC200548, and in part by the LiaoNing Revitalization Talents Program under Grant XLYC1802100.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
We promise that this manuscript is the authors’ original work and has not been published nor has it been submitted simultaneously elsewhere. All authors have checked the manuscript and have agreed to the submission.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Yang, L., Jia, J., Chen, J. et al. Online reliability optimization for URLLC in HetNets: a DQN approach. Neural Comput & Applic 33, 7271–7290 (2021). https://doi.org/10.1007/s00521-020-05492-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-020-05492-4