Skip to main content
Springer Nature Link
Log in

Satellite Integration into 5G: Deep Reinforcement Learning for Network Selection

  • Research Article
  • Published:
Machine Intelligence Research Aims and scope Submit manuscript

Abstract

This paper proposes a deep-Q-network (DQN) controller for network selection and adaptive resource allocation in heterogeneous networks, developed on the ground of a Markov decision process (MDP) model of the problem. Network selection is an enabling technology for multi-connectivity, one of the core functionalities of 5G. For this reason, the present work considers a realistic network model that takes into account path-loss models and intra-RAT (radio access technology) interference. Numerical simulations validate the proposed approach and show the improvements achieved in terms of connection acceptance, resource allocation, and load balancing. In particular, the DQN algorithm has been tested against classic reinforcement learning one and other baseline approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. 3GPP TR 38.811 Study on New Radio (NR) to Support Non-terrestrial Networks, Technical Report. 3GPP, France 2017.

  2. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller. Playing Atari with deep reinforcement learning, [Online], Available: https://arxiv.org/abs/1312.5602v1, 2013.

    Google Scholar 

  3. K. S. S. Anupama, S. S. Gowri, B. P. Rao. A comparative study of outranking MADM algorithms in network selection. In Proceedings of the 2nd International Conference on Computing Methodologies and Communication, IEEE, Erode, India, pp. 904–907, 2018. DOI: https://doi.org/10.1109/ICCMC.2018.8487931.

    Google Scholar 

  4. Y. F. Zhong, H. Q. Wang, H. W. Lv. A cognitive wireless networks access selection algorithm based on MADM. Ad Hoc Networks, vol. 109, Article number 102286, 2020. DOI: https://doi.org/10.1016/j.adhoc.2020.102286.

  5. S. Radouche, C. Leghris, A. Adib. MADM methods based on utility function and reputation for access network selection in a multi-access mobile network environment. In Proceedings of International Conference on Wireless Networks and Mobile Communications, IEEE, Rabat, Morocco, 2017. DOI: https://doi.org/10.1109/WINCOM.2017.8238177.

    Google Scholar 

  6. Q. Y. Song, A. Jamalipour. Network selection in an integrated wireless LAN and UMTS environment using mathematical modeling and computing techniques. IEEE Wireless Communications, vol. 12, no. 3, pp. 42–48, 2005. DOI: https://doi.org/10.1109/mwc.2005.1452853.

    Article  Google Scholar 

  7. T. Ding, L. Liang, M. Yang, H. Q. Wu. Multiple attribute decision making based on cross-evaluation with uncertain decision parameters. Mathematical Problems in Engineering, vol. 2016, Article number 4313247, 2016. DOI: https://doi.org/10.1155/2016/4313247.

  8. R. K. Goyal, S. Kaushal, A. K. Sangaiah. The utility based non-linear fuzzy AHP optimization model for network selection in heterogeneous wireless networks. Applied Soft Computing, vol. 67, pp. 800–811, 2018. DOI: https://doi.org/10.1016/j.asoc.2017.05.026.

    Article  Google Scholar 

  9. X. Y. Yan, P. Dong, T. Zheng, H. K. Zhang. Fuzzy and utility based network selection for heterogeneous networks in high-speed railway. Wireless Communications and Mobile Computing, vol. 2017, Article number 4967438, 2017. DOI: https://doi.org/10.1155/2017/4967438.

  10. M. M. R. Mou, M. Z. Chowdhury. Service aware fuzzy logic based handover decision in heterogeneous wireless networks. In Proceedings of International Conference on Electrical, Computer and Communication Engineering, IEEE, Cox’s Bazar, Bangladesh, pp. 686–691, 2017. DOI: https://doi.org/10.1109/ECACE.2017.7912992.

    Google Scholar 

  11. A. Wilson, A. Lenaghan, R. Malyan. Optimising wireless access network selection to maintain QoS in heterogeneous wireless environments. In Proceedings of International Symposium on Wireless Personal Multimedia Communications, Aalborg, Denmark. 1236–1240, 2005.

    Google Scholar 

  12. R. Trestian, O. Ormond, G. M. Muntean. Game theory-based network selection: Solutions and challenges. IEEE Communications Surveys & Tutorials, vol. 14, no. 4, pp. 1212–1231, 2012. DOI: https://doi.org/10.1109/surv.2012.010912.00081.

    Article  Google Scholar 

  13. J. Antoniou, A. Pitsillides. 4G converged environment: Modeling network selection as a game. In Proceedings of the 16th IST Mobile and Wireless Communications Summit, IEEE, Budapest, Hungary, 2007. DOI: https://doi.org/10.1109/IST-MWC.2007.4299242.

    Google Scholar 

  14. T. Rahman, M. Z. Chowdhury, Y. M. Jang. Radio access network selection mechanism based on hierarchical modelling and game theory. In Proceedings of International Conference on Information and Communication Technology Convergence, IEEE, Jeju, Korea, pp. 126–131, 2016. DOI: https://doi.org/10.1109/ICTC.2016.7763451.

    Google Scholar 

  15. L. Rajesh, K. B. Bagan, B. Ramesh. User demand wireless network selection using game theory. In Proceedings of International Conference on Nano-electronics, Circuits & Communication Systems, Jharkhand, India, pp.39–53, 2017. DOI: https://doi.org/10.1007/978-981-10-2999-8_4.

    Google Scholar 

  16. Meenakshi, N. P. Singh. A comparative study of cooperative and non-cooperative game theory in network selection. In Proceedings of International Conference on Computational Techniques in Information and Communication Technologies, IEEE, New Delhi, India, pp. 612–617, 2016. DOI: https://doi.org/10.1109/ICCTICT.2016.7514652.

    Google Scholar 

  17. R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, Cambridge, UK: MIT Press, 1998.

    MATH  Google Scholar 

  18. Z. H. Zhang, X. F. Jiang, H. S. Xi. Optimal content placement and request dispatching for cloud-based video distribution services. International Journal of Automation and Computing, vol. 13, no. 6, pp. 529–540, 2016. DOI: https://doi.org/10.1007/s11633-016-1025-z.

    Article  Google Scholar 

  19. F. S. Lin, B. Q. Yin, J. Huang, X. M. Wu. Admission control with elastic QoS for video on demand systems. International Journal of Automation and Computing, vol. 9, no. 5, pp. 467–473, 2012. DOI: https://doi.org/10.1007/s11633-012-0668-7.

    Article  Google Scholar 

  20. Z. Y. Du, C. X. Wang, Y. M. Sun, G. F. Wu. Context-aware indoor VLC/RF heterogeneous network selection: Reinforcement learning with knowledge transfer. IEEE Access, vol. 6, pp. 33275–33284, 2018. DOI: https://doi.org/10.1109/access.2018.2844882.

    Article  Google Scholar 

  21. Y. Yang, Y. Wang, K. Y. Liu, N. Zhang, S. S. Gu, Q. Y. Zhang. Deep reinforcement learning based online network selection in CRNs with multiple primary networks. IEEE Transactions on Industrial Informatics, vol. 16, no. 12, pp. 7691–7699, 2020. DOI: https://doi.org/10.1109/tii.2020.2971735.

    Article  Google Scholar 

  22. D. D. Nguyen, H. X. Nguyen, L. B. White. Reinforcement learning with network-assisted feedback for heterogeneous RAT selection. IEEE Transactions on Wireless Communications, vol. 16, no. 9, pp. 6062–6076, 2017. DOI: https://doi.org/10.1109/twc.2017.2718526.

    Article  Google Scholar 

  23. F. Liberati, A. Giuseppi, A. Pietrabissa, V. Suraci, A. Di Giorgio, M. Trubian, D. Dietrich, P. Papadimitriou, F. Delli Priscoli. Stochastic and exact methods for service mapping in virtualized network infrastructures. International Journal of Network Management, vol. 27, no. 6, Article number e1985, 2017. DOI: https://doi.org/10.1002/nem.1985.

    Google Scholar 

  24. X. W. Wang, J. D. Li, L. X. Wang, C. G. Yang, Z. Han. Intelligent user-centric network selection: A model-driven reinforcement learning framework. IEEE Access, vol. 7, pp. 21645–21661, 2019. DOI: https://doi.org/10.1109/access.2019.2898205.

    Article  Google Scholar 

  25. K. S. Shin, G. H. Hwang, O. Jo. Distributed reinforcement learning scheme for environmentally adaptive IoT network selection. Electronics Letters, vol. 56, no. 9, pp. 462–464, 2020. DOI: https://doi.org/10.1049/el.2019.3891.

    Article  Google Scholar 

  26. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2016.

    Google Scholar 

  27. Y. B. Zhou, Z. M. Fadlullah, B. M. Mao, N. Kato. A deep-learning-based radio resource assignment technique for 5G ultra dense networks. IEEE Network, vol. 32, no. 6, pp. 28–34, 2018. DOI: https://doi.org/10.1109/MNET.2018.1800085.

    Article  Google Scholar 

  28. B. M. Mao, F. X. Tang, Y. Kawamoto, N. Kato. Optimizing computation offloading in satellite-UAV-served 6G IoT: A deep learning approach. IEEE Network, vol. 35, no. 4, pp. 102–108, 2021. DOI: https://doi.org/10.1109/MNET.011.2100097.

    Article  Google Scholar 

  29. E. De Santis. Trunk96/wireless-network-simulator, [Online], Available: https://github.com/trunk96/wireless-network-simulator, 2022.

    Google Scholar 

  30. F. D. Priscoli, A. Giuseppi, F. Liberati, A. Pietrabissa. Traffic steering and network selection in 5G networks based on reinforcement learning. In Proceedings of European Control Conference, IEEE, St. Petersburg, Russia, pp. 595–601, 2020. DOI: https://doi.org/10.23919/ECC51009.2020.9143837.

    Google Scholar 

  31. 5G; NR; Physical Channels and Modulation, ETSI TS 138 211 v15.2.0. 3GPP, 2018.

  32. Final report for COST Action 231, [Online], Available: http://www.lx.it.pt/cost231/final_report.htm, 2022.

  33. G. Maral, M. Bousquet, Z. L. Sun. Satellite Communications Systems: Systems, Techniques and Technology. 6th ed., Hoboken, USA: Wiley, 2020. DOI: https://doi.org/10.1002/9781119673811.

    Book  Google Scholar 

Download references

Acknowledgements

This work was supported by the European Commission in the framework of the H2020 EU-Korea project 5G-ALLSTAR (5G AgiLe and fLexible integration of SaTellite And cellulaR, https://www.5g-allstar.eu) (No. 815323). The authors acknowledge all their colleagues of the Consortium for the Research in Automation and Telecommunication (CRAT) team working on the project for their fruitful discussions and confrontations.

Author information

Authors and Affiliations

  1. Department of Computer, Control and Management Engineering “Antonio Ruberti”, University of Rome La Sapienza, Rome, 00185, Italy

    Emanuele De Santis, Alessandro Giuseppi, Antonio Pietrabissa, Michael Capponi & Francesco Delli Priscoli

Authors
  1. Emanuele De Santis
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Alessandro Giuseppi
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Antonio Pietrabissa
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Michael Capponi
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Francesco Delli Priscoli
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Emanuele De Santis.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Emanuele De Santis received the B. Sc. degree in automatic control and M. Sc. degree in engineering in computer science with specialization in communication networks control from Sapienza University of Rome, Italy in 2017 and 2019, where he is currently a Ph. D. degree candidate in automatic control. He participated in the H2020 projects 5G-ALLSTAR and 5G-Solutions and in the European Space Agency (ESA) project ARIES. He is a student member of IEEE.

His research interests include power and communication network control, artificial intelligence, and optimal control.

Alessandro Giuseppi received the B. Sc. degree in computer and automation engineering, the M. Sc. degree in control engineering and the Ph. D. degree in automatica from University of Rome La Sapienza, Italy in 2014, 2016 and 2019, respectively, where he is currently a postdoctoral researcher in automatic control. Since 2016, he has participated in five European and national research projects. He is a member of IEEE.

His research interests include network control and intelligent systems.

Antonio Pietrabissa received the M. Sc. degree in electronics engineering and the Ph. D. degree in systems engineering from Sapienza University of Rome, Italy in 2000 and 2004, respectively. He is an associate professor at Sapienza University of Rome, Italy. He has participated in about 20 European and national research projects. He is a senior member of IEEE.

His research interests include the application of systems and control theory to the analysis and control of networks.

Michael Capponi received the M. Sc. degree in communication networks control from Sapienza University of Rome, Italy in 2020. Now, he works in a company in the field of computer science.

His research interests include reinforcement learning applications to communication networks.

Francesco Delli Priscoli received the M. Sc. degree in electronics engineering and the Ph. D. degree in systems engineering from University of Rome, Italy in 1986 and 1991, respectively. From 1986 to 1991, he was with Telespazio, Italy. Since 1991, he has been with University of Rome, Italy, where, at present, he is a full professor of automatic control, control of autonomous multiagent systems, and control of communication and energy networks. He is an Associate Editor of Control Engineering Practice and a Member of the IFAC Technical Committee on Networked Systems. He was/is the Scientific Responsible with University of Rome, for 40 projects funded by the European Union and by the European Space Agency. He is a member of IEEE.

His research interests include closed-loop multiagent learning techniques in advanced communication and energy networks.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

De Santis, E., Giuseppi, A., Pietrabissa, A. et al. Satellite Integration into 5G: Deep Reinforcement Learning for Network Selection. Mach. Intell. Res. 19, 127–137 (2022). https://doi.org/10.1007/s11633-022-1326-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-022-1326-3

Keywords