research-article

Deep Q-learning-enabled Deployment of Aerial Base Stations in the Presence of Mobile Users

Authors:

Nahid Parvaresh,

Burak KantarciAuthors Info & Claims

MobiWac '22: Proceedings of the 20th ACM International Symposium on Mobility Management and Wireless Access

Pages 73 - 80

https://doi.org/10.1145/3551660.3560909

Published: 24 October 2022 Publication History

Abstract

Uncrewed aerial vehicle-mounted base stations (UAV-BS) have recently attracted significant attention in order to assist ground base stations (BSs) and provide Internet access to users. UAV-BSs benefit from their mobility nature in the air and are able to constantly move towards the locations where the demand is higher. However, finding the optimal location of UAV-BSs and maintaining it is an NP-hard problem that has no deterministic solution in polynomial time. In this paper, we exploit reinforcement learning (RL) in order to solve the optimization problem of UAV-BSs and find their optimal location in the presence of mobile User Equipment (UEs). We consider UAV-BS as the agent of RL and deploy two algorithms, i.e. Q-learning and deep Q-learning in order to solve the location optimization problem of UAV-BSs. Through simulations, we show that the proposed DQL model with a continuous state space including the mobility information of users can effectively adapt to the environmental changes and improve the user data rate by 46%, packet loss ratio by 70%, and transmission delay by 60%.

References

[1]

L. Gupta, R. Jain, G. Vaszkun, Survey of important issues in uav communication networks, IEEE Communications Surveys and Tutorials 18 (2) (Apr. 2016).

Digital Library

[2]

J. Lyu, Y. Zeng, R. Zhang, T. J. Lim, Placement optimization of uav-mounted mobile base stations, Computer Networks 21 (3) (Mar. 2017).

[3]

R. Ruby, K. Wu, Q.-V. Pham, B. M. ElHalawany, Aiding a disaster spot via an uav-based mobile af relay: Joint trajectory and power optimization, in: MobiWac 2020 - Proceedings of the 18th ACM Symposium on Mobility Management and Wireless Access, ACM, 2020.

[4]

E. Kalantari, M. Z. Shakir, H. Yanikomeroglu, A. Yongacoglu, Backhaul-aware robust 3D drone placement in 5G wireless networks, in: 2017 IEEE International Conference on Commun. Workshops, Networking and Commun., IEEE, 2017, pp. 109--114.

[5]

Y. Zeng, R. Zhang, T. J. Lim, Wireless communications with unmanned aerial vehicles: Opportunities and challenges, IEEE Communications Magazine 54 (5) (May 2016).

Digital Library

[6]

A. Fotouhi, H. Qiang, M. Ding, M. Hassan, Survey on uav cellular communications: Practical aspects, standardization advancements, regulation, and security challenges, IEEE Communications Surveys and Tutorials 21 (4) (Mar. 2019).

[7]

W. Shi, J. Li, W. Xu, H. Zhou, N. Zhang, S. Zhang, X. Shen, Multiple drone-cell deployment analyses and optimization in drone assisted radio access networks, IEEE Access 6 (Feb. 2018).

[8]

P. V. Klaine, J. P. B. Nadas, R. D. Souza, M. A. Imran, Distributed drone base station positioning for emergency cellular networks using reinforcement learning, cognitive computation 10 (6) (May 2018).

[9]

Y. Cao, L. Zhang, Y.-C. Liang, Deep reinforcement learning for multi-user access control in uav networks, in: ICC 2019 - 2019 IEEE International Conference on Communications (ICC), IEEE, 2019.

[10]

X. Liu, Y. Liu, Y. Chen, Reinforcement learning in multiple-uav networks: Deployment and movement design, IEEE Transactions on Vehicular Technology 68 (8) (Jun. 2019).

[11]

X. Zhong, Y. Huo, X. Dong, Z. Liang, Deep q-network based dynamic movement strategy in a uav-assisted network, in: 2020 IEEE 92nd Vehicular Technology Conference (VTC2020- Fall), IEEE, 2020.

[12]

H. V. Abeywickrama, Y. He, E. Dutkiewicz, B. A. Jayawickrama, M. Mueck, A reinforcement learning approach for fair user coverage using uav mounted base stations under energy constraints, IEEE Open Journal of Vehicular Technology 1 (Feb. 2020).

[13]

M. Zhang, S. Fu, Q. Fan, Joint 3d deployment and power allocation for uav-bs: A deep reinforcement learning approach, IEEE Wireless Communications Letters 10 (10) (Jul. 2021).

[14]

T. Camp, J. Boleng, V. Davies, A survey of mobility models for ad hoc network research, Wireless Communications and Mobile Computing 2 (5) (Sep. 2002).

[15]

A. Al-Hourani, S. Kandeepan, S. Lardner, Optimal lap altitude for maximum coverage, IEEE Wireless Communications Letters 3 (6) (Jul. 2014).

[16]

R. Ghanavi, E. Kalantari, M. Sabbaghian, H. Yanikomeroglu, A. Yongacoglu, Efficient 3d aerial base station placement considering users mobility by reinforcement learning, in: 2018 IEEE Wireless Communications and Networking Conference, IEEE, 2018.

Digital Library

[17]

X. Xi, X. Cao, P. Yang, J. Chen, T. Quek, D. Wu, Joint user association and uav location optimization for uav-aided communications, IEEE Wireless Communications Letters 8 (6) (Aug. 2019).

[18]

M. Alzenad, A. El-Keyi, F. Lagum, H. Yanikomeroglu, 3-d placement of an unmanned aerial vehicle base station (uav-bs) for energy-efficient maximal coverage, IEEE Wireless Communications Letters 6 (4) (Aug. 2017).

[19]

C. J. C. H. Watkins, P. Dayan, Q-learning, Machine Learning 8 (3) (May 1992).

[20]

I. J. Sledge, J. C. Príncipe, Balancing exploration and exploitation in reinforcement learning using a value of information criterion, in: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2017.

Digital Library

[21]

N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P. Wang, Y.-C. Liang, D. I. Kim, Applications of deep reinforcement learning in communications and networking- a survey, IEEE Communications Surveys and Tutorials 21 (4) (2019).

Digital Library

[22]

B. Jang, M. Kim, G. Harerimana, J. W. Kim, Q-learning algorithms: A comprehensive classification and applications, IEEE Access 7 (Sep 2019).

[23]

K. A. andMarc Peter Deisenroth, M. Brundage, A. A. Bharath, Deep reinforcement learning, IEEE Signal Processing Magazine 34 (6) (Nov. 2017).

[24]

C. H. Liu, X. Ma, X. Gao, J. Tang, Distributed energy-efficient multi-uav navigation for long-term communication coverage by deep reinforcement learning, IEEE Transactions on Mobile Computing 19 (6) (Jun. 2020).

[25]

Q. Li, M. Ding, C. Ma, C. Liu, Z. Lin, Y.-C. Liang, A reinforcement learning based user association algorithm for uav networks, in: 2018 28th International Telecommunication Networks and Applications Conference (ITNAC), IEEE, 2018.

[26]

J. Wu, P. Yu, L. Feng, F. Zhou, W. Li, X. Qiu, 3d aerial base station position planning based on deep q-network for capacity enhancement, in: 2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), IEEE, 2019.

Cited By

Benning OLohan PKantarci B(2024)Actor-Critic Deep Reinforcement Learning for 3D UAV Base Station Positioning and Delay Reduction2024 IEEE 10th World Forum on Internet of Things (WF-IoT)10.1109/WF-IoT62078.2024.10811425(1-6)Online publication date: 10-Nov-2024
https://doi.org/10.1109/WF-IoT62078.2024.10811425
Zhu RBoukerche ALi PYang Q(2024)A Traffic-Aware Trust Model Based on Edge Computing for Underwater Wireless Sensor NetworksICC 2024 - IEEE International Conference on Communications10.1109/ICC51166.2024.10622709(2390-2395)Online publication date: 9-Jun-2024
https://doi.org/10.1109/ICC51166.2024.10622709
Svaigen ABoukerche ARuiz LLoureiro A(2024)SkyCloaking: A UAV-Assisted Privacy-Preserving Strategy for Location-Based Service UsersICC 2024 - IEEE International Conference on Communications10.1109/ICC51166.2024.10622488(2580-2585)Online publication date: 9-Jun-2024
https://doi.org/10.1109/ICC51166.2024.10622488
Show More Cited By

Index Terms

Deep Q-learning-enabled Deployment of Aerial Base Stations in the Presence of Mobile Users
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Reinforcement learning
    2. Machine learning algorithms
      1. Dynamic programming for Markov decision processes
        Q-learning
        Temporal difference learning

Recommendations

Deep Reinforcement Learning: From Q-Learning to Deep Q-Learning
Neural Information Processing
Abstract
As the two hottest branches of machine learning, deep learning and reinforcement learning both play a vital role in the field of artificial intelligence. Combining deep learning with reinforcement learning, deep reinforcement learning is a method ...
Backward Q-learning: The combination of Sarsa algorithm and Q-learning

Reinforcement learning (RL) has been applied to many fields and applications, but there are still some dilemmas between exploration and exploitation strategy for action selection policy. The well-known areas of reinforcement learning are the Q-learning ...
Deep reinforcement learning collision avoidance using policy gradient optimisation and Q-learning

Usage of trust region policy optimisation (TRPO) and proximal policy optimisation (PPO) 'children of policy gradient optimisation method' and deep Q-learning network (DQN) in Lidar-based differential robots are proposed using Turtlebot and OpenAI's ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiWac '22: Proceedings of the 20th ACM International Symposium on Mobility Management and Wireless Access

October 2022

134 pages

ISBN:9781450394802

DOI:10.1145/3551660

General Chair:
Frank Li
University of Agder, Norway
,
Program Chair:
Rodolfo W.L. Coutinho
Concordia University, Canada

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSIM: ACM Special Interest Group on Simulation and Modeling

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MSWiM '22

Sponsor:

SIGSIM

MSWiM '22: Int'l ACM Conference on Modeling Analysis and Simulation of Wireless and Mobile Systems

October 24 - 28, 2022

Quebec, Montreal, Canada

Acceptance Rates

MobiWac '22 Paper Acceptance Rate 16 of 50 submissions, 32%;

Overall Acceptance Rate 83 of 272 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
136
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)6

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Benning OLohan PKantarci B(2024)Actor-Critic Deep Reinforcement Learning for 3D UAV Base Station Positioning and Delay Reduction2024 IEEE 10th World Forum on Internet of Things (WF-IoT)10.1109/WF-IoT62078.2024.10811425(1-6)Online publication date: 10-Nov-2024
https://doi.org/10.1109/WF-IoT62078.2024.10811425
Zhu RBoukerche ALi PYang Q(2024)A Traffic-Aware Trust Model Based on Edge Computing for Underwater Wireless Sensor NetworksICC 2024 - IEEE International Conference on Communications10.1109/ICC51166.2024.10622709(2390-2395)Online publication date: 9-Jun-2024
https://doi.org/10.1109/ICC51166.2024.10622709
Svaigen ABoukerche ARuiz LLoureiro A(2024)SkyCloaking: A UAV-Assisted Privacy-Preserving Strategy for Location-Based Service UsersICC 2024 - IEEE International Conference on Communications10.1109/ICC51166.2024.10622488(2580-2585)Online publication date: 9-Jun-2024
https://doi.org/10.1109/ICC51166.2024.10622488
Svaigen ABoukerche ARuiz LLoureiro A(2023)DissIdent: A Dissimilarity-based Approach for Improving the Identification of Unknown UAVs2023 IEEE 34th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)10.1109/PIMRC56721.2023.10293825(1-6)Online publication date: 5-Sep-2023
https://doi.org/10.1109/PIMRC56721.2023.10293825
Parvaresh NKantarci B(2023)A Continuous Actor–Critic Deep Q-Learning-Enabled Deployment of UAV Base Stations: Toward 6G Small Cells in the Skies of Smart CitiesIEEE Open Journal of the Communications Society10.1109/OJCOMS.2023.32512974(700-712)Online publication date: 2023
https://doi.org/10.1109/OJCOMS.2023.3251297
Anany MElmesalawy MIbrahim IEl-Haleem A(2023)Deep Reinforcement Learning Algorithms for Location Optimization in Multi-RAT UAV-Assisted Heterogeneous Networks2023 5th Novel Intelligent and Leading Emerging Sciences Conference (NILES)10.1109/NILES59815.2023.10296805(396-401)Online publication date: 21-Oct-2023
https://doi.org/10.1109/NILES59815.2023.10296805
Bine LBoukerche ARuiz LLoureiro A(2023)ARAT: An Altitude-Based Routing Protocol for Hybrid Aerial-Terrestrial NetworksICC 2023 - IEEE International Conference on Communications10.1109/ICC45041.2023.10279654(334-339)Online publication date: 28-May-2023
https://doi.org/10.1109/ICC45041.2023.10279654
Bine LBoukerche ARuiz LLoureiro A(2023)IoDMixAd Hoc Networks10.1016/j.adhoc.2023.103204148:COnline publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1016/j.adhoc.2023.103204
Svaigen ABoukerche ARuiz LLoureiro A(2023)Trajectory MattersAd Hoc Networks10.1016/j.adhoc.2023.103179146:COnline publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1016/j.adhoc.2023.103179

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten