Abstract
Multi-access edge computing (MEC) presents computing services at the edge of networks to address the enormous processing requirements of intelligent applications. Due to the maneuverability of unmanned aerial vehicles (UAVs), they can be used as temporal aerial edge nodes for providing edge services to ground users in MEC. However, MEC environment is usually dynamic and complicated. It is a challenge for multiple UAVs to select appropriate service strategies. Besides, most of existing works study UAV-MEC with the assumption that the flight heights of UAVs are fixed; i.e., the flying is considered to occur with reference to a two-dimensional plane, which neglects the importance of the height. In this paper, with consideration of the co-channel interference, an optimization problem of energy efficiency is investigated to maximize the number of fulfilled tasks, where multiple UAVs in a three-dimensional space collaboratively fulfill the task computation of ground users. In the formulated problem, we try to obtain the optimal flight and sub-channel selection strategies for UAVs and schedule strategies for tasks. Based on the multi-agent deep deterministic policy gradient (MADDPG) algorithm, we propose a curiosity-driven and twin-networks-structured MADDPG (CTMADDPG) algorithm to solve the formulated problem. It uses the inner reward to facilitate the state exploration of agents, avoiding convergence at the sub-optimal strategy. Furthermore, we adopt the twin critic networks for update stabilization to reduce the probability of Q value overestimation. The simulation results show that CTMADDPG is outstanding in maximizing the energy efficiency of the whole system and outperforms the other benchmarks.
摘要
针对智能应用算力处理需求, 多接入边缘计算 (multi-access edge computing, MEC) 在网络边缘为其提供计算服务. 无人机 (unmanned aerial vehicle, UAV) 具有良好机动性, 可在MEC中作为临时空中边缘节点为地面用户提供边缘服务. 然而, MEC环境复杂且动态可变, 如何为多台无人机制定合适的服务策略具有一定挑战. 此外, 现有很多UAV-MEC相关工作均假定无人机飞行高度固定, 即飞行在二维平面内, 忽略了飞行高度的重要性. 在同信道干扰存在的前提下, 本文通过优化能效实现任务完成量的最大化, 多台无人机在三维空间中共同协作为地面用户提供任务计算服务. 为实现能效优化目标, 最大化任务完成量并最小化飞行能耗, 须制定最优飞行策略、 子信道选择策略以及任务调度策略. 基于多智能体深度确定性策略梯度算法 (multi-agent deep deterministic policy gradient, MADDPG), 本文提出好奇心驱动和双网络结构的多智能体深度确定性策略梯度算法 (curiosity-driven and twin-networks-structured MADDPG, CTMADDPG) 解决上述优化问题, 通过内部奖励促进智能体的状态探索, 避免收敛于次优策略. 同时, 利用双批评家网络降低Q值高估概率, 实现稳定更新. 仿真结果表明CTMADDPG算法在最大化整个系统能效方面表现突出, 优于其他基准测试算法.
This is a preview of subscription content, log in via an institution to check access.
Data availability
The data that support the findings of this study are available from the corresponding authors upon reasonable request.
References
Al-Hourani A, Kandeepan S, Lardner S, 2014. Optimal LAP altitude for maximum coverage. IEEE Wirel Commun Lett, 3(6):569–572. https://doi.org/10.1109/LWC.2014.2342736
Ashraf Ateya AA, Muthanna A, Kirichek R, et al., 2019. Energy- and latency-aware hybrid offloading algorithm for UAVs. IEEE Access, 7:37587–37600. https://doi.org/10.1109/ACCESS.2019.2905249
Badia AP, Sprechmann P, Vitvitskyi A, et al., 2020. Never give up: learning directed exploration strategies. Proc 8th Int Conf on Learning Representations.
Chakrabarty A, Langelaan J, 2009. Energy maps for longrange path planning for small- and micro-UAVs. AIAA Guidance, Navigation, and Control Conf, Article 6113. https://doi.org/10.2514/6.2009-6113
Dai C, Zhu K, Hossain E, 2022. Multi-agent deep reinforcement learning for joint decoupled user association and trajectory design in full-duplex multi-UAV networks. IEEE Trans Mob Comput, 22(10):6056–6070. https://doi.org/10.1109/TMC.2022.3188473
Dai ZJ, Zhang Y, Zhang WC, et al., 2022. A multi-agent collaborative environment learning method for UAV deployment and resource allocation. IEEE Trans Signal Inform Process Netw, 8:120–130. https://doi.org/10.1109/TSIPN.2022.3150911
Ding CF, Wang JB, Cheng M, et al., 2023. Dynamic transmission and computation resource optimization for dense LEO satellite assisted mobile-edge computing. IEEE Trans Commun, 71(5):3087–3102. https://doi.org/10.1109/TCOMM.2023.3253721
Fujimoto S, van Hoof H, Meger D, 2018. Addressing function approximation error in actor-critic methods. Proc 35th Int Conf on Machine Learning, p.1587–1596.
Gu XH, Zhang GA, Wang MX, et al., 2021. UAV-aided energy-efficient edge computing networks: security offloading optimization. IEEE Int Things J, 9(6):4245–4258. https://doi.org/10.1109/JIOT.2021.3103391
Ji JQ, Zhu K, Yi CY, et al., 2021. Energy consumption minimization in UAV-assisted mobile-edge computing systems: joint resource allocation and trajectory design. IEEE Int Things J, 8(10):8570–8584. https://doi.org/10.1109/JIOT.2020.3046788
Jiang FB, Wang KZ, Dong L, et al., 2020. Deep-learning-based joint resource scheduling algorithms for hybrid MEC networks. IEEE Int Things J, 7(7):6252–6265. https://doi.org/10.1109/JIOT.2019.2954503
Joo S, Kang HG, Kang J, 2021. CoSMoS: cooperative sky-ground mobile edge computing system. IEEE Trans Veh Technol, 70(8):8373–8377. https://doi.org/10.1109/TVT.2021.3094584
Lakew DS, Tran AT, Dao NN, et al., 2023. Intelligent offloading and resource allocation in heterogeneous aerial access IoT networks. IEEE Int Things J, 10(7):5704–5718. https://doi.org/10.1109/JIOT.2022.3161571
Liao ZF, Ma YB, Huang JW, et al., 2021. HOTSPOT: a UAV-assisted dynamic mobility-aware offloading for mobile-edge computing in 3-D space. IEEE Int Things J, 8(13):10940–10952. https://doi.org/10.1109/JIOT.2021.3051214
Liu JF, Li LX, Yang FC, et al., 2019. Minimization of offloading delay for two-tier UAV with mobile edge computing. Proc 15th Int Wireless Communications & Mobile Computing Conf, p.1534–1538. https://doi.org/10.1109/IWCMC.2019.8766778
Liu Q, Shi L, Sun LL, et al., 2020. Path planning for UAV-mounted mobile edge computing with deep reinforcement learning. IEEE Trans Veh Technol, 69(5):5723–5728. https://doi.org/10.1109/TVT.2020.2982508
Liu XY, Xu C, Yu HB, et al., 2022. Multi-agent deep reinforcement learning for end—edge orchestrated resource allocation in industrial wireless networks. Front Inform Technol Electron Eng, 23(1):47–60. https://doi.org/10.1631/FITEE.2100331
Mei HB, Yang K, Liu Q, et al., 2020. Joint trajectory-resource optimization in UAV-enabled edge-cloud system with virtualized mobile clone. IEEE Int Things J, 7(7):5906–5921. https://doi.org/10.1109/JIOT.2019.2952677
Tun YK, Park YM, Tran NH, et al., 2021. Energy-efficient resource management in UAV-assisted mobile edge computing. IEEE Commun Lett, 25(1):249–253. https://doi.org/10.1109/LCOMM.2020.3026033
Wang JR, Liu KY, Pan JP, 2020. Online UAV-mounted edge server dispatching for mobile-to-mobile edge computing. IEEE Int Things J, 7(2):1375–1386. https://doi.org/10.1109/JIOT.2019.2954798
Wang JZ, Zhang XL, He XS, et al., 2023. Bandwidth allocation and trajectory control in UAV-assisted IoV edge computing using multiagent reinforcement learning. IEEE Trans Reliab, 72(2):599–608. https://doi.org/10.1109/TR.2022.3192020
Wang L, Wang KZ, Pan CH, et al., 2021. Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing. IEEE Trans Cogn Commun Netw, 7(1):73–84. https://doi.org/10.1109/TCCN.2020.3027695
Wang LY, Zhang HX, Guo SS, et al., 2022. Deployment and association of multiple UAVs in UAV-assisted cellular networks with the knowledge of statistical user position. IEEE Trans Wirel Commun, 21(8):6553–6567. https://doi.org/10.1109/TWC.2022.3150429
Wang ZQ, Rong HG, Jiang HB, et al., 2022. A load-balanced and energy-efficient navigation scheme for UAV-mounted mobile edge computing. IEEE Trans Netw Sci Eng, 9(5):3659–3674. https://doi.org/10.1109/TNSE.2022.3188670
Wu SL, Xu WJ, Wang FY, et al., 2022. Distributed federated deep reinforcement learning based trajectory optimization for air-ground cooperative emergency networks. IEEE Trans Veh Technol, 71(8):9107–9112. https://doi.org/10.1109/TVT.2022.3175592
Xia WC, Zhu YX, De Simone L, et al., 2022. Multiagent collaborative learning for UAV enabled wireless networks. IEEE J Sel Areas Commun, 40(9):2630–2642. https://doi.org/10.1109/JSAC.2022.3191329
Xu S, Zhang XY, Li CG, et al., 2022. Deep reinforcement learning approach for joint trajectory design in multi-UAV IoT networks. IEEE Trans Veh Technol, 71(3):3389–3394. https://doi.org/10.1109/TVT.2022.3144277
Xu Y, Zhang TK, Loo J, et al., 2021. Completion time minimization for UAV-assisted mobile-edge computing systems. IEEE Trans Veh Technol, 70(11):12253–12259. https://doi.org/10.1109/TVT.2021.3112853
Xue NS, 2014. Design and Optimization of Lithium-Ion Batteries for Electric-Vehicle Applications. PhD Thesis, The University of Michigan, Ann Arbor, United States.
Yang L, Yao HP, Wang JJ, et al., 2020. Multi-UAV-enabled load-balance mobile-edge computing for IoT networks. IEEE Int Things J, 7(8):6898–6908. https://doi.org/10.1109/JIOT.2020.2971645
Yin ZY, Lin Y, Zhang YJ, et al., 2022. Collaborative multi-agent reinforcement learning aided resource allocation for UAV anti-jamming communication. IEEE Int Things J, 9(23):23995–24008. https://doi.org/10.1109/JIOT.2022.3188833
Yu Y, Bu XY, Yang K, et al., 2021. UAV-aided low latency multi-access edge computing. IEEE Trans Veh Technol, 70(5):4955–4967. https://doi.org/10.1109/TVT.2021.3072065
Yu Z, Gong YM, Gong SM, et al., 2020. Joint task offloading and resource allocation in UAV-enabled mobile edge computing. IEEE Int Things J, 7(4):3147–3159. https://doi.org/10.1109/JIOT.2020.2965898
Zhang L, Ansari N, 2020. Latency-aware IoT service provisioning in UAV-aided mobile-edge computing networks. IEEE Int Things J, 7(10):10573–10580. https://doi.org/10.1109/JIOT.2020.3005117
Zhao N, Ye ZY, Pei YY, et al., 2022. Multi-agent deep reinforcement learning for task offloading in UAV-assisted mobile edge computing. IEEE Trans Wirel Commun, 21(9):6949–6960. https://doi.org/10.1109/TWC.2022.3153316
Zheng LL, Chen JR, Wang JH, et al., 2021. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Proc 34th Int Conf on Neural Information Processing Systems, p.3757–3769.
Zhong RK, Liu X, Liu YW, et al., 2022. Multi-agent reinforcement learning in NOMA-aided UAV networks for cellular offloading. IEEE Trans Wirel Commun, 21(3):1498–1512. https://doi.org/10.1109/TWC.2021.3104633
Zhou FH, Wu YP, Hu RQ, et al., 2018. Computation rate maximization in UAV-enabled wireless-powered mobile-edge computing systems. IEEE J Sel Areas Commun, 36(9):1927–1941. https://doi.org/10.1109/JSAC.2018.2864426
Author information
Authors and Affiliations
Contributions
Yang LI and Ziling WEI designed the research. Yang LI processed the data. Yang LI and Jinshu SU drafted the paper. Baokang ZHAO helped organize the paper. Yang LI, Ziling WEI, Jinshu SU, and Baokang ZHAO revised and finalized the paper.
Corresponding authors
Ethics declarations
All the authors declare that they have no conflict of interest.
Additional information
Project supported by the National Natural Science Foundation of China (Nos. 62202486 and U22B2005)
List of supplementary materials
1 Preliminary details of the MADDPG algorithm
2 Flying actions in a 3D space
3 Details of the function \(K(o_{n}^{-t+1},f_{i})\)
4 Parameter setting
5 Training rewards with N = 3
6 Convergence rewards with N = 3
Fig. S1 Flying actions in a 3D space
Fig. S2 Training rewards of MIADDPG, MADDPG, and CTMADDPG with N = 3
Fig. S3 Reward results in different intensities Table S1 Summary of the key notations Table S2 Algorithm parameters Table S3 Simulation values
Supplementary materials for
Rights and permissions
About this article
Cite this article
Li, Y., Wei, Z., Su, J. et al. A multi-agent collaboration scheme for energy-efficient task scheduling in a 3D UAV-MEC space. Front Inform Technol Electron Eng 25, 824–838 (2024). https://doi.org/10.1631/FITEE.2300393
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2300393
Key words
- Multi-access edge computing
- Multi-agent reinforcement learning
- Unmanned aerial vehicles
- Task scheduling