Skip to main content
SpringerLink
Log in

Deep Deterministic Policy Gradient Algorithm for Space/Aerial-Assisted Computation Offloading

  • Conference paper
  • First Online:
Communications and Networking (ChinaCom 2021)

Included in the following conference series:

  • 1078 Accesses

Abstract

Space-air-ground integrated network (SAGIN) has been envisioned as a promising architecture and computation offloading is a challenging issue, with the growing demand for computation-intensive applications in remote area. In this paper, we investigate a SAGIN edge computing architecture considering the energy consumption and delay of computation offloading, in which ground users can determine whether take advantage of edge server mounted on the unmanned aerial vehicle and satellite for partial offloading or not. Specifically, the optimization problem of minimizing the total cost is formulated as a Markov decision process, and we proposed a deep reinforcement learning-based method to derive the near-optimal policy, adopting the deep deterministic policy gradient (DDPG) algorithm to handle the large state space and continuous action space. Finally, simulation results demonstrate that the partial offloading scheme learned from proposed algorithm can substantially reduce the user devices’ total cost as compared to other greedy policies, and its performance is better than the binary offloading scheme learned from Deep Q-learning algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Zhang, Z., et al.: 6G wireless networks: vision, requirements, architecture, and key technologies. IEEE Veh. Technol. Mag. 14(3), 28–41 (2019)

    Article  Google Scholar 

  2. Liu, J., Shi, Y., Fadlullah, Z.M., Kato, N.: Space-air-ground integrated network: a survey. IEEE Commun. Surv. Tutor. 20(4), 2714–2741 (2018)

    Article  Google Scholar 

  3. Jiang, W., Han, B., Habibi, M.A., Schotten, H.D.: The road towards 6G: a comprehensive survey. IEEE Open J. Commun. Soc. 2, 334–366 (2021)

    Article  Google Scholar 

  4. Abbas, N., Zhang, Y., Taherkordi, A., Skeie, T.: Mobile edge computing: a survey. IEEE Internet Things J. 5(1), 450–465 (2018)

    Article  Google Scholar 

  5. Khayyat, M., et al.: Multilevel service-provisioning-based autonomous vehicle applications. Sustainability 12, 2497 (2020)

    Google Scholar 

  6. Mach, P., Becvar, Z.: Mobile edge computing: a survey on architecture and computation offloading. IEEE Commun. Surv. Tutor.. 19(3), 1628–1656 (2017)

    Article  Google Scholar 

  7. Lyu, X., et al.: Optimal schedule of mobile edge computing for internet of things using partial information. IEEE J. Sel. Areas Commun. 35(11), 2606–2615 (2017)

    Article  Google Scholar 

  8. Min, M., Xiao, L., Chen, Y., Cheng, P., Wu, D., Zhuang, W.: Learning-based computation offloading for IoT devices with energy harvesting. IEEE Trans. Veh. Technol. 68(2), 1930–1941 (2019)

    Article  Google Scholar 

  9. Elgendy, I.A., Zhang, W.-Z., He, H., Gupta, B.B., Abd El-Latif, A.A.: Joint computation offloading and task caching for multi-user and multi-task MEC systems: reinforcement learning-based algorithms. Wirel. Netw. 27(3), 2023–2038 (2021). https://doi.org/10.1007/s11276-021-02554-w

    Article  Google Scholar 

  10. Chen, Z., Wang, X.: Decentralized computation offloading for multi-user mobile edge computing: a deep reinforcement learning approach. EURASIP J. Wirel. Commun. Netw. 2020(1), 1–21 (2020). https://doi.org/10.1186/s13638-020-01801-6

    Article  Google Scholar 

  11. Zhou, C., et al.: Delay-aware IoT task scheduling in space-air-ground integrated network. In: 2019 IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA, 2019, pp. 1–6. IEEE (2019)

    Google Scholar 

  12. Xu, F., Yang, F., Zhao, C., Wu, S.: Deep reinforcement learning based joint edge resource management in maritime network. China Commun. 17(5), 211–222 (2020)

    Article  Google Scholar 

  13. Zhou, C., et al.: Deep reinforcement learning for delay-oriented IoT task scheduling in space-air-ground integrated network. IEEE Trans. Wirel. Commun. 20(2), 911–925 (2021)

    Article  Google Scholar 

  14. Dinh, T.H., Niyato, D., Hung, N.T.: Optimal energy allocation policy for wireless networks in the sky. In: 2015 IEEE International Conference on Communications (ICC), London, UK, 2015, pp. 3204–3209. IEEE (2015)

    Google Scholar 

  15. Tang, Q., Fei, Z., Li, B., Han, Z.: Computation offloading in LEO satellite networks with hybrid cloud and edge computing. IEEE Internet Things J. 11, 9164–9176 (2021)

    Article  Google Scholar 

  16. Cheng, N., et al.: Space/Aerial-assisted computing offloading for IoT applications: a learning-based approach. IEEE J. Sel. Areas Commun. 37(5), 1117–1129 (2019)

    Article  Google Scholar 

  17. Saleem, U., Liu, Y., Jangsher, S., Li, Y.: Performance guaranteed partial offloading for mobile edge computing. In: GLOBECOM (2018)

    Google Scholar 

  18. Wu, Q., Zeng, Y., Zhang, R.: Joint trajectory and communication design for multi-UAV enabled wireless networks. IEEE Trans. Wirel. Commun. 17(3), 2109–2121 (2018)

    Article  Google Scholar 

  19. Bi, S., Zhang, Y.: Computation rate maximization for wireless powered mobile-edge computing with binary computation offloading. IEEE Trans. Wirel. Commun. 17(6), 4177–4190 (2018)

    Article  Google Scholar 

  20. Guo, S., Xiao, B., Yang, Y., Yang, Y.: Energy-efficient dynamic offloading and resource scheduling in mobile cloud computing. In: IEEE International Conference on Computer Communications, pp. 1–9 (2016)

    Google Scholar 

  21. Zhang, N., Liang, H., Cheng, N., Tang, Y., Mark, J.W., Shen, X.S.: Dynamic spectrum access in multi-channel cognitive radio networks. IEEE J. Sel. Areas Commun. 32(11), 2053–2064 (2014)

    Article  Google Scholar 

  22. Hosseini, N., Jamal, H., Haque, J., Magesacher, T., Matolak, D.W.: UAV command and control, navigation and surveillance: a review of potential 5G and satellite systems. In: IEEE Aerospace Conference, pp. 1–10 (2019)

    Google Scholar 

  23. Shi, W., et al.: Multiple drone-cell deployment analyses and optimization in drone assisted radio access networks. IEEE Access 6, 12518–1252 (2018)

    Article  Google Scholar 

  24. Al-Hourani, A., Kandeepan, S., Lardner, S.: Optimal LAP altitude for maximum coverage. IEEE Wirel. Commun. Lett. 3(6), 569–572 (2014)

    Article  Google Scholar 

  25. Bor-Yaliniz, R.I., El-Keyi, A., Yanikomeroglu, H.: Efficient 3-D placement of an aerial base station in next generation cellular networks. In: IEEE ICC, pp. 1–5 (2016)

    Google Scholar 

  26. Kanellopoulos, S.A., Kourogiorgas, C.I., Panagopoulos, A.D., Livieratos, S.N., Chatzarakis, G.E.: Channel model for satellite communication links above 10GHz based on Weibull distribution. IEEE Commun. Lett. 18(4), 568–571 (2014)

    Article  Google Scholar 

  27. Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. Computer Science (2015)

    Google Scholar 

Download references

Ackonwledgment

This work is supported by the National Natural Science Foundation of China under grant (61761014), Guangxi Natural Science Foundation (2018GXNSFBA281131), Ministry of Education Key Laboratory of Cognitive Radio and Information Processing (CRKL190109).

Author information

Authors and Affiliations

  1. Guilin University of Electronic Technology, Guilin, 541004, China

    Jielin Fu, Lei Liang, Yanlong Li & Junyi Wang

Authors
  1. Jielin Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanlong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanlong Li .

Editor information

Editors and Affiliations

  1. Shanghai University, Shanghai, China

    Honghao Gao

  2. Fudan University, Shanghai, China

    Jun Wun

  3. Zhejiang University, Hangzhou, China

    Jianwei Yin

  4. Tsinghua University, Beijing, China

    Feifei Shen

  5. Xidian University, X'ian, China

    Yulong Shen

  6. Hangzhou Dianzi University, Hangzhou, China

    Jun Yu

Rights and permissions

Reprints and permissions

Copyright information

© 2022 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fu, J., Liang, L., Li, Y., Wang, J. (2022). Deep Deterministic Policy Gradient Algorithm for Space/Aerial-Assisted Computation Offloading. In: Gao, H., Wun, J., Yin, J., Shen, F., Shen, Y., Yu, J. (eds) Communications and Networking. ChinaCom 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 433. Springer, Cham. https://doi.org/10.1007/978-3-030-99200-2_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-99200-2_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-99199-9

  • Online ISBN: 978-3-030-99200-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation