Abstract
Mobile Edge Computing (MEC) has been regarded as a promising technology to satisfy the growing demand for resource-intensive applications in vehicle networks. Content caching and delivery, a critical problem in MEC, has attracted much research attention in the past decade. However, most existing caching schemes in the vehicle network scenario still confront two challenges: 1) High mobility of vehicles results in unstable connectivity; 2) Fairly massive state spaces of existing schemes have become their obstacles to good scalability. To address these challenges, we propose a hierarchical reinforcement learning (HRL)-based mobility-aware content caching and delivery policy for vehicle networks. First of all, we formulate the caching and delivery problem as a Markov decision process (MDP) problem. Our aim is to minimize the time-averaged transmission cost in the vehicle network scenario. To address the curse of dimensionality, we decompose the joint optimization of content caching and delivery into the vehicle side and RSU side subproblems. DDPG and Double-DQN are applied to address these two subtasks. Furthermore, an LSTM-based location prediction module is built to mine the mobility patterns of vehicles. Experimental studies and analysis, which are conducted on a real-world dataset, demonstrate that our approach outperforms other baseline schemes in terms of transmission cost and convergence speed.
This work is partially supported by Longyan Industry-Education Integration Project of Xiamen University (20210302) and the Natural Science Foundation of Guangdong (2021A1515011578).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abbas, N., Zhang, Y., Taherkordi, A., Skeie, T.: Mobile edge computing: a survey. IEEE Internet Things J. 5(1), 450–465 (2018). https://doi.org/10.1109/JIOT.2017.2750180
Barto, A.G., Mahadevan, S.: Recent advances in hierarchical reinforcement learning. Discrete Event Dyn Syst. 13(1), 41–77 (2003). https://doi.org/10.1023/A:1022140919877
Hu, B., Fang, L., Cheng, X., Yang, L.: Vehicle-to-vehicle distributed storage in vehicular networks. In: 2018 IEEE International Conference on Communications (ICC), pp. 1–6 (2018). https://doi.org/10.1109/ICC.2018.8422220
Hu, Y.C., Patel, M., Sabella, D., Sprecher, N., Young, V.: Mobile edge computing - a key technology towards 5G. ETSI White Pap. 11(11), 1–16 (2015)
Jiang, W., Feng, G., Qin, S.: Optimal cooperative content caching and delivery policy for heterogeneous cellular networks. IEEE Trans. Mob. Comput. 16(5), 1382–1393 (2017). https://doi.org/10.1109/TMC.2016.2597851
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. In: 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, Conference Track Proceedings (2016). http://arxiv.org/abs/1509.02971
Majidi, F., Khayyambashi, M.R., Barekatain, B.: HFDRL: An intelligent dynamic cooperate cashing method based on hierarchical federated deep reinforcement learning in edge-enabled IoT. IEEE Int. Things J. 9(2), 1402–1413 (2022). https://doi.org/10.1109/JIOT.2021.3086623
Narayanan, A., Verma, S., Ramadan, E., Babaie, P., Zhang, Z.: Deepcache: a deep learning based framework for content caching. In: Proceedings of the 2018 Workshop on Network Meets AI ML, pp. 48–53 (2018). https://doi.org/10.1145/3229543.3229555
Nomikos, N., Zoupanos, S., Charalambous, T., Krikidis, I.: A survey on reinforcement learning-aided caching in heterogeneous mobile edge networks. IEEE Access 10, 4380–4413 (2022). https://doi.org/10.1109/ACCESS.2022.3140719
Pateria, S., Subagdja, B., Tan, A.h., Quek, C.: Hierarchical reinforcement learning: A comprehensive survey. ACM Comput. Surv. 54(5), 1-35 (2021). https://doi.org/10.1145/3453160
Qian, Y., Wang, R., Wu, J., Tan, B., Ren, H.: Reinforcement learning-based optimal computing and caching in mobile edge network. IEEE J. Sel. Areas Commun. 38(10), 2343–2355 (2020). https://doi.org/10.1109/JSAC.2020.3000396
Qiao, G., Leng, S., Maharjan, S., Zhang, Y., Ansari, N.: Deep reinforcement learning for cooperative content caching in vehicular edge computing and networks. IEEE Int. Things J. 7(1), 247–257 (2020). https://doi.org/10.1109/JIOT.2019.2945640
Qin, Z., Xian, Y., Zhang, D.: A neural networks based caching scheme for mobile edge networks: Poster abstract. In: Proceedings of the 17th Conference on Embedded Networked Sensor Systems, pp. 408–409 (2019). https://doi.org/10.1145/3356250.3361961
Rappaport, T.S., et al.: Wireless communications: principles and practice, vol. 2. prentice hall PTR New Jersey (1996)
Song, C., Xu, W., Wu, T., Yu, S., Zeng, P., Zhang, N.: QoE-driven edge caching in vehicle networks based on deep reinforcement learning. IEEE Trans. Veh. Technol. 70(6), 5286–5295 (2021). https://doi.org/10.1109/TVT.2021.3077072
Sun, Y., Chen, Z., Liu, H.: Delay analysis and optimization in cache-enabled multi-cell cooperative networks. In: 2016 IEEE Global Communications Conference (GLOBECOM), pp. 1–7 (2016). https://doi.org/10.1109/GLOCOM.2016.7841723
Sun, Y., Cui, Y., Liu, H.: Joint pushing and caching for bandwidth utilization maximization in wireless networks. IEEE Trans. Commun. 67(1), 391–404 (2019). https://doi.org/10.1109/TCOMM.2018.2858791
Szepesvári, C.: Joint pushing and caching for bandwidth utilization maximization in wireless networks. Synth. Lect. Artif. Intell. Mach. Learn. 4(1), 1–103 (2010). https://doi.org/10.2200/S00268ED1V01Y201005AIM009
Tsai, K.C., Wang, L., Han, Z.: Mobile social media networks caching with convolutional neural network. In: 2018 IEEE Wireless Communications and Networking Conference Workshops (WCNCW), pp. 83–88 (2018). https://doi.org/10.1109/WCNCW.2018.8368988
Van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence. vol. 30 (2016). https://doi.org/10.1609/aaai.v30i1.10295
Yao, J., Ansari, N.: Joint content placement and storage allocation in c-rans for IoT sensing service. IEEE Int. Things J. 6(1), 1060–1067 (2019). https://doi.org/10.1109/JIOT.2018.2866947
Yu, Z., Hu, J., Min, G., Zhao, Z., Miao, W., Hossain, M.S.: Mobility-aware proactive edge caching for connected vehicles using federated learning. IEEE Trans. Intell. Transp. Syst. 22(8), 5341–5351 (2021). https://doi.org/10.1109/TITS.2020.3017474
Zong, T., Li, C., Lei, Y., Li, G., Cao, H., Liu, Y.: Cocktail edge caching: ride dynamic trends of content popularity with ensemble learning. In: IEEE INFOCOM 2021 - IEEE Conference on Computer Communications, pp. 1–10 (2021). https://doi.org/10.1109/INFOCOM42981.2021.9488910
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, L., Lai, Y., Yang, F. (2023). Hierarchical Reinforcement Learning-Based Mobility-Aware Content Caching and Delivery Policy for Vehicle Networks. In: Meng, W., Lu, R., Min, G., Vaidya, J. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2022. Lecture Notes in Computer Science, vol 13777. Springer, Cham. https://doi.org/10.1007/978-3-031-22677-9_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-22677-9_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22676-2
Online ISBN: 978-3-031-22677-9
eBook Packages: Computer ScienceComputer Science (R0)