Skip to main content

Advertisement

Log in

Privacy-preserving cooperative hierarchical caching approach based on federated deep reinforcement learning for vehicular edge computing

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Vehicular edge computing (VEC) supports real-time vehicular application services. However, accurate prediction and caching of popular content in roadside units (RSUs) while safeguarding user privacy is challenging. This study proposes a privacy-preserving cooperative hierarchical caching approach based on federated deep reinforcement learning (PCFR) for VEC. An asynchronous federated learning algorithm based on improved differential privacy is proposed, which considers the vehicle locations and movement directions and reasonably limits the local-update norm, improving the global model prediction accuracy while protecting user privacy. To address spatiotemporal variations in content popularity, a proposed attention-weighted asynchronous actor-critic collaborative caching algorithm extracts and weights key state features to optimize the collaborative cache content and its distribution location, enhancing the overall caching efficiency. In simulation, the PCFR scheme outperforms other caching schemes. With a 400-MB cache capacity, the PCFR scheme improves the cache hit rate by approximately 50.0% and reduces the content access delay by approximately 28.0%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Algorithm 2
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Data availability

The datasets analyzed during the current study were all derived from the following public domain resources [https://grouplens.org/datasets/movielens/; https://grouplens.org/datasets/hetrec-2011/].

References

  1. Xiao H, Zhang X, Hu Z, Zheng M, Liang Y (2024) A collaborative cache allocation strategy for performance and link cost in mobile edge computing. J Supercomput 80(15):22885–22912. https://doi.org/10.1007/s11227-024-06310-7

    Article  MATH  Google Scholar 

  2. Zhao J, Huang F, Liao L, Zhang Q (2024) Blockchain-based trust management model for vehicular ad hoc networks. IEEE Internet Things J 11(5):8118–8132. https://doi.org/10.1109/JIOT.2023.3318597

    Article  MATH  Google Scholar 

  3. Tewari P, Tripathi S (2023) An energy efficient routing scheme in internet of things enabled wsn: neuro-fuzzy approach. J Supercomput 79(10):11134–11158. https://doi.org/10.1007/s11227-023-05091-9

    Article  MATH  Google Scholar 

  4. Jeong J, Guo S, Gu Y, He T, Du DH (2012) Trajectory-based statistical forwarding for multihop infrastructure-to-vehicle data delivery. IEEE Trans Mob Comput 11(10):1523–1537. https://doi.org/10.1109/TMC.2011.189

    Article  MATH  Google Scholar 

  5. Yin L, Cao G (2006) Supporting cooperative caching in ad hoc networks. IEEE Trans Mob Comput 5(1):77–89. https://doi.org/10.1109/TMC.2006.15

    Article  MATH  Google Scholar 

  6. Singh P, Hazarika B, Singh K, Pan C, Huang WJ, Li CP (2024) Drl-based federated learning for efficient vehicular caching management. IEEE Internet Things J 11(21):34156–34171. https://doi.org/10.1109/JIOT.2024.3417265

    Article  Google Scholar 

  7. Anh Nguyen DT, Cheng J, Nguyen DT, Nedić A (2023) Crowdcache: A decentralized game-theoretic framework for mobile edge content sharing. In: 2023 21st International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), pp 350–357, https://doi.org/10.23919/WiOpt58741.2023.10349816

  8. Huang Y, Song X, Ye F, Yang Y, Li X (2020) Fair and efficient caching algorithms and strategies for peer data sharing in pervasive edge computing environments. IEEE Trans Mob Comput 19(4):852–864. https://doi.org/10.1109/TMC.2019.2902090

    Article  MATH  Google Scholar 

  9. Liu J, Zhu B, Wang F, Jin Y, Zhang W, Xu Z, Cui S (2023) Cav3: Cache-assisted viewport adaptive volumetric video streaming. In: 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pp 173–183, https://doi.org/10.1109/VR55154.2023.00033

  10. Li Y, Ma H, Wang L, Mao S, Wang G (2022) Optimized content caching and user association for edge computing in densely deployed heterogeneous networks. IEEE Trans Mob Comput 21(6):2130–2142. https://doi.org/10.1109/TMC.2020.3033563

    Article  MATH  Google Scholar 

  11. Zeng F, Zhang K, Wu L, Wu J (2023) Efficient caching in vehicular edge computing based on edge-cloud collaboration. IEEE Trans Veh Technol 72(2):2468–2481. https://doi.org/10.1109/TVT.2022.3213130

    Article  MATH  Google Scholar 

  12. Yu Z, Hu J, Min G, Wang Z, Miao W, Li S (2022) Privacy-preserving federated deep learning for cooperative hierarchical caching in fog computing. IEEE Internet Things J 9(22):22246–22255. https://doi.org/10.1109/JIOT.2021.3081480

    Article  MATH  Google Scholar 

  13. Zhao M, Nakhai MR (2024) A unified federated deep q learning caching scheme for scalable collaborative edge networks. IEEE Transactions on Mobile Computing pp 1–12. https://doi.org/10.1109/TMC.2024.3382824

  14. Tian A, Feng B, Zhou H, Huang Y, Sood K, Yu S, Zhang H (2023) Efficient federated drl-based cooperative caching for mobile edge networks. IEEE Trans Netw Serv Manage 20(1):246–260. https://doi.org/10.1109/TNSM.2022.3198074

    Article  MATH  Google Scholar 

  15. Wang S, Wu Q, Fan Q, Fan P, Wang J (2023) Mobility-aware asynchronous federated learning for edge-assisted vehicular networks. In: ICC 2023 - IEEE International Conference on Communications, pp 3621–3626, https://doi.org/10.1109/ICC45041.2023.10278823

  16. Zhao J, Huang C, Wang W, Xie R, Dong R, Matwin S (2023) Local differentially private federated learning with homomorphic encryption. J Supercomput 79(17):19365–19395. https://doi.org/10.1007/s11227-023-05378-x

    Article  Google Scholar 

  17. Yang X, Huang W, Ye M (2023) Dynamic personalized federated learning with adaptive differential privacy. Adv Neural Inf Process Syst 36:72181–72192

    MATH  Google Scholar 

  18. Du R, Wu J, Gao Y (2024) Dual-q network deep reinforcement learning-based computation offloading method for industrial internet of things. J Supercomput 80(17):25590–25615. https://doi.org/10.1007/s11227-024-06425-x

    Article  Google Scholar 

  19. Cao T, Zhang Z, Wang X, Xiao H, Xu C (2024) Ptcc: A privacy-preserving and trajectory clustering-based approach for cooperative caching optimization in vehicular networks. IEEE Trans Sustain Comput 9(4):615–630. https://doi.org/10.1109/TSUSC.2024.3350386

    Article  MATH  Google Scholar 

  20. Liu M, Li D, Wu H, Lyu F, Shen X (2022) Real-time search-driven caching for sensing data in vehicular networks. IEEE Internet Things J 9(14):12219–12230. https://doi.org/10.1109/JIOT.2021.3134964

    Article  MATH  Google Scholar 

  21. Wu H, Jin J, Ma H, Xing L (2024) Federation-based deep reinforcement learning cooperative cache in vehicular edge networks. IEEE Internet Things J 11(2):2550–2560. https://doi.org/10.1109/JIOT.2023.3292374

    Article  MATH  Google Scholar 

  22. Liu Y, Mao B (2023) On a novel content edge caching approach based on multi-agent federated reinforcement learning in internet of vehicles. In: 2023 32nd Wireless and Optical Communications Conference (WOCC), pp 1–5, https://doi.org/10.1109/WOCC58016.2023.10139417

  23. Somesula MK, Rout RR, Somayajulu D (2023) Greedy cooperative cache placement for mobile edge networks with user preferences prediction and adaptive clustering. Ad Hoc Netw 140:103051. https://doi.org/10.1016/j.adhoc.2022.103051

    Article  MATH  Google Scholar 

  24. Yang S, Fan S, Deng G, Tian H (2019) Local content cloud based cooperative caching placement for edge caching. In: 2019 IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), pp 1–6, https://doi.org/10.1109/PIMRC.2019.8904138

  25. Li Q, Wang X, Wang D (2019) Optimal d2d cooperative caching system in sdn based wireless network. In: 2019 IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), pp 1–7, https://doi.org/10.1109/PIMRC.2019.8904149

  26. Jin Y, Liu J, Wang F, Cui S (2023) Ebublio: Edge-assisted multiuser \(360^\circ\) video streaming. IEEE Internet Things J 10(17):15408–15419. https://doi.org/10.1109/JIOT.2023.3263885

    Article  Google Scholar 

  27. Yu Z, Hu J, Min G, Zhao Z, Miao W, Hossain MS (2021) Mobility-aware proactive edge caching for connected vehicles using federated learning. IEEE Trans Intell Transp Syst 22(8):5341–5351. https://doi.org/10.1109/TITS.2020.3017474

    Article  MATH  Google Scholar 

  28. Li D, Zhang H, Li T, Ding H, Yuan D (2023) Community detection and attention-weighted federated learning based proactive edge caching for d2d-assisted wireless networks. IEEE Trans Wireless Commun 22(11):7287–7303. https://doi.org/10.1109/TWC.2023.3249756

    Article  MATH  Google Scholar 

  29. Wang W, Zhao Y, Wu Q, Fan Q, Zhang C, Li Z (2022) Asynchronous federated learning based mobility-aware caching in vehicular edge computing. In: 2022 14th International Conference on Wireless Communications and Signal Processing (WCSP), pp 1–5, https://doi.org/10.1109/WCSP55476.2022.10039430

  30. Zhou X, Liang W, Kawai A, Fueda K, She J, Wang KIK (2024) Adaptive segmentation enhanced asynchronous federated learning for sustainable intelligent transportation systems. IEEE Trans Intell Transp Syst 25(7):6658–6666. https://doi.org/10.1109/TITS.2024.3362058

    Article  MATH  Google Scholar 

  31. Cao XK, Wang CD, Lai JH, Huang Q, Chen CLP (2023) Multiparty secure broad learning system for privacy preserving. IEEE Trans Cybernet 53(10):6636–6648. https://doi.org/10.1109/TCYB.2023.3235496

    Article  MATH  Google Scholar 

  32. Nguyen CH, Saputra YM, Hoang DT, Nguyen DN, Nguyen VD, Xiao Y, Dutkiewicz E (2024) Encrypted data caching and learning framework for robust federated learning-based mobile edge computing. IEEE/ACM Trans Netw 32(3):2705–2720. https://doi.org/10.1109/TNET.2024.3365815

    Article  MATH  Google Scholar 

  33. Jiang X, Liu B, Qin J, Zhang Y, Qian J (2022) Fedncf: Federated neural collaborative filtering for privacy-preserving recommender system. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp 1–8, https://doi.org/10.1109/IJCNN55064.2022.9892909

  34. Cheng A, Wang P, Zhang XS, Cheng J (2022) Differentially private federated learning with local regularization and sparsification. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 10112–10121, https://doi.org/10.1109/CVPR52688.2022.00988

  35. Gao X, Sun Y, Chen H, Xu X, Cui S (2024) Joint computing, pushing, and caching optimization for mobile-edge computing networks via soft actor-critic learning. IEEE Internet Things J 11(6):9269–9281. https://doi.org/10.1109/JIOT.2023.3323433

    Article  MATH  Google Scholar 

  36. Qiao D, Guo S, Liu D, Long S, Zhou P, Li Z (2022) Adaptive federated deep reinforcement learning for proactive content caching in edge computing. IEEE Trans Parallel Distrib Syst 33(12):4767–4782. https://doi.org/10.1109/TPDS.2022.3201983

    Article  MATH  Google Scholar 

  37. Li C, Zhang Y, Luo Y (2023) A federated learning-based edge caching approach for mobile edge computing-enabled intelligent connected vehicles. IEEE Trans Intell Transp Syst 24(3):3360–3369. https://doi.org/10.1109/TITS.2022.3224395

    Article  MATH  Google Scholar 

  38. Yang W, Liu Z (2024) Efficient vehicular edge computing: a novel approach with asynchronous federated and deep reinforcement learning for content caching in vec. IEEE Access 12:13196–13212. https://doi.org/10.1109/ACCESS.2024.3355462

    Article  Google Scholar 

  39. Singh P, Hazarika B, Singh K, Pan C, Huang WJ, Li CP (2024) Drl-based federated learning for efficient vehicular caching management. IEEE Internet Things J 11(21):34156–34171. https://doi.org/10.1109/JIOT.2024.3417265

    Article  Google Scholar 

  40. 3GPP A (2016) Study on lte-based v2x services. 3rd Generation Partnership Project (3GPP), TR 36885

  41. Chen J, Wu H, Yang P, Lyu F, Shen X (2020) Cooperative edge caching with location-based and popular contents for vehicular networks. IEEE Trans Veh Technol 69(9):10291–10305. https://doi.org/10.1109/TVT.2020.3004720

    Article  MATH  Google Scholar 

  42. Lee HS, Lee JW (2021) Adaptive transmission scheduling in wireless networks for asynchronous federated learning. IEEE J Sel Areas Commun 39(12):3673–3687. https://doi.org/10.1109/JSAC.2021.3118353

    Article  MATH  Google Scholar 

  43. Wu Q, Wang W, Fan P, Fan Q, Zhu H, Letaief KB (2024) Cooperative edge caching based on elastic federated and multi-agent deep reinforcement learning in next-generation networks. IEEE Trans Netw Serv Manage 21(4):4179–4196. https://doi.org/10.1109/TNSM.2024.3403842

    Article  MATH  Google Scholar 

  44. Harper FM, Konstan JA (2015) The movielens datasets: history and context. Acm Trans Interact Intell Syst (tiis) 5(4):1–19

    MATH  Google Scholar 

  45. Wu Q, Zhao Y, Fan Q, Fan P, Wang J, Zhang C (2023) Mobility-aware cooperative caching in vehicular edge computing based on asynchronous federated and deep reinforcement learning. IEEE J Sel Topics Signal Process 17(1):66–81. https://doi.org/10.1109/JSTSP.2022.3221271

    Article  MATH  Google Scholar 

  46. Cantador I, Brusilovsky P, Kuflik T (2011) 2nd workshop on information heterogeneity and fusion in recommender systems (hetrec 2011). In: Proceedings of the 5th ACM conference on Recommender systems. ACM, New York, NY, USA, RecSys 2011

Download references

Acknowledgements

This work was supported by Songshan Laboratory (Project No. 232102210154), the Pre-research Project SongShan Laboratory (Project No. YYJC022022001), and the Major Science and Technology Projects in Henan Province (Project No. 241110210200).

Author information

Authors and Affiliations

Authors

Contributions

Yangxi Mu worked in investigation, conceptualization, methodology, simulation, validation, writing—original draft, visualization, and writing—review & editing. Mengyang He helped in conceptualization, methodology, validation, writing—original draft, and writing—review & editing. Bing Hao helped in methodology, validation, and writing—review & editing.

Corresponding author

Correspondence to Mengyang He.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Consent to Participate

The authors declare that they agree to participate.

Consent for Publication

The authors declare that they agree to publish.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Proof of theorem 1

Proof of theorem 1

Theorem 1

(The Gaussian mechanism satisfies \((\epsilon , \delta )\)-differential privacy) Given privacy parameters \(\epsilon > 0\) and \(\delta \in (0,1)\), the Gaussian mechanism \(\mathcal {M}(D) = f(D) + \mathcal {N}(0, \sigma ^2 I)\) satisfies \((\epsilon , \delta )\)-DP when the noise standard deviation \(\sigma\) satisfies

$$\begin{aligned} \sigma \ge \frac{\sqrt{2 \ln (1.25/\delta )} \Delta f}{\epsilon } \end{aligned}$$
(A1)

Here, \(\Delta f\) is the \(L_2\)-sensitivity of function f(x), representing the maximum output difference between any two adjacent datasets D and \(D'\); i.e., \(\Delta f = \max _{D, D'} \Vert f(D) - f(D') \Vert _2\).

Proof of Theorem 1

Assuming that the Gaussian mechanism is \(\mathcal {M}(D) = f(D) + \mathcal {N}(0, \sigma ^2 I)\), for any output value z, its probability density function is:

$$\begin{aligned} p(\mathcal {M}(D) = z) = \frac{1}{\sqrt{2\pi \sigma ^2}} \exp \left( -\frac{(z - f(D))^2}{2\sigma ^2} \right) \end{aligned}$$
(A2)

Similarly, the probability density function of \(D'\) is:

$$\begin{aligned} p(\mathcal {M}(D') = z) = \frac{1}{\sqrt{2\pi \sigma ^2}} \exp \left( -\frac{(z - f(D'))^2}{2\sigma ^2} \right) \end{aligned}$$
(A3)

By comparing their output probabilities, we have:

$$\begin{aligned} \frac{p(\mathcal {M}(D) = z)}{p(\mathcal {M}(D') = z)} = \exp \left( \frac{(z - f(D'))^2 - (z - f(D))^2}{2\sigma ^2} \right) \end{aligned}$$
(A4)

Simplifying this expression gives:

$$\begin{aligned} \frac{p(\mathcal {M}(D) = z)}{p(\mathcal {M}(D') = z)} = \exp \left( \frac{2z(f(D) - f(D')) + f(D')^2 - f(D)^2}{2\sigma ^2} \right) \end{aligned}$$
(A5)

Considering the properties of the Gaussian distribution, when the values of f(D) or \(f(D')\) approach z, the maximum value is:

$$\begin{aligned} \max _z \frac{p(\mathcal {M}(D) = z)}{p(\mathcal {M}(D') = z)} = \exp \left( \frac{\Vert f(D) - f(D')\Vert _2^2}{2\sigma ^2} \right) \end{aligned}$$
(A6)

AS the maximum difference between D and \(D'\) is \(\Vert f(D) - f(D')\Vert _2 \le \Delta f\), it follows that:

$$\begin{aligned} \frac{p(\mathcal {M}(D) = z)}{p(\mathcal {M}(D') = z)} \le \exp \left( \frac{(\Delta f)^2}{2\sigma ^2} \right) \le e^{\epsilon } \end{aligned}$$
(A7)

To further meet the relaxed condition of \((\epsilon , \delta )\)-differential privacy, the privacy loss function \(\mathcal {L}(o; \mathcal {M}, D, D')\) is introduced to analyze the privacy leakage. Here,

$$\begin{aligned} \mathcal {L}(o; \mathcal {M}, D, D') = \log \left( \frac{\Pr [\mathcal {M}(D) = o]}{\Pr [\mathcal {M}(D') = o]} \right) \end{aligned}$$
(A8)

This function measures the difference in the algorithm output distribution on adjacent datasets. When this function is combined with the Rényi differential privacy method, solution of the high-order moments of the privacy loss function derives the upper bound for the noise standard deviation required to satisfy the \((\epsilon , \delta )\)-differential privacy condition:

$$\begin{aligned} \sigma \ge \frac{\sqrt{2 \ln (1.25/\delta )} \Delta f}{\epsilon } \end{aligned}$$
(A1)

This formula incorporates the function sensitivity \(\Delta f\), privacy budget \(\epsilon\), and relaxation parameter \(\delta\), ensuring more stringent privacy protection within the permissible privacy disclosure range. \(\hfill\square\)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mu, Y., Hao, B. & He, M. Privacy-preserving cooperative hierarchical caching approach based on federated deep reinforcement learning for vehicular edge computing. J Supercomput 81, 532 (2025). https://doi.org/10.1007/s11227-025-07058-4

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-025-07058-4

Keywords