Abstract
With extensive applications and remarkable performance, deep reinforcement learning is becoming one of the most important technologies that researchers have been focusing on. Many applications have used reinforcement learning, such as robotics, recommendation systems, and healthcare systems. These systems could collect data about the environment or users, which may contain sensitive information that posed a real risk when these data were disclosed. In this work, we aim to preserve the privacy of the data used in deep reinforcement learning with Double Deep-Q-Network in continuous space by adopting the differentially private SGD method to inject a noise to the gradient. In our experiment, we used a different amount of noise on two separate settings to demonstrate how effective of using this method.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Abadi, M., et al.: Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318 (2016)
Allen, M., Monks, T.: Integrating deep reinforcement learning networks with health system simulations. arXiv preprint arXiv:2008.07434 (2020)
Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep reinforcement learning: a brief survey. IEEE Signal Process. Mag. 34(6), 26–38 (2017)
Balle, B., Gomrokchi, M., Precup, D.: Differentially private policy evaluation. In: International Conference on Machine Learning, pp. 2130–2138. PMLR (2016)
Boufous, O.: Deep reinforcement learning for complete coverage path planning in unknown environments (2020)
Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., Naor, M.: Our data, ourselves: privacy via distributed noise generation. In: Vaudenay, S. (ed.) EUROCRYPT 2006. LNCS, vol. 4004, pp. 486–503. Springer, Heidelberg (2006). https://doi.org/10.1007/11761679_29
Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) TCC 2006. LNCS, vol. 3876, pp. 265–284. Springer, Heidelberg (2006). https://doi.org/10.1007/11681878_14
Lanham, M.: Hands-on reinforcement learning for games: implementing self-learning agents in games using artificial intelligence techniques (2020)
Lapan, M.: Deep Reinforcement Learning Hands-On - Second Edition. Packt Publishing Ltd (2020)
Ma, P., Wang, Z., Zhang, L., Wang, R., Zou, X., Yang, T.: Differentially private reinforcement learning. In: Zhou, J., Luo, X., Shen, Q., Xu, Z. (eds.) ICICS 2019. LNCS, vol. 11999, pp. 668–683. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-41579-2_39
Pan, X., Wang, W., Zhang, X., Li, B., Yi, J., Song, D.: How you act tells a lot: privacy-leaking attack on deep reinforcement learning. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 368–376 (2019)
Rosset, C., Jose, D., Ghosh, G., Mitra, B., Tiwary, S.: Optimizing query evaluations using reinforcement learning for web search. In: The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1193–1196 (2018)
Shen, S., Zhu, T., Ye, D., Wang, M., Zuo, X., Zhou, A.: A novel differentially private advising framework in cloud server environment. Concurr. Comput. Pract. Exp. e5932 (2020)
Song, S., Chaudhuri, K., Sarwate, A.D.: Stochastic gradient descent with differentially private updates. In: 2013 IEEE Global Conference on Signal and Information Processing, pp. 245–248. IEEE (2013)
Tang, X., Zhu, L., Shen, M., Du, X.: When homomorphic cryptosystem meets differential privacy: training machine learning classifier with privacy protection. arXiv preprint arXiv:1812.02292 (2018)
Tossou, A.C., Dimitrakakis, C.: Achieving privacy in the adversarial multi-armed bandit. arXiv preprint arXiv:1701.04222 (2017)
Van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30 (2016)
Wang, B., Hegde, N.: Privacy-preserving q-learning with functional noise in continuous spaces. In: Advances in Neural Information Processing Systems, pp. 11327–11337 (2019)
Ye, D., Zhu, T., Shen, S., Zhou, W., Yu, P.: Differentially private multi-agent planning for logistic-like problems. IEEE Trans. Dependable Secure Comput., 1 (2020). https://doi.org/10.1109/TDSC.2020.3017497
Ye, D., Zhu, T., Zhou, W., Philip, S.Y.: Differentially private malicious agent avoidance in multiagent advising learning. IEEE Trans. Cybern. 50(10), 4214–4227 (2019)
Zhu, T., Li, G., Zhou, W., Philip, S.Y.: Differentially private data publishing and analysis: a survey. IEEE Trans. Knowl. Data Eng. 29(8), 1619–1638 (2017)
Zhu, T., Philip, S.Y.: Applying differential privacy mechanism in artificial intelligence. In: 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pp. 1601–1609. IEEE (2019)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Abahussein, S., Cheng, Z., Zhu, T., Ye, D., Zhou, W. (2022). Privacy-Preserving in Double Deep-Q-Network with Differential Privacy in Continuous Spaces. In: Long, G., Yu, X., Wang, S. (eds) AI 2021: Advances in Artificial Intelligence. AI 2022. Lecture Notes in Computer Science(), vol 13151. Springer, Cham. https://doi.org/10.1007/978-3-030-97546-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-97546-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-97545-6
Online ISBN: 978-3-030-97546-3
eBook Packages: Computer ScienceComputer Science (R0)