Abstract
DRL (Deep Reinforcement Learning) has been widely used in the field of robotic manipulation. The accuracy of DRL depends on large amounts of data for training. However, training data is distributed among different organizations and is difficult to share due to information security and privacy concerns. In order to solve the security problem of robotic manipulation and improve the accuracy of robotic manipulation, a new privacy-protecting federated reinforcement learning scheme, called PFRL, was proposed. Through PFRL, the robot learns the global reinforcement model safely through the local reinforcement model with the help of the cloud, and well protects the sensitive data of each robot. Specifically, with a secure multi-party vector aggregation algorithm, all local models are encrypted by their owners before being sent to the cloud, and can be directly operated without decryption. And in order to prevent malicious cloud servers or robots to damage the correctness of training, we have integrated non-interactive zero-knowledge proof tools to provide the auditability of collaborative training processes. Detailed security analysis shows that PFRL can resist various known security threats. Moreover, PFRL has been tested in Mujoco simulation environment, and a large number of simulations show that PFRL is feasible.
Supported by National Key Research and Development Program of China (2019YFD1100901), Natural Science Foundation of Shaanxi (Grant No. 2013JM8022).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013)
Deisenroth, M.P., Neumann, G., Peters, J.: A survey on policy search for robotics. Found. Trends Robot. 2(1–2), 1–142 (2013)
Kober, J., Peters, J.R.: Policy search for motor primitives in robotics. Mach. Learn. 84, 171–203 (2009)
McMahan, H.B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pp. 1273–1282. AISTATS (2017)
Son, C., Ristenpart, T., Shmatikov, V.: Machine learning models that remember too much. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 587–601, ACM (2017)
Melis, L., Song, C., De Cristofaro, E., and Shmatikov, V.: Inference attacks against collaborative learning. arXiv preprint arXiv:1805.04049 (2018)
Hitaj, B., Ateniese, G., Perez-Cruz, F.: Deep models under the gan: information leakage from collaborative deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 603–618. ACM (2017)
Shokri, R., Shmatikov, V.: Privacy-preserving deep learning. In: Allerton Conference on Communication, Control, and Computing, pp. 909–910 (2015)
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., Shmatikov, V.: How to backdoor federated learning. arXiv preprint arXiv:1807.00459 (2018)
Schoenmakers, B., Veeningen, M.: Universally verifiable multiparty computation from threshold homomorphic crypto-systems. In: International Conference on Applied Cryptography and Network Security, pp. 3–22. Springer (2015)
Paillier, P.: Public-key crypto-systems based on composite degree residuosity classes. In: Advances in Cryptology - EUROCRYPT 1999, International Conference on the Theory and Application of Cryptographic Techniques, pp. 223–238 (1999)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhu, X., Li, H. (2021). Privacy-Preserving and Auditable Federated Deep Reinforcement Learning for Robotic Manipulation. In: Yang, M., Chen, C., Liu, Y. (eds) Network and System Security. NSS 2021. Lecture Notes in Computer Science(), vol 13041. Springer, Cham. https://doi.org/10.1007/978-3-030-92708-0_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-92708-0_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92707-3
Online ISBN: 978-3-030-92708-0
eBook Packages: Computer ScienceComputer Science (R0)