skip to main content
10.1145/3587716.3587815acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicmlcConference Proceedingsconference-collections
research-article

Preserving model privacy for Federated Reinforcement Learning in Complementary Environments

Published: 07 September 2023 Publication History

Abstract

Federated reinforcement learning (FRL) uses data from multiple partners interacting with the environment to train a global decision model while maintaining data privacy. In specific situations, it is necessary to protect not only data privacy but also model privacy. That is, our objective is to collaboratively train a global model that can be used privately and independently by the initiator. We refer to this need for model privacy preservation as model monopolization. It allows the task initiator to ensure the privacy of the model during the training process and to own the trained model alone. To the best of our knowledge, model monopolization protection in Federated Reinforcement Learning has not been studied. We propose monopolized federated reinforcement learning in complementary environments via function transformation. We also provide contribution metrics to ensure the sustainability of monopolized federated reinforcement learning and to provide a quantitative basis for task initiators to give back to task participants. The experimental results show that our method achieved model monopolization by training agents in the Grid-world.

References

[1]
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. 2020. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics. PMLR, 2938–2948.
[2]
Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning attacks against support vector machines. arXiv preprint arXiv:1206.6389 (2012).
[3]
Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, and Qiang Yang. 2019. Secureboost: A lossless federated learning framework. arXiv preprint arXiv:1901.08755 (2019).
[4]
Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. 2018. Stochastic activation pruning for robust adversarial defense. arXiv preprint arXiv:1803.01442 (2018).
[5]
Jacob Dumford and Walter Scheirer. 2018. Backdooring convolutional neural networks via targeted weight perturbations. In 2020 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 1–9.
[6]
Cynthia Dwork, Aaron Roth, 2014. The algorithmic foundations of differential privacy.Foundations and Trends in Theoretical Computer Science 9, 3-4 (2014), 211–407.
[7]
Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thorne. 2017. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv preprint arXiv:1711.10677 (2017).
[8]
Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. 2011. Adversarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence. 43–58.
[9]
Yujie Ji, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang. 2018. Model-reuse attacks on deep learning systems. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. 349–363.
[10]
Qi Jia, Linke Guo, Zhanpeng Jin, and Yuguang Fang. 2018. Preserving model privacy for machine learning in distributed systems. IEEE Transactions on Parallel and Distributed Systems 29, 8 (2018), 1808–1822.
[11]
Jakub Konečnỳ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016).
[12]
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses. Springer, 273–294.
[13]
Saeed Mahloujifar, Mohammad Mahmoody, and Ameer Mohammed. 2018. Multi-party Poisoning through Generalized p -Tampering. arXiv preprint arXiv:1809.03474 (2018).
[14]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics. PMLR, 1273–1282.
[15]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529–533.
[16]
Nishat I Mowla, Nguyen H Tran, Inshil Doh, and Kijoon Chae. 2020. AFRL: Adaptive federated reinforcement learning for intelligent jamming defense in FANET. Journal of Communications and Networks 22, 3 (2020), 244–258.
[17]
Chetan Nadiger, Anil Kumar, and Sherine Abdelhak. 2019. Federated reinforcement learning for fast personalization. In 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE). IEEE, 123–127.
[18]
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 2016. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP). IEEE, 582–597.
[19]
Protection Regulation. 2018. General data protection regulation. Intouch 25 (2018).
[20]
Zexue Wang, Qidong Wan, Yangmei Qin, Senqing Fan, and Zeyi Xiao. 2021. Research on intelligent algorithm for alerting vehicle impact based on multi-agent deep reinforcement learning. Journal of Ambient Intelligence and Humanized Computing 12, 1 (2021), 1337–1347.
[21]
Chengliang Zhang, Junzhe Xia, Baichen Yang, Huancheng Puyang, Wei Wang, Ruichuan Chen, Istemi Ekin Akkus, Paarijaat Aditya, and Feng Yan. 2021. Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGX. arXiv preprint arXiv:2105.01281 (2021).
[22]
Hankz Hankui Zhuo, Wenfeng Feng, Qian Xu, Qiang Yang, and Yufeng Lin. 2019. Federated reinforcement learning. arXiv preprint arXiv:1901.08277 1 (2019).
[23]
Minhui Zou, Yang Shi, Chengliang Wang, Fangyu Li, WenZhan Song, and Yu Wang. 2018. Potrojan: powerful neural-level trojan designs in deep learning models. arXiv preprint arXiv:1802.03043 (2018).

Cited By

View all
  • (2024)Blockchain-Enabled Federated Reinforcement Learning (B-FRL) Model for Privacy Preservation Service in IoT SystemsWireless Personal Communications10.1007/s11277-024-11411-w136:4(2545-2571)Online publication date: 3-Jul-2024

Index Terms

  1. Preserving model privacy for Federated Reinforcement Learning in Complementary Environments

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Other conferences
      ICMLC '23: Proceedings of the 2023 15th International Conference on Machine Learning and Computing
      February 2023
      619 pages
      ISBN:9781450398411
      DOI:10.1145/3587716
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 07 September 2023

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. federated reinforcement learning
      2. model monopolization
      3. model privacy

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      ICMLC 2023

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)22
      • Downloads (Last 6 weeks)2
      Reflects downloads up to 20 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Blockchain-Enabled Federated Reinforcement Learning (B-FRL) Model for Privacy Preservation Service in IoT SystemsWireless Personal Communications10.1007/s11277-024-11411-w136:4(2545-2571)Online publication date: 3-Jul-2024

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media