skip to main content
research-article

Intrinsic Performance Influence-based Participant Contribution Estimation for Horizontal Federated Learning

Published: 22 September 2022 Publication History

Abstract

The rapid development of modern artificial intelligence technique is mainly attributed to sufficient and high-quality data. However, in the data collection, personal privacy is at risk of being leaked. This issue can be addressed by federated learning, which is proposed to achieve efficient model training among multiple data providers without direct data access and aggregation. To encourage more parties owning high-quality data to participate in the federated learning, it is important to evaluate and reward the participant contribution in a reasonable, robust, and efficient manner. To achieve this goal, we propose a novel contribution estimation method: Intrinsic Performance Influence-based Contribution Estimation (IPICE). In particular, the class-level intrinsic performance influence is adopted as the contribution estimation criteria in IPICE, and a neural network is employed to exploit the non-linear relationship between the performance change and estimated contribution. Extensive experiments are conducted on various datasets, and the results demonstrate that IPICE is more accurate and stable than the counterpart in various data distribution settings. The computational complexity is significantly reduced in our IPICE, especially when a new party joins the federation. IPICE assigns small contributions to bad/garbage data and thus prevent them from participating and deteriorating the learning ecosystem.

References

[1]
Vito Walter Anelli, Yashar Deldjoo, Tommaso Di Noia, and Antonio Ferrara. 2019. Towards effective device-aware federated learning. InProceedings of the International Conference of the Italian Association for Artificial Intelligence. 477–491.
[2]
Sebastian Caldas, Jakub Konecný, H. Brendan McMahan, and Ameet Talwalkar. 2018. Expanding the reach of federated learning by reducing client resource requirements. arXiv:1812.07210. Retrieved from https://arxiv.org/abs/1812.07210.
[3]
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečnỳ, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. 2018. Leaf: A benchmark for federated settings. arXiv:1812.01097. Retrieved from https://arxiv.org/abs/1812.01097.
[4]
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. 2017. EMNIST: an extension of MNIST to handwritten letters. arXiv:1702.05373. Retrieved from https://arxiv.org/abs/1702.05373.
[5]
R. Dennis Cook. 1977. Detection of influential observation in linear regression. Technometrics (1977).
[6]
Boi Faltings, Goran Radanovic, Ronald Brachman, and Peter Stone. 2017. Game Theory for Data Science: Eliciting Truthful Information.
[7]
Dashan Gao, Ce Ju, Xiguang Wei, Yang Liu, Tianjian Chen, and Qiang Yang. 2019. HHHFL: Hierarchical heterogeneous horizontal federated learning for electroencephalography. arXiv:1909.05784. Retrieved from https://arxiv.org/abs/1909.05784.
[8]
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. 2018. Federated learning for mobile keyboard prediction. arXiv:1811.03604. Retrieved from https://arxiv.org/abs/1811.03604.
[9]
Florian Hartmann, Sunah Suh, Arkadiusz Komarzewski, Tim D. Smith, and Ilana Segall. 2019. Federated learning for ranking browser history suggestions. arXiv:1911.11807. Retrieved from https://arxiv.org/abs/1911.11807.
[10]
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. 2015. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 595–604.
[11]
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the effects of non-identical data distribution for federated visual classification. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’19).
[12]
Di Jiang, Yuanfeng Song, Yongxin Tong, Xueyang Wu, Weiwei Zhao, Qian Xu, and Qiang Yang. 2019. Federated topic modeling. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1071–1080.
[13]
Yihan Jiang, Jakub Konecný, Keith Rush, and Sreeram Kannan. 2019. Improving federated learning personalization via model agnostic meta learning. InProceedings of the Annual Conference on Neural Information Processing Systems (NIPS’19).
[14]
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, et al. 2021. Advances and open problems in federated learning. Found. Trends Mach. Learn. 14, 1 (2021).
[15]
Jiawen Kang, Zehui Xiong, Dusit Niyato, Shengli Xie, and Junshan Zhang. 2019. Incentive mechanism for reliable federated learning: A joint optimization approach to combining reputation and contract theory. IEEE IoT J. 6, 6 (2019), 10700–10714.
[16]
Jiawen Kang, Zehui Xiong, Dusit Niyato, Han Yu, Ying-Chang Liang, and Dong In Kim. 2019. Incentive design for efficient federated learning in Mobile networks: A contract theory approach. In Proceedings of the IEEE VTS Asia Pacific Wireless Communications Symposium (APWCS’19). 1–5.
[17]
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Ananda Theertha Suresh, Dave Bacon, and Peter Richtárik. 2018. Federated learning: Strategies for improving communication efficiency. arXiv:1610.05492. Retrieved from https://arxiv.org/abs/1610.05492.
[18]
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3D object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 554–561.
[19]
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. 2020. Fair resource allocation in federated learning. In Proceedings of the International Conference on Learning Representations (ICLR’20).
[20]
Xinchen Liu, Wu Liu, Tao Mei, and Huadong Ma. 2018. PROVID: Progressive and multimodal vehicle reidentification for large-scale urban surveillance. IEEE Trans. Multimedia 20, 3 (2018), 645–658.
[21]
Yang Liu, Tianjian Chen, and Qiang Yang. 2018. Secure federated transfer learning. arXiv:1812.03337. Retrieved from https://arxiv.org/abs/1812.03337.
[22]
Songtao Lu, Yawen Zhang, Yunlong Wang, and Christina Mack. 2019. Learn electronic health records by fully decentralized federated learning.arXiv:1912.01792. Retrieved from https://arxiv.org/abs/1912.01792.
[23]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. 1273–1282.
[24]
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. 2016. Communication-efficient learning of deep networks from decentralized data. arXiv:1602.05629. Retrieved from https://arxiv.org/abs/1602.05629.
[25]
M.-E. Nilsback and A. Zisserman. 2008. Automated flower classification over a large number of classes. In Proceedings of the 6th Indian Conference on Computer Vision, Graphics & Image Processing. 722–729.
[26]
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. 2020. Federated adversarial domain adaptation. In Proceedings of the 8th International Conference on Learning Representations(ICLR’20).
[27]
Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. 2016. Performance measures and a data set for multi-target, multi-camera tracking. In Proceedings of the European Conference on Computer Vision Workshop on Benchmarking Multi-Target Tracking.
[28]
Lloyd S. Shapley. 1953. A value for n-person games.Ann. Math. Stud. 28 (1953), 307–317.
[29]
Yexuan Shi, Yongxin Tong, Zhiyang Su, Di Jiang, Zimu Zhou, and Wenbin Zhang. 2020. Federated topic discovery: A semantic consistent approach. Missouri Rev. (2020), 1–1.
[30]
Hao Luo, Wei Jiang, Youzhi Gu, Fuxu Liu, Xingyu Liao, Shenqi Lai, and Jianyang Gu. 2019. A strong baseline and batch normalization neck for deep person re-identification. IEEE Trans. Multimedia 22, 10 (2019), 2597–2609.
[31]
Tianshu Song, Yongxin Tong, and Shuyue Wei. 2019. Profit allocation for federated learning. In Proceedings of the IEEE International Conference on Big Data (Big Data’19). 2577–2586.
[32]
Gan Sun, Yang Cong, Jiahua Dong, Qiang Wang, and Ji Liu. 2020. Data poisoning attacks on federated machine learning. arXiv:2004.10020. Retrieved from https://arxiv.org/abs/2004.10020.
[33]
Ananda Theertha Suresh, Brendan McMahan, Peter Kairouz, and Ziteng Sun. 2019. Can you really backdoor federated learning. arXiv:1911.07963. Retrieved from https://arxiv.org/abs/1911.07963.
[34]
Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. 2020. Data poisoning attacks against federated learning systems. In Proceedings of the European Symposium on Research in Computer Security. Springer, 480–501.
[35]
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011. The caltech-ucsd birds-200-2011 dataset. California Institute of Technology.
[36]
Guan Wang. 2019. Interpret federated learning with shapley values.arXiv preprint arXiv:1905.04519. Retrieved from https://arxiv.org/abs/1905.04519.
[37]
Guan Wang, Charlie Xiaoqian Dang, and Ziye Zhou. 2019. Measure contribution of participants in federated learning. In Proceedings of the IEEE International Conference on Big Data (Big Data’19). 2597–2604.
[38]
Yansheng Wang, Yongxin Tong, and Dingyuan Shi. 2020. Federated latent dirichlet allocation: A local differential privacy based framework. Proc. AAAI Conf. Artif. Intell. 34, 4 (2020), 6283–6290.
[39]
Shuyue Wei, Yongxin Tong, Zimu Zhou, and Tianshu Song. 2020. Efficient and fair data valuation for horizontal federated learning. InFederated Learning, 139–152.
[40]
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. 2020. DBA: Distributed backdoor attacks against federated learning. In Proceedings of the International Conference on Learning Representations (ICLR’20).
[41]
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol. 10, 2 (2019), 1–19.
[42]
Xin Yao, Tianchi Huang, Rui-Xiao Zhang, Ruiyu Li, and Lifeng Sun. 2019. Federated learning with unbiased gradient aggregation and controllable meta updating.Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’19).
[43]
Zhedong Zheng, Liang Zheng, and Yi Yang. 2017. Unlabeled samples generated by GAN improve the person re-identification baseline in vitro. In Proceedings of the IEEE International Conference on Computer Vision.
[44]
Zhaohua Zheng, Yize Zhou, Yilong Sun, Zhang Wang, Boyi Liu, and Keqiu Li. 2021. Applications of federated learning in smart cities: Recent advances, taxonomy, and open challenges. Connect. Sci. (2021), 1–28.
[45]
Hankz Hankui Zhuo, Wenfeng Feng, Qian Xu, Qiang Yang, and Yufeng Lin. 2019. Federated reinforcement learning. arXiv:1901.08277. Retrieved from https://arxiv.org/abs/1901.08277.

Cited By

View all
  • (2024)Defending Federated Recommender Systems Against Untargeted Attacks: A Contribution-Aware Robust Aggregation SchemeACM Transactions on Knowledge Discovery from Data10.1145/3706112Online publication date: 28-Nov-2024
  • (2024)A Survey of Trustworthy Federated Learning: Issues, Solutions, and ChallengesACM Transactions on Intelligent Systems and Technology10.1145/367818115:6(1-47)Online publication date: 23-Jul-2024
  • (2024)Privacy-Preserving Federated Interpretability2024 IEEE International Conference on Big Data (BigData)10.1109/BigData62323.2024.10825590(7592-7601)Online publication date: 15-Dec-2024
  • Show More Cited By

Index Terms

  1. Intrinsic Performance Influence-based Participant Contribution Estimation for Horizontal Federated Learning

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Intelligent Systems and Technology
      ACM Transactions on Intelligent Systems and Technology  Volume 13, Issue 6
      December 2022
      468 pages
      ISSN:2157-6904
      EISSN:2157-6912
      DOI:10.1145/3560231
      • Editor:
      • Huan Liu
      Issue’s Table of Contents

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 22 September 2022
      Online AM: 22 March 2022
      Accepted: 28 February 2022
      Revised: 17 October 2021
      Received: 03 September 2020
      Published in TIST Volume 13, Issue 6

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Federated learning
      2. participant contribution estimation
      3. neural network

      Qualifiers

      • Research-article
      • Refereed

      Funding Sources

      • National Natural Science Foundation of China
      • PKU-NTU Joint Research Institute (JRI) sponsored by a donation from the Ng Teng Fong Charitable Foundation

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)123
      • Downloads (Last 6 weeks)8
      Reflects downloads up to 18 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Defending Federated Recommender Systems Against Untargeted Attacks: A Contribution-Aware Robust Aggregation SchemeACM Transactions on Knowledge Discovery from Data10.1145/3706112Online publication date: 28-Nov-2024
      • (2024)A Survey of Trustworthy Federated Learning: Issues, Solutions, and ChallengesACM Transactions on Intelligent Systems and Technology10.1145/367818115:6(1-47)Online publication date: 23-Jul-2024
      • (2024)Privacy-Preserving Federated Interpretability2024 IEEE International Conference on Big Data (BigData)10.1109/BigData62323.2024.10825590(7592-7601)Online publication date: 15-Dec-2024
      • (2023)Meta-Learning-Based Spatial-Temporal Adaption for Coldstart Air Pollution PredictionInternational Journal of Intelligent Systems10.1155/2023/37345572023Online publication date: 1-Jan-2023
      • (2023)Recent Trends in Task and Motion Planning for Robotics: A SurveyACM Computing Surveys10.1145/358313655:13s(1-36)Online publication date: 13-Jul-2023
      • (2023)Extended Research on the Security of Visual Reasoning CAPTCHAIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2023.323840820:6(4976-4992)Online publication date: 20-Jan-2023
      • (2023)PPCE: Privacy-Preserving Contribution Evaluation for Fairness-Aware Federated Learning2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00077(474-480)Online publication date: 17-Dec-2023
      • (2023)HyperFed: Free-riding Resistant Federated Learning with Performance-based Reputation Mechanism and Adaptive Aggregation using Hypernetworks2023 10th International Conference on Dependable Systems and Their Applications (DSA)10.1109/DSA59317.2023.00025(126-134)Online publication date: 10-Aug-2023
      • (2022)Residual stacked gated recurrent unit with encoder–decoder architecture and an attention mechanism for temporal traffic predictionSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-022-07230-526:17(8617-8633)Online publication date: 1-Sep-2022

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      Full Text

      HTML Format

      View this article in HTML Format.

      HTML Format

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media