Abstract
Federated learning can complete the neural network model training without uploading users’ private data. However, the deep leakage from gradients (DLG) and the compensatory reconstruction attack (CRA) can reconstruct the training data according to the gradients uploaded by users. We propose an efficient federated convolutional neural network scheme with differential privacy to solve this problem. By adding Gaussian noise to the fully connected layers of the convolutional neural network, the attacker cannot identify the critical gradients that cause privacy leakage. The cumulative privacy loss is tracked using the analytical moments accountant technique. We conduct extensive experiments on the MNIST and CIFAR10 datasets to evaluate our defense algorithm. After selecting appropriate parameters, the results show that our defense algorithm can defend against DLG and CRA while maintaining a high model accuracy.
Supported by the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDC02040400.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
McMahan, H.B., Moore, E., Ramage, D., Hampson, S., et al.: Communication-efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629 (2016)
Degan, E., et al.: Application of federated learning in medical imaging. In: Ludwig, H., Baracaldo, N. (eds.) Federated Learning, pp. 483–497. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-96896-0_22
Hitaj, B., Ateniese, G., Perez-Cruz, F.: Deep models under the GAN: information leakage from collaborative deep learning. In: 24th Proceedings of the Conference on Computer and Communications Security, pp. 603–618. ACM, New York (2017)
Zhu, L., Han, S.: Deep leakage from gradients. In: Yang, Q., Fan, L., Yu, H. (eds.) Federated Learning. LNCS (LNAI), vol. 12500, pp. 17–31. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63076-8_2
Zhao, B., Konda, R.M., Hakan, B.: iDLG: improved deep leakage from gradients. arXiv preprint arXiv:2001.02610 (2020)
Aono, Y., Hayashi, T., Wang, L., et al.: Privacy-preserving deep learning via additively homomorphic encryption. IEEE Trans. Inf. Forensics Secur. 13(5), 1333–1345 (2017)
Zhang, Q., Jing, S., Zhao, C., Zhang, B., Chen, Z.: Efficient federated learning framework based on multi-key homomorphic encryption. In: Barolli, L. (ed.) 3PGCIC 2021. LNNS, vol. 343, pp. 88–105. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-89899-1_10
Bonawitz, K., Ivanov, V., Kreuter, B., et al.: Practical secure aggregation for privacy-preserving machine learning. In: 16th Proceedings of the Conference on Computer and Communications Security, pp. 1175–1191. ACM SIGSAC, Dallas (2017)
Xu, R., Baracaldo, N., Zhou, Y., Anwar, A., Ludwig, H.: HybridAlpha: an efficient approach for privacy-preserving federated learning. In: 12th Proceedings of ACM Workshop on Artificial Intelligence and Security, pp. 13–23. ACM (2019)
Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) TCC 2006. LNCS, vol. 3876, pp. 265–284. Springer, Heidelberg (2006). https://doi.org/10.1007/11681878_14
Zhang, D., Chen, X., Wang, D., et al.: A survey on collaborative deep learning and privacy-preserving. In: 3th International Conference on Data Science in Cyberspace, pp. 652–658. IEEE, Guangzhou (2018)
Long, G., Shen, T., Tan, Y., Gerrard, L., Clarke, A., Jiang, J.: Federated learning for privacy-preserving open innovation future on digital health. In: Chen, F., Zhou, J. (eds.) Humanity Driven AI, pp. 113–133. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-72188-6_6
McMahan, H.B., Moore, E., Ramage, D., et al.: Federated learning of deep networks using model averaging. arXiv preprint arXiv:1602.05629 (2016)
Zhang, S., Zhang, C., You, Z., et al: Asynchronous stochastic gradient descent for DNN training. In: International Conference on Acoustics, Speech and Signal Processing, pp. 6660–6663. IEEE, Florence (2013)
Shokri, R., Shmatikov, V.: Privacy-preserving deep learning. In: 22nd Conference on Computer and Communications Security, pp. 1310–1321. ACM SIGSAC, New York (2015)
Mi, H., et al.: Collaborative deep learning across multiple data centers. Sci. China Inf. Sci. 63(8), 182102 (2020). https://doi.org/10.1007/s11432-019-2705-2
Augenstein, S., et al.: Generative models for effective ml on private, decentralized datasets. arXiv preprint arXiv:1911.06679 (2019)
Huang, X., Ding, Y., Jiang, Z.L., Qi, S., Wang, X., Liao, Q.: DP-FL: a novel differentially private federated learning framework for the unbalanced data. World Wide Web 23(4), 2529–2545 (2020). https://doi.org/10.1007/s11280-020-00780-4
Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., Naor, M.: Our data, ourselves: privacy via distributed noise generation. In: Vaudenay, S. (ed.) EUROCRYPT 2006. LNCS, vol. 4004, pp. 486–503. Springer, Heidelberg (2006). https://doi.org/10.1007/11761679_29
Mironov, I.: Rényi differential privacy. In: 30th Computer Security Foundations Symposium, pp. 263–275. IEEE, Santa Barbara (2017)
Wang, Y.X., Balle, B., Kasiviswanathan, S.-P.: Subsampled Rényi differential privacy and analytical moments accountant. In: 22nd International Conference on Artificial Intelligence and Statistics, pp. 1226–1235. PMLR, Naha (2019)
Sun, J., Li, A., Wang, B., et al.: Soteria: provable defense against privacy leakage in federated learning from representation perspective. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 9311–9319. IEEE, virtual (2021)
Brendan, M.H., Ramage, D., Talwar, K., Zhang, L.: Learning differentially private recurrent language models. In: 6nd International Conference on Learning Representations, Vancouver (2018)
Lecun, Y., Bottou, L.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, Citeseer (2009)
Li, A., Sun, J., Wang, B., et al.: LotteryFL: personalized and communication-efficient federated learning with lottery ticket hypothesis on non-IID datasets. arXiv preprint arXiv:2008.03371 (2020)
Abadi, M., et al.: Deep learning with differential privacy. In: Proceedings of the Conference on Computer and Communications Security, pp. 308–318. ACM SIGSAC, Vienna (2016)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, D., Chen, X., Shi, J. (2022). An Efficient Federated Convolutional Neural Network Scheme with Differential Privacy. In: Chen, J., He, D., Lu, R. (eds) Emerging Information Security and Applications. EISA 2022. Communications in Computer and Information Science, vol 1641. Springer, Cham. https://doi.org/10.1007/978-3-031-23098-1_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-23098-1_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23097-4
Online ISBN: 978-3-031-23098-1
eBook Packages: Computer ScienceComputer Science (R0)