Abstract
Deep learning models are often incompetent to privacy attacks, resulting in the leakage of private data. Recently, Differentially-Private Stochastic Gradient Descent (DP-SGD) has emerged as a prime method for training deep learning models with rigorous privacy guarantee, and has been widely adopted in both academic and industrial research. However, using the DP-SGD optimizer will make the model converge slower and worse, so improving the utility of the model while maintaining privacy becomes a challenge. In non-private training, setting momentum to the SGD optimizer is a common method to improve the utility of the model, but the performance of this method in DP-SGD optimizer is not yet known. In this paper, we empirically study the impact of momentum setting on the optimization of DP-SGD models. With extensive experiments, we were able to gain some fresh insights and proposed a method to dynamically set the momentum for DP-SGD to achieve better utility. The results showd that we achieved the new state-of-the-art on MNIST, Fashion-MNIST, CIFAR-10 and Imagenette datasets without any modification of differential-privacy analysis.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Settings for the Experiments in This Section Can Be Found in Section 5.
- 2.
Opacus is an open source library provided by Facebook that implements DP-SGD in the Pytorch framework.
References
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)
Zheng, W., Yan, L., Gou, C., Wang, F.-Y.: Fighting fire with fire: a spatial-frequency ensemble relation network with generative adversarial learning for adversarial image classification. In: Int. J. Intell. Syst. 36(5), 2081–2121 (2021)
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., Khudanpur, S.: Recurrent neural network based language model. In: Proceedings of Interspeech (2010)
Deng, L., Hinton, G., Kingsbury, B.: New types of deep neural network learning for speech recognition and related applications: an overview. In: Proceedings of ICASSP’13 (2013)
Shokri, R., Stronati, M., Song, C., Shmatikov, V.: Membership inference attacks against machine learning models. In: Security and Privacy (SP), 2017 IEEE Symposium on IEEE, pp. 3–18 (2017)
Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., Backes, M.: Ml-leaks: model and data independent membership inference attacks and defenses on machine learning models. In: arXiv preprint arXiv:1806.01246, 2018
Zhu, L., Liu, Z., Han, S. : Deep leakage from gradients. In: Advances in Neural Information Processing Systems, pp. 14 747–14 756 (2019)
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., Zhang, L.: Deep learning with differential privacy. In: Conference on Computer and Communications Security (2016)
Bertino, E., Shulman, H., Waidner, M. (eds.): ESORICS 2021. LNCS, vol. 12973. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88428-4
Tramer, F., Boneh, D.: Differentially private learning needs better features (or much more data). In: International Conference on Learning Representations (2021)
Pichapati, V., Suresh, A. T., Yu, F. X., Reddi, S. J., Kumar, S.: Adaclip: adaptive clipping for private SGD. arXiv preprint arXiv:1908.07643 (2019)
Andrew, G., Thakkar, O., McMahan, H. B., Ramaswamy, S.: Differentially private learning with adaptive clipping. In: NeurIPS (2021)
Gong, M., Pan, K., Xie, Y., Qin, A. K., Tang, Z.: Preserving differential privacy in deep neural networks with relevance-based adaptive noise imposition. Neural Netw. 125, 131–141 (2020)
Xu, Z., Shi, S., Liu, A. X., Zhao, J., Chen, L.: An adaptive and fast convergent approach to differentially private deep learning. In: IEEE INFOCOM 2020-IEEE Conference on Computer Communications, pp. 1867–1876 IEEE (2020)
Papernot, N., Thakurta, A., Song, S., Chien, S., Erlingsson, U.: Tempered sigmoid activations for deep learning with differential privacy. arXiv preprint arXiv:2007.14191 (2020)
Leino, K., Fredrikson, M.: Stolen memories: leveraging model memorization for calibrated white-box membership inference. In: 29th USENIX Security Symposium (USENIX Security 20), pp. 1605–1622 (2020)
Jayaraman, B., Evans, D.: Evaluating differentially private machine learning in practice. In: 28th USENIX Security Symposium (USENIX Security 19), pp. 1895–1912 (2019)
van der Veen, K. L., Seggers, R., Bloem, P., Patrini, G.: Three tools for practical differential privacy. In: NeurIPS 2018 Workshop (2018)
Yu, D., Zhang, H., Chen, W.: Improve the gradient perturbation approach for differentially private optimization. In: NeurIPS 2018 Workshop (2018)
Du, J., Li, S., Feng, M., Chen, S.: Dynamic differential-privacy preserving SGD. arXiv preprint arXiv:2111.00173 (2021)
Zhang, X., Ding, J., Wu, M., Wong, STC.: Adaptive privacy preserving deep learning algorithms for medical data. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2021)
Chen, J., Wolfe, C., Li, Z., Kyrillidis, A.: Demon: improved neural network training with momentum decay. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 3958–3962 (2022)
Bai, J., Ren, Y., Zhang, J.: Adaptive momentum with discriminative weight for neural network stochastic optimization. Int. J. Intell. Syst. 37, 6531-6554 (2022)
Keskar, N. S., Socher, R.: Improving generalization performance by switching from Adam to SGD. arXiv preprint arXiv:1712.07628 (2017)
Liu, Y., Gao Y., Yin, W.: An improved analysis of stochastic gradient descent with momentum. arXiv preprint arXiv:2007.07989
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
DP-SGD
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lin, G. et al. (2023). Dynamic Momentum for Deep Learning with Differential Privacy. In: Xu, Y., Yan, H., Teng, H., Cai, J., Li, J. (eds) Machine Learning for Cyber Security. ML4CS 2022. Lecture Notes in Computer Science, vol 13656. Springer, Cham. https://doi.org/10.1007/978-3-031-20099-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-20099-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20098-4
Online ISBN: 978-3-031-20099-1
eBook Packages: Computer ScienceComputer Science (R0)