Abstract
Differential privacy (DP) has been widely used in many domains of statistics and deep learning (DL), such as protecting the parameters of DL models. The framework Private Aggregation of Teacher Ensembles (PATE) is a popular solution for privacy protection that effectively avoids membership inference attacks in model training. However, in Trusted Industrial Data Matrix (TDM) where privacy budgets are constrained and information sharing between models is required, existing works using PATE have two issues. First, the data utility is reduced due to the overfitting problem resulting from insufficient knowledge transfer from teachers to students. Second, teachers cannot share information, thus creating an information silo problem. In this paper, we first proposed the Personalized Voting-based PATE framework (PV-PATE) in TDM to solve the above-mentioned issues. It includes Teacher Credibility that reduces sensitivity by changing voting weights and an Adaptive Voting mechanism based on teachers voting. In addition, we propose a Model Sharing mechanism to achieve model cloning and elimination. We conduct extensive experiments on MNIST dataset and SVHN dataset to demonstrate that our approach achieves not only outstanding learning performance but also provides strong privacy guarantees.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadi, M., et alL.: Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318 (2016)
Bagdasaryan, E., Poursaeed, O., Shmatikov, V.: Differential privacy has disparate impact on model accuracy. Adv. Neural Inform. Process. Syst. 32 (2019)
Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) TCC 2006. LNCS, vol. 3876, pp. 265–284. Springer, Heidelberg (2006). https://doi.org/10.1007/11681878_14
Jordon, J., Yoon, J., van der Schaar, M.: Differentially private bagging: Improved utility and cheaper privacy than subsample-and-aggregate. Adv. Neural Inform. Process. Syst. 32 (2019)
Jorgensen, Z., Yu, T., Cormode, G.: Conservative or liberal? personalized differential privacy. In: 2015 IEEE 31St International Conference on Data Engineering, pp. 1023–1034. IEEE (2015)
Mahmoud, A.A., Shawabkeh, T.A., Salameh, W.A., Al Amro, I.: Performance predicting in hiring process and performance appraisals using machine learning. In: 2019 10th International Conference on Information and Communication Systems (ICICS), pp. 110–115. IEEE (2019)
Mironov, I.: Rényi differential privacy. In: 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263–275. IEEE (2017)
Miyato, T., Maeda, S.i., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Analy. Mach. Intell. 41(8), 1979–1993 (2018)
Otto, B., et al.: Reference architecture model for the industrial data space (2017)
Papernot, N., Abadi, M., Erlingsson, U., Goodfellow, I., Talwar, K.: Semi-supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755 (2016)
Papernot, N., Song, S., Mironov, I., Raghunathan, A., Talwar, K., Erlingsson, Ú.: Scalable private learning with pate. arXiv preprint arXiv:1802.08908 (2018)
Shokri, R., Stronati, M., Song, C., Shmatikov, V.: Membership inference attacks against machine learning models. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 3–18. IEEE (2017)
Song, S., Chaudhuri, K., Sarwate, A.D.: Stochastic gradient descent with differentially private updates. In: 2013 IEEE Global Conference on Signal and Information Processing, pp. 245–248. IEEE (2013)
Tramer, F., Boneh, D.: Differentially private learning needs better features (or much more data). arXiv preprint arXiv:2011.11660 (2020)
Uniyal, A., et al.: Dp-sgd vs pate: which has less disparate impact on model accuracy? arXiv preprint arXiv:2106.12576 (2021)
Wiens, J., Shenoy, E.S.: Machine learning for healthcare: on the verge of a major shift in healthcare epidemiology. Clin. Infect. Dis. 66(1), 149–153 (2018)
Wu, B., et al.: P3sgd: patient privacy preserving sgd for regularizing deep cnns in pathological image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2099–2108 (2019)
Yu, D., Kamath, G., Kulkarni, J., Yin, J., Liu, T.Y., Zhang, H.: Per-instance privacy accounting for differentially private stochastic gradient descent. arXiv preprint arXiv:2206.02617 (2022)
Acknowledgments
This work was supported by the National Key R &D Program of China under Grant No. 2020YFB1710200, and the National Natural Science Foundation of China under Grant No. 62072136.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Hu, H. et al. (2024). PV-PATE: An Improved PATE for Deep Learning with Differential Privacy in Trusted Industrial Data Matrix. In: Song, X., Feng, R., Chen, Y., Li, J., Min, G. (eds) Web and Big Data. APWeb-WAIM 2023. Lecture Notes in Computer Science, vol 14333. Springer, Singapore. https://doi.org/10.1007/978-981-97-2387-4_32
Download citation
DOI: https://doi.org/10.1007/978-981-97-2387-4_32
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-2386-7
Online ISBN: 978-981-97-2387-4
eBook Packages: Computer ScienceComputer Science (R0)