Publication Type
Journal Article
Version
acceptedVersion
Publication Date
12-2020
Abstract
Deep learning (DL) has been widely applied to achieve promising results in many fields, but it still exists various privacy concerns and issues. Applying differential privacy (DP) to DL models is an effective way to ensure privacy-preserving training and classification. In this paper, we revisit the DP stochastic gradient descent (DP-SGD) method, which has been used by several algorithms and systems and achieved good privacy protection. However, several factors, such as the sequence of adding noise, the models used etc., may impact its performance with various degrees. We empirically show that adding noise first and clipping second will not only significantly achieve high accuracy, but also accelerate convergence. Rigorous experiments have been conducted on three different datasets to train two popular DL models, Convolutional Neural Network (CNN) and Long and Short-Term Memory (LSTM). For the CNN, the accuracy rate can be increased by 3%, 8% and 10% on average for the respective datasets, and the loss value is reduced by 18%, 14% and 22% on average. For the LSTM, the accuracy rate can be increased by 18%, 13% and 12% on average, and the loss value can be reduced by 55%, 25% and 23% on average. Meanwhile, we have compared the performance of our proposed method with a state-of-the-art SGD-based technique. The results show that under the premise of a reasonable clipping threshold, the proposed method not only has better performance, but also achieve ideal privacy protection effects. The proposed alternative can be applied to many existing privacy preserving solutions.
Keywords
Differential privacy, Privacy preserving, Deep learning, Stochastic gradient descent (SGD)
Discipline
Databases and Information Systems | Information Security
Research Areas
Data Science and Engineering
Publication
Computers & Security
Volume
99
First Page
1
Last Page
16
ISSN
0167-4048
Identifier
10.1016/j.cose.2020.102061
Publisher
Elsevier
Embargo Period
12-9-2020
Citation
LIN, Ying; BAO, Ling-Yan; LI, Ze-Minghui; SI, Shu-Sheng; and CHU, Chao-Hsien.
Differential privacy protection over deep learning: An investigation of its impacted factors. (2020). Computers & Security. 99, 1-16.
Available at: https://ink.library.smu.edu.sg/sis_research/5402
Copyright Owner and License
Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1016/j.cose.2020.102061