Privacy-Preserving Stochastic Gradient Descent with Multiple Distributed Trainers

Abstract

Assume that there are L local datasets distributed among L owners (also called trainers hereafter). The problem is as follows: the owners wish to apply a machine learning method over the combined dataset of all to obtain the best possible learning output; but do not want to publicly share the local datasets due to privacy concerns. In this paper we design a system solving the problem in which stochastic gradient descent (SGD) algorithm is used as the machine learning method, as SGD is at the heart of recent deep learning techniques. Our system differs from existing work by following features: (1) we do not share the gradients in SGD but share the weight parameters; and (2) we use symmetric encryption to protect the weight parameters against an honest-but-curious server used as a common place for storage. Therefore, we are able to avoid information leakage of local data to the server; and the efficiency of our system is kept reasonably compared to the original SGD over the combined dataset. Finally, we experiment over a real dataset to verify the practicality of our system.