Privacy-Preserving Distributed Machine Learning Based on Secret Sharing

Abstract

Machine Learning has been widely applied in practice, such as disease diagnosis, target detection. Commonly, a good model relies on massive training data collected from different sources. However, the collected data might expose sensitive information. To solve the problem, researchers have proposed many excellent methods that combine machine learning with privacy protection technologies, such as secure multiparty computation (MPC), homomorphic encryption (HE), and differential privacy. In the meanwhile, some other researchers proposed distributed machine learning which allows the clients to store their data locally but train a model collaboratively. The first kind of methods focuses on security, but the performance and accuracy remain to be improved, while the second provides higher accuracy and better performance but weaker security, for instance, the adversary can launch membership attacks from the gradients’ updates in plaintext.

In this paper, we join secret sharing to distributed machine learning to achieve reliable performance, accuracy, and high-level security. Next, we design, implement, and evaluate a practical system to jointly learn an accurate model under semi-honest and servers-only malicious adversary security, respectively. And the experiments show our protocols achieve the best overall performance as well.

Appendices

A Proof of Correctness

1.1 A.1 Lemma 1

Proof

Suppose we have two secrets, $s_0$ and $s_1$, and we share both in Shamir’s Secret Sharing scheme with two polynomial-functions

$$\begin{aligned} \begin{aligned} f(x)=a_0 + a_1\cdot x +...+a_{t-1}\cdot x^{t-1}\mod p\\ g(x)=b_0 + b_1\cdot x +...+b_{t-1}\cdot x^{t-1}\mod p \end{aligned} \end{aligned}$$

(12)

where $f(0)=a_0 = s_0$, $g(0)=b_0=s_1$ and p is a large prime.

In order to compute the shares, we can evaluate f(x) and g(x) at n different points $f(x_0),f(x_1),...,f(x_{n-1})$ and $g(x_0),g(x_1),...,g(x_{n-1})$ respectively.

Then we will turn to getting the shares of $s_0 + s_1$. We define a new polynomial-function

$$\begin{aligned} \begin{aligned} h(x) = (a_0+b_0) + (a_1+b_1)\cdot x +... +(a_{t-1}+b_{t-1})\cdot x^{t-1}\mod p \end{aligned} \end{aligned}$$

(13)

Obviously, h(x) is a polynomial-function of degree $t-1$ with t coefficients and $h(0) = s_0+ s_1$. On the one hand, $h(x_i)$ is the shares for $s_0+s_1$, and on the other hand, we can confirm

$$\begin{aligned} \begin{aligned} h(x_i)&= (a_0+b_0) + (a_1+b_1)\cdot x_i +...+ (a_{t-1}+b_{t-1})\cdot x_i^{t-1} \\ {}&=(a_0 + a_1\cdot x_i\,+...+\,a_{t-1}\cdot x_i^{t-1}) + (b_0 + b_1\cdot x_i\,+... +\,b_{t-1}\cdot x_i^{t-1})\\ {}&= f(x_i) + g(x_i) \mod p,\ 0\le i\le n-1 \end{aligned} \end{aligned}$$

(14)

So that the shares of $s_0+s_1$ can be computed by adding the corresponding shares of $s_0$ and $s_1$.

1.2 A.2 Lemma2

Proof

Suppose we have $x_0,x_1,...,x_{n-1}$ and a secret key $\alpha$. We could the compute the MAC of $x_i$

$$\begin{aligned} \begin{aligned} \varvec{\delta }(x_i) = \alpha \cdot x_i\mod p,\ 0\le i\le n-1 \end{aligned} \end{aligned}$$

(15)

Then we can also compute the MAC of $x_i$’s sum:

$$\begin{aligned} \begin{aligned} \varvec{\delta }(\sum \limits _{i=0}^{n-1} x_i) = \alpha \cdot (\sum \limits _{i=0}^{n-1} x_i)\mod p \end{aligned} \end{aligned}$$

(16)

Then it is easy to confirm

$$\begin{aligned} \begin{aligned} \varvec{\delta }(\sum \limits _{i=0}^{n-1} x_i)&= (\alpha \cdot x_0)+(\alpha \cdot x_1)+...+(\alpha \cdot x_{n-1}) \mod p\\&= \varvec{\delta }(x_0) +\varvec{\delta }(x_1) + ... + \varvec{\delta }(x_{n-1}) \mod p\\&= \sum \limits _{i=0}^{n-1} \varvec{\delta }(x_i) \mod p \end{aligned} \end{aligned}$$

(17)

For a more concrete proof, please refer to [12].

B Accuracy and Performance for Linear Regression and MLP

1.1 B.1 Linear Regression

See Fig. 6.

Fig. 6.

Experimental results of linear regression. (a) is for accuracy, (b) is for performance.

1.2 B.2 MLP

See Fig. 7.

Fig. 7.

Experimental results of MLP. (a) is for accuracy, (b) is for performance.