Stochastic ADMM Based Distributed Machine Learning with Differential Privacy

Abstract

While embracing various machine learning techniques to make effective decisions in the big data era, preserving the privacy of sensitive data poses significant challenges. In this paper, we develop a privacy-preserving distributed machine learning algorithm to address this issue. Given the assumption that each data provider owns a dataset with different sample size, our goal is to learn a common classifier over the union of all the local datasets in a distributed way without leaking any sensitive information of the data samples. Such an algorithm needs to jointly consider efficient distributed learning and effective privacy preservation. In the proposed algorithm, we extend stochastic alternating direction method of multipliers (ADMM) in a distributed setting to do distributed learning. For preserving privacy during the iterative process, we combine differential privacy and stochastic ADMM together. In particular, we propose a novel stochastic ADMM based privacy-preserving distributed machine learning (PS-ADMM) algorithm by perturbing the updating gradients, that provide differential privacy guarantee and have a low computational cost. We theoretically demonstrate the convergence rate and utility bound of our proposed PS-ADMM under strongly convex objective. Through our experiments performed on real-world datasets, we show that PS-ADMM outperforms other differentially private ADMM algorithms under the same differential privacy guarantee.

A Appendix

The approximate gradient $\varvec{g}_i^s$ can be written as $\varvec{g}_i^s = \varvec{b}_i^s + \varvec{q}_i^s$, where

1.1 A.1 Proof of Lemma 1

Proof

Since each $l_{im }(\varvec{x})$ is convex, G-Lipschitz and has $L_m$-Lipschitz continuous gradient, for any $\varvec{x}_1$ and $\varvec{x}_2,$ there exists $L_m > 0$ such that

$$\begin{aligned} l_{im}(\varvec{x}_1) \le l_{im}(\varvec{x}_2) + (\varvec{x}_2 - \varvec{x}_1)^T\nabla l_{im}(\varvec{x}_2) + \frac{L_m}{2} \Vert \varvec{x}_2 - \varvec{x}_1\Vert ^2. \end{aligned}$$

We can see that $f_i(\varvec{x})$ is $v _{f}$-smooth, with $f_i(\varvec{x}_1) \le f_i(\varvec{x}_2) +(\varvec{x}_2 - \varvec{x}_1)^T\nabla f_i(\varvec{x}_1) + \frac{v_{f }}{2} \Vert \varvec{x}_2 - \varvec{x}_1\Vert ^2$, where $v _{f} = \max _{m}L_m .$ Then, we can have

$$\begin{aligned}&\Vert \nabla L_i(\varvec{x}_1) -\nabla L_i(\varvec{x}_2) \Vert = \Vert \nabla f_i(\varvec{x}_1) - \nabla f_i(\varvec{x}_2) + \rho (\varvec{x}_1 - \varvec{x}_2)\Vert \\&\le \Vert \nabla f_i(\varvec{x}_1) - \nabla f_i(\varvec{x}_2) \Vert + \Vert \rho (\varvec{x}_1 - \varvec{x}_2)\Vert \le (v_{f }+ \rho )\Vert \varvec{x}_1 -\varvec{x}_2\Vert \le v_{L }\Vert \varvec{x}_1 -\varvec{x}_2\Vert , \end{aligned}$$

where we let $v_{L } \ge v_{f }+ \rho$. Thus, $L_i(\varvec{x})$ and $\hat{L}_m(\varvec{x})$ are $v_{L }$-smooth. Moreover, it is obvious to see that $L_i(\varvec{x})$ is $\mu _L$-strongly convex with $\mu _L \le \mu _f+\rho$.

1.2 A.2 Basic Lemmas

Lemma 2

The variance of $\varvec{g}_i^s$ satisfies

Proof

Notice that

Hence, the variance of $\varvec{g}_i^s$ can be bounded as

where the first inequality uses $\Vert a + b\Vert ^2 \le 2\Vert a\Vert ^2 + 2 \Vert b\Vert ^2$ and the second inequality uses $\mathbb {E}\Vert \varvec{x}_i - \mathbb {E}\varvec{x}_i\Vert ^2 = \mathbb {E}\Vert \varvec{x}_i\Vert ^2 - \Vert \mathbb {E}\varvec{x}_i\Vert ^2 \le \mathbb {E}\Vert \varvec{x}_i\Vert ^2$.

Lemma 3

For $0<\eta < \frac{1}{2 v_{L}}$, we have

Proof

$$\begin{aligned}&L_i(\varvec{v}_{i}^{s+1})\le L_i(\varvec{v}_i^s) + (\varvec{v}_i^{s+1}-\varvec{v}_i^s)^T\nabla L_i(\varvec{v}_i^s) + \frac{v_{L }}{2}\Vert \varvec{v}_i^{s+1}-\varvec{v}_i^s\Vert ^2. \end{aligned}$$

Taking expectation on both sides, we obtain

Then, we have

By choosing $\eta < 1/{(2v_{L })}$, we get

Lemma 4

Proof

We have $\mathbb {E}(\varvec{b}_i^s) = \nabla f_i(\varvec{v}_i^s)$ and this leads to

$$\begin{aligned} \mathbb {E} \Vert \varvec{v}_i^{s+1} - \varvec{x}_i\Vert ^2&\le \Vert \varvec{v}_i^s - \varvec{x}_i\Vert ^2 - 2\eta (\varvec{v}_i^s -\varvec{x}_i)^T \mathbb {E}(\varvec{g}_i^s) + \eta ^2 \mathbb {E}(\Vert \varvec{g}_i^s \Vert ^2) \\&\le \Vert \varvec{v}_i^s - \varvec{x}_i\Vert ^2 - 2\eta (\varvec{v}_i^s -\varvec{x}_i)^T \mathbb {E}(\nabla f_i(\varvec{v}_i^s) + \varvec{q}_i^s) + \eta ^2 \mathbb {E}(\Vert \varvec{g}_i^s \Vert ^2) \\&\le \Vert \varvec{v}_i^s - \varvec{x}_i\Vert ^2 - 2\eta (\varvec{v}_i^s -\varvec{x}_i)^T\nabla L_i (\varvec{v}_i^s) + \eta ^2 \mathbb {E}(\Vert \varvec{g}_i^s \Vert ^2). \end{aligned}$$

Then, we have

According to Lemma 3, we obtain

Lemma 5

$$\begin{aligned}&g(\varvec{z}^{k+1}) - g(\varvec{z})- \sum _{i = 1}^N (\varvec{z}^{k+1} - \varvec{z})^T\varvec{\alpha }_i^{k+1} \\&\le \frac{\rho }{2}( \Vert \varvec{z}^{k } - \varvec{z}\Vert ^2-\Vert \varvec{z}^{k+1} - \varvec{z}^k\Vert ^2-\Vert \varvec{z}^{k+1} - \varvec{z}\Vert ^2) \end{aligned}$$

where $\varvec{\alpha }_i^{k+1} = \varvec{\lambda }_i^k +\rho (\varvec{x}_i^{k+1}-\varvec{z}^k).$

Proof

By deriving the optimal conditions of the minimization problem in (5), we have

$$\begin{aligned} g(\varvec{z}^{k+1}) - g(\varvec{z}) \le -(\varvec{z}^{k+1}-\varvec{z})^T\sum _{i= 1}^N[-\varvec{\lambda }_i^{k}+ \rho (\varvec{x}_i^{k+1} -\varvec{z}^{k+1}) ]. \end{aligned}$$

Then, by using the notation $\varvec{\alpha }_i^{{ k+1 }} = \varvec{\lambda }_i^k +\rho (\varvec{x}_i^{k+1}-\varvec{z}^k)$, we obtain

$$\begin{aligned}&g(\varvec{z}^{k+1}) - g(\varvec{z})- \sum _{i = 1}^N (\varvec{z}^{k+1} - \varvec{z})^T\varvec{\alpha }_i^{k+1} \le \rho (\varvec{z}^{k}-\varvec{z}^{k+1})^T (\varvec{z}^{k+1} -\varvec{z}) )\\&\le \frac{\rho }{2} (\Vert \varvec{z}^{k } - \varvec{z}\Vert ^2-\Vert \varvec{z}^{k+1} - \varvec{z}^k\Vert ^2-\Vert \varvec{z}^{k+1} - \varvec{z}\Vert ^2). \end{aligned}$$

Lemma 6

$$\begin{aligned}&(\varvec{\alpha }_i^{k+1} - \varvec{\alpha }_i)^T[-(\varvec{x}_i^{k+1} - \varvec{z}^{k+1})] \\&\le \frac{1}{2\rho }( \Vert \varvec{\lambda }_i^k-\varvec{\alpha }_i \Vert ^2 - \Vert \varvec{\lambda }_i^{k+1}-\varvec{\alpha }_i \Vert ^2) + \frac{\rho }{2} \Vert \varvec{z}^k - \varvec{z}^{k+1}\Vert ^2 \end{aligned}$$

where $\varvec{\alpha }_i^{k+1} = \varvec{\lambda }_i^k +\rho (\varvec{x}_i^{k+1}-\varvec{z}^k).$

Proof

$$\begin{aligned}&(\varvec{\alpha }_i^{k+1} - \varvec{\alpha }_i )^T[-(\varvec{x}_i^{k+1} - \varvec{z}^{k+1})] =\frac{1}{\rho }(\varvec{\alpha }_i^{k+1} - \varvec{\alpha }_i )^T(\varvec{\lambda }_i^k - \varvec{\lambda }_i^{k+1})\\&= \frac{1}{2\rho }(\Vert \varvec{\alpha }_i^{k+1}-\varvec{\lambda }_i^{k+1}\Vert ^2 - \Vert \varvec{\alpha }_i^{k+1}-\varvec{\lambda }_i^{k}\Vert ^2+ \Vert \varvec{\lambda }_i^k-\varvec{\alpha }_i \Vert ^2 - \Vert \varvec{\lambda }_i^{k+1}-\varvec{\alpha }_i \Vert ^2)\\&\le \frac{1}{2\rho }( \Vert \varvec{\lambda }_i^k-\varvec{\alpha }_i \Vert ^2 - \Vert \varvec{\lambda }_i^{k+1}-\varvec{\alpha }_i \Vert ^2) + \frac{\rho }{2} \Vert \varvec{z}^k - \varvec{z}^{k+1}\Vert ^2. \end{aligned}$$

Lemma 7

Assume $f_i(\cdot )$ be $\mu _f$-strongly convex, and let $\varvec{x}_i^{k+1}$, $\varvec{z}^k$ and $\varvec{\lambda }_i^k$ be generated by the proposed algorithm. For $\eta$ satisfies , $1-\frac{\rho \xi }{2}-\frac{\mu _f\xi }{4}+\frac{4\eta ^2 v_L^2 S}{1-2\eta v_L} \le \frac{S \eta \mu _f}{2}$, the following holds if

$$\begin{aligned} \mathbb {E}[f_i(\varvec{x}_i^{k+1}) - f_i(\varvec{x}_i)+ (\varvec{x}_i^{k+1} - \varvec{x}_i)^T\varvec{\alpha }_i^{{ k+1 }}]&\le \frac{\mu _f }{4}[\Vert \varvec{x}_i^k - \varvec{x}_i\Vert ^2-\Vert \varvec{x}_i^{k+1} - \varvec{x}_i\Vert ^2]\\&+ \frac{2\eta }{1-2\eta v_L} (\sigma ^2)_k^sp \end{aligned}$$

where $\varvec{\alpha }_i^{k+1} = \varvec{\lambda }_i^k +\rho (\varvec{x}_i^{k+1}-\varvec{z}^k).$

Proof

Using Lemma 4 and the strong convexity of $L_i(\varvec{v}_i)$, we have

where $\zeta = 2\eta - \frac{4\eta }{1-2\eta v_L}$, $\xi =\frac{4\eta }{1-2\eta v_L}$.

Then, we obtain

where we apply Lemma 2 and $L_i(\varvec{v}_i^s) - L_i(\varvec{x}_i) = f_i(\varvec{v}_i^s) - f_i(\varvec{x}_i) + (\varvec{v}_i^s - \varvec{x}_i)^T\varvec{q}_i^s -\frac{\rho }{2}\Vert \varvec{v}_i^s - \varvec{x}_i\Vert ^2$ to obtain the inequality. Hence, we choose $\eta \le \frac{4\mu _L-4\rho -3\mu _f}{8v_L^2+2\mu _fv_L}$ so that $1-\frac{\rho \xi }{2}-\frac{\mu _f\xi }{4} \ge 1+\frac{4\eta ^2 v_L^2}{1-2\eta v_L}-\frac{\mu _L\xi }{2}-\frac{\mu _f\zeta }{4}$. We take and we know that $f_i(\varvec{v}_i^s) - f_i(\varvec{x}_i) + (\varvec{v}_i^s - \varvec{x}_i)^T\varvec{q}_i^s$ is convex in $\varvec{v}_i^s$. By using the Jensen’s inequality, we have

where Summing from $s = 0,1,2,...,S-1$ and using we obtain

$$\begin{aligned}&2S\eta \mathbb {E}[f_i(\varvec{x}_i^{k+1}) - f_i(\varvec{x}_i)+ (\varvec{x}_i^{k+1} - \varvec{x}_i)^T\varvec{\alpha }_i^{{ k+1 }}+\frac{\mu _f }{4}\Vert \varvec{x}_i^{k+1} - \varvec{x}_i\Vert ^2] \\&\le \frac{2\eta ^2S}{1-2\eta v_L} (\sigma ^2)_k^sp +(1-\frac{\rho \xi }{2}-\frac{\mu _f\xi }{4} +\frac{4\eta ^2 v_L^2 S}{1-2\eta v_L})\Vert \varvec{x}_i^k - \varvec{x}_i\Vert ^2, \end{aligned}$$

where $\varvec{\alpha }_i^{{ k+1 }} = \varvec{\lambda }_i^k +\rho (\varvec{x}_i^{k+1}-\varvec{z}^k).$

Thus, we have

$$\begin{aligned}&\mathbb {E}[f_i(\varvec{x}_i^{k+1}) - f_i(\varvec{x}_i)+ (\varvec{x}_i^{k+1} - \varvec{x}_i)^T\varvec{\alpha }_i^{{ k+1 }}] \\&\le \frac{1}{2S\eta }(1-\frac{\rho \xi }{2}-\frac{\mu _f\xi }{4} +\frac{4\eta ^2 v_L^2 S}{1-2\eta v_L})\Vert \varvec{x}_i^k - \varvec{x}_i\Vert ^2 - \frac{\mu _f }{4}\mathbb {E}\Vert \varvec{x}_i^{k+1} - \varvec{x}_i\Vert ^2 \\&~~~+\frac{2\eta }{1-2\eta v_L} (\sigma ^2)_k^sp \\&\le \frac{\mu _f }{4}[\Vert \varvec{x}_i^k - \varvec{x}_i\Vert ^2-\Vert \varvec{x}_i^{k+1} - \varvec{x}_i\Vert ^2] + \frac{2\eta }{1-2\eta v_L} (\sigma ^2)_k^sp, \end{aligned}$$

where we assume $1-\frac{\rho \xi }{2}-\frac{\mu _f\xi }{4} +\frac{4\eta ^2 v_L^2 S}{1-2\eta v_L} \le \frac{ S\eta \mu _f }{2}.$

1.3 A.3 Proof of Theorem 3

Proof

Combining Lemmas 7, 5 and 6 together and using the convergence criterion (9), we let $\varvec{w}_i^{k+1} = (\varvec{x}_i^{k+1}; \varvec{z}^{k+1}; \varvec{\alpha }_i^{k+1})$, and . For any $\varvec{w}= (\varvec{x}_i ;\varvec{z};\varvec{\lambda }_i ),$ we have

$$\begin{aligned}&\mathbb {E}[P(\varvec{u}^{k+1})-P(\varvec{u}) +\sum _{i=1}^N (\varvec{w}^{k+1} - \varvec{w})^T F(\varvec{w}^{k+1})]\\&\le \mathbb {E}\Bigg \{ \sum _{i=1}^N f_i(\varvec{x}_i^{k+1}) + g(\varvec{z}^{k+1})-\sum _{i=1}^N f_i(\varvec{x}_i) - g(\varvec{z}) \\&~~~+ \sum _{i=1}^N \begin{pmatrix} \varvec{x}_i^{k+1} - \varvec{x}_i\\ \varvec{z}^{k+1} - \varvec{z}\\ \varvec{\alpha }_i^{k+1} - \varvec{\alpha }_i \end{pmatrix} ^T \begin{pmatrix} \varvec{\alpha }_i^{k+1} \\ -\varvec{\alpha }_i^{k+1} \\ -(\varvec{x}_i^{k+1} - \varvec{z}^{k+1}) \end{pmatrix} \Bigg \}\\&\le \sum _{i=1}^N\Bigg \{\frac{\mu _f }{4}[\Vert \varvec{x}_i^k - \varvec{x}_i\Vert ^2-\Vert \varvec{x}_i^{k+1} - \varvec{x}_i\Vert ^2] + \frac{\rho }{2} (\Vert \varvec{z}^{k } - \varvec{z}\Vert ^2-\Vert \varvec{z}^{k+1} - \varvec{z}\Vert ^2)\\&~~~+\frac{1}{2\rho }( \Vert \varvec{\lambda }_i^k-\varvec{\alpha }_i \Vert ^2 - \Vert \varvec{\lambda }_i^{k+1}-\varvec{\alpha }_i \Vert ^2)+ \frac{2\eta }{1-2\eta v_L} (\sigma ^2)_k^sp\Bigg \}, \end{aligned}$$

where $F(\varvec{w}) =$ $\begin{pmatrix} \varvec{\alpha }_i \\ -\varvec{\alpha }_i \\ -(\varvec{x}_i - \varvec{z}) \end{pmatrix}$.

Summing the inequality over $k = 0,1,2,...,K-1$ and using the Jensen’s inequality, we get

where , and If we take $\varvec{x}= \varvec{x}^*,\varvec{z}= \varvec{z}^*,$ and , we have

1.4 A.4 Proof of Theorem 4

By choosing $\eta$, which satisfies condition in Theorem 3, and $S = O(\frac{v_f}{\mu _f})$, we can make $A = \frac{\mu _f }{4 } \Vert \varvec{x}_i^0 - \varvec{x}_i^*\Vert ^2+ \frac{\rho }{2 } \Vert \varvec{z}^{0 } - \varvec{z}^*\Vert ^2 +\frac{1}{2\rho } (\Vert \varvec{\lambda }_i^0\Vert ^2+ \tau _i^2)$ a constant.

Then, we have

Thus, if we choose $K = O\left( \frac{M \epsilon }{G}\sqrt{\frac{\mu _f}{v_f p \ln (1/\delta )}}\right) ,$ we have