Differentially-Private “Draw and Discard” Machine Learning: Training Distributed Model from Enormous Crowds

Abstract

The setting of our problem is a distributed architecture facing an enormous user set, where events are repeating and evolving over time, and we want to absorb the stream of events into the model: first local model, then absorb it in the global one, and also care about user privacy. Naturally, we learn a phenomenon which happens distributedly in many places (like malware spread over smartphones, user behavior to operation and UX of an app, or other such events). To this end, we considered a configuration where the learning server is built to deal with the possibly high frequency high-volume environment in a natural distributed fashion, while taking care of statistical convergence and privacy properties of the setting as well. We propose a novel framework for privacy-preserving client-distributed machine learning. It is based on the desire to allow differential privacy guarantees in the local model of privacy in a way that satisfies systems constraints using high number of asynchronous client-server communication (i.e., not much coordination among separate clients, which is a simple communication model to implement, which in some settings already exist, i.e., in apps facing users), and provides attractive model learning properties.

We develop a generic randomized learning algorithm “Draw and Discard” because it relies on random sampling and discarding of models for load distribution and scalability, which also provides additional server-side privacy protections and improved model quality through averaging. The model is general and we show its applicability to Generalized Linear models. We analyze the statistical stability and privacy guarantees provided by our approach against faults and against several types of adversaries. We then showcase experimental results. Our framework (first reported in [28]) has been experimentally deployed in a real industrial setting. We view the result as an initial combination of ML and of distributed systems, and we believe it poses numerous directions for further developments.

Appendices

Appendix

A Variance Stabilization Proof and Other Proofs

Theorem 1 Let $B = (B_1, \ldots , B_k)$ be a vector of k random variables (weights) with mean $\mu$ and variance $\frac{k}{2}\sigma ^2$. Selecting one of the weights at random, adding noise with mean 0 and variance $\sigma ^2$ and putting it back with replacement does not change the expected intra-model variance of B (i.e., weights remain distributed with variance $\frac{k}{2}\sigma ^2$).

Proof

We use the Law of Total Variance

$$ V(B) = E(V(B|X)) + V(E(B|X)) $$

three times as we think of B as a mixture of mixtures. In the first mixture, the expectation is taken over the distribution of whether the same or different model was updated ($X \in \{j \rightarrow j, j \rightarrow j'\}$). In the inner mixture, the expectation is taken over the partitioning of the k weights themselves (Z).

Total variance, therefore, is equal to

$$\begin{aligned} V(B)= & {} \frac{1}{k}V(B|j \rightarrow j) + \frac{k-1}{k} V(B | j \rightarrow j') \\&+ V(E(B|X)). \end{aligned}$$

Because we add noise with mean 0, in either case, the mean of B does not change, so $V(E(B|X)) = 0$.

Replacing the same weight partitions the k weights into two sets, a single weight updated and the rest of the weights. Partition means do not change, and the variance increases due to added noise in the first partition (mixture component). Thus, $V(B | j \rightarrow j)$ is given as

$$\begin{aligned} V(B| j \rightarrow j)= & {} \frac{1}{k}\left( \frac{k}{2} + 1\right) \sigma ^2 + \frac{k-1}{k}\frac{k}{2}\sigma ^2 \\= & {} \left( \frac{1}{2} + \frac{1}{k} + \frac{k-1}{2}\right) \sigma ^2 \\= & {} \left( \frac{k}{2} + \frac{1}{k}\right) \sigma ^2. \end{aligned}$$

Replacing a different weight partitions the weights space into 2 and $(k-2)$ subsets. Unlike in the first case, V(E(B|Z)) is non-zero due to a single weight essentially replicated twice in the first partition ($B_1$ has a mean of 0, but $B_2$ has a mean of $B_1$). After the update, the overall mean of B under Z becomes

$$\begin{aligned} \mu _{Z}= & {} \frac{2\mu _1 + (k-2)\mu }{k}, \end{aligned}$$

where $\mu _1$ is the mean of the model selected and model replaced and has a distribution with mean $\mu$ and variance $\frac{k}{2}\sigma ^2$. Note that the mean of B over sampling in X is still 0 as $E(B_1) = 0$ over X.

Then $V(B|j \rightarrow j')$

$$\begin{aligned}= & {} \frac{2}{k}\frac{1}{2}\sigma ^2 + \frac{k-2}{k} \frac{k}{2}\sigma ^2 \\&+ \frac{2E[(\mu _1 - \mu _{B_i})^2] + (k-2)E[(\mu -\mu _{B_i})^2]}{k-1} \\= & {} \frac{1}{k}\sigma ^2 + \frac{k-2}{2}\sigma ^2 \\&+ \frac{2}{k-1}\Bigl (\frac{k-2}{k}\Bigr )^2E[(\mu _1 - \mu )^2] \\&+ \frac{k-2}{k-1}\Bigl (\frac{2}{k}\Bigr )^2E[(\mu _1 - \mu )^2] \\= & {} \frac{1}{k}\sigma ^2 + \frac{k-2}{2}\sigma ^2 + \frac{2}{k-1}\left( \frac{k-2}{k}\right) ^2\frac{k}{2}\sigma ^2 \\&+ \frac{k-2}{k-1}\left( \frac{2}{k}\right) ^2\frac{k}{2}\sigma ^2 \\= & {} \left( \frac{1}{k} + \frac{k-2}{2} + \frac{(k-2)^2}{k(k-1)} + \frac{2(k-2)}{k(k-1)}\right) \sigma ^2 \\= & {} \frac{2k - 2 + k(k-2)(k-1) + 2(k-2)^2 + 4k - 8}{2k(k-1)}\sigma ^2 \\= & {} \frac{k^3 - k^2 - 2}{2k(k-1)}\sigma ^2. \end{aligned}$$

Note that the variance component must be computed with $k-1$ and not k because of the finite nature of k in this case.

Putting it all together,

$$\begin{aligned} V(B)= & {} \frac{1}{k}V(B|j \rightarrow j) + \frac{k-1}{k} V(B | j \rightarrow j') \\= & {} \frac{1}{k}\left( \frac{k}{2} + \frac{1}{k}\right) \sigma ^2 + \frac{k-1}{k}\frac{k^3 - k^2 - 2}{2k(k-1)}\sigma ^2 \\= & {} \left( \frac{1}{2} + \frac{1}{k^2} + \frac{k^3 - k^2 - 2}{2k^2} \right) \sigma ^2 \\= & {} \frac{k}{2}\sigma ^2. \end{aligned}$$

Lemma 1 DDML discards $1-\frac{1}{k}$ updates long-term.

Proof

Consider a Markov process on the states $0, 1, \ldots , k$, where each state represents the number of models in which a particular update can be found. Denote by $p_i$ - the probability to go from i to $i-1$ or $i+1$. In DDML, $p_i = \frac{(k-i)i}{k^2}$, for $1 \le i \le k-1$, and $p_0 = 0, p_k = 1$.

Let $q_i$ be the probability of eventually ending up at state 0 if you start in state i. By the set-up of DDML, for $1< i<k-1$ we have:

$q_i = p_iq_{i-1} + p_iq_{i+1}+(1-2p_i)q_i,$ or $2q_i = q_{i-1}+q_{i+1}$.

We also know that

• $q_0=1, p_0=0$, $q_k=0, p_k=1$,

• $q_1 = p_1q_0+p_1q_2+(1-2p_1)q_1$,

• $q_{k-1} = p_{k-1}q_{k-2} + p_{k-1}q_k+(1-2p_{k-1})q_{k-1}$

Summing equations for $1 \le i \le k-1$ we have

$2q_1 + \cdots + 2 q_{k-1} = q_0 + q_1 + 2(q_2+\cdots +q_{k-2})+q_{k-1}+q_{k}$ Simplifying: $q_1+q_{k-1} = q_0+q_k$ or

$$\begin{aligned} q_1+q_{k-1} = 1 \end{aligned}$$

(5)

On the other hand,

• $q_{k-2} = 2q_{k-1}$,

• $q_{k-3} = 2q_{k-2} - q_{k-1} = 3q_{k-1}$

• $q_{k-4}=2q_{k-3}-q_{k-2} = 4q_{k-1}$

• $\cdots$

  $$\begin{aligned} q_1 = 2q_2-q_3 = (k-1)q_{k-1} \end{aligned}$$

  (6)

Combining (5) and (6), we have $(k-1)q_{k-1}+q_{k-1} = 1$ or $q_{k-1}=\frac{1}{k}$ and $q_1 = 1-\frac{1}{k}$. Since DDML is set-up that each particular contribution is initially in state 1, this completes the proof.

Theorem 2 With high probability, DDML guarantees a user $(\epsilon _T, \delta _T)$-differential privacy against adversary III, where $\epsilon _T = \frac{\epsilon }{\sqrt{2T}} \sqrt{\ln \left( \frac{1}{2\delta _T}\right) }$ and $\delta _T$ is an arbitrary small constant (typically chosen as O(1/ number of users)). T is the number of updates made to the model instance between when a user submitted his instance update and when the adversary observes the instances. The statement holds if T is sufficiently large.

Proof

We rely on the result from concurrent and independent work of [15] obtained in a different context to analyze the privacy amplification in this case. Specifically, their result states that for any contractive noisy process, privacy amplification is no worse than that for the identity contraction, which we analyze below.

The sum of T random variables drawn independently from the Laplace distribution with mean 0 will tend toward a normal distribution for sufficiently large T, by the Central Limit Theorem. In DDML’s case with Laplace noise, the variance of each random variable is $\frac{8\gamma ^2}{\epsilon ^2}$, therefore, if we assume that the adversary observes the model instance after T updates to it, the variance of the noise added will be $T \cdot \frac{8\gamma ^2}{\epsilon ^2}$. This corresponds to Gaussian with scale $\sigma = \frac{2\sqrt{2T}\gamma }{\epsilon }$.

Lemma 1 from [19] states that for points in p-dimensional space that differ by at most w in $\ell _2$, addition of noise drawn from $N^p(0, \sigma ^2_T)$, where $\sigma _T \ge w \frac{\sqrt{2\left( \ln \left( \frac{1}{2\delta _T}\right) +\epsilon _T\right) }}{\epsilon _T}$ and $\delta _T < \frac{1}{2}$ ensures $(\epsilon _T, \delta _T)$ differential privacy. We use the result of Lemma 1 from [19], rather than the more commonly referenced result from Theorem A.1 of [11], because the latter result holds only for $\epsilon _T \le 1$, which is not the privacy loss used in most practical applications.

We now ask the question: what approximate differential privacy guarantee is achieved by DDML against adversary III? To answer this, fix a desired level of $\delta _T$ and use the approximation obtained from the Central Limit theorem to solve for the $\epsilon _T$.

$$\begin{aligned} \frac{2\sqrt{2T}\gamma }{\epsilon } \ge w \frac{\sqrt{2\left( \ln \left( \frac{1}{2\delta _T}\right) +\epsilon _T\right) }}{\epsilon _T} \end{aligned}$$

$$\begin{aligned} T \cdot \frac{8\gamma ^2}{\epsilon ^2} \ge w^2 \frac{2\left( \ln \left( \frac{1}{2\delta _T}\right) +\epsilon _T\right) }{\epsilon ^2_T} \end{aligned}$$

$$\begin{aligned} T \cdot \frac{4\gamma ^2}{\epsilon ^2} \cdot \epsilon ^2_T - w^2 \epsilon _T - w^2 \ln \left( \frac{1}{2\delta _T}\right) \ge 0 \end{aligned}$$

Solving the quadratic inequality, we have:

$$\begin{aligned} D = w^4+4T \cdot \frac{4\gamma ^2}{\epsilon ^2} w^2 \ln \left( \frac{1}{2\delta _T}\right) \end{aligned}$$

$$\begin{aligned} \epsilon _T \ge \frac{w^2 + \sqrt{D}}{2T \cdot \frac{4\gamma ^2}{\epsilon ^2} } = \frac{\epsilon ^2 w^2}{8\gamma ^2T} \left[ 1 + \sqrt{1+16T\frac{\gamma ^2}{\epsilon ^2 w^2} \ln \left( \frac{1}{2\delta _T}\right) }\right] \end{aligned}$$

For large T, the additive term of 1 under the square root is negligible, so we have:

$$\begin{aligned} \epsilon _T \approx \frac{\epsilon ^2 w^2}{8\gamma ^2T} 4\sqrt{T} \frac{\gamma }{\epsilon w} \sqrt{\ln \left( \frac{1}{2\delta _T}\right) } = \frac{\epsilon w}{2\gamma \sqrt{T}} \sqrt{\ln \left( \frac{1}{2\delta _T}\right) } \end{aligned}$$

In DDML, $w=\sqrt{2}\gamma$, therefore,

$$ \epsilon _T \approx \frac{\epsilon }{\sqrt{2T}} \sqrt{\ln \left( \frac{1}{2\delta _T}\right) } $$