Measured Continuous Greedy with Differential Privacy

Abstract

In the paper, we design a privacy algorithm for maximizing a general submodular set function over a down-monotone family of subsets, which includes some typical and important constraints such as matroid and knapsack constraints. The technique is inspired by the measured continuous greedy (MCG) which compensates for the difference between the residual increase of elements at a given point and the gradient of it by distorting the original direction with a multiplicative factor. It directly makes the continuous greedy approach fit to the problem of maximizing a non-monotone submodular function. We generate the MCG algorithm in the framework of differential privacy. It is accepted as a robust mathematical guarantee and can provide the protection to sensitive and personal data. We propose a 1/e-approximation algorithm for the general submodular function. Moreover, for monotone submodular objective functions, our algorithm achieves an approximation ratio that depends on the density of the polytope defined by the problem at hand, which is always at least as good as the previously known best approximation ratio of $1-1/e$.

Appendix: Missing Proofs

Theorem 3.1

Algorithm 1 preserves $O(\epsilon \cdot d_{\mathcal {P}}^2)$-differential privacy.

Proof

Let D and $D^{\prime }$ be two neighboring datasets and $f_D$, $f_{D^{\prime }}$ be their associated functions. For a fixed $\mathbf {y}_t\in C_{\rho }$, we consider the relative probability of Algorithm 1 (denoted by M) choosing $\mathbf {y}_t$ at time step t given multilinear extensions of $f_D$ and $f_{D^{\prime }}$. Let $M_t(f_D|\mathbf {x}_t)$ denote the output of M at time step t given dataset D and point $\mathbf {x}_t$. Similarly, $M_t(f_{D^{\prime }}|\mathbf {x}_t)$ denotes the output of M at time step t given dataset $D^{\prime }$ and point $\mathbf {x}_t$. Further, write $d_\mathbf {y}=\langle \mathbf {y},\nabla f_D(\mathbf {x}_t)\rangle$ and $d^{\prime }_\mathbf {y}=\langle \mathbf {y},\nabla f_D(\mathbf {x}_t)\rangle$. We have

$$\begin{aligned} \frac{\Pr [M_t(f_D|\mathbf {x}_t)=\mathbf {y}_t]}{\Pr [M_t(f_{D^{\prime }}|\mathbf {x}_t)=\mathbf {y}_t]} = \frac{\exp (\epsilon ^{\prime }\cdot d_{\mathbf {y}_t})}{\exp (\epsilon ^{\prime }\cdot d^{\prime }_{\mathbf {y}_t})} \cdot \frac{\sum _{\mathbf {y}\in C_{\rho }}\exp (\epsilon ^{\prime }\cdot d^{\prime }_{\mathbf {y}})}{\sum _{\mathbf {y}\in C_{\rho }}\exp (\epsilon ^{\prime }\cdot d_{\mathbf {y}})}. \end{aligned}$$

For the first factor, we have

$$\begin{aligned}&\frac{\exp (\epsilon ^{\prime }\cdot d_{\mathbf {y}_t})}{\exp (\epsilon ^{\prime }\cdot d^{\prime }_{\mathbf {y}_t})} \\= & {} \exp \left( \epsilon ^{\prime }(d_{\mathbf {y}_t} - d^{\prime }_{\mathbf {y}_t}) \right) \\= & {} \exp \left( \epsilon ^{\prime }(\langle \mathbf {y}_t, \nabla f_D(\mathbf {x}_t)-\nabla f_{D^{\prime }}(\mathbf {x}_t)\rangle ) \right) \\\le & {} \exp \left( \epsilon ^{\prime }\Vert \mathbf {y}_t\Vert _1 \Vert \nabla f_D(\mathbf {x}_t)-\nabla f_{D^{\prime }}(\mathbf {x}_t)\Vert _{\infty } \right) \\= & {} \exp \left( \epsilon ^{\prime }\sum _{e\in \mathcal {X}}\mathbf {y}_t(e)\cdot \left( \max _{e\in \mathcal {X}}\mathbb {E}_{R\sim \mathbf {x}_t}\left[ f_D(R\cup \{e\})-f_D(R)-f_{D^{\prime }}(R\cup \{e\})+f_{D^{\prime }}(R) \right] \right) \right) \\\le & {} \exp (O(\epsilon ^{\prime }\cdot md_{\mathcal {P}} \cdot 2\varDelta )) = \exp (O(\epsilon \cdot d_{\mathcal {P}})). \end{aligned}$$

Note that the last inequality holds since $\mathbf {y}_t$ is a member of the polytope $\mathcal {P}$ and by definition we have $\sum _{e\in \mathcal {X}}a_{i,e}\mathbf {y}_t(e)\le b_i$ and $d_{\mathcal {P}}=\min _{1\le i\le m}\frac{b_i}{\sum _{e\in \mathcal {X}}a_{i,e}}$. Moreover, recall that $f_D$ is $\varDelta$-sensitive.

For the second factor, let us write $\beta _{\mathbf {y}} = d^{\prime }_{\mathbf {y}} - d_{\mathbf {y}}$ to be the deficit of the probabilities of choosing direction $\mathbf {y}$ in instances $f_{D^{\prime }}$ and $f_D$. Then, we have

$$\begin{aligned} \frac{\sum _{\mathbf {y}\in C_{\rho }}\exp (\epsilon ^{\prime }\cdot d^{\prime }_{\mathbf {y}})}{\sum _{\mathbf {y}\in C_{\rho }}\exp (\epsilon ^{\prime }\cdot d_{\mathbf {y}})}= & {} \frac{\sum _{\mathbf {y}\in C_{\rho }}\exp (\epsilon ^{\prime }\cdot \beta _{\mathbf {y}})\exp (\epsilon ^{\prime }\cdot d_{\mathbf {y}})}{\sum _{\mathbf {y}\in C_{\rho }}\exp (\epsilon ^{\prime }\cdot d_{\mathbf {y}})} \\= & {} \mathbb {E}_{\mathbf {y}}[\exp (\epsilon ^{\prime }\cdot \beta _{\mathbf {y}})] \le \exp (O(\epsilon ^{\prime }\cdot md_{\mathcal {P}}\cdot 2\varDelta )) \\= & {} \exp \left( O(\epsilon \cdot d_{\mathcal {P}})\right) . \end{aligned}$$

   $\square$

Lemma 3.1

For every time $0\le t\le T$, I(t) is the sampled vector by Algorithm 1 and $I^\prime (t)$ is the solution of the linear programming in measured continuous greedy algorithm. Then,

$$\begin{aligned} I(t)w(t)(1-\mathbf {y}(t))\ge & {} F(\mathbf {y}(t) \vee \mathbf {1}_{OPT}) - F(\mathbf {y}(t)) - O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) . \end{aligned}$$

Proof

$$\begin{aligned} w(t)\cdot \mathbf {1}_{OPT}= & {} \sum _{e\in OPT}w_e(t) = \sum _{e\in OPT}[F(\mathbf {y}(t)\vee \mathbf {1}_e) - F(\mathbf {y}(t))] \\= & {} \mathbb {E}\left[ \sum _{e\in OPT} f(R(\mathbf {y}(t)) + e) - f(R(\mathbf {y}(t))) \right] \\\ge & {} \mathbb {E}\left[ f(R(\mathbf {y}(t))\cup OPT) - f(R(\mathbf {y}(t))) \right] = F(\mathbf {y}(t) \vee \mathbf {1}_{OPT}) - F(\mathbf {y}(t)), \end{aligned}$$

where the inequality is followed by submodularity.

Since $\mathbf {1}_{OPT}\in \mathcal {P}$, we get from Algorithm 1

$$\begin{aligned} I(t)^\prime \cdot w(t) \ge F(\mathbf {y}(t) \vee \mathbf {1}_{OPT}) - F(\mathbf {y}(t)). \end{aligned}$$

Hence,

$$\begin{aligned}&\sum _{e\in \mathcal {X}} I^\prime _e(t)\cdot (1-\mathbf {y}_e(t))\cdot \partial _e F(\mathbf {y}(t)) \\= & {} \sum _{e\in \mathcal {X}} (1-\mathbf {y}_e(t))\cdot I^\prime _e(t)\cdot [F(\mathbf {y}(t)\vee \mathbf {1}_e) - F(\mathbf {y}(t)\wedge \mathbf {1}_{\hat{e}})] \\= & {} \sum _{e\in \mathcal {X}} I^\prime _e(t) \cdot [F(\mathbf {y}(t)\vee \mathbf {1}_e) - F(\mathbf {y}(t))] = I^\prime (t)\cdot w(t) \\\ge & {} F(\mathbf {y}(t)\vee \mathbf {1}_{OPT}) - F(\mathbf {y}(t)). \end{aligned}$$

Recall we define a neighboring feasible field, i.e., the $\rho$-covering of $\mathcal {P}$. And we get the followings by the Theorem 2.2 of exponential mechanism:

$$\begin{aligned} I(t)w(t)(1-\mathbf {y}(t))\ge & {} \sum _{e\in \mathcal {X}}I^\prime _e(t)\cdot (1-\mathbf {y}_e(t))\cdot \partial _e F(\mathbf {y}(t)) - O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) \\\ge & {} F(\mathbf {y}(t) \vee \mathbf {1}_{OPT}) - F(\mathbf {y}(t)) - O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) . \end{aligned}$$

   $\square$

Lemma 3.2

For every time $0\le t < T$,

$$\begin{aligned}&F(\mathbf {y}(t+\delta )) - F(\mathbf {y}(t)) \\\ge & {} \delta \cdot \left[ F(\mathbf {y}(t) \vee \mathbf {1}_{OPT}) - F(\mathbf {y}(t)) - O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) \right] - O(n^3\delta ^2)\cdot f(OPT). \end{aligned}$$

Proof

$$\begin{aligned}&F(\mathbf {y}(t+\delta )) - F(\mathbf {y}(t)) \\\ge & {} (\mathbf {y}(t+\delta )-\mathbf {y}(t))\cdot \partial F(\mathbf {y}(t)) - O(n^3\delta ^2)\cdot f(OPT) \\= & {} \delta \cdot I(t)(1-\mathbf {y}(t))w(t) - O(n^3\delta ^2)\cdot f(OPT) \\\ge & {} \delta \cdot \left[ F(\mathbf {y}(t) \vee \mathbf {1}_{OPT}) - F(\mathbf {y}(t)) - O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) \right] - O(n^3\delta ^2)\cdot f(OPT), \end{aligned}$$

where the first and last inequalities are given by Lemma 2.2 and Lemma 3.1. And the algorithm makes the equality hold.    $\square$

Lemma 3.3

For every $0\le t\le T$,

$$\begin{aligned} g(t)\le F(\mathbf {y}(t)) + O(n^3\delta )\cdot tf(OPT) + \delta O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) . \end{aligned}$$

Proof

Assume the big O notation in Lemma 3.3 to be $cn^3\delta ^2$. Prove by induction on t that $g(t)\le F(\mathbf {y}(t)) + cn^3\delta t f(OPT)$. For $t=0$, $g(0)=0\le F(\mathbf {y}(0))$. Assume that the claim holds for some t. Then

$$\begin{aligned} g(t+\delta )= & {} (1-\delta )g(t) + \delta f(OPT) \\\le & {} (1-\delta )\left[ F(\mathbf {y}(t) + cn^3\delta t f(OPT)) \right] + \delta e^{-t} f(OPT) \\= & {} F(\mathbf {y}(t)) + \delta [e^{-t}f(OPT) - F(\mathbf {y}(t))] + c(1-\delta )n^3\delta t f(OPT) \\\le & {} F(\mathbf {y}(t+\delta )) + cn^3\delta ^2 f(OPT) + c(1-\delta )n^3\delta t f(OPT) + \delta O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) \\\le & {} F(\mathbf {y}(t+\delta )) + cn^3\delta (t+\delta ) f(OPT) + \delta O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) , \end{aligned}$$

where the inductive assumption and Lemma 3.3 give the first two inequalities and the last one is hold by $\delta \in [0,1]$.    $\square$

Lemma 3.4

For every time $0\le t\le T$, $g(t)\ge h(t)$.

Proof

The proof is by induction on t. For $t=0$, $g(0)=0=h(0)$. Assume that the lemma holds for some t. Then, we can easily get

$$\begin{aligned} h(t+\delta )= & {} h(t) + \int _t^{t+\delta } h^\prime (l)dl = h(t) + f(OPT)\cdot \int _t^{t+\delta } e^{-l}(1-l)dl \\\le & {} h(t) + f(OPT)\cdot \delta e^{-t}(1-t) = (1-\delta )h(t) + \delta e^{-t}\cdot f(OPT) \\\le & {} (1-\delta )g(t) + \delta e^{-t}\cdot f(OPT) = g(t+\delta ). \end{aligned}$$

   $\square$

Corollary 3.1

$F(\mathbf {y}(t)) \ge \left[ Te^{-T} - o(1) \right] \cdot f(OPT) - \delta O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right)$.

Proof

By Lemma 3.3 and Lemma 3.4,

$$\begin{aligned} F(\mathbf {y}(T))\ge & {} g(T)-O(n^3\delta )\cdot T\cdot f(OPT) - \delta O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) \\\ge & {} h(T)-O(n^3\delta )\cdot f(OPT) - \delta O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) \\= & {} \left[ Te^{-T} - O(n^3\delta ) \right] \cdot f(OPT) - \delta O\left( \sqrt{\epsilon }+\frac{2\varDelta \ln n}{\epsilon ^3}\right) . \end{aligned}$$

Recall that $\delta \le n^{-5}$, hence, $O(n^3\delta )=o(1)$ and the proof is complete.    $\square$