A Simple Algorithm for Estimating Distribution Parameters from $n$-Dimensional Randomized Binary Responses

Abstract

Randomized response is attractive for privacy preserving data collection because the provided privacy can be quantified by means such as differential privacy. However, recovering and analyzing statistics involving multiple dependent randomized binary attributes can be difficult, posing a significant barrier to use. In this work, we address this problem by identifying and analyzing a family of response randomizers that change each binary attribute independently with the same probability. Modes of Google’s Rappor randomizer as well as applications of two well-known classical randomized response methods, Warner’s original method and Simmons’ unrelated question method, belong to this family. We show that randomizers in this family transform multinomial distribution parameters by an iterated Kronecker product of an invertible and bisymmetric $2\times 2$ matrix. This allows us to present a simple and efficient algorithm for obtaining unbiased maximum likelihood parameter estimates for $k$-way marginals from randomized responses and provide theoretical bounds on the statistical efficiency achieved. We also describe the efficiency – differential privacy tradeoff. Importantly, both randomization of responses and the estimation algorithm are simple to implement, an aspect critical to technologies for privacy protection and security.

A Proofs

We start by making a key observation.

Observation 1: Consider the $2^n \times 2^n$ matrix $C$. If we let entry $C_{ix', jy'} = \eta ((ix') \oplus (jy')) = 2^{\eta (i \oplus j)} \;\eta (x' \oplus y'),$ we get that $C_{x,y} = 1^{n - |x \oplus y|} \; 2^{|x \oplus y|}$. Since we can write $C = J_1 \otimes J_2 \otimes \cdots \otimes J_n$ where $J_k$ is the $2 \times 2$ matrix $J$ such that $J_{i,j} = 2^{i \oplus j}$, i.e., $J = \begin{pmatrix} 1 &{} 2 \\ 2 &{} 1 \end{pmatrix},$ we get that $D_{x,y} = a^{n - |x \oplus y|} \; b^{|x \oplus y|}$ for $D = C_{a,b}(n) = B_1 \otimes B_2 \otimes \cdots \otimes B_n$ where $B_k = \begin{pmatrix} a &{} b \\ b &{} a \end{pmatrix}.$

Proof of Proposition 1: Note that we can write

$$\begin{aligned}&{\text {Tr}}({\text {cov}}(\hat{\pi }(m))) = \sum _x {\text {Var}}(\hat{\pi }_x(m)), \text { } {\text {Tr}}({\text {cov}}(\hat{\pi }^*(m))) = \sum _x {\text {Var}}(\hat{\pi }_x^*(m))\\&{\text {Var}}(\hat{\pi }_x(m)) = m^{-1} F(x,\pi ), \text { } {\text {Var}}(\hat{\pi }^*_x(m)) = m^{-1} G(x,\pi ) \end{aligned}$$

where $F$ and $G$ are functions independent of $m$. Then for any positive integer $m$,

$$ L(m) = \frac{m^{-1} \sum _x F(x,\pi )}{m^{-1} \sum _x G(x,\pi )} = \frac{\sum _x F(x,\pi )}{\sum _x G(x,\pi )} = L. $$

and

$$\begin{aligned} \sum _x {\text {Var}}(\hat{\pi }_x(\alpha Lm))&= \sum _x \alpha ^{-1} m^{-1} L^{-1} F(x,\pi ) \le m^{-1} L^{-1} \sum _x F(x, \pi )\\&= \sum _x m^{-1} G(x, \pi ) = \sum _x {\text {Var}}(\hat{\pi }^*_x(m)). \end{aligned}$$

Furthermore,

$$\begin{aligned} {\text {E}}\left( \Vert \hat{\pi }(m) - \pi \Vert _2^2\right)&= {\text {E}}\left( \sum _x (\hat{\pi }_x(m) - p_x)^2\right) = \sum _x {\text {E}}\left( (\hat{\pi }_x(m) - p_x)^2\right) \\&= \sum _x {\text {Var}}(\hat{\pi }_x(m)), \end{aligned}$$

and similarly ${\text {E}}\left( \Vert \hat{\pi }^*(m) - \pi \Vert _2^2\right) = \sum _x{\text {Var}}(\hat{\pi }^*_x(m))$.   $\square$

Proof of Proposition 2: The proposition follows directly from Theorem 1.   $\square$

Proof of Proposition 3: We first note that for $i \in \{0, 1, \ldots , 2^n - 1\}$ we have that $\varsigma ((2^n - 1) - i) = \mathbf {1} \oplus \varsigma (i)$. From this and that $\oplus$ commutes, we get

• 1. $|\varsigma (i) \oplus \varsigma (n - i)| = n$, and

• 2. $|\varsigma (i) \oplus \varsigma (j)| = |\varsigma (2^n - 1 - j) \oplus \varsigma (2^n - 1 - i)|$.

The above and that the entry $C_{a,b}(n)_{i,j} = g(|\varsigma (i) \oplus \varsigma (j)|, a, b)$ for some $g$, the proposition follows.   $\square$

Proof of Theorem 2: The first equation follows directly from Observation 1. We have that $C_{a,b}(1)$ is invertible if $a^2 \ne b^2$. From this and that $(A \otimes B) = (A^{-1} \otimes B^{-1})$ we complete the proof.   $\square$

Proof of Lemma 1: From Sect. 3.2 we have that

$$\begin{aligned} {\text {cov}}(\hat{\pi }(m))&= m^{-1} \left( C^{-1} \text {diag}(C\pi ) {C^{-1}}^T - \pi \pi ^T\right) . \end{aligned}$$

By properties of the trace of matrix products and symmetry of $C^{-1}$,

$$\begin{aligned}&{\text {Tr}}\left( m^{-1} \left( C^{-1} \text {diag}(C\pi ) {C^{-1}}^T - \pi \pi ^T\right) \right) \\&= m^{-1}\left( {\text {Tr}}\left( C^{-1} \text {diag}(C\pi ) {C^{-1}}^T\right) - {\text {Tr}}(\pi \pi ^T) \right) \\&= m^{-1}\left( {\text {Tr}}({C^{-1}}{C^{-1}} \text {diag}(C \pi )) - s \right) \end{aligned}$$

From $(A \otimes B)(C \otimes D) = (AC) \otimes (BD)$ it follows that $C_{a}(n) C_{a}(n) = C_{a^2 + (1-a)^2}(n)$. From this and Theorem 2 and Corollary 1 we get that the entry $(C^{-1}C^{-1})_{0,0} = f(n, a)$ where

$$ f(n, a) = \left( \frac{a^2+(1-a)^2}{(2a-1)^2}\right) ^n. $$

Furthermore, from Proposition 3 the diagonal entries of $C^{-1}C^{-1}$ are all $f(n,a)$. Combining this, that ${\text {Tr}}(AB) = \sum _{i,j}(A \odot B^T)_{i,j}$, and $\sum _x C_x \pi = 1$,

$$\begin{aligned} {\text {Tr}}(C^{-1}C^{-1} \text {diag}(C \pi ))&= \sum _x f(n,a) C_x \pi = f(n,a) \sum _x C_x \pi = f(n,a), \end{aligned}$$

and consequently, ${\text {Tr}}({\text {cov}}(\hat{\pi }_x(m))) = m^{-1} \left( f(n,a) - s\right)$.   $\square$

Proof of Theorem 3: We have that

$$\begin{aligned} {\text {Tr}}({\text {cov}}(\hat{\pi }^*))&= m^{-1}{\text {Tr}}(\text {diag}(\pi ) - \pi \pi ^T) = m^{-1}(1 - s). \end{aligned}$$

From Lemma 1 and Proposition 1 we get that $L= f_L(s) = \frac{c - s}{(1 - s)}$ for $c = \left( \frac{a^2+(1-a)^2}{(2a-1)^2}\right) ^n.$ From $0 \le p_x \le 1$ and $\sum _x p_x = 1$, $s$ has a minimum when $p_x = 1/2^n$ for all $x$, and maximum when $p_x = 1$ for a fixed $x$, and $p_y = 0$ for $y \ne x$. These values are then $\frac{2^n}{(2^n)^2} = 2^{-n}$ and $1$, respectively. The $m$’th derivative of $f_L(s) = \frac{c-s}{1-s}$ wrt. $0 \le s < 1$ is $f_L^{(m)}(s) = \frac{m!}{(1-s)^m} \left( f_L(s) - 1 \right)$. The loss $f_L$ therefore achieves its minimum at $f_L(2^{-n})$.   $\square$

Proof of Proposition 4 (sketch): The $m$’th derivative of $f_L(s) = \frac{c-s}{1-s}$ wrt. $0 \le s < 1$ is $f_L^{(m)}(s) = \frac{m!}{(1-s)^m} \left( f_L(s) - 1 \right)$. Since $c \ge 1$, $f^{(m)}_L \ge 0$ for all $m > 0$. In particular, we have that $f_L$ is convex, as is $f_L^{(m)}$ for all $m$. Using the expectation for a first order Taylor approximation for convex $f_L$ we have that for random variable $S$

where $\lambda = \max _{x \in \mathcal {I}} f^{(2)}_L(x) \ge 0$ for suitable interval $\mathcal {I}$. Dividing $(*)$ by $f_L({\text {E}}(S)) = L(n)$, we get

$$ 1 \le \frac{{\text {E}}(f_L(S))}{f_L({\text {E}}(S))} \le 1 + \delta , $$

where

$$ \delta = \frac{\lambda {\text {Var}}(S)}{2 f_L({\text {E}}(S))}. $$

Let $S = \pi ^T\pi$. Recalling that $c = c(a)^n$ and expanding both numerator and denominator at $n=3$ (where the minimum occurs since $f_L$ is increasing and ${\text {Var}}(S)$ and ${\text {E}}(S)$ are both decreasing in $n$), we see that $\delta (n) \in O(2^{-3n})$. Applying Chebyshev’s inequality, we have that $P(S \ge {\text {E}}(S) + 10 {\text {Var}}(S)^{\frac{1}{2}}) \le 0.01$. Evaluating $\delta$ at ${\text {E}}(S) + 10 {\text {Var}}(S)^{\frac{1}{2}}$ and $n=3$, we arrive at the numerical bound.   $\square$

Proof of Proposition 5: Let the computation of $Z \otimes R$ require $t_f(n^2)$ time for $2\times 2$ matrix $Z$ and $R$ of size $n \times n$. Then we can compute $R_{a,b}(n)$ at a time cost of $t(n) = t_f(2^{2(n-1)}) + t(n-1) = \sum _{i = 0}^n t_f(2^{2i}) = \sum _{i = 0}^n t_f(4^i).$ Letting $t_f(n) = k 4 n$ for some $k$, then $t(n) = 4k\sum _{i=0}^n 2^{2i} = 4k \sum _{i=0}^n 4^{i} = 4k (1 + \frac{1-4^n}{1-4}) = 4k (1 + \frac{4^n-1}{3}).$ Now we have that $t(n) = O(4^n) = O({2^n}^2) = O(|R_{a,b}(n)|)$. In other words, the singly recursive algorithm $R_{a,b}(n)$ is linear in the time in the number of elements of the output matrix as we can perform $t_f$ in linear time in the size of input $R$, in fact we can expect that the Kronecker product can be implemented with $k \le 3$, due to reading, multiplication, and writing.   $\square$