A Recommender System for Efficient Implementation of Privacy Preserving Machine Learning Primitives Based on FHE

Abstract

With the increased dependence on cloud computing, there is growing concern for privacy of data that is stored and processed on third party cloud service providers. Of many solutions that achieve privacy preserving computations, fully homomorphic encryption (FHE) is a promising direction. FHE has several applications that can be used to perform computations on encrypted data without decrypting them. In this paper, we focus on realizing privacy preserving machine learning (PPML) using FHE. Our prime motivation behind choosing PPML is the increased use of machine learning algorithms on end-user’s data for predictions or classification, where privacy of end-user’s data is at stake. Given the importance of PPML and FHE, we formulate a recommender system that enables machine learning experts who are new to cryptography to efficiently realize a machine learning application in privacy preserving manner. We formulate the recommender system as a multi objective multi constraints optimization problem along with a simpler single objective multi constraint optimization problem. We solve this optimization using TOPSIS based on experimental analysis performed on three prominent FHE libraries HElib, SEAL and HEAAN from the PPML perspective. We present the observations on the performance parameters such as elapsed time and memory usage for the primitive machine learning algorithms such as linear regression and logistic regression. We also discuss the technical issues in making the FHE schemes practically deployable and give insights into selection of parameters to efficiently implement PPML algorithms. We observe that our estimates for matrix multiplication and linear regression correlate with the experimental analysis when assessed using an optimizer. The proposed recommendation system can be used in FHE compilers to facilitate optimal implementation of PPML applications.

Regular Submission, CSCML 2020.

Appendix

In this section, we describe the data movement in slots in packed linear regression algorithm.

1.1 Packed Linear Regression

Input matrix X is encrypted in a single ciphertext where $m \times n < N/2$. The model values w are encoded and encrypted in a single ciphertext by duplicating the values m times.

$$ X_{m \times n}= \begin{bmatrix} x_{11} &{} \dots &{} x_{1n} \\ x_{21} &{} \dots &{} x_{2n} \\ . &{} \dots &{} . \\ . &{} \dots &{} . \\ x_{m1} &{} \dots &{} x_{mn} \\ \end{bmatrix}_{m \times n} w_{n \times 1}= \begin{bmatrix} w_{1} \\ w_{2} \\ . \\ . \\ w_{n} \\ \end{bmatrix}_{n \times 1} $$

$$ \texttt {Enc}[X_{m \times n}]= \texttt {Enc}\, \begin{bmatrix} x_{11} &{} \dots &{} x_{1n} \\ x_{21} &{} \dots &{} x_{2n} \\ . &{} \dots &{} . \\ . &{} \dots &{} . \\ x_{m1} &{} \dots &{} x_{mn} \\ \end{bmatrix}_{m \times n} \texttt {Enc}[w_{n \times 1}]= \texttt {Enc}\, \begin{bmatrix} w_{1} &{} w_{2} &{} \dots w_{n} \\ w_{1} &{} w_{2} &{} \dots w_{n} \\ . \\ . \\ w_{1} &{} w_{2} &{} \dots w_{n} \\ \end{bmatrix}_{m \times n} $$

The subscripts are written only to provide clarity of how values in slots need to be mapped for operations to be done correctly.

The output matrix is encoded by placing each $y_i$ value at index multiples of n.

$$ Y_{m \times 1}= \begin{bmatrix} y_{1} \\ y_{2} \\ . \\ . \\ y_{m} \\ \end{bmatrix}_{m \times 1} \implies \texttt {Enc}[Y_{m \times 1}]= \texttt {Enc}\, \begin{bmatrix} y_{1} &{} 0 &{} \dots &{} 0 \\ y_{2} &{} 0 &{} \dots &{} 0 \\ . \\ . \\ y_{m} &{} 0 &{} \dots &{} 0 \\ \end{bmatrix}_{m \times n} $$

Now we begin the computation of following LR algorithm:

$$\theta _{update}=\theta - \alpha [Y - wX]X^{T}$$

• Computing wX. Multiplying two ciphertexts X and w multiplies values in each slots respectively. To complete the multiplication, rotate the slots $n-1$ times and add to the original result.

  To remove the garbage values in other slots, multiply by 1 matrix defined as follows:

  $$ \texttt {Enc}[wX]= \texttt {Enc}\, \begin{bmatrix} x_{1} \times W &{} * &{} \dots &{} * \\ x_{2} \times W &{} * &{} \dots &{} * \\ . \\ . \\ x_{m} \times W &{} * &{} \dots &{} * \\ \end{bmatrix}_{m \times n} \times \texttt {Enc}\, \begin{bmatrix} 1 &{} 0 &{} \dots &{} 0 \\ 1 &{} 0 &{} \dots &{} 0 \\ . \\ . \\ 1 &{} 0 &{} \dots &{} 0 \\ \end{bmatrix}_{m \times n} = \texttt {Enc}\, \begin{bmatrix} x_{1} \times W &{} 0 &{} \dots &{} 0 \\ x_{2} \times W &{} 0 &{} \dots &{} 0 \\ . \\ . \\ x_{m} \times W &{} 0 &{} \dots &{} 0 \\ \end{bmatrix}_{m \times n} $$

• Compute $Y-wX$. Since the $y_i$ values are in their respective slots, we can proceed with subtraction operation.

  The result of the subtraction is as follows:

  $$ \texttt {Enc}\, \begin{bmatrix} A_{1} &{} 0 &{} \dots &{} 0 \\ A_{2} &{} 0 &{} \dots &{} 0 \\ . \\ . \\ A_{m} &{} 0 &{} \dots &{} 0 \\ \end{bmatrix}_{m \times n} where\ A_{i}=y_{i}-(x_{i} \times W) $$

• Compute $[Y-wX] \times X^{T}$. As we can see, there is a mismatch in the values in each slots of $Y-wX$ and $X^T$ and we cannot simply proceed with the multiplication. Now we have to move the data between the slots to get the following transformation:

  $$ \texttt {Enc}\, \begin{bmatrix} A_{1} &{} A_{2} &{} \dots &{} A_{m} \\ A_{1} &{} A_{2} &{} \dots &{} A_{m} \\ . \\ . \\ A_{1} &{} A_{2} &{} \dots &{} A_{m} \\ \end{bmatrix}_{n \times m} $$

  This is done using the following steps which takes much computation time. Let $\texttt {B=(Y-wX)}$, $\texttt {sum} \leftarrow \texttt {Enc(0)}$, $\texttt {A} \leftarrow \texttt {Enc(0)}$. Let set(i) denote function which sets $i^{th}$ slot of vector as 1 and remaining as 0 and gets an encryption of this vector

  

  set(i) is used to get the $A_i$ value and then it is rotated to the appropriate location and added to sum. This completes getting all m values together. The we again rotate these values and add n − 1 times to complete the transformation. Efficient on-the fly packing will be useful to solve this issue of data movement in between operations.

  $$ \texttt {Enc}\, \begin{bmatrix} A_{1} \times x_{11} &{} A_{2} \times x_{21} &{} \dots &{} A_{m} \times x_{m1} \\ A_{1} \times x_{12} &{} A_{2} \times x_{22} &{} \dots &{} A_{m} \times x_{m2} \\ . \\ . \\ A_{1} \times x_{1n} &{} A_{2} \times x_{2n} &{} \dots &{} A_{m} \times x_{mn} \\ \end{bmatrix}_{n \times m} $$

  Now we perform rotation of matrix to complete multiplication and multiply with 1 matrix to get:

  $$ \texttt {Enc}\, \begin{bmatrix} A_{1} \times X_{1} &{} 0 &{} \dots &{} 0 \\ A_{2} \times X_{2} &{} 0 &{} \dots &{} 0 \\ . &{} 0 &{} \dots &{} 0 \\ . &{} 0 &{} \dots &{} 0 \\ A_{n} \times X_{n} &{} 0 &{} \dots &{} 0 \\ \end{bmatrix}_{n \times m} where\ A_{i} X_{i} = \sum _{j=1}^{m}{A_{j}x_{jk}\ for\ k \in [1,n]} $$

• Multiply $[Y-wX]X^{T}$ with $\alpha$, the learning rate Now the values of gradient have to be mapped to respective slots for theta updation as follows:

  $$ \theta _{update}= \texttt {Enc}\, \begin{bmatrix} w_{1} - A_{1} X_{1} \alpha &{} w_{2} - A_{2} X_{2} \alpha &{} \dots &{} w_{n} - A_{n} X_{n} \alpha \\ w_{1} - A_{1} X_{1} \alpha &{} w_{2} - A_{2} X_{2} \alpha &{} \dots &{} w_{n} - A_{n} X_{n} \alpha \\ . &{} . &{} \dots &{} . \\ . &{} . &{} \dots &{} . \\ w_{1} - A_{1} X_{1} \alpha &{} w_{2} - A_{2} X_{2} \alpha &{} \dots &{} w_{n} - A_{n} X_{n} \alpha \\ \end{bmatrix}_{m \times n} $$

The algorithm is repeated until $\theta$ converges.