A Novel Evaluation Metric for Synthetic Data Generation

Abstract

Differentially private algorithmic synthetic data generation (SDG) solutions take input datasets $D_p$ consisting of sensitive, private data and generate synthetic data $D_s$ with similar qualities. The importance of such solutions is increasing both because more and more people realize how much data is collected about them and used in machine learning contexts, as well as a consequence of newly introduced data privacy regulations, e.g. the EU’s General Data Protection Regulation (GDPR). We aim to develop a novel and composite SDG evaluation metric which takes into account macro-statistical dataset similarities and data utility in machine learning tasks against privacy boundaries of the synthetic data. We formalize the mathematical foundations for quantitatively measuring both the statistical similarities and the data utility of synthetic data. We use two well-known datasets containing (potentially) personally identifiable information as inputs ($D_p$) and existing SDG algorithms PrivBayes and DPGroupFields to generate synthetic data ($D_s$) based on them. We then test our evaluation metric for different values of privacy budget $\epsilon$. Based on our experiments we conclude that the proposed composite evaluation metric is appropriate for quantitatively measuring the quality of synthetic data generated by different SDG solutions and possesses an expected sensitivity to various privacy budget values.