pPATE: A Pragmatic Private Aggregation of Teacher Ensembles Framework by Sparse Vector Technique Based Differential Privacy, Paillier Cryptosystem and Human-in-the-loop

Abstract

With advances of deep learning models, Artificial Intelligence (AI) has been applied into various fields to aid human. Some domains where sensitive data with privacy concerns are pivotal, for example medical care, are no exception. Dealing with that, a private learning framework satisfying differential privacy - a gold standard to protect privacy, namely Private Aggregation of Teacher Ensembles (PATE) has gained popularity. However, this framework needs to train a large number of models in disjoint private training datasets, thereby in plethora of cases, it can not be leveraged. In this paper, we propose pPATE - a pragmatical framework that is based on PATE but it uses a sparse vector technique to achieve differential privacy and demonstrate that with small manual efforts of human (expert) in the development loop, our solution can train privacy-preserving models that have approximate accuracy as ground-truth models. Moreover, we extend PATE framework pragmatically in a distributed setting so that it not only aggregates privately but also secures confidentiality and privacy when multi-parties collaborate.