Poisoning-Attack Detection Using an Auto-encoder for Deep Learning Models

Abstract

Modern Deep Learning DL models can be trained in various ways, including incremental learning. The idea is that a user whose model has been trained on his own data will perform better on new data. The model owner can share its model with other users, who can then train it on their data and return it to the model owner. However, these users can perform poisoning attacks PA by modifying the model’s behavior in the attacker’s favor. In the context of incremental learning, we are interested in detecting a DL model for image classification that has undergone a poisoning attack. To perform such attacks, an attacker can, for example, modify the labels of some training data, which is then used to fine-tune the model in such a way that the attacked model will incorrectly classify images similar to the attacked images, while maintaining good classification performance on other images. As a countermeasure, we propose a poisoned model detector that is capable of detecting various types of PA attacks. This technique exploits the reconstruction error of a machine learning-based auto-encoder AE trained to model the distribution of the activation maps from the second-to-last layer of the model to protect. By analyzing AE reconstruction errors for some given inputs, we demonstrate that a PA can be distinguished from a fine-tuning operation that can be used to improve classification performance. We demonstrate the performance of our method on a variety of architectures and in the context of a DL model for mass cancer detection in mammography images.

This work was partly supported by the Joint Laboratory SePEMeD, ANR-13-LAB2-0006-01, and the French ANR via the European program “Preservation of R&D employment in the framework of the French recovery plan” under the reference ANR-21-PRRD-0027-01.