We present a privacy preserving machine learning method for images that separates task-relevant information from task-irrelevant information. Our primary hypothesis is that by revealing the minimal number of pixels required for a task we can provide the most privacy preserving guarantees. Specifically, we propose an adversarial method that masks out task-irrelevant information from an image for preserving privacy. The proposed method only uses task-specific label information and no privacy annotations such as identity of the subject, gender, race, etc., are required. We validate the proposed method on face attribute prediction on the CelebA dataset and emotion recognition on the FER+ dataset, showing that we can preserve visual privacy with little degradation in the task performance.