Xiaoyun Xu
ABSTRACT
Vision Transformer (ViT) shows superior performance on various tasks, but, similar to other deep learning techniques, it is vulnerable to adversarial attacks. Due to the differences between ViT and traditional CNNs, previous works designed new adversarial training methods as defenses according to the design of ViT, such as blocking attention to individual patches or dropping embeddings with low attention. However, these methods usually focus on fine-tuning stage or the training of the model itself. Improving robustness at the pre-training stage, especially with lower overhead, has yet to be thoroughly investigated. This paper proposes a novel method, Adv-MAE, which increases adversarial robustness by masked adversarial pre-training without a penalty to performance on clean data. We design a simple method to generate adversarial perturbation for the autoencoder, as the autoencoder does not provide classification results. Then, we use masked inputs with perturbation to conduct adversarial training for the autoencoder. The pre-trained autoencoder can be used to build a ViT with better robustness. Our experimental results show that, when using adversarial fine-tuning, Adv-MAE offers better accuracy under adversarial attack than the non-adversarial pre-training method (3.46% higher on CIFAR-10, 1.12% higher on Tiny ImageNet). It also shows better accuracy on clean data (4.94% higher on CIFAR-10, 1.74% higher on Tiny ImageNet), meaning Adv-MAE does not deteriorate performance on clean inputs. In addition, masked pre-training also shows much lower time consumption at each training epoch.
- Yutong Bai, Jieru Mei, Alan L Yuille, and Cihang Xie. 2021. Are Transformers more robust than CNNs?. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 26831--26843.Google Scholar
- Philipp Benz, Soomin Ham, Chaoning Zhang, Adil Karjauv, and In So Kweon. 2021. Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs. British Machine Vision Conference (BMVC) (2021).Google Scholar
- Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. 2021. Understanding Robustness of Transformers for Image Classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 10231--10241.Google ScholarCross Ref
- Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. 2020. Adversarial robustness: From self-supervised pre-training to fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 699--708.Google ScholarCross Ref
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=YicbFdNTTyGoogle Scholar
- Yonggan Fu, Shunyao Zhang, Shang Wu, Cheng Wan, and Yingyan Lin. 2021. Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?. In International Conference on Learning Representations.Google Scholar
- George Gondim-Ribeiro, Pedro Tabacof, and Eduardo Valle. 2018. Adversarial attacks on variational autoencoders. arXiv preprint arXiv:1806.04646 (2018).Google Scholar
- Jindong Gu, Volker Tresp, and Yao Qin. 2022. Are Vision Transformers Robust to Patch Perturbations?. In Computer Vision - ECCV 2022, Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer Nature Switzerland, Cham, 404--421.Google ScholarDigital Library
- Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 16000--16009.Google ScholarCross Ref
- Dan Hendrycks, Kimin Lee, and Mantas Mazeika. 2019. Using pre-training can improve model robustness and uncertainty. In International conference on machine learning. PMLR, 2712--2721.Google Scholar
- Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. 2020. Robust Pre-Training by Adversarial Contrastive Learning. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 16199--16210.Google Scholar
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=rJzIBfZAbGoogle Scholar
- Yichuan Mo, Dongxian Wu, Yifei Wang, Yiwen Guo, and Yisen Wang. 2022. When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture. In NeurIPS.Google Scholar
- Tianyu Pang, Min Lin, Xiao Yang, Jun Zhu, and Shuicheng Yan. 2022. Robustness and Accuracy Could be Reconcilable by (Proper) Definition. In International Conference on Machine Learning (ICML).Google Scholar
- Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Robustness May Be at Odds with Accuracy. In International Conference on Learning Representations.Google Scholar
- Shaoru Wang, Jin Gao, Zeming Li, Xiaoqin Zhang, and Weiming Hu. 2023. A Closer Look at Self-Supervised Lightweight Vision Transformers. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 35624--35641. https://proceedings.mlr.press/v202/wang23e.htmlGoogle Scholar
- Boxi Wu, Jindong Gu, Zhifeng Li, Deng Cai, Xiaofei He, and Wei Liu. 2022. Towards Efficient Adversarial Training on Vision Transformers. In Computer Vision - ECCV 2022, Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer Nature Switzerland, Cham, 307--325.Google Scholar
Index Terms
- Poster: Boosting Adversarial Robustness by Adversarial Pre-training
Recommendations
A hybrid adversarial training for deep learning model and denoising network resistant to adversarial examples
AbstractDeep neural networks (DNNs) are vulnerable to adversarial attacks that generate adversarial examples by adding small perturbations to the clean images. To combat adversarial attacks, the two main defense methods used are denoising and adversarial ...
GAN-Based Fusion Adversarial Training
Knowledge Science, Engineering and ManagementAbstractIn the field of artificial intelligence security, adversarial machine learning has made breakthroughs. However, it is still vulnerable to attacks under a wide variety of adversarial samples, and adversarial training is a very effective method ...
Learning Activation Functions for Adversarial Attack Resilience in CNNs
Artificial Intelligence and Soft ComputingAbstractAdversarial attacks on convolutional neural networks (CNNs) have been a serious concern in recent years, as they can cause CNNs to produce inaccurate predictions. Through our analysis of training CNNs with adversarial examples, we discovered that ...
Comments