Abstract
Adversarial training (AT) has been shown to be one of the most effective ways to protect deep neural networks (DNNs) from adversarial attacks . However, the phenomenon of robust overfitting, that is, the robustness will drop sharply at a certain stage, always exists in the AT process. In order to obtain a robust model, it is important to reduce this robust generalization gap. In this paper, we delve into robust overfitting from a new perspective. We observe that consistency regularization, a popular technique in semi-supervised learning, has similar goals to AT and can help mitigate robust overfitting. We empirically verify this observation and find that most previous solutions are implicitly linked to consistency regularization. Inspired by this, we introduce a new AT solution that integrates consistency regularization and mean teacher (MT) strategy into AT. Specifically, we introduce a teacher model derived from the average weights of the student models in the training step. We then design a consistency loss function to make the predicted distribution of the student model on adversarial samples consistent with the predicted distribution of the teacher model on clean samples. Experiments show that our proposed method can effectively mitigate robust overfitting and improve the robustness of DNN models against common adversarial attacks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Athalye, A., Carlini, N., Wagner, D.: Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples. In: International Conference on Machine Learning, pp. 274–283. PMLR (2018)
Carlini, N., Wagner, D.A.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, 22–26 May 2017, pp. 39–57. IEEE Computer Society (2017)
Carmon, Y., Raghunathan, A., Schmidt, L., Liang, P., Duchi, J.C.: Unlabeled data improves adversarial robustness. arXiv preprint arXiv:1905.13736 (2019)
Chen, T., Zhang, Z., Liu, S., Chang, S., Wang, Z.: Robust overfitting may be mitigated by properly learned smoothening. In: International Conference on Learning Representations (2020)
Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: ICML (2020)
Dong, Y., et al.: Exploring memorization in adversarial training. arXiv preprint arXiv:2106.01606 (2021)
Goodfellow, I., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572
Goodfellow, I., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572 (2014)
Huang, L., Zhang, C., Zhang, H.: Self-adaptive training: beyond empirical risk minimization. In: Advances in Neural Information Processing Systems, vol. 33 (2020)
Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 (2016)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017)
Papernot, N., McDaniel, P., Wu, X., Jha, S., Swami, A.: Distillation as a defense to adversarial perturbations against deep neural networks. In: 2016 IEEE symposium on security and privacy (SP), pp. 582–597. IEEE (2016)
Qiu, H., Dong, T., Zhang, T., Lu, J., Memmi, G., Qiu, M.: Adversarial attacks against network intrusion detection in IoT systems. IEEE Internet Things J. 8(13), 10327–10335 (2021)
Qiu, M., Qiu, H.: Review on image processing based adversarial example defenses in computer vision. In: 2020 IEEE 6th Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing, (HPSC) and IEEE International Conference on Intelligent Data and Security (IDS), pp. 94–99. IEEE (2020)
Rice, L., Wong, E., Kolter, Z.: Overfitting in adversarially robust deep learning. In: International Conference on Machine Learning, pp. 8093–8104. PMLR (2020)
Szegedy, C.: Intriguing properties of neural networks, arXiv preprint arXiv:1312.6199
Tack, J., Yu, S., Jeong, J., Kim, M., Hwang, S.J., Shin, J.: Consistency regularization for adversarial robustness. arXiv preprint arXiv:2103.04623 (2021)
Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. arXiv preprint arXiv:1703.01780 (2017)
Tramer, F., Carlini, N., Brendel, W., Madry, A.: On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002.08347 (2020)
Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial robustness requires revisiting misclassified examples. In: ICLR (2020)
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: International Conference on Machine Learning, pp. 7472–7482. PMLR (2019)
Zhang, S., Gao, H., Rao, Q.: Defense against adversarial attacks by reconstructing images. IEEE Trans. Image Process. 30, 6117–6129 (2021)
Acknowledgments
This work was supported in part by the National Nature Science Foundation of China under Grant 61972306 and in part by the Zhejiang Laboratory under Grant 2021KD0AB03.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, S., Gao, H., Zhou, Y., Wu, Z., Tang, Y. (2022). Consistency Regularization Helps Mitigate Robust Overfitting in Adversarial Training. In: Memmi, G., Yang, B., Kong, L., Zhang, T., Qiu, M. (eds) Knowledge Science, Engineering and Management. KSEM 2022. Lecture Notes in Computer Science(), vol 13370. Springer, Cham. https://doi.org/10.1007/978-3-031-10989-8_58
Download citation
DOI: https://doi.org/10.1007/978-3-031-10989-8_58
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-10988-1
Online ISBN: 978-3-031-10989-8
eBook Packages: Computer ScienceComputer Science (R0)