Abstract
Catastrophic overfitting (CO) poses a significant challenge to fast adversarial training (FastAT), particularly at large perturbation scales, leading to dramatic reductions in adversarial test accuracy. Our analysis of existing FastAT methods shows that CO is accompanied by abrupt and irregular fluctuations in loss convergence, indicating that a stable training dynamic is key to preventing CO. Therefore, we propose a training model that uses the Douglas-Rachford (DR) splitting technique to ensure a balanced and consistent training progression, effectively counteracting CO. The DR splitting technique, known for its ability to solve complex optimization problems, offering a distinct advantage over classical FastAT methods by providing a smoother loss convergence. This is achieved without resorting to complex regularization or incurring the computational costs associated with double backpropagation, presenting an efficient solution to enhance adversarial robustness. Our comprehensive evaluation conducted across standard datasets, demonstrates that our DR splitting-based model not only improves adversarial robustness but also achieves this with remarkable efficiency compared to various FastAT methods. This efficiency is particularly observed under conditions involving long training schedules and large adversarial perturbations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Addepalli, S., Jain, S., Sriramanan, G., Venkatesh Babu, R.: Scaling adversarial training to large perturbation bounds. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13665, pp. 301–316. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20065-6_18
Andriushchenko, M., Croce, F., Flammarion, N., Hein, M.: Square attack: a query-efficient black-box adversarial attack via random search. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12368, pp. 484–501. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58592-1_29
Andriushchenko, M., Flammarion, N.: Understanding and improving fast adversarial training. Adv. Neural. Inf. Process. Syst. 33, 16048–16059 (2020)
Applegate, D., Hinder, O., Lu, H., Lubin, M.: Faster first-order primal-dual methods for linear programming using restarts and sharpness. Math. Program. 201(1), 133–184 (2023)
Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 39–57 (2017)
Chambolle, A., Contreras, J.P.: Accelerated Bregman primal-dual methods applied to optimal transport and Wasserstein barycenter problems. SIAM J. Math. Data Sci. 4(4), 1369–1395 (2022)
Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: International Conference on Machine Learning, pp. 2206–2216 (2020)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Dong, Y., et al.: Boosting adversarial attacks with momentum. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9185–9193 (2018)
Golgooni, Z., Saberi, M., Eskandar, M., Rohban, M.H.: ZeroGrad: mitigating and explaining catastrophic overfitting in FGSM adversarial training. arXiv preprint arXiv:2103.15476 (2021)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: International Conference on Learning Representations (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part IV. LNCS, vol. 9908, pp. 630–645. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_38
He, Z., Li, T., Chen, S., Huang, X.: Investigating catastrophic overfitting in fast adversarial training: a self-fitting perspective. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2313–2320 (2023)
Jia, X., et al.: Prior-guided adversarial initialization for fast adversarial training. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13664, pp. 567–584. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19772-7_33
de Jorge Aranda, P., et al.: Make some noise: reliable and efficient single-step adversarial training. Adv. Neural. Inf. Process. Syst. 35, 12881–12893 (2022)
Kurakin, A., Goodfellow, I., Bengio, S.: Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236 (2016)
Li, T., Wu, Y., Chen, S., Fang, K., Huang, X.: Subspace adversarial training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13409–13418 (2022)
Liang, J., Fadili, J., Peyré, G.: Local convergence properties of Douglas-Rachford and alternating direction method of multipliers. J. Optim. Theory Appl. 172, 874–913 (2017)
Lindbäck, J., Wang, Z., Johansson, M.: Bringing regularized optimal transport to lightspeed: a splitting method adapted for GPUs. Adv. Neural Inf. Process. Syst. 36 (2023)
Liu, X., Chakraborty, S., Sun, Y., Huang, F.: Rethinking adversarial policies: a generalized attack formulation and provable defense in RL. In: International Conference on Learning Representations (2024)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations (2018)
Mai, V.V., Lindbäck, J., Johansson, M.: A fast and accurate splitting method for optimal transport: analysis and implementation. In: International Conference on Learning Representations (2022)
Moosavi-Dezfooli, S.M., Fawzi, A., Uesato, J., Frossard, P.: Robustness via curvature regularization, and vice versa. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9078–9086 (2019)
Park, G.Y., Lee, S.W.: Reliably fast adversarial training via latent adversarial perturbation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7758–7767 (2021)
Poon, C., Liang, J.: Trajectory of alternating direction method of multipliers and adaptive acceleration. Adv. Neural Inf. Process. Syst. 32 (2019)
Qin, C., et al.: Adversarial robustness through local linearization. Adv. Neural Inf. Process. Syst. 32 (2019)
Rocamora, E.A., Liu, F., Chrysos, G.G., Olmos, P.M., Cevher, V.: Efficient local linearity regularization to overcome catastrophic overfitting. In: International Conference on Learning Representations (2024)
Shaeiri, A., Nobahari, R., Rohban, M.H.: Towards deep learning models resistant to large perturbations. arXiv preprint arXiv:2003.13370 (2020)
Shafahi, A., et al.: Adversarial training for free! Adv. Neural Inf. Process. Syste. 32 (2019)
Song, C., He, K., Wang, L., Hopcroft, J.E.: Improving the generalization of adversarial training with domain adaptation. In: International Conference on Learning Representations (2019)
Sriramanan, G., Addepalli, S., Baburaj, A., et al.: Guided adversarial attack for evaluating and enhancing adversarial defenses. Adv. Neural. Inf. Process. Syst. 33, 20297–20308 (2020)
Sriramanan, G., Addepalli, S., Baburaj, A., et al.: Towards efficient and effective adversarial training. Adv. Neural. Inf. Process. Syst. 34, 11821–11833 (2021)
Themelis, A., Patrinos, P.: Douglas-Rachford splitting and ADMM for nonconvex optimization: tight convergence results. SIAM J. Optim. 30(1), 149–181 (2020)
Tsiligkaridis, T., Roberts, J.: Understanding and increasing efficiency of Frank-Wolfe adversarial training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 50–59 (2022)
Wong, E., Rice, L., Kolter, J.Z.: Fast is better than free: revisiting adversarial training. In: International Conference on Learning Representations (2020)
Wu, D., Xia, S.T., Wang, Y.: Adversarial weight perturbation helps robust generalization. Adv. Neural Inf. Process. Syst. (2020)
Xie, Y., Li, Z., Shi, C., Liu, J., Chen, Y., Yuan, B.: Enabling fast and universal audio adversarial attack using generative model. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14129–14137 (2021)
Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: International Conference on Machine Learning, pp. 7472–7482 (2019)
Zhang, Y., Zhang, G., Khanduri, P., Hong, M., Chang, S., Liu, S.: Revisiting and advancing fast adversarial training through the lens of bi-level optimization. In: International Conference on Machine Learning, pp. 26693–26712 (2022)
Zhao, M., Zhang, L., Kong, Y., Yin, B.: Fast adversarial training with smooth convergence. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4720–4729 (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zareapoor, M., Shamsolmoali, P. (2025). Rethinking Fast Adversarial Training: A Splitting Technique to Overcome Catastrophic Overfitting. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15136. Springer, Cham. https://doi.org/10.1007/978-3-031-73229-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-73229-4_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-73228-7
Online ISBN: 978-3-031-73229-4
eBook Packages: Computer ScienceComputer Science (R0)