Abstract
Deep neural networks are vulnerable to adversarial examples, which exploit imperceptible perturbations to mislead classifiers. To improve adversarial robustness, recent methods have focused on estimating mutual information (MI). However, existing MI estimators struggle to provide stable and reliable estimates in high-dimensional data. To this end, we propose a Copula Entropic MI Estimator (CE\(^2\)) to address these limitations. CE\(^2\) leverages copula entropy to estimating MI in high dimensions, allowing target models to harness information from both clean and adversarial examples to withstand attacks. Our empirical experiments demonstrate that CE\(^2\) achieves a trade-off between variance and bias in MI estimates, resulting in stable and reliable estimates. Furthermore, the defense algorithm based on CE\(^2\) significantly enhances adversarial robustness against multiple attacks. The experimental results underscore the effectiveness of CE\(^2\) and its potential for improving adversarial robustness.
This work was supported in part by the National Natural Science Foundation of China (Grant No. 62006097, U1836218), in part by the Natural Science Foundation of Jiangsu Province (Grant No. BK20200593) and in part by the China Postdoctoral Science Foundation (Grant No. 2021M701456).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Belghazi, M.I., et al.: Mutual information neural estimation. In: ICML. Proceedings of Machine Learning Research, vol. 80, pp. 530–539. PMLR (2018)
Carlini, N., Wagner, D.A.: Towards evaluating the robustness of neural networks. In: IEEE Symposium on Security and Privacy, pp. 39–57. IEEE Computer Society (2017)
Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: ICML. Proceedings of Machine Learning Research, vol. 119, pp. 2206–2216. PMLR (2020)
Ding, G.W., Lui, K.Y.C., Jin, X., Wang, L., Huang, R.: On the sensitivity of adversarial robustness to input data distributions. In: ICLR (Poster). OpenReview.net (2019)
Duan, Y., Lu, J., Zheng, W., Zhou, J.: Deep adversarial metric learning. IEEE Trans. Image Process. 29, 2037–2051 (2020)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: ICLR (Poster) (2015)
Hu, C., Li, Y., Feng, Z., Wu, X.: Attention-guided evolutionary attack with elastic-net regularization on face recognition. Pattern Recogn. 109760 (2023)
Hu, C., Xu, H.Q., Wu, X.J.: Substitute meta-learning for black-box adversarial attack. IEEE Sig. Process. Lett. 29, 2472–2476 (2022). https://doi.org/10.1109/LSP.2022.3226118
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examples are not bugs, they are features. In: NeurIPS, pp. 125–136 (2019)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1106–1114 (2012)
Li, H., Wu, X., Kittler, J.: MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Trans. Image Process. 29, 4733–4746 (2020)
Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks. In: CVPR, pp. 510–519. Computer Vision Foundation/IEEE (2019)
Liao, F., Liang, M., Dong, Y., Pang, T., Hu, X., Zhu, J.: Defense against adversarial attacks using high-level representation guided denoiser. In: CVPR, pp. 1778–1787. Computer Vision Foundation/IEEE Computer Society (2018)
Lin, S., et al.: Towards optimal structured CNN pruning via generative adversarial learning. In: CVPR, pp. 2790–2799. Computer Vision Foundation/IEEE (2019)
Linsker, R.: Self-organization in a perceptual network. Computer 21(3), 105–117 (1988)
Ma, J., Sun, Z.: Mutual information is copula entropy. CoRR abs/0808.0845 (2008)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: ICLR (Poster). OpenReview.net (2018)
McAllester, D., Stratos, K.: Formal limitations on the measurement of mutual information. In: AISTATS. Proceedings of Machine Learning Research, vol. 108, pp. 875–884. PMLR (2020)
Naseer, M., Khan, S.H., Hayat, M., Khan, F.S., Porikli, F.: A self-supervised approach for adversarial robustness. In: CVPR, pp. 259–268. Computer Vision Foundation/IEEE (2020)
Nguyen, X., Wainwright, M.J., Jordan, M.I.: Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory 56(11), 5847–5861 (2010)
Noshad, M., Zeng, Y., III, A.O.H.: Scalable mutual information estimation using dependence graphs. In: ICASSP, pp. 2962–2966. IEEE (2019)
van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. CoRR abs/1807.03748 (2018)
Paninski, L.: Estimation of entropy and mutual information. Neural Comput. 15(6), 1191–1253 (2003)
Papernot, N., McDaniel, P.D., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: EuroS &P, pp. 372–387. IEEE (2016)
Poole, B., Ozair, S., van den Oord, A., Alemi, A.A., Tucker, G.: On variational bounds of mutual information. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 5171–5180. PMLR (2019)
Ravens, B.: An introduction to copulas. Technometrics 42(3), 317 (2000)
Rony, J., Hafemann, L.G., Oliveira, L.S., Ayed, I.B., Sabourin, R., Granger, E.: Decoupling direction and norm for efficient gradient-based L2 adversarial attacks and defenses. In: CVPR, pp. 4322–4330. Computer Vision Foundation/IEEE (2019)
Rubenstein, P.K., Bousquet, O., Djolonga, J., Riquelme, C., Tolstikhin, I.O.: Practical and consistent estimation of f-divergences. In: NeurIPS, pp. 4072–4082 (2019)
Shi, Y., Liao, B., Chen, G., Liu, Y., Cheng, M., Feng, J.: Understanding adversarial behavior of DNNs by disentangling non-robust and robust components in performance metric. CoRR abs/1906.02494 (2019)
Sklar, M.J.: Fonctions de repartition a n dimensions et Leurs Marges (1959)
Song, J., Ermon, S.: Understanding the limitations of variational mutual information estimators. In: ICLR. OpenReview.net (2020)
Tian, C., Xu, Y., Li, Z., Zuo, W., Fei, L., Liu, H.: Attention-guided CNN for image denoising. Neural Netw. 124, 117–129 (2020)
Tong, J., Chen, T., Wang, Q., Yao, Y.: Few-shot object detection via understanding convolution and attention. In: Yu, S., et al. (eds.) PRCV 2022. LNCS, vol. 13534, pp. 674–687. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-18907-4_52
Wang, M., Deng, W.: Deep face recognition: a survey. Neurocomputing 429, 215–244 (2021)
Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial robustness requires revisiting misclassified examples. In: ICLR. OpenReview.net (2020)
Wu, K., Wang, A.H., Yu, Y.: Stronger and faster Wasserstein adversarial attacks. In: ICML. Proceedings of Machine Learning Research, vol. 119, pp. 10377–10387. PMLR (2020)
Zhang, H., Yu, Y., Jiao, J., Xing, E.P., Ghaoui, L.E., Jordan, M.I.: Theoretically principled trade-off between robustness and accuracy. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 7472–7482. PMLR (2019)
Zhang, W., Gou, Y., Jiang, Y., Zhang, Y.: Adversarial VAE with normalizing flows for multi-dimensional classification. In: Yu, S., et al. (eds.) PRCV 2022. LNCS, vol. 13534, pp. 205–219. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-18907-4_16
Zhao, Z., Zheng, P., Xu, S., Wu, X.: Object detection with deep learning: a review. IEEE Trans. Neural Netw. Learn. Syst. 30(11), 3212–3232 (2019)
Zhou, D., et al.: Improving adversarial robustness via mutual information estimation. In: ICML. Proceedings of Machine Learning Research, vol. 162, pp. 27338–27352. PMLR (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, L., Hu, C., Wu, XJ. (2024). CE\(^2\): A Copula Entropic Mutual Information Estimator for Enhancing Adversarial Robustness. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14428. Springer, Singapore. https://doi.org/10.1007/978-981-99-8462-6_14
Download citation
DOI: https://doi.org/10.1007/978-981-99-8462-6_14
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8461-9
Online ISBN: 978-981-99-8462-6
eBook Packages: Computer ScienceComputer Science (R0)