Skip to main content
Springer Nature Link
Log in

CE\(^2\): A Copula Entropic Mutual Information Estimator for Enhancing Adversarial Robustness

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14428))

Included in the following conference series:

  • 833 Accesses

Abstract

Deep neural networks are vulnerable to adversarial examples, which exploit imperceptible perturbations to mislead classifiers. To improve adversarial robustness, recent methods have focused on estimating mutual information (MI). However, existing MI estimators struggle to provide stable and reliable estimates in high-dimensional data. To this end, we propose a Copula Entropic MI Estimator (CE\(^2\)) to address these limitations. CE\(^2\) leverages copula entropy to estimating MI in high dimensions, allowing target models to harness information from both clean and adversarial examples to withstand attacks. Our empirical experiments demonstrate that CE\(^2\) achieves a trade-off between variance and bias in MI estimates, resulting in stable and reliable estimates. Furthermore, the defense algorithm based on CE\(^2\) significantly enhances adversarial robustness against multiple attacks. The experimental results underscore the effectiveness of CE\(^2\) and its potential for improving adversarial robustness.

This work was supported in part by the National Natural Science Foundation of China (Grant No. 62006097, U1836218), in part by the Natural Science Foundation of Jiangsu Province (Grant No. BK20200593) and in part by the China Postdoctoral Science Foundation (Grant No. 2021M701456).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Belghazi, M.I., et al.: Mutual information neural estimation. In: ICML. Proceedings of Machine Learning Research, vol. 80, pp. 530–539. PMLR (2018)

    Google Scholar 

  2. Carlini, N., Wagner, D.A.: Towards evaluating the robustness of neural networks. In: IEEE Symposium on Security and Privacy, pp. 39–57. IEEE Computer Society (2017)

    Google Scholar 

  3. Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: ICML. Proceedings of Machine Learning Research, vol. 119, pp. 2206–2216. PMLR (2020)

    Google Scholar 

  4. Ding, G.W., Lui, K.Y.C., Jin, X., Wang, L., Huang, R.: On the sensitivity of adversarial robustness to input data distributions. In: ICLR (Poster). OpenReview.net (2019)

    Google Scholar 

  5. Duan, Y., Lu, J., Zheng, W., Zhou, J.: Deep adversarial metric learning. IEEE Trans. Image Process. 29, 2037–2051 (2020)

    Article  Google Scholar 

  6. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: ICLR (Poster) (2015)

    Google Scholar 

  7. Hu, C., Li, Y., Feng, Z., Wu, X.: Attention-guided evolutionary attack with elastic-net regularization on face recognition. Pattern Recogn. 109760 (2023)

    Google Scholar 

  8. Hu, C., Xu, H.Q., Wu, X.J.: Substitute meta-learning for black-box adversarial attack. IEEE Sig. Process. Lett. 29, 2472–2476 (2022). https://doi.org/10.1109/LSP.2022.3226118

    Article  Google Scholar 

  9. Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examples are not bugs, they are features. In: NeurIPS, pp. 125–136 (2019)

    Google Scholar 

  10. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1106–1114 (2012)

    Google Scholar 

  11. Li, H., Wu, X., Kittler, J.: MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Trans. Image Process. 29, 4733–4746 (2020)

    Article  Google Scholar 

  12. Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks. In: CVPR, pp. 510–519. Computer Vision Foundation/IEEE (2019)

    Google Scholar 

  13. Liao, F., Liang, M., Dong, Y., Pang, T., Hu, X., Zhu, J.: Defense against adversarial attacks using high-level representation guided denoiser. In: CVPR, pp. 1778–1787. Computer Vision Foundation/IEEE Computer Society (2018)

    Google Scholar 

  14. Lin, S., et al.: Towards optimal structured CNN pruning via generative adversarial learning. In: CVPR, pp. 2790–2799. Computer Vision Foundation/IEEE (2019)

    Google Scholar 

  15. Linsker, R.: Self-organization in a perceptual network. Computer 21(3), 105–117 (1988)

    Article  Google Scholar 

  16. Ma, J., Sun, Z.: Mutual information is copula entropy. CoRR abs/0808.0845 (2008)

    Google Scholar 

  17. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: ICLR (Poster). OpenReview.net (2018)

    Google Scholar 

  18. McAllester, D., Stratos, K.: Formal limitations on the measurement of mutual information. In: AISTATS. Proceedings of Machine Learning Research, vol. 108, pp. 875–884. PMLR (2020)

    Google Scholar 

  19. Naseer, M., Khan, S.H., Hayat, M., Khan, F.S., Porikli, F.: A self-supervised approach for adversarial robustness. In: CVPR, pp. 259–268. Computer Vision Foundation/IEEE (2020)

    Google Scholar 

  20. Nguyen, X., Wainwright, M.J., Jordan, M.I.: Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory 56(11), 5847–5861 (2010)

    Article  MathSciNet  Google Scholar 

  21. Noshad, M., Zeng, Y., III, A.O.H.: Scalable mutual information estimation using dependence graphs. In: ICASSP, pp. 2962–2966. IEEE (2019)

    Google Scholar 

  22. van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. CoRR abs/1807.03748 (2018)

    Google Scholar 

  23. Paninski, L.: Estimation of entropy and mutual information. Neural Comput. 15(6), 1191–1253 (2003)

    Article  Google Scholar 

  24. Papernot, N., McDaniel, P.D., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: EuroS &P, pp. 372–387. IEEE (2016)

    Google Scholar 

  25. Poole, B., Ozair, S., van den Oord, A., Alemi, A.A., Tucker, G.: On variational bounds of mutual information. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 5171–5180. PMLR (2019)

    Google Scholar 

  26. Ravens, B.: An introduction to copulas. Technometrics 42(3), 317 (2000)

    Article  Google Scholar 

  27. Rony, J., Hafemann, L.G., Oliveira, L.S., Ayed, I.B., Sabourin, R., Granger, E.: Decoupling direction and norm for efficient gradient-based L2 adversarial attacks and defenses. In: CVPR, pp. 4322–4330. Computer Vision Foundation/IEEE (2019)

    Google Scholar 

  28. Rubenstein, P.K., Bousquet, O., Djolonga, J., Riquelme, C., Tolstikhin, I.O.: Practical and consistent estimation of f-divergences. In: NeurIPS, pp. 4072–4082 (2019)

    Google Scholar 

  29. Shi, Y., Liao, B., Chen, G., Liu, Y., Cheng, M., Feng, J.: Understanding adversarial behavior of DNNs by disentangling non-robust and robust components in performance metric. CoRR abs/1906.02494 (2019)

    Google Scholar 

  30. Sklar, M.J.: Fonctions de repartition a n dimensions et Leurs Marges (1959)

    Google Scholar 

  31. Song, J., Ermon, S.: Understanding the limitations of variational mutual information estimators. In: ICLR. OpenReview.net (2020)

    Google Scholar 

  32. Tian, C., Xu, Y., Li, Z., Zuo, W., Fei, L., Liu, H.: Attention-guided CNN for image denoising. Neural Netw. 124, 117–129 (2020)

    Article  Google Scholar 

  33. Tong, J., Chen, T., Wang, Q., Yao, Y.: Few-shot object detection via understanding convolution and attention. In: Yu, S., et al. (eds.) PRCV 2022. LNCS, vol. 13534, pp. 674–687. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-18907-4_52

    Chapter  Google Scholar 

  34. Wang, M., Deng, W.: Deep face recognition: a survey. Neurocomputing 429, 215–244 (2021)

    Article  Google Scholar 

  35. Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial robustness requires revisiting misclassified examples. In: ICLR. OpenReview.net (2020)

    Google Scholar 

  36. Wu, K., Wang, A.H., Yu, Y.: Stronger and faster Wasserstein adversarial attacks. In: ICML. Proceedings of Machine Learning Research, vol. 119, pp. 10377–10387. PMLR (2020)

    Google Scholar 

  37. Zhang, H., Yu, Y., Jiao, J., Xing, E.P., Ghaoui, L.E., Jordan, M.I.: Theoretically principled trade-off between robustness and accuracy. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 7472–7482. PMLR (2019)

    Google Scholar 

  38. Zhang, W., Gou, Y., Jiang, Y., Zhang, Y.: Adversarial VAE with normalizing flows for multi-dimensional classification. In: Yu, S., et al. (eds.) PRCV 2022. LNCS, vol. 13534, pp. 205–219. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-18907-4_16

    Chapter  Google Scholar 

  39. Zhao, Z., Zheng, P., Xu, S., Wu, X.: Object detection with deep learning: a review. IEEE Trans. Neural Netw. Learn. Syst. 30(11), 3212–3232 (2019)

    Article  Google Scholar 

  40. Zhou, D., et al.: Improving adversarial robustness via mutual information estimation. In: ICML. Proceedings of Machine Learning Research, vol. 162, pp. 27338–27352. PMLR (2022)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China

    Lin Liu, Cong Hu & Xiao-Jun Wu

Authors
  1. Lin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiao-Jun Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cong Hu .

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, L., Hu, C., Wu, XJ. (2024). CE\(^2\): A Copula Entropic Mutual Information Estimator for Enhancing Adversarial Robustness. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14428. Springer, Singapore. https://doi.org/10.1007/978-981-99-8462-6_14

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8462-6_14

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8461-9

  • Online ISBN: 978-981-99-8462-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation