Skip to main content
Springer Nature Link
Log in

SLIM: Spuriousness Mitigation with Minimal Human Annotations

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15104))

Included in the following conference series:

  • 287 Accesses

Abstract

Recent studies highlight that deep learning models often learn spurious features mistakenly linked to labels, compromising their reliability in real-world scenarios where such correlations do not hold. Despite the increasing research effort, existing solutions often face two main challenges: they either demand substantial annotations of spurious attributes, or they yield less competitive outcomes with expensive training when additional annotations are absent. In this paper, we introduce SLIM, a cost-effective and performance-targeted approach to reducing spurious correlations in deep learning. Our method leverages a human-in-the-loop protocol featuring a novel attention labeling mechanism with a constructed attention representation space. SLIM significantly reduces the need for exhaustive additional labeling, requiring human input for fewer than \(3\%\) of instances. By prioritizing data quality over complicated training strategies, SLIM curates a smaller yet more feature-balanced data subset, fostering the development of spuriousness-robust models. Experimental validations across key benchmarks demonstrate that SLIM competes with or exceeds the performance of leading methods while significantly reducing costs. The SLIM framework thus presents a promising path for developing reliable models more efficiently. Our code is available in https://github.com/xiweix/SLIM.git/.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ahmed, F., Bengio, Y., Van Seijen, H., Courville, A.: Systematic generalisation with group invariant predictions. In: International Conference on Learning Representations (2020)

    Google Scholar 

  2. Bontempelli, A., Teso, S., Tentori, K., Giunchiglia, F., Passerini, A.: Concept-level debugging of part-prototype networks. In: The Eleventh International Conference on Learning Representations (2023). https://openreview.net/forum?id=oiwXWPDTyNk

  3. Chen, Z., Deng, Y., Wu, Y., Gu, Q., Li, Y.: Towards understanding the mixture-of-experts layer in deep learning. In: Advances in Neural Information Processing Systems, vol. 35, pp. 23049–23062 (2022)

    Google Scholar 

  4. Christie, G., Fendley, N., Wilson, J., Mukherjee, R.: Functional map of the world. In: CVPR (2018)

    Google Scholar 

  5. Codella, N.C.F., et al.: Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 168–172 (2018). https://doi.org/10.1109/ISBI.2018.8363547

  6. Creager, E., Jacobsen, J.H., Zemel, R.: Environment inference for invariant learning. In: International Conference on Machine Learning, pp. 2189–2200. PMLR (2021)

    Google Scholar 

  7. Deng, Y., Yang, Y., Mirzasoleiman, B., Gu, Q.: Robust learning with progressive data expansion against spurious correlation. In: Thirty-Seventh Conference on Neural Information Processing Systems (2023). https://openreview.net/forum?id=9QEVJ9qm46

  8. Dosovitskiy, A., et al.: An image is worth \(16 \times 16\) words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  9. Gao, Y., Sun, T.S., Zhao, L., Hong, S.R.: Aligning eyes between humans and deep neural network through interactive attention alignment. Proc. ACM Hum.-Comput. Interact. 6(CSCW2) (2022). https://doi.org/10.1145/3555590

  10. Gao, Y., Sun, T.S., Zhao, L., Hong, S.R.: Aligning eyes between humans and deep neural network through interactive attention alignment. Proc. ACM Hum.-Comput. Interact. 6(CSCW2), 1–28 (2022)

    Google Scholar 

  11. He, H., Garcia, E.A.: Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21(9), 1263–1284 (2009)

    Article  Google Scholar 

  12. Jelassi, S., Li, Y.: Towards understanding how momentum improves generalization in deep learning. In: International Conference on Machine Learning, pp. 9965–10040. PMLR (2022)

    Google Scholar 

  13. Joshi, S., Yang, Y., Xue, Y., Yang, W., Mirzasoleiman, B.: Towards mitigating spurious correlations in the wild: A benchmark & a more realistic dataset. arXiv preprint arXiv:2306.11957 (2023)

  14. Kirichenko, P., Izmailov, P., Wilson, A.G.: Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937 (2022)

  15. Lee, S., Payani, A., Chau, D.H.P.: Towards mitigating spurious correlations in image classifiers with simple yes-no feedback. In: AI and HCI Workshop at the 40th International Conference on Machine Learning (2023)

    Google Scholar 

  16. Liang, W., Zou, J.: Metashift: a dataset of datasets for evaluating contextual distribution shifts and training conflicts. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=MTex8qKavoS

  17. Liu, E.Z., et al.: Just train twice: Improving group robustness without training group information. In: International Conference on Machine Learning, pp. 6781–6792. PMLR (2021)

    Google Scholar 

  18. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)

    Google Scholar 

  19. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)

    Google Scholar 

  20. McInnes, L., Healy, J., Melville, J.: UMAP: uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)

  21. Nam, J., Cha, H., Ahn, S., Lee, J., Shin, J.: Learning from failure: de-biasing classifier from biased classifier. In: Advances in Neural Information Processing Systems, vol. 33, pp. 20673–20684 (2020)

    Google Scholar 

  22. Nam, J., Kim, J., Lee, J., Shin, J.: Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. arXiv preprint arXiv:2204.02070 (2022)

  23. Petsiuk, V., Das, A., Saenko, K.: Rise: randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421 (2018)

  24. Rao, S., Böhle, M., Parchami-Araghi, A., Schiele, B.: Using explanations to guide models. arXiv preprint arXiv:2303.11932 (2023)

  25. Rieger, L., Singh, C., Murdoch, W.J., Yu, B.: Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020. JMLR.org (2020)

    Google Scholar 

  26. Ross, A.S., Hughes, M.C., Doshi-Velez, F.: Right for the right reasons: training differentiable models by constraining their explanations. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, pp. 2662–2670 (2017). https://doi.org/10.24963/ijcai.2017/371

  27. Sagawa, S., Koh, P.W., Hashimoto, T.B., Liang, P.: Distributionally robust neural networks. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=ryxGuJrFvS

  28. Schramowski, P., et al.: Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nat. Mach. Intell. 2(8), 476–486 (2020)

    Article  Google Scholar 

  29. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)

    Google Scholar 

  30. Sohoni, N., Dunnmon, J., Angus, G., Gu, A., Ré, C.: No subclass left behind: fine-grained robustness in coarse-grained classification problems. In: Advances in Neural Information Processing Systems, vol. 33, pp. 19339–19352 (2020)

    Google Scholar 

  31. Srivastava, M., Hashimoto, T., Liang, P.: Robustness to spurious correlations via human annotations. In: International Conference on Machine Learning, pp. 9109–9119. PMLR (2020)

    Google Scholar 

  32. Taghanaki, S.A., Choi, K., Khasahmadi, A.H., Goyal, A.: Robust representation learning via perceptual similarity metrics. In: International Conference on Machine Learning, pp. 10043–10053. PMLR (2021)

    Google Scholar 

  33. Wang, H., et al.: Score-CAM: score-weighted visual explanations for convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 24–25 (2020)

    Google Scholar 

  34. Wu, S., Yuksekgonul, M., Zhang, L., Zou, J.: Discover and cure: concept-aware mitigation of spurious correlation. In: Proceedings of the 40th International Conference on Machine Learning, ICML 2023. JMLR.org (2023)

    Google Scholar 

  35. Xiao, K., Engstrom, L., Ilyas, A., Madry, A.: Noise or signal: the role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994 (2020)

  36. Xuan, X., Ono, J.P., Gou, L., Ma, K.L., Ren, L.: AttributionScanner: a visual analytics system for metadata-free data-slicing based model validation. arXiv preprint arXiv:2401.06462 (2024)

  37. Xue, Y., Payani, A., Yang, Y., Mirzasoleiman, B.: Eliminating spurious correlations from pre-trained models via data mixing. arXiv preprint arXiv:2305.14521 (2023)

  38. Yan, S., et al.: Towards trustable skin cancer diagnosis via rewriting model’s decision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11568–11577 (2023)

    Google Scholar 

  39. Yang, Y., Gan, E., Karolina Dziugaite, G., Mirzasoleiman, B.: Identifying spurious biases early in training through the lens of simplicity bias. In: Proceedings of The 27th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 238, pp. 2953–2961. PMLR (2024)

    Google Scholar 

  40. Yang, Y., Nushi, B., Palangi, H., Mirzasoleiman, B.: Mitigating spurious correlations in multi-modal models during fine-tuning. In: Proceedings of the 40th International Conference on Machine Learning, ICML 2023. JMLR.org (2023)

    Google Scholar 

  41. Zhang, M., Sohoni, N.S., Zhang, H.R., Finn, C., Ré, C.: Correct-n-contrast: a contrastive approach for improving robustness to spurious correlations. arXiv preprint arXiv:2203.01517 (2022)

  42. Zhang, X., He, Y., Xu, R., Yu, H., Shen, Z., Cui, P.: Nico++: towards better benchmarking for domain generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16036–16047 (2023)

    Google Scholar 

  43. Zhou, D., Bousquet, O., Lal, T., Weston, J., Schölkopf, B.: Learning with local and global consistency. In: Advances in Neural Information Processing Systems, vol. 16 (2003)

    Google Scholar 

Download references

Acknowledgements

This research is supported in part by Bosch Research and the National Institute of Health with grants P41-EB032840 and R01CA270454. H.-T. Lin is partially supported by the National Taiwan University Center for Data Intelligence via NTU-112L900901 and the Ministry of Science and Technology in Taiwan via MOST 112-2628-E-002-030.

Author information

Authors and Affiliations

  1. University of California, Davis, USA

    Xiwei Xuan, Ziquan Deng & Kwan-Liu Ma

  2. National Taiwan University, Taipei, Taiwan

    Hsuan-Tien Lin

Authors
  1. Xiwei Xuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ziquan Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hsuan-Tien Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kwan-Liu Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiwei Xuan .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Hessen, Germany

    Stefan Roth

  4. Princeton University, Palo Alto, CA, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 13553 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xuan, X., Deng, Z., Lin, HT., Ma, KL. (2025). SLIM: Spuriousness Mitigation with Minimal Human Annotations. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15104. Springer, Cham. https://doi.org/10.1007/978-3-031-72952-2_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72952-2_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72951-5

  • Online ISBN: 978-3-031-72952-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics