Skip to main content
Springer Nature Link
Log in

The Co-12 Recipe for Evaluating Interpretable Part-Prototype Image Classifiers

  • Conference paper
  • First Online:
Explainable Artificial Intelligence (xAI 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1901))

Included in the following conference series:

  • 1381 Accesses

Abstract

Interpretable part-prototype models are computer vision models that are explainable by design. The models learn prototypical parts and recognise these components in an image, thereby combining classification and explanation. Despite the recent attention for intrinsically interpretable models, there is no comprehensive overview on evaluating the explanation quality of interpretable part-prototype models. Based on the Co-12 properties for explanation quality as introduced in [42] (e.g., correctness, completeness, compactness), we review existing work that evaluates part-prototype models, reveal research gaps and outline future approaches for evaluation of the explanation quality of part-prototype models. This paper, therefore, contributes to the progression and maturity of this relatively new research field on interpretable part-prototype models. We additionally provide a “Co-12 cheat sheet” that acts as a concise summary of our findings on evaluating part-prototype models.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    In ProtoPNet, only 2% of the prototypes have no overlap with an object, since ProtoPNet uses cropped images which makes it less likely to entirely miss the object.

References

  1. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., Kim, B.: Sanity checks for saliency maps. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems (2018)

    Google Scholar 

  2. Barredo Arrieta, A., et al.: Explainable Artificial Intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58 (2020)

    Google Scholar 

  3. Biederman, I.: Recognition-by-components: a theory of human image understanding. Psychol. Rev. 94(2), 115 (1987)

    Article  Google Scholar 

  4. Borowski, J., et al.: Exemplary natural images explain CNN activations better than state-of-the-art feature visualization. In: Proceedings of the International Conference on Learning Representations (2021)

    Google Scholar 

  5. Brendel, W., Bethge, M.: Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet. In: Proceedings of the International Conference on Learning Representations (2019)

    Google Scholar 

  6. Bruckert, S., Finzel, B., Schmid, U.: The next generation of medical decision support: a roadmap toward transparent expert companions. Front. Artif. Intell. 3, 507973 (2020)

    Article  Google Scholar 

  7. Carloni, G., Berti, A., Iacconi, C., Pascali, M.A., Colantonio, S.: On the applicability of prototypical part learning in medical images: breast masses classification using ProtoPNet. In: Rousseau, J.J., Kapralos, B. (eds.) ICPR 2022. LNCS, vol. 13643, pp. 539–557. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-37660-3_38

    Chapter  Google Scholar 

  8. Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., Su, J.: This looks like that: deep learning for interpretable image recognition. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems, pp. 8928–8939 (2019)

    Google Scholar 

  9. Clement, T., Kemmerzell, N., Abdelaal, M., Amberg, M.: XAIR: a systematic metareview of explainable AI (XAI) aligned to the software development process. Mach. Learn. Knowl. Extr. 5(1), 78–108 (2023)

    Article  Google Scholar 

  10. Colin, J., Fel, T., Cadene, R., Serre, T.: What I cannot predict, I do not understand: a human-centered evaluation framework for explainability methods. In: Advances in Neural Information Processing Systems (2022)

    Google Scholar 

  11. Das, S., Xu, P., Dai, Z., Endert, A., Ren, L.: Interpreting deep neural networks through prototype factorization. In: Fatta, G.D., Sheng, V.S., Cuzzocrea, A., Zaniolo, C., Wu, X. (eds.) Proceedings of the International Conference on Data Mining Workshops, ICDM Workshops, pp. 448–457. IEEE (2020)

    Google Scholar 

  12. Doshi-Velez, F., Kim, B.: Considerations for evaluation and generalization in interpretable machine learning. In: Escalante, H.J., et al. (eds.) Explainable and Interpretable Models in Computer Vision and Machine Learning. TSSCML, pp. 3–17. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98131-4_1

    Chapter  Google Scholar 

  13. Ehsan, U., et al.: The who in explainable AI: how AI background shapes perceptions of AI explanations. arXiv preprint arXiv:2107.13509 (2021)

  14. Fong, R., Vedaldi, A.: Net2Vec: quantifying and explaining how concepts are encoded by filters in deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8730–8738. IEEE Computer Society (2018)

    Google Scholar 

  15. Gautam, S., Höhne, M.M.-C., Hansen, S., Jenssen, R., Kampffmeyer, M.: This looks more like that: enhancing self-explaining models by prototypical relevance propagation. Pattern Recogn. 136, 109172 (2023)

    Article  Google Scholar 

  16. Ghorbani, A., Wexler, J., Zou, J.Y., Kim, B.: Towards automatic concept-based explanations. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems, pp. 9273–9282 (2019)

    Google Scholar 

  17. Goodman, B., Flaxman, S.: European union regulations on algorithmic decision-making and a “right to explanation’’. AI Mag. 38(3), 50–57 (2017)

    Google Scholar 

  18. Goyal, Y., Feder, A., Shalit, U., Kim, B.: Explaining classifiers with causal concept effect (CaCE). arXiv preprint arXiv:1907.07165 (2019)

  19. Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., Lee, S.: Counterfactual visual explanations. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the International Conference on Machine Learning, ICML 2019, Long Beach, California, USA, 9–15 June 2019, vol. 97, pp. 2376–2384. PMLR (2019)

    Google Scholar 

  20. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. (CSUR) 51(5), 1–42 (2018)

    Article  Google Scholar 

  21. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the International Conference on Machine Learning, vol. 70, pp. 1321–1330. PMLR (2017)

    Google Scholar 

  22. Hase, P., Bansal, M.: Evaluating explainable AI: which algorithmic explanations help users predict model behavior? In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J.R. (eds.) Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, 5–10 July 2020, pp. 5540–5552. Association for Computational Linguistics (2020)

    Google Scholar 

  23. Hase, P., Xie, H., Bansal, M.: The out-of-distribution problem in explainability and search methods for feature importance explanations. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 3650–3666. Curran Associates Inc. (2021)

    Google Scholar 

  24. He, Y., Shen, Z., Cui, P.: Towards non-IID image classification: a dataset and baselines. Pattern Recogn. 110, 107383 (2021)

    Article  Google Scholar 

  25. Hoffman, R.R., Mueller, S.T., Klein, G., Litman, J.: Metrics for explainable AI: challenges and prospects. arXiv:1812.04608 [cs] (2019)

  26. Hoffmann, A., Fanconi, C., Rade, R., Kohler, J.: This looks like that... does it? Shortcomings of latent space prototype interpretability in deep networks. arXiv preprint arXiv:2105.02968 (2021)

  27. Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., Baesens, B.: An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decis. Support Syst. 51(1) (2011)

    Google Scholar 

  28. Jacovi, A., Goldberg, Y.: Towards faithfully interpretable NLP systems: how should we define and evaluate faithfulness? In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J.R. (eds.) Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, 5–10 July 2020, pp. 4198–4205. Association for Computational Linguistics (2020)

    Google Scholar 

  29. Jeyakumar, J.V., Noor, J., Cheng, Y., Garcia, L., Srivastava, M.B.: How can I explain this to you? An empirical study of deep neural network explanation methods. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems (2020)

    Google Scholar 

  30. Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., Wortman Vaughan, J.: Interpreting interpretability: understanding data scientists’ use of interpretability tools for machine learning. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–14. Association for Computing Machinery, New York (2020)

    Google Scholar 

  31. Kim, S.S.Y., Meister, N., Ramaswamy, V.V., Fong, R., Russakovsky, O.: HIVE: evaluating the human interpretability of visual explanations. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13672, pp. 280–298. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19775-8_17

    Chapter  Google Scholar 

  32. Kulesza, T., Burnett, M., Wong, W.-K., Stumpf, S.: Principles of explanatory debugging to personalize interactive machine learning. In: Proceedings of the International Conference on Intelligent User Interfaces, pp. 126–137. Association for Computing Machinery, New York (2015)

    Google Scholar 

  33. Lakkaraju, H., Leskovec, J.: Confusions over time: an interpretable Bayesian model to characterize trends in decision making. In: Lee, D.D., Sugiyama, M., von Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, pp. 3261–3269 (2016)

    Google Scholar 

  34. Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 951–958 (2009)

    Google Scholar 

  35. Liu, Y., Khandagale, S., White, C., Neiswanger, W.: Synthetic benchmarks for scientific research in explainable machine learning. In: NeurIPS Datasets and Benchmarks Track (2021)

    Google Scholar 

  36. Michel, A., Jha, S.K., Ewetz, R.: A survey on the vulnerability of deep neural networks against adversarial attacks. Progress Artif. Intell. 11, 131–141 (2021). https://doi.org/10.1007/s13748-021-00269-9

    Article  Google Scholar 

  37. Mohammadjafari, S., Cevik, M., Thanabalasingam, M., Basar, A.: Using ProtoPNet for interpretable Alzheimer’s disease classification. In: Canadian Conference on AI (2021)

    Google Scholar 

  38. Montavon, G.: Gradient-based vs. propagation-based explanations: an axiomatic comparison. In: Samek, W., Montavon, G., Vedaldi, A., Hansen, L.K., Müller, K.-R. (eds.) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. LNCS (LNAI), vol. 11700, pp. 253–265. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28954-6_13

    Chapter  Google Scholar 

  39. Nakka, K.K., Salzmann, M.: Towards robust fine-grained recognition by maximal separation of discriminative features. In: Ishikawa, H., Liu, C.-L., Pajdla, T., Shi, J. (eds.) ACCV 2020. LNCS, vol. 12627, pp. 391–408. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-69544-6_24

    Chapter  Google Scholar 

  40. Nauta, M., Jutte, A., Provoost, J., Seifert, C.: This looks like that, because ... explaining prototypes for interpretable image recognition. In: Kamp, M., et al. (eds.) ECML PKDD 2021. CCIS, vol. 1524, pp. 441–456. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-93736-2_34

    Chapter  Google Scholar 

  41. Nauta, M., Schlötterer, J., van Keulen, M., Seifert, C.: PIP-Net: patch-based intuitive prototypes for interpretable image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2744–2753 (2023)

    Google Scholar 

  42. Nauta, M., et al.: From anecdotal evidence to quantitative evaluation methods: a systematic review on evaluating explainable AI. ACM Comput. Surv. (2023)

    Google Scholar 

  43. Nauta, M., van Bree, R., Seifert, C.: Neural prototype trees for interpretable fine-grained image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14933–14943 (2021)

    Google Scholar 

  44. Nguyen, G., Kim, D., Nguyen, A.: The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems, vol. 34, pp. 26422–26436. Curran Associates Inc. (2021)

    Google Scholar 

  45. Papenmeier, A., Kern, D., Englebienne, G., Seifert, C.: It’s complicated: the relationship between user trust, model accuracy and explanations in AI. ACM Trans. Comput.-Hum. Interact. 29(4) (2022)

    Google Scholar 

  46. Pearce, T., Brintrup, A., Zhu, J.: Understanding softmax confidence and uncertainty. arXiv preprint arXiv:2106.04972 (2021)

  47. Rong, Y., Leemann, T., Borisov, V., Kasneci, G., Kasneci, E.: A consistent and efficient evaluation strategy for attribution methods. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. (eds.) Proceedings of the International Conference on Machine Learning, vol. 162, pp. 18770–18795. PMLR (2022)

    Google Scholar 

  48. Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1(5), 206–215 (2019)

    Article  Google Scholar 

  49. Rymarczyk, D., Struski, Ł, Górszczak, M., Lewandowska, K., Tabor, J., Zieliński, B.: Interpretable image classification with differentiable prototypes assignment. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, pp. 351–368. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19775-8_21

    Chapter  Google Scholar 

  50. Rymarczyk, D., Struski, L., Tabor, J., Zieliński, B.: ProtoPShare: prototypical parts sharing for similarity discovery in interpretable image classification. In: Proceedings of the ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1420–1430. Association for Computing Machinery, New York (2021)

    Google Scholar 

  51. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: IEEE International Conference on Computer Vision, ICCV, pp. 618–626. IEEE Computer Society (2017)

    Google Scholar 

  52. Singh, G.: Think positive: an interpretable neural network for image recognition. Neural Netw. 151, 178–189 (2022)

    Article  Google Scholar 

  53. Singh, G., Yow, K.-C.: An interpretable deep learning model for Covid-19 detection with chest X-ray images. IEEE Access 9, 85198–85208 (2021)

    Article  Google Scholar 

  54. Sinhamahapatra, P., Heidemann, L., Monnet, M., Roscher, K.: Towards human-interpretable prototypes for visual assessment of image classification models. arXiv preprint arXiv:2211.12173 (2022)

  55. Summers, C., Dinneen, M.J.: Nondeterminism and instability in neural network optimization. In: Meila, M., Zhang, T. (eds.) Proceedings of the International Conference on Machine Learning, vol. 139, pp. 9913–9922. PMLR (2021)

    Google Scholar 

  56. Vilone, G., Longo, L.: Notions of explainability and evaluation approaches for explainable artificial intelligence. Inf. Fusion 76 (2021)

    Google Scholar 

  57. Wang, C., et al.: Knowledge distillation to ensemble global and interpretable prototype-based mammogram classification models. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13433, pp. 14–24. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16437-8_2

    Chapter  Google Scholar 

  58. Wang, J., Liu, H., Wang, X., Jing, L.: Interpretable image recognition by constructing transparent embedding space. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 895–904 (2021)

    Google Scholar 

  59. Williams, P.A., Jenkins, J., Valacich, J., Byrd, M.D.: Measuring actual behaviors in HCI research-a call to action and an example. AIS Trans. Hum.-Comput. Interact. 9(4), 339–352 (2017)

    Article  Google Scholar 

  60. Xie, W., Li, X.-H., Cao, C.C., Zhang, N.L.: ViT-CX: causal explanation of vision transformers. arXiv preprint arXiv:2211.03064 (2022)

  61. Xu-Darme, R., Quénot, G., Chihani, Z., Rousset, M.-C.: Sanity checks and improvements for patch visualisation in prototype-based image classification. working paper or preprint (2023)

    Google Scholar 

  62. Xue, M., et al.: ProtoPFormer: concentrating on prototypical parts in vision transformers for interpretable image recognition. arXiv preprint arXiv:2208.10431 (2022)

  63. Yang, M., Kim, B.: Benchmarking attribution methods with relative feature importance. CoRR, abs/1907.09701 (2019)

    Google Scholar 

  64. Yeh, C., Kim, B., Arik, S.Ö., Li, C., Pfister, T., Ravikumar, P.: On completeness-aware concept-based explanations in deep neural networks. In: Advances in Neural Information Processing Systems (2020)

    Google Scholar 

  65. Zhou, J., Gandomi, A.H., Chen, F., Holzinger, A.: Evaluating the quality of machine learning explanations: a survey on methods and metrics. Electronics 10(5) (2021)

    Google Scholar 

  66. Zhuang, D., Zhang, X., Song, S., Hooker, S.: Randomness in neural network training: characterizing the impact of tooling. In: Marculescu, D., Chi, Y., Wu, C. (eds.) Proceedings of the Machine Learning and Systems, vol. 4, pp. 316–336 (2022)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Twente, Enschede, The Netherlands

    Meike Nauta

  2. University of Duisburg-Essen, Duisburg, Germany

    Christin Seifert

  3. University of Marburg, Marburg, Germany

    Christin Seifert

Authors
  1. Meike Nauta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christin Seifert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Meike Nauta or Christin Seifert .

Editor information

Editors and Affiliations

  1. Technological University Dublin, Dublin, Ireland

    Luca Longo

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nauta, M., Seifert, C. (2023). The Co-12 Recipe for Evaluating Interpretable Part-Prototype Image Classifiers. In: Longo, L. (eds) Explainable Artificial Intelligence. xAI 2023. Communications in Computer and Information Science, vol 1901. Springer, Cham. https://doi.org/10.1007/978-3-031-44064-9_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-44064-9_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-44063-2

  • Online ISBN: 978-3-031-44064-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics