Skip to main content
Springer Nature Link
Log in

Assessing Heuristic Machine Learning Explanations with Model Counting

  • Conference paper
  • First Online:
Theory and Applications of Satisfiability Testing – SAT 2019 (SAT 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11628))

  • 1792 Accesses

Abstract

Machine Learning (ML) models are widely used in decision making procedures in finance, medicine, education, etc. In these areas, ML outcomes can directly affect humans, e.g. by deciding whether a person should get a loan or be released from prison. Therefore, we cannot blindly rely on black box ML models and need to explain the decisions made by them. This motivated the development of a variety of ML-explainer systems, including LIME and its successor \({\textsc {Anchor}}\). Due to the heuristic nature of explanations produced by existing tools, it is necessary to validate them. We propose a SAT-based method to assess the quality of explanations produced by \({\textsc {Anchor}}\). We encode a trained ML model and an explanation for a given prediction as a propositional formula. Then, by using a state-of-the-art approximate model counter, we estimate the quality of the provided explanation as the number of solutions supporting it.

This work was supported by FCT grants ABSOLV (PTDC/CCI-COM/28986/2017), FaultLocker (PTDC/CCI-COM/29300/2017), SAFETY (SFRH/BPD/120315/2016), SAMPLE (CEECIND/04549/2017), National Research Foundation Singapore under its AI Singapore Programme AISG-RP-2018-005 and NUS ODPRT Grant R-252-000-685-133.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    In the training phase, there is an additional hard tanh layer after batch normalization but it is redundant in the inference phase.

  2. 2.

    https://github.com/marcotcr/anchor-experiments.

References

  1. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I.J., Hardt, M., Kim, B.: Sanity checks for saliency maps. In: NeurIPS, pp. 9525–9536 (2018)

    Google Scholar 

  2. Alvarez-Melis, D., Jaakkola, T.S.: Towards robust interpretability with self-explaining neural networks. In: NeurIPS, pp. 7786–7795 (2018)

    Google Scholar 

  3. Shih, A., Darwiche, A., Choi, A.: Verifying binarized neural networks by local automaton learning. In: VNN (2019)

    Google Scholar 

  4. Biere, A., Heule, M., van Maaren, H., Walsh, T. (eds.): Handbook of Satisfiability, Frontiers in Artificial Intelligence and Applications, vol. 185. IOS Press (2009)

    Google Scholar 

  5. Carter, J.L., Wegman, M.N.: Universal classes of hash functions. In: Proceedings of STOC, pp. 106–112. ACM (1977)

    Google Scholar 

  6. Chakraborty, S., Meel, K.S., Vardi, M.Y.: A scalable approximate model counter. In: Proceedings of CP, pp. 200–216 (2013)

    Google Scholar 

  7. Chakraborty, S., Meel, K.S., Vardi, M.Y.: Improving approximate counting for probabilistic inference: from linear to logarithmic sat solver calls. In: Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), July 2016

    Google Scholar 

  8. Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: KDD, pp. 785–794. ACM (2016)

    Google Scholar 

  9. Dagum, P., Karp, R., Luby, M., Ross, S.: An optimal algorithm for Monte Carlo estimation. SIAM J. Comput. 29(5), 1484–1496 (2000)

    Article  MathSciNet  Google Scholar 

  10. Darwiche, A., Marquis, P.: A knowledge compilation map. J. Artif. Intell. Res. 17, 229–264 (2002). https://doi.org/10.1613/jair.989

    Article  MathSciNet  MATH  Google Scholar 

  11. Dreossi, T., Ghosh, S., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: A formalization of robustness for deep neural networks. CoRR abs/1903.10033 (2019). http://arxiv.org/abs/1903.10033

  12. Ermon, S., Gomes, C.P., Sabharwal, A., Selman, B.: Taming the curse of dimensionality: discrete integration by hashing and optimization. In: Proceedings of ICML, pp. 334–342 (2013)

    Google Scholar 

  13. Frosst, N., Hinton, G.E.: Distilling a neural network into a soft decision tree. In: Besold, T.R., Kutz, O. (eds.) Proceedings of the First International Workshop on Comprehensibility and Explanation in AI and ML 2017 co-located with 16th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2017), Bari, Italy, 16–17 November 2017. CEUR Workshop Proceedings, vol. 2071. CEUR-WS.org (2017)

    Google Scholar 

  14. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks. In: NIPS, pp. 4107–4115 (2016)

    Google Scholar 

  15. Ignatiev, A., Narodytska, N., Marques-Silva, J.: Abduction-based explanations for machine learning models. In: AAAI (2019)

    Google Scholar 

  16. Ivrii, A., Malik, S., Meel, K.S., Vardi, M.Y.: On computing minimal independent support and its applications to sampling and counting. Constraints 21(1), 41–58 (2016). https://doi.org/10.1007/s10601-015-9204-z

    Article  MathSciNet  MATH  Google Scholar 

  17. Kohavi, R.: Scaling up the accuracy of naive-bayes classifiers: a decision-tree hybrid. In: KDD, pp. 202–207 (1996)

    Google Scholar 

  18. Lagniez, J.M., Marquis, P.: An improved decision-DNNF compiler. In: IJCAI, pp. 667–673 (2017)

    Google Scholar 

  19. Leofante, F., Narodytska, N., Pulina, L., Tacchella, A.: Automated verification of neural networks: advances, challenges and perspectives. CoRR abs/1805.09938 (2018). http://arxiv.org/abs/1805.09938

  20. Li, O., Liu, H., Chen, C., Rudin, C.: Deep learning for case-based reasoning through prototypes: a neural network that explains its predictions. In: AAAI, pp. 3530–3537 (2018)

    Google Scholar 

  21. Montavon, G., Samek, W., Müller, K.: Methods for interpreting and understanding deep neural networks. Digital Sig. Process. 73, 1–15 (2018)

    Article  MathSciNet  Google Scholar 

  22. Muise, C., McIlraith, S.A., Beck, J.C., Hsu, E.: DSHARP: Fast d-DNNF Compilation with sharpSAT. In: Canadian Conference on Artificial Intelligence (2012)

    Google Scholar 

  23. Narodytska, N., Kasiviswanathan, S.P., Ryzhyk, L., Sagiv, M., Walsh, T.: Verifying properties of binarized deep neural networks. In: AAAI, pp. 6615–6624 (2018)

    Google Scholar 

  24. Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you?: explaining the predictions of any classifier. In: KDD, pp. 1135–1144 (2016)

    Google Scholar 

  25. Ribeiro, M.T., Singh, S., Guestrin, C.: Anchors: high-precision model-agnostic explanations. In: AAAI (2018)

    Google Scholar 

  26. Ross, A.S., Doshi-Velez, F.: Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In: AAAI, pp. 1660–1669 (2018)

    Google Scholar 

  27. Ross, A.S., Hughes, M.C., Doshi-Velez, F.: Right for the right reasons: training differentiable models by constraining their explanations. In: IJCAI, pp. 2662–2670 (2017)

    Google Scholar 

  28. Sang, T., Beame, P., Kautz, H.: Performing Bayesian inference by weighted model counting. In: Proceedings of AAAI, pp. 475–481 (2005)

    Google Scholar 

  29. Schmidt, P., Witte, A.D.: Predicting recidivism in north carolina, 1978 and 1980. Inter-University Consortium for Political and Social Research (1988). https://www.ncjrs.gov/App/Publications/abstract.aspx?ID=115306

  30. Shih, A., Choi, A., Darwiche, A.: A symbolic approach to explaining Bayesian network classifiers. In: IJCAI, pp. 5103–5111 (2018)

    Google Scholar 

  31. Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. CoRR abs/1312.6034 (2013). http://arxiv.org/abs/1312.6034

  32. Sinz, C.: Towards an optimal CNF encoding of Boolean cardinality constraints. In: CP, pp. 827–831 (2005)

    Google Scholar 

  33. Soos, M., Meel, K.S.: Bird: Engineering an efficient CNF-XOR SAT solver and its applications to approximate model counting. In: Proceedings of AAAI Conference on Artificial Intelligence (AAAI), Jan 2019

    Google Scholar 

  34. Soos, M., Nohl, K., Castelluccia, C.: Extending SAT solvers to cryptographic problems. In: Kullmann, O. (ed.) SAT 2009. LNCS, vol. 5584, pp. 244–257. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02777-2_24

    Chapter  Google Scholar 

  35. Thurley, M.: SharpSAT: counting models with advanced component caching and implicit BCP. In: Proceedings of SAT, pp. 424–429 (2006)

    Google Scholar 

  36. Toda, S.: PP is as hard as the polynomial-time hierarchy. SIAM J. Comput. 20(5), 865–877 (1991)

    Article  MathSciNet  Google Scholar 

  37. Valiant, L.: The complexity of enumeration and reliability problems. SIAM J. Comput. 8(3), 410–421 (1979)

    Article  MathSciNet  Google Scholar 

  38. Wu, M., Hughes, M.C., Parbhoo, S., Zazzi, M., Roth, V., Doshi-Velez, F.: Beyond sparsity: tree regularization of deep models for interpretability. In: AAAI, pp. 1670–1678 (2018)

    Google Scholar 

  39. Zhang, Q., Yang, Y., Wu, Y.N., Zhu, S.: Interpreting CNNs via decision trees. CoRR abs/1802.00121 (2018), http://arxiv.org/abs/1802.00121

Download references

Author information

Authors and Affiliations

  1. VMware Research, Palo Alto, CA, USA

    Nina Narodytska

  2. Rice University, Houston, USA

    Aditya Shrotri

  3. National University of Singapore, Singapore, Singapore

    Kuldeep S. Meel

  4. Faculty of Science, University of Lisbon, Lisbon, Portugal

    Alexey Ignatiev & Joao Marques-Silva

  5. ISDCT SB RAS, Irkutsk, Russia

    Alexey Ignatiev

Authors
  1. Nina Narodytska
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aditya Shrotri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kuldeep S. Meel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexey Ignatiev
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Joao Marques-Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nina Narodytska .

Editor information

Editors and Affiliations

  1. University of Lisbon, Lisbon, Portugal

    Mikoláš Janota

  2. University of Lisbon, Lisbon, Portugal

    Inês Lynce

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Narodytska, N., Shrotri, A., Meel, K.S., Ignatiev, A., Marques-Silva, J. (2019). Assessing Heuristic Machine Learning Explanations with Model Counting. In: Janota, M., Lynce, I. (eds) Theory and Applications of Satisfiability Testing – SAT 2019. SAT 2019. Lecture Notes in Computer Science(), vol 11628. Springer, Cham. https://doi.org/10.1007/978-3-030-24258-9_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-24258-9_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-24257-2

  • Online ISBN: 978-3-030-24258-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation