Abstract
How can we accurately classify feature-based data such that the learned model and results are more interpretable? Interpretability is beneficial in various perspectives, such as in checking for compliance with exiting knowledge and gaining insights from decision processes. To gain in both accuracy and interpretability, we propose a novel tree-structured classifier called Gaussian Soft Decision Trees (GSDT). GSDT is characterized by multi-branched structures, Gaussian mixture-based decisions, and a hinge loss with path regularization. The three key features make it learn short trees where the weight vector of each node is a prototype for data that mapped to the node. We show that GSDT results in the best average accuracy compared to eight baselines. We also perform an ablation study of the various structures of covariance matrix in the Gaussian mixture nodes in GSDT and demonstrate the interpretability of GSDT in a case study of classification in a breast cancer dataset.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
References
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)
Chang, Y., Hsieh, C., Chang, K., Ringgaard, M., Lin, C.: Training and testing low-degree polynomial data mappings via linear SVM. J. Mach. Learn. Res. 11, 1471–1490 (2010)
Dogan, Ü., Glasmachers, T., Igel, C.: A unified view on multi-class support vector classification. J. Mach. Learn. Res. 17, 45:1–45:32 (2016)
Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., Garcia, R.: Incorporating second-order functional knowledge for better option pricing. In: NIPS, pp. 472–478. MIT Press (2000)
Eckart, B., Kim, K., Kautz, J.: HGMR: hierarchical gaussian mixtures for adaptive 3D registration. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 730–746. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01267-0_43
Frosst, N., Hinton, G.E.: Distilling a neural network into a soft decision tree. In: AI*IA. CEUR Workshop Proceedings, vol. 2071. CEUR-WS.org (2017)
Harville, D.A.: Matrix algebra from a statistician’s perspective (1998)
Hinton, G.E., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015). CoRR abs/1503.02531
Hofmann, T., Schölkopf, B., Smola, A.J.: Kernel methods in machine learning. Ann. Statistics 36, 1171–1220 (2008)
Irsoy, O., Alpaydin, E.: Autoencoder trees. In: ACML, vol. 45, pp. 378–390 (2015)
Irsoy, O., Yildiz, O.T., Alpaydin, E.: Soft decision trees. In: ICPR (2012)
Jordan, M.I., Jacobs, R.A.: Hierarchical mixtures of experts and the EM algorithm. Neural Comput. 6(2), 181–214 (1994)
Kim, B., Khanna, R., Koyejo, O.: Examples are not enough, learn to criticize! criticism for interpretability. In: NIPS (2016)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)
Liu, M., Chang, E., Dai, B.Q.: Hierarchical Gaussian mixture model for speaker verification. In: Seventh International Conference on Spoken Language Processing (2002)
Maaten, L.V.D., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)
Magdon-Ismail, M., Purnell, J.T.: Approximating the covariance matrix of gmms with low-rank perturbations. Int. J. Data Min. Model. Manag. 4(2), 107–122 (2012)
Miller, T.: Explanation in artificial intelligence: insights from the social sciences. Artif. Intell 267, 1–38 (2018)
Olech, L.P., Paradowski, M.: Hierarchical gaussian mixture model with objects attached to terminal and non-terminal dendrogram nodes. In: CORES (2015)
Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Yoo, J., Sael, L.: Edit: interpreting ensemble models via compact soft decision trees. In: ICDM, pp. 1438–1443 (2019)
Zhu, J., Hastie, T.: Kernel logistic regression and the import vector machine. In: NIPS, pp. 1081–1088 (2001)
Acknowledgments
Publication of this article has been funded by the Basic Science Research Program through the National Research Foundation of Korea (2018R1A1A3A0407953, 2018R1A5A1060031).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Yoo, J., Sael, L. (2021). Gaussian Soft Decision Trees for Interpretable Feature-Based Classification. In: Karlapalem, K., et al. Advances in Knowledge Discovery and Data Mining. PAKDD 2021. Lecture Notes in Computer Science(), vol 12713. Springer, Cham. https://doi.org/10.1007/978-3-030-75765-6_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-75765-6_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-75764-9
Online ISBN: 978-3-030-75765-6
eBook Packages: Computer ScienceComputer Science (R0)