Abstract
Misclassification with high confidence can cause great harm, especially in high-risk tasks. This is because when faced with unfamiliar samples which outside the distribution of known samples, classifiers are prone to give overly high prediction confidence. In this paper, we propose a classification framework with effective confidence estimation based on a reduced dimensional space (subspace). Intuitively, our method is designed to give low confidence to “unfamiliar” or out-of-domain test samples, so to “know what we do not know”. The effectiveness of this method is supported by a real-world Stroke diagnosis data set. We use multiple metrics for evaluation, including Brier score, calibration curve, and expected calibration error (ECE).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 (2017)
Li, Z., Hoiem, D.: Improving confidence estimates for unfamiliar examples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2686–2695. IEEE, San Francisco (2020)
Wang, S., Nie, F., Chang, X., Li, X., Sheng, Quan Z., Yao, L.: Uncovering locally discriminative structure for feature analysis. In: Frasconi, P., Landwehr, N., Manco, G., Vreeken, J. (eds.) ECML PKDD 2016. LNCS (LNAI), vol. 9851, pp. 281–295. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46128-1_18
Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems, pp. 5574–5584, Long Beach (2017)
Hein, M., Andriushchenko, M., Bitterwolf, J.: Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 41–50. IEEE, San Francisco (2019)
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. arXiv preprint arXiv:1706.04599 (2017)
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 (2016)
Jiang, H., Kim, B., Guan, M., Gupta, M.: To trust or not to trust a classifier. In: Advances in Neural Information Processing Systems, pp. 5541–5552, Montreal, Canada (2018)
Mandelbaum, A., Weinshall, D.: Distance-based confidence score for neural network classifiers. arXiv preprint arXiv:1709.09844 (2017)
Elhamifar, E., Vidal, R.: Sparse subspace clustering. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790–2797. IEEE (2009)
Vaswani, N., Bouwmans, T., Javed, S., Narayanamurthy, P.: Robust subspace learning: robust PCA, robust subspace tracking, and robust subspace recovery. IEEE Signal Process. Mag. 35(4), 32–55 (2018)
Bröcker, J., Smith, L.A.: Increasing the reliability of reliability diagrams. Weather Forecast. 22(3), 651–661 (2007)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, X., Yu, D., Lai, G. (2020). SS-AOE: Subspace Based Classification Framework for Avoiding Over-Confidence Errors. In: Yang, X., Wang, CD., Islam, M.S., Zhang, Z. (eds) Advanced Data Mining and Applications. ADMA 2020. Lecture Notes in Computer Science(), vol 12447. Springer, Cham. https://doi.org/10.1007/978-3-030-65390-3_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-65390-3_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-65389-7
Online ISBN: 978-3-030-65390-3
eBook Packages: Computer ScienceComputer Science (R0)