Abstract
Current machine learning algorithms perform well on many problem domains, but in risk-sensitive decision making, for example in medicine and finance, common evaluation methods that give overall assessments of models fail to gain trust among experts, as they do not provide any information about single predictions. We continue the previous work on approaches for evaluating the reliability of single classifications where we focus on methods that are model independent. These methods have been shown to be successful in their narrow fields of application, so we constructed a testing methodology to evaluate these methods in straightforward, general-use test cases. For the evaluation, we had to derive a statistical reference function, which enables comparison between the reliability estimators and the model’s own predictions. We compare five different approaches and evaluate them on a simple neural network with several artificial and real-world domains. The results indicate that reliability estimators CNK and LCV can be used to improve the model’s predictions.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Asuncion, A., Newman, D.J.: UCI Machine Learning Repository (2007), http://archive.ics.uci.edu/ml/
Bosnić, Z., Kononenko, I.: Comparison of approaches for estimating reliability of individual regression predictions. Data Knowl. Eng. 67(3), 504–516 (2008)
Kanji, G.K.: 100 statistical tests. SAGE Publications, Thousand Oaks (2006)
Kukar, M., Kononenko, I.: Reliable classifications with machine learning. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 1–8. Springer, Heidelberg (2002)
Kukar, M.: Quality assessment of individual classifications in machine learning and data mining. Knowledge and Information Systems 9(3), 364–384 (2006)
Ripley, B.D.: Pattern Recognition and Neural Networks, Cambridge (1996)
R Development Core Team: A Language and Environment for Statistical Computing. In: R Foundation for Statistical Computing, Vienna (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Pevec, D., Štrumbelj, E., Kononenko, I. (2011). Evaluating Reliability of Single Classifications of Neural Networks. In: Dobnikar, A., Lotrič, U., Šter, B. (eds) Adaptive and Natural Computing Algorithms. ICANNGA 2011. Lecture Notes in Computer Science, vol 6593. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20282-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-642-20282-7_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-20281-0
Online ISBN: 978-3-642-20282-7
eBook Packages: Computer ScienceComputer Science (R0)