Abstract
We address the problem of determination of the size of the test set which can can guarantee statistically significant results in classifier error estimation and in selection of the best classifier from a given set. We focus on the case of the 0-1 valued loss function and we provide one and two sides optimal bounds for Validation (known also as Hold-Out Estimate and Train-and-Test Method). We also calculate the smallest sample size, necessary for obtaining the bound for given estimation accuracy and reliability of estimation, and we present the results in tables. Finally, we propose strategies for classifier design using the bounds derived.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Cucker, F., Smale, S.: On the mathematical foundations of learning. Bulletin of AMS 39, 1–49 (2001)
Duda, R., Hart, P., Stock, D.: Pattern Classification. John Wiley & Sons, Inc., Chichester (2001)
Friedman, J.H., Hastie, T., Tibshirani, R.: Statistical Learning: Data Mining, Inference, and Prediction. Springer, Heidelberg (2001)
Fukunaga, K., Hayes, R.R.: Effects of Sample Size in Classifier Design. IEEE Transactions on Pattern Analysis and Machine Intelligence 11(8), 873–885 (1989)
Fukunaga, K., Hayes, R.R.: Estimation of Classifier Performance. IEEE Transactions on Pattern Analysis and Machine Intelligence 11(10), 1087–1101 (1989)
Guyon, I., Makhoul, J., Schwartz, R., Vapnik, V.: What size test set gives good error rate estimates? IEEE Pattern Analysis and Machine Intelligence 20, 52–64 (1998)
Hoeffding, W.: Probability Inequalities for Sums of Bounded Random Variables. JASA 58, 13–30
Jaworski, W.: Bounds for Validation. Fundamenta Informaticae (to appear)
Michie, D., Spiegelhalter, D., Taylor, C. (eds.): Machine Learning, Neural and Statistical Classification. John Wiley & Sons, Inc, Chichester (2001)
Vapnik, V.N.: Statistical Learning Theory. Wiley, New York (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Jaworski, W. (2005). Model Selection and Assessment for Classification Using Validation. In: Ślęzak, D., Wang, G., Szczuka, M., Düntsch, I., Yao, Y. (eds) Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing. RSFDGrC 2005. Lecture Notes in Computer Science(), vol 3641. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11548669_50
Download citation
DOI: https://doi.org/10.1007/11548669_50
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-28653-0
Online ISBN: 978-3-540-31825-5
eBook Packages: Computer ScienceComputer Science (R0)