Abstract
Obtaining accurate bounds of the probability of overfitting is a fundamental question in statistical learning theory. In this paper we propose exact combinatorial bounds for the family of classifiers making a lattice. We use some lattice properties to derive the probability of overfitting for a set of classifiers represented by concepts. The extent of a concept, in turn, matches the set of objects correctly classified by the corresponding classifier. Conducted experiments illustrate that the proposed bounds are consistent with the Monte Carlo bounds.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bax, E.: Similar classifiers and VC error bounds. CalTech-CS-TR-97-14 (1997)
Botov, P.V.: Exact estimates of the probability of overfitting for multidimensional modeling families of algorithms. Pattern Recogn. Image Anal. 21(1), 52–65 (2011)
Bousquet, O., Elisseeff, A.: Algorithmic stability and generalization performance. In: Proceedings of the 2000 Conference on Advances in Neural Information Processing Systems, vol. 13, pp. 196–202. MIT Press (2001)
Bousquet, O., Elisseeff, A.: Stability and generalization. J. Mach. Learn. Res. 2, 499–526 (2002)
Chernoff, H.: A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. In: The Annals of Mathematical Statistics, pp. 493–507 (1952)
Ganter, B., Grigoriev, P.A., Kuznetsov, S.O., Samokhin, M.V.: Concept-based data mining with scaled labeled graphs. In: Wolff, K.E., Pfeiffer, H.D., Delugach, H.S. (eds.) ICCS-ConceptStruct 2004. LNCS, vol. 3127, pp. 94–108. Springer, Heidelberg (2004). doi:10.1007/978-3-540-27769-9_6
Ganter, B., Kuznetsov, S.O.: Hypotheses and version spaces. In: Ganter, B., Moor, A., Lex, W. (eds.) ICCS-ConceptStruct 2003. LNCS, vol. 2746, pp. 83–95. Springer, Heidelberg (2003). doi:10.1007/978-3-540-45091-7_6
Ganter, B., Wille, R.: Formal Concept Analysis: Mathematical Foundations. Springer, Heidelberg (1999)
Kashnitsky, Y., Ignatov, D.I.: Can FCA-based recommender system suggest a proper classifier? In: Proceedings of the International Workshop “What can FCA do for Artificial Intelligence?” (FCA4AI at ECAI 2014), CEUR Workshop Proceedings, pp. 17–26 (2014)
Kearns, M.J., Schapire, R.E.: Efficient distribution-free learning of probabilistic concepts. In: Proceedings of 31st Annual Symposium on Foundations of Computer Science, pp. 382–391. IEEE (1990)
Koltchinskii, V.: Rademacher penalties and structural risk minimization. IEEE Trans. Inf. Theor. 47(5), 1902–1914 (2001)
Koltchinskii, V., Panchenko, D.: Rademacher processes and bounding the risk of function learning. In: Giné, E., Mason, D.M., Wellner, J.A. (eds.) High Dimensional Probability II, vol. 47, pp. 443–457. Springer, Heidelberg (2000)
Koltchinskii, V., Panchenko, D.: Empirical margin distributions and bounding the generalization error of combined classifiers. Ann. Stat. 30(1), 1–50 (2002)
Kuznetsov, S.O., Poelmans, J.: Knowledge representation and processing with formal concept analysis. Wiley Interdisc. Rev.: Data Min. Knowl. Discov. 3(3), 200–215 (2013)
Langford, J.: Quantitatively tight sample complexity bounds. Ph.D. thesis/Carnegie Mellon thesis (2002)
McAllester, D.A.: Pac-Bayesian model averaging. In: Proceedings of the Twelfth Annual Conference on Computational Learning Theory, pp. 164–170. ACM (1999)
McDiarmid, C.: On the method of bounded differences. Surv. Comb. 141(1), 148–188 (1989)
Philips, P., et al.: Data-dependent analysis of learning algorithms (2005)
Seeger, M.: Pac-Bayesian generalisation error bounds for Gaussian process classification. J. Mach. Learn. Res. 3, 233–269 (2002)
Sill, J.: Monotonicity and connectedness in learning systems. Ph.D. thesis, California Institute of Technology (1998)
Talagrand, M.: Sharper bounds for Gaussian and empirical processes. Ann. Probab. 22(1), 28–76 (1994)
Talagrand, M.: Concentration of measure and isoperimetric inequalities in product spaces. Publications Mathématiques de l’Institut des Hautes Etudes Scientifiques 81(1), 73–205 (1995)
Tolstikhin, I.O.: The probability of overfitting for the compact and sparse sets of predictors. In: Intelligent Data Processing: Theory and Applications: IDP-8, pp. 83–86 (2010)
Vapnik, V.N., Chervonenkis, A.Y.: On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab. Appl. 16(2), 264–280 (1971)
Vorontsov, K.V.: Combinatorial probability and the tightness of generalization bounds. Pattern Recogn. Image Anal. 18(2), 243–259 (2008)
Vorontsov, K.V.: Exact combinatorial bounds on the probability of overfitting for empirical risk minimization. Pattern Recogn. Image Anal. 20(3), 269–285 (2010)
Vorontsov, K.V., Frey, A.I., Sokolov, E.A.: Computable combinatorial overfitting bounds. Mach. Learn. 1, 6 (2013)
Acknowledgments
The authors thank Daniel Borchmann, Dmitry Ignatov and Konstantin Vorontsov for discussion and helpful comments. This paper was prepared within the framework of the Basic Research Program at the National Research University Higher School of Economics (HSE) and supported within the framework of a subsidy by the Russian Academic Excellence Project ‘5–100’.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Makhalova, T., Kuznetsov, S.O. (2017). On Overfitting of Classifiers Making a Lattice. In: Bertet, K., Borchmann, D., Cellier, P., Ferré, S. (eds) Formal Concept Analysis. ICFCA 2017. Lecture Notes in Computer Science(), vol 10308. Springer, Cham. https://doi.org/10.1007/978-3-319-59271-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-59271-8_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-59270-1
Online ISBN: 978-3-319-59271-8
eBook Packages: Computer ScienceComputer Science (R0)