Abstract
The previous results describing the generalization ability of Empirical Risk Minimization (ERM) algorithm are usually based on the assumption of independent and identically distributed (i.i.d.) samples. In this paper we go far beyond this classical framework by establishing the first exponential bound on the rate of uniform convergence of the ERM algorithm with V-geometrically ergodic Markov chain samples, as the application of the bound on the rate of uniform convergence, we also obtain the generalization bounds of the ERM algorithm with V-geometrically ergodic Markov chain samples and prove that the ERM algorithm with V-geometrically ergodic Markov chain samples is consistent. The main results obtained in this paper extend the previously known results of i.i.d. observations to the case of V-geometrically ergodic Markov chain samples.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bartlett, P.L., Lugosi, G.: An inequality for uniform deviations of sample averages from their means. Stat. Probab. Lett. 4, 55–62 (1999)
Bousquet, O.: New approaches to statistical learning theory. Ann. Inst. Stat. Math. 55, 371–389 (2003)
Chen, D.R., Wu, Q., Ying, Y.M., Zhou, D.X.: Support vector machine soft margin clossifiers: error analysis. J. Mach. Learn. Res. 5, 1143–1175 (2004)
Cucker, F., Smale, S.: On the mathematical foundations of learning. Bull. Am. Math. Soc. 39, 1–49 (2001)
Cucker, F., Smale, S.: Best choices for regularization parameters in learning theory: on the bias-variance problem. Found. Comput. Math. 2, 413–428 (2002)
Cucker, F., Zhou, D.X.: Learning theory: An approximation theory viewpoint. Cambridge University Press, Cambridge (2007)
Devroye, L.: Bounds for the uniform deviation of empirical measures. J. Multivar. Anal. 12, 72–79 (1982)
Gamarnik, D.: Extension of the PAC framework to finite and countable Markov chains. IEEE Trans. Inf. Theory 49, 338–345 (2003)
Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58, 13–30 (1963)
Modha, S., Masry, E.: Minimum complexity regression estimation with weakly dependent observations. IEEE Trans. Inf. Theory 42, 2133–2145 (1996)
Meyn, S.P., Tweedie, R.L.: Markov chains and stochastic stability. Springer (1993)
Smale, S., Zhou, D.X.: Estimating the approximation error in learning theory. Anal. Appl. 1, 17–41 (2003)
Smale, S., Zhou, D.X.: Shannon sampling and function reconstruction from point values. Bull. Am. Math. Soc. 41, 279–305 (2004)
Smale, S., Zhou, D.X.: Online learning with Markov sampling. Anal. Appl. 7, 87–113 (2009)
Steinwart, I., Christmann, A.: Fast learning from non-i.i.d. observations. Adv. Neural Inf. Process. Syst. 22, 1768–1776 (2009)
Steinwart, I., Hush, D., Scovel, C.: Learning from dependent observations. Multivariate Anal. 100, 175–194 (2009)
Sun, H.W., Wu, Q.: Regularized least square regression with dependent samples. Adv. Comput. Math. 32, 175–189 (2010)
Talagrand, M.: Sharper bounds for Gaussian and empirical processes. Ann. Probab. 22, 28–76 (1994)
Vidyasagar, M.: Learning and generalization with applications to neural networks, 2nd edn. Springer, London (2003)
Vapnik, V.: Statistical learning theory. John Wiley, New York (1998)
Wu, Q., Zhou, D.X.: SVM soft margin classifiers: linear programming versus quadratic programming. Neural Comput. 17, 1160–1187 (2005)
Xu, Y.L., Chen, D.R.: Learning rates of regularized regression for exponentially strongly mixing sequence. J. Statist. Plann. 138, 2180–2189 (2008)
Yu, B.: Rates of convergence for empirical processes of stationary mixing sequences. Ann. Probab. 22, 94–114 (1994)
Zhou, D.X.: Capacity of reproducing kernel spaces in learning theory. IEEE Trans. Inf. Theory 49, 1743–1752 (2003)
Zou, B., Li, L.Q.: The performance bounds of learning machines based on exponentially strongly mixing sequence. Comput. Math. Appl. 53, 1050–1058 (2007)
Zou, B., Li, L.Q., Xu, Z.B.: The generalization performance of ERM algorithm with strongly mixing observations. Mach. Learn. 75, 275–295 (2009)
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Ding Xuan Zhou.
This work is supported in part by National 973 project (2007CB311002), NSFC key project (70501030), NSFC project (61070225), FHEC (Q20091003) and China Postdoctoral Science Foundation (20080440190, 200902592).
Rights and permissions
About this article
Cite this article
Zou, B., Xu, Z. & Chang, X. Generalization bounds of ERM algorithm with V-geometrically Ergodic Markov chains. Adv Comput Math 36, 99–114 (2012). https://doi.org/10.1007/s10444-011-9182-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10444-011-9182-7