Abstract
This paper shows that near optimal rates of aggregation and adaptation to unknown sparsity can be simultaneously achieved via ℓ1 penalized least squares in a nonparametric regression setting. The main tool is a novel oracle inequality on the sum between the empirical squared loss of the penalized least squares estimate and a term reflecting the sparsity of the unknown regression function.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Baraud, Y.: Model selection for regression on a random design. ESAIM Probability & Statistics 7, 127–146 (2002)
Birgé, L.: Model selection for Gaussian regression with random design. Prépublication n. 783, Laboratoire de Probabilités et Modèles Aléatoires, Universités Paris 6 - Paris 7 (2002), http://www.proba.jussieu.fr/mathdoc/preprints/index.html#2002
Birgé, L., Massart, P.: Minimum contrast estimators on sieves: Exponential bounds and rates of convergence. Bernouilli 4, 329–375 (1998)
Bunea, F., Tsybakov, A., Wegkamp, M.H.: Aggregation for Gaussian regression (preprint, 2005), http://www.stat.fsu.edu/~wegkamp
Bunea, F., Wegkamp, M.: Two stage model selection procedures in partially linear regression. The Canadian Journal of Statistics 22, 1–14 (2004)
Candes, E., Tao,T.: The Dantzig selector: statistical estimation when p is much larger than n (preprint, 2005)
Catoni, O.: Statistical Learning Theory and Stochastic Optimization. In: Ecole d’Eté de Probabilités de Saint-Flour 2001. Lecture Notes in Mathematics, Springer, N.Y (2004)
Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81, 425–455 (1994)
Györfi, L., Kohler, M., Krzyżak, A., Walk, H.: A Distribution-Free Theory of Nonparametric Regression. Springer, N.Y (2002)
Juditsky, A., Nemirovski, A.: Functional aggregation for nonparametric estimation. Annals of Statistics 28, 681–712 (2000)
Kerkyacharian, G., Picard, D.: Tresholding in learning theory. Prépublication n.1017, Laboratoire de Probabilités et Modèles Aléatoires, Universités Paris 6 - Paris 7, http://www.proba.jussieu.fr/mathdoc/preprints/index.html#2005
Koltchinskii, V.: Model selection and aggregation in sparse classification problems. Oberwolfach Reports: Meeting on Statistical and Probabilistic Methods of Model Selection, (October 2005) (to appear)
Nemirovski, A.: Topics in Non-parametric Statistics. In: Ecole d’Eté de Probabilités de Saint-Flour XXVIII. Lecture Notes in Mathematics, vol. 1738, Springer, N.Y (2000)
Tibshirani, R.: Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, Series B. 58, 267–288 (1996)
Tsybakov, A.B.: Optimal rates of aggregation. In: Schölkopf, B., Warmuth, M.K. (eds.) COLT/Kernel 2003. LNCS (LNAI), vol. 2777, pp. 303–313. Springer, Heidelberg (2003)
Wegkamp, M.H.: Model selection in nonparametric regression. Annals of Statistics 31, 252–273 (2003)
Yang, Y.: Combining different procedures for adaptive regression. J.of Multivariate Analysis 74, 135–161 (2000)
Yang, Y.: Aggregating regression procedures for a better performance. Bernoulli 10, 25–47 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bunea, F., Tsybakov, A.B., Wegkamp, M.H. (2006). Aggregation and Sparsity Via ℓ1 Penalized Least Squares. In: Lugosi, G., Simon, H.U. (eds) Learning Theory. COLT 2006. Lecture Notes in Computer Science(), vol 4005. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11776420_29
Download citation
DOI: https://doi.org/10.1007/11776420_29
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-35294-5
Online ISBN: 978-3-540-35296-9
eBook Packages: Computer ScienceComputer Science (R0)