Abstract
This paper shows that near optimal rates of aggregation and adaptation to unknown sparsity can be simultaneously achieved via ℓ1 penalized least squares in a nonparametric regression setting. The main tool is a novel oracle inequality on the sum between the empirical squared loss of the penalized least squares estimate and a term reflecting the sparsity of the unknown regression function.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Baraud, Y.: Model selection for regression on a random design. ESAIM Probability & Statistics 7, 127–146 (2002)
Birgé, L.: Model selection for Gaussian regression with random design. Prépublication n. 783, Laboratoire de Probabilités et Modèles Aléatoires, Universités Paris 6 - Paris 7 (2002), http://www.proba.jussieu.fr/mathdoc/preprints/index.html#2002
Birgé, L., Massart, P.: Minimum contrast estimators on sieves: Exponential bounds and rates of convergence. Bernouilli 4, 329–375 (1998)
Bunea, F., Tsybakov, A., Wegkamp, M.H.: Aggregation for Gaussian regression (preprint, 2005), http://www.stat.fsu.edu/~wegkamp
Bunea, F., Wegkamp, M.: Two stage model selection procedures in partially linear regression. The Canadian Journal of Statistics 22, 1–14 (2004)
Candes, E., Tao,T.: The Dantzig selector: statistical estimation when p is much larger than n (preprint, 2005)
Catoni, O.: Statistical Learning Theory and Stochastic Optimization. In: Ecole d’Eté de Probabilités de Saint-Flour 2001. Lecture Notes in Mathematics, Springer, N.Y (2004)
Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81, 425–455 (1994)
Györfi, L., Kohler, M., Krzyżak, A., Walk, H.: A Distribution-Free Theory of Nonparametric Regression. Springer, N.Y (2002)
Juditsky, A., Nemirovski, A.: Functional aggregation for nonparametric estimation. Annals of Statistics 28, 681–712 (2000)
Kerkyacharian, G., Picard, D.: Tresholding in learning theory. Prépublication n.1017, Laboratoire de Probabilités et Modèles Aléatoires, Universités Paris 6 - Paris 7, http://www.proba.jussieu.fr/mathdoc/preprints/index.html#2005
Koltchinskii, V.: Model selection and aggregation in sparse classification problems. Oberwolfach Reports: Meeting on Statistical and Probabilistic Methods of Model Selection, (October 2005) (to appear)
Nemirovski, A.: Topics in Non-parametric Statistics. In: Ecole d’Eté de Probabilités de Saint-Flour XXVIII. Lecture Notes in Mathematics, vol. 1738, Springer, N.Y (2000)
Tibshirani, R.: Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, Series B. 58, 267–288 (1996)
Tsybakov, A.B.: Optimal rates of aggregation. In: Schölkopf, B., Warmuth, M.K. (eds.) COLT/Kernel 2003. LNCS (LNAI), vol. 2777, pp. 303–313. Springer, Heidelberg (2003)
Wegkamp, M.H.: Model selection in nonparametric regression. Annals of Statistics 31, 252–273 (2003)
Yang, Y.: Combining different procedures for adaptive regression. J.of Multivariate Analysis 74, 135–161 (2000)
Yang, Y.: Aggregating regression procedures for a better performance. Bernoulli 10, 25–47 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bunea, F., Tsybakov, A.B., Wegkamp, M.H. (2006). Aggregation and Sparsity Via ℓ1 Penalized Least Squares. In: Lugosi, G., Simon, H.U. (eds) Learning Theory. COLT 2006. Lecture Notes in Computer Science(), vol 4005. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11776420_29
Download citation
DOI: https://doi.org/10.1007/11776420_29
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-35294-5
Online ISBN: 978-3-540-35296-9
eBook Packages: Computer ScienceComputer Science (R0)