Abstract
Lasso proved to be an extremely successful technique for simultaneous estimation and variable selection. However lasso has two major drawbacks. First, it does not enforce any grouping effect and secondly in some situation lasso solutions are inconsistent for variable selection. To overcome this inconsistency adaptive lasso is proposed where adaptive weights are used for penalizing different coefficients. Recently a doubly regularized technique namely elastic net is proposed which encourages grouping effect i.e. either selection or omission of the correlated variables together. However elastic net is also inconsistent. In this paper we study adaptive elastic net which does not have this drawback. In this article we specially focus on the grouped selection property of adaptive elastic net along with its model selection complexity. We also shed some light on the bias-variance tradeoff of different regularization methods including adaptive elastic net. An efficient algorithm was proposed in the line of LARS-EN, which is then illustrated with simulated as well as real life data examples.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bondell, H.D., Reich, B.J.: Simultaneous regression shrinkage, variable selection and clustering of predictors with OSCAR. Biometrics 64, 115–123 (2008)
Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81, 425–455 (1994)
Efron, B.: The estimation of prediction error: covariance penalties and cross-validation. J. Am. Stat. Assoc. 99, 619–632 (2004)
Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Stat. 32, 407–499 (2004)
Fan, L., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96, 1348–1360 (2001)
Ghosh, S.: Adaptive Elastic Net: A Doubly Regularized method for variable selection to Achieve Oracle Properties. Tech. Rep. pr07-01, available at http://www.math.iupui.edu/research/preprints.php, IUPUI (2007)
Grandvalet, Y., Canu, S.: Outcomes of the equivalence of adaptive ridge with least absolute shrinkage. In: Advances in Neural Information Processing Systems II, pp. 445–451. MIT Press, Cambridge (1998)
Hoerl, A., Kennard, R.: Ridge regression: biased estimation for non-orthogonal problems. Technometrics 12, 55–67 (1970)
Huang, J., Ma, S., Zhang, C.-H.: Adaptive lasso for sparse high-dimensional regression models. Stat. Sin. 18, 1603–1618 (2008)
Jia, J., Yu, B.: On model selection consistency of the elastic net when p≫n. Stat. Sin. 20, 595–611 (2010)
Knight, K., Fu, W.: Asymptotic for lasso-type estimators. Ann. Stat. 28, 1356–1378 (2000)
Leng, C., Lin, Y., Wahba, G.: A note on inconsistency of lasso and related procedure. Stat. Sin. 16, 1273–1284 (2006)
Marquardt, W.D.: Generalized inverses, ridge regression, biased linear estimation, and nonlinear estimation. Technometrics 12, 591–612 (1970)
Stamey, T.A., Kabalin, J.N., McNeal, J.E., Johnstone, I.M., Freiha, F., Redwine, E.A., Yang, N.: Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate: II. Radical prostatectomy treated patients. J. Urol. 141, 1076–1083 (1989)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. (B) 58, 267–288 (1996)
Tibshirani, R., Saunders, M., Rosset, S., Zhu, Z., Knight, K.: Sparsity and smoothness via the fused lasso. J. R. Stat. Soc. (B) 67, 91–108 (2005)
Tutz, G., Ulbricht, J.: Penalized regression with correlation based penalty. Stat. Comput. 19, 239–253 (2009)
Wahba, G., Lin, X., Gao, F., Xiang, D., Klein, R., Klein, B.: The bias-variance tradeoff and the randomized GACV. In: Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II, pp. 620–626 (1999)
Wang, J., Li, G., Tsai, C.-L.: Regression coefficient and autoregressive order shrinkage and selection via the lasso. J. R. Stat. Soc. (B) 69, 63–78 (2007)
Zhang, H.H., Lu, W.: Adaptive Lasso for Cox’s proportional hazards model. Biometrika 94, 691–703 (2007)
Zou, H.: The adaptive lasso and its oracle properties. J. Am. Stat. Assoc. 101, 1418–1429 (2006)
Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. (B) 67, 301–320 (2005)
Zou, H., Zhang, H.: On the adaptive elastic-net with a diverging number of parameters. Ann. Stat. (2008, to appear)
Zou, H., Hastie, T., Tibshirani, R.: On the degrees of freedom of the lasso. Ann. Stat. 35, 2173–2192 (2007)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ghosh, S. On the grouped selection and model complexity of the adaptive elastic net. Stat Comput 21, 451–462 (2011). https://doi.org/10.1007/s11222-010-9181-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11222-010-9181-4