Model selection using PRESS statistic

Abstract

The most popularly used statistic $R^2$ has a fundamental weakness in model building: it favors adding more predictors to the model because $R^2$ can only increase. In effect, additional predictors start fitting noise to the data. Other measures used in selecting a regression model such as $R^2_{adj}$, AIC, SBC, and Mallow’s $C_p$ does not guarantee that the model selected will also make better prediction of future values. To avoid this, data scientists withhold a percentage of the data for validation purposes. The PRESS statistic does something similar by withholding each observation in calculating its own predicted value. In this paper, we investigated the behavior of $R^2_{PRESS}$, and how it performs compared to other criterion in model selection in the presence of unnecessary predictors. Using simulated data, we found $R^2_{PRESS}$ has generally performed best in selecting the true model as the best model for prediction among the model selection measures considered.