Variable selection for categorical response: a comparative study

Abstract

Variable selection is a well-studied problem in linear regression, but the existing works mostly deal with continuous responses. However, in many applications, we come across data with categorical responses. In the classical (frequentist) approach there exists penalized regression methods (e.g. logistic Lasso) which can be used for variable selection when we have a categorical response, and a large number of predictors. In this paper, we compare the performance of three alternative approaches for handling data with a single categorical response and multiple continuous (or count) predictors. In addition to the well-known logistic Lasso, we consider a model-based Bayesian approach, and a model-free approach for variable selection. We consider a binary response, and a response with three categories. Through extensive simulation studies we compare the performance of these three competing methods. We observe that the model-based methods can often accurately identify the important predictors, but sometimes fail to detect the unimportant ones. Also the model-based approaches are computationally expensive whereas the model-free approach is extremely fast. For misspecified models, the model-free method really outperforms in prediction. However, when the predictors are correlated (moderately or substantially) then the model-based methods perform better than the model-free method. We analyse the well-known Pima Indian Diabetes dataset for illustrating the effectiveness of three competing methods under consideration.