Penalized factor mixture analysis for variable selection in clustered data

Abstract

A model-based clustering approach which contextually performs dimension reduction and variable selection is presented. Dimension reduction is achieved by assuming that the data have been generated by a linear factor model with latent variables modeled as Gaussian mixtures. Variable selection is performed by shrinking the factor loadings though a penalized likelihood method with an L1 penalty. A maximum likelihood estimation procedure via the EM algorithm is developed and a modified BIC criterion to select the penalization parameter is illustrated. The effectiveness of the proposed model is explored in a Monte Carlo simulation study and in a real example.

Introduction

Model-based clustering is recently receiving a wide interest in statistics. According to this approach if a sample of observations arises from some underlying populations of unknown proportions, its distributional form is specified in each of the underlying populations and the purpose is to decompose the sample into its mixture components (Mclachlan and Peel, 2000). For quantitative data each mixture component is usually modeled as a multivariate Gaussian distribution.

When model-based clustering is performed on a large number of observed variables, it is well known that Gaussian mixture models represent an over-parameterized solution as, besides the mixing weights, it is required to estimate the mean vector and the covariance matrix for each component (Mclachlan and Peel, 2000). In order to avoid over-parameterized solutions associated with very computationally intensive procedures, some strategies have been proposed in the statistical literature aimed at parameterizing the generic component-covariance matrix (see Banfield and Raftery (1993)) or at performing dimensional reduction in each component through latent variables (see for instance Ghahramani and Hilton (1997) and Mclachlan et al. (2003)).

Since clustering methods are strongly derailed by the presence of non-informative variables, there has recently also been an increasing interest in variable selection for model-based clustering, mostly within the Bayesian framework (see Liu et al. (2003) and Hoff (2005)), but with contributes also in the frequentist one (see, for instance, Pan and Shen (2007)). The main idea of these proposals is to parameterize the mean vector of the $k$th cluster as ${\mu }_k=\mu +{\delta }_k$, where $\mu$ is the global mean. If some components of ${\delta }_k$ are 0, then the corresponding attributes are non-informative to clustering, at least as far as the cluster location is concerned. A different approach, followed by Raftery and Dean (2006), considers a stepwise variable selection procedure based on the Bayesian Information Criterion (BIC).

In this paper we address both issues simultaneously by assuming that the data have been generated by a linear factor model with latent variables modeled as Gaussian mixtures and by shrinking the factor loadings, resorting to a penalized likelihood method with an L1 penalty.

The paper is organized as follows. In Section 2 we first briefly review the standard model-based clustering and present the dimension reduction approach; afterwards we propose an implementation with an $L_1$ penalty resulting in soft thresholding on the estimated factor loadings and thus realizing automatic variable selection. In Section 3 the EM algorithm for obtaining the penalized model estimates and a modified BIC criterion for selecting the penalization parameter are illustrated. Section 4 shows the experimental results: the proposed model is first evaluated on a Monte Carlo simulation study and then on a real example on thyroid classification with added irrelevant variables.