Sparse inverse covariance learning of conditional Gaussian mixtures for multiple-output regression

Abstract

We consider the task of multiple-output regression where both input and output are high-dimensional. Due to the limited amount of training samples compared to data dimensions, properly imposing loose statistical dependency in learning a regression model is crucial for reliable prediction accuracy. The sparse inverse covariance learning of conditional Gaussian random fields has been recently emerging to achieve this goal, shown to exhibit superior performance to non-sparse approaches. However, one of its main drawbacks is the strong assumption of linear Gaussianity in modeling the input-output relationship. For certain application domains, the assumption might be too restricted and less powerful in representation, and consequently, prediction based on the wrong models can result in suboptimal performance. In this paper, we extend the idea of sparse learning to a non-Gaussian model, especially the powerful conditional Gaussian mixture. For this latent-variable model, we propose a novel sparse inverse covariance learning algorithm based on the expectation-maximization lower-bound optimization technique. It is shown that each M-step reduces to solving the regular sparse inverse covariance estimation of linear Gaussian models, in conjunction with estimating sparse logistic regression. We demonstrate the improved prediction performance of the proposed algorithm over exisitng methods on several datasets.