A robust adaptive Lasso estimator for the independent contamination model

Highlights

• Robust variable selection and parameter estimation in the presence of outliers.

• A few highly contaminated predictors can cause existing robust estimators to break down.

• The Independent Contamination Model is a relatively new and realistic model for outliers.

• Combining a robust loss function with a sparsity and robustness inducing penalty term.

• Determining the temporal releases of the European Tracer Experiment (ETEX).

Abstract

The Lasso has become a benchmark method for simultaneous parameter estimation and variable selection in regression analysis. It is based on the least-squares estimator and, therefore, suffers from the presence of outliers. Robust Lasso methods combine the objective function of a robust estimator with ℓ₁-penalization. We address robustness for cases in which the number of observations is smaller (or not much larger) than the number of predictors. Further, we assume that the regression matrix may contain cellwise outliers. In such settings, even a few highly contaminated predictors can cause existing robust methods that are based on the commonly used rowwise contamination model to break down. Therefore, we propose a new adaptive Lasso type regularization. It takes into account cellwise outlyingness in the regression matrix and uses this information for robust variable selection. The proposed regularization term is integrated into the objective function of the MM-estimator, which yields the proposed MM-Robust Weighted Adaptive Lasso (MM-RWAL). A performance comparison to existing robust Lasso estimators is provided using Monte Carlo experiments. Further, the MM-RWAL is applied to determine the temporal releases of the European Tracer Experiment (ETEX) at the source location. This sparse and ill-conditioned linear inverse problem contains cellwise and rowwise outliers.

Introduction

Many of today’s signal processing problems can be formulated as a linear regression model

$$\mathbf{y}=\mathbf{X}\mathit{\beta }+\mathbf{u},$$

where $\mathbf{X}\in {\mathbb{R}}^{n\times p}$ is the predictor matrix, $\mathbf{u}\in {\mathbb{R}}^{n\times 1}$ is the error vector $\mathbf{y}\in {\mathbb{R}}^{n\times 1}$ is the response vector, and $\mathit{\beta }\in {\mathbb{R}}^{p\times 1}$ is the unknown parameter vector.

The presence of outliers and impulsive noise in linear regression problems has been reported in applications as diverse as wireless communication, ultrasonic systems, computer vision, electric power systems, automated detections of defects, biomedical signal analysis, genomics and the estimation of the temporal releases of a pollutant to the atmosphere. See [1], [2], [3], [4], [5], [6], [7], [8], [9], [10] and references therein. It is well-known that violations of the Gaussian noise assumption cause a drastic performance drop for the commonly used least-squares estimator (LSE) [11], [12], [13]

$${\widehat{\mathit{\beta }}}_{\text{LSE}}=\underset{\mathit{\beta }}{\text{arg}\text{min}}\parallel \mathbf{y}-\mathbf{X}\mathit{\beta }{\parallel }_2^2.$$

This has led to the development of robust estimators. For decades, the vast majority of robust linear regression estimators has focused on robustness against rowwise contamination. Under the so-called Tukey-Huber contamination model (THCM) [12], even for high-breakdown regression estimators, such as the LTS-, S-, MM-, and τ-estimators [1], [12], only a minority of the rows of X may be contaminated. In [14], Rousseuw and Van den Bossche state that the outlying rows paradigm is no longer sufficient for modern high-dimensional data sets. It often happens that most data cells (entries) in a row are regular and just a few of them are anomalous.

The case that independent cells of X are outliers is referred to as the independent contamination model (ICM) [15], [16], [17]. Only recently, the first cellwise robust regression estimation methods have been developed [16], [17]. Developing new cellwise robust estimators is essential for solving many real-world problems. For example, the estimation of the spatio-temporal emissions of a pollutant, given noisy observations, can be formulated as a linear inverse problem with the help of an atmospheric dispersion model [7]. The data of the European Tracer Experiment (ETEX) which was conducted in Monterfil, Brittany in 1994, where Perfluorocarbon (PFC) tracers were released into the atmosphere, for instance, contains both cellwise and rowwise outliers.

Additionally to the robustness considerations, atmospheric inverse problems, like many other problems in signal processing, require finding sparse solutions to under-determined, or ill-conditioned, linear systems of equations. For example, handling large data sets in terms of model interpretation, including the case where the number of explanatory variables p is larger than the sample size n, requires penalized estimators, such as the classical least absolute shrinkage and selection operator (Lasso) [18]

$${\widehat{\mathit{\beta }}}_{\mathrm{Lasso}}=\underset{\mathit{\beta }}{\text{arg}\text{min}}{\parallel \mathbf{y}-\mathbf{X}\mathit{\beta }\parallel }_2^2+\lambda {\parallel \mathit{\beta }\parallel }_1.$$

with $\lambda \in {\mathbb{R}}^{+}$.

Many other regularizations have been proposed [19], [20], [21], [22], [23]. In this paper the focus lies on Lasso estimation, to select a robust and interpretable model in high-dimensional settings. Zou [24] showed that the Lasso variable selection can be inconsistent, so that the oracle properties do not hold and proposed the adaptive Lasso

$${\widehat{\mathit{\beta }}}_{\mathrm{Lasso}}^{\mathrm{ad}}=\underset{\mathit{\beta }}{\text{arg}\text{min}}{\parallel \mathbf{y}-\mathbf{X}\mathit{\beta }\parallel }_2^2+\lambda \sum _{j=1}^p{\widehat{w}}_j\left|{\beta }_j\right|,$$

where ${\widehat{w}}_j=1/{\left|{\widehat{\beta }}_j\right|}^{\gamma }$ (γ > 0) are non-negative weights depending on $\widehat{\mathit{\beta }},$ which is a $\sqrt{n}$-consistent estimator of β. For a thorough discussion on consistency of estimators, see [25].

Just like the LSE, the Lasso and the adaptive Lasso rely on the Gaussian noise assumption and are sensitive to outliers. In recent years, some robust and regularized approaches have been proposed that replace the penalized square objective function by a penalized bounded objective function [8], [10], [26], [27], [28]. These methods, however, again, rely on the THCM.

Especially, in cases where the number of predictors p exceeds the number of rows n, it becomes more and more likely that a few highly contaminated predictors force the THCM-based estimators to flag all data points as outliers, which makes it impossible to draw any inferences from the data. This scenario is illustrated in Fig. 1, where a predictor matrix with n data points and p variables is depicted. In this example, the first predictor contains two outliers and the second predictor contains three outliers. For classical robust estimators such as the least trimmed squares estimator (LTS-estimator) [29] or the MM-estimator [12] and their sparsity inducing counterparts namely the Sparse LTS-estimator [27] and the Sparse MM-estimator [28], the depicted predictor matrix appears to be fully contaminated. The reason for this is that for these estimators a single contaminated cell in the predictor matrix leads to flagging the whole corresponding data point as an outlier which is then removed (LTS-estimator and Sparse LTS-estimator) or downweighted (MM-estimator and Sparse MM-estimator). This is depicted in the lower part of Fig. 1.

Such scenarios require methods that flag highly contaminated predictors and decrease their likelihood of being selected as active variables by Lasso type estimators.

Original Contributions: We propose and analyze a robustly weighted and adaptive Lasso type regularization term, which takes into account cellwise outliers for model selection. The proposed regularization term is integrated into the objective function of the MM-estimator, which yields the proposed MM-Robust Weighted Adaptive Lasso (MM-RWAL). The RWAL penalty can easily be integrated into the objective function of other robust estimators. A performance comparison to existing robust Lasso estimators is provided using Monte Carlo experiments. Further, a real-data application of estimating the sparse non-negative spatio-temporal emissions of a pollutant, given noisy observations $\mathbf{y}$ and an imprecisely estimated ill-conditioned and sparse dispersion model X, is considered. This example contains both cellwise and rowwise outliers.

Notation: Scalars are denoted by lowercase letters, e.g., x, column vectors by bold-faced lowercase letters, e.g. x, matrices by bold-faced uppercase letters, e.g. X, sets are denoted by calligraphic letters, e.g. $\mathcal{X}$ with associated cardinality $\left|\mathcal{X}\right|$. The jth column of a matrix X is denoted by x_j while (x)_{i: j} denotes the vector that contains the entries i to j of vector x. The ith element of vector x is denoted by x_i, ${\mathbf{I}}_p$ is the p-dimensional identity-matrix, 0_p is the p-dimensional all-zeros vector and diag(x) forms a matrix that contains the entries of x as its diagonal. $\widehat{\mathit{\beta }}$ refers to the estimator (or estimate) of the parameter vector β, ( · )^⊤ is the transpose operator. The derivative of a function f with respect to its argument is abbreviated by f′. P(X) is the probability of event X. Bin(1, ϵ) denotes the binomial distribution with one trial and a success probability of ϵ. Convergence to the normal distribution with mean vector μ and covariance matrix Σ is denoted by $\stackrel{d}{\to }\mathcal{N}(\mathbf{0},\mathit{\Sigma })$.

Organization: Section 2 discusses the Tukey-Huber and Independent Contamination models and motivates the use of cellwise robust methods. Section 3 introduces the proposed estimator and provides algorithms to compute the estimates. Section 4 provides numerical experiments, while Section 5 contains a real-data application of source estimation for an atmospheric inverse problem. Finally, Section 6 concludes the paper with a brief outlook on future work.