Fast robust regression algorithms for problems with Toeplitz structure

Abstract

The problem of computing an approximate solution of an overdetermined system of linear equations is considered. The usual approach to the problem is least squares, in which the 2-norm of the residual is minimized. This produces the minimum variance unbiased estimator of the solution when the errors in the observations are independent and normally distributed with mean 0 and constant variance. It is well known, however, that the least squares solution is not robust if outliers occur, i.e., if some of the observations are contaminated by large error. In this case, alternate approaches have been proposed which judge the size of the residual in a way that is less sensitive to these components. These include the Huber M-function, the Talwar function, the logistic function, the Fair function, and the ${\ell }_1$ norm. New algorithms are proposed to compute the solution to these problems efficiently, in particular, when the matrix A has small displacement rank. Matrices with small displacement rank include matrices that are Toeplitz, block-Toeplitz, block-Toeplitz with Toeplitz blocks, Toeplitz plus Hankel, and a variety of other forms. For exposition, only Toeplitz matrices are considered, but the ideas apply to all matrices with small displacement rank. Algorithms are also presented to compute the solution efficiently when a regularization term is included to handle the case when the matrix of the coefficients is ill-conditioned or rank-deficient. The techniques are illustrated on a problem of FIR system identification.

Introduction

Consider the approximation problem

$$\mathit{Ax}\approx b\text{,}$$

where $A\in {\mathbb{R}}^{m\times n}$ and $b\in {\mathbb{R}}^m$ ($m⩾n$) are given and $x\in {\mathbb{R}}^n$ is to be determined. We define the residual

$$r=b-\mathit{Ax}\text{.}$$

The usual approach to the problem is least squares, in which we minimize the 2-norm of the residual over all choices of x. This produces the minimum variance unbiased estimator of the solution when the errors in the observation b are independent and normally distributed with mean 0 and constant variance.

It is well known, however, that the least squares solution is not robust if outliers occur, i.e., if some of the components of b are contaminated by large error. In this case, alternate approaches have been proposed which judge the size of the residual in a way that is less sensitive to these components. For example, in order to reduce the influence of the outliers, we might replace the least squares problem

$$\underset{x}{\min }\parallel r{\parallel }_2\text{,}$$

by

$$\underset{x}{\min }\sum _{i=1}^m\rho (r_i(x))\text{,}$$

subject to $r=b-\mathit{Ax}$, where $\rho$ is a given function. This robust regression problem has been extensively studied. Taking $\rho (z)=z^2/2$ gives the ordinary linear least squares problem. Other possible functions include:

In Section 2, we consider how the solution to weighted problems can be computed efficiently, in particular, when the matrix A has small displacement rank (Kailath and Sayed, 1999, Kailath and Sayed, 1995). Matrices with small displacement rank include matrices that are Toeplitz, block-Toeplitz, block-Toeplitz with Toeplitz blocks (BTTB), Toeplitz plus Hankel, and a variety of other forms. This structure has been effectively exploited in solving least squares problems (Kailath and Sayed, 1999, Ng, 1996), weighted least squares problems (Benzi and Ng, 2006), total least squares problems (Kalsi and O’Leary, 2006; Mastronardi et al., 2001a, Mastronardi et al., 2004), and regression using other norms (Pruessner and O’Leary, 2003), so we now extend these ideas to robust regression. For exposition, in Section 2 we focus on Toeplitz matrices, but the ideas apply to all matrices with small displacement rank. In Section 3, we also show how to compute the solution efficiently when we include a regularization term in case the matrix A is ill-conditioned or rank-deficient. Section 4 extends these results to robust regression problems. Experiments illustrating these ideas are presented in Section 5, and Section 6 provides our conclusions.