Error rates for multivariate outlier detection

Abstract

Multivariate outlier identification requires the choice of reliable cut-off points for the robust distances that measure the discrepancy from the fit provided by high-breakdown estimators of location and scatter. Multiplicity issues affect the identification of the appropriate cut-off points. It is described how a careful choice of the error rate which is controlled during the outlier detection process can yield a good compromise between high power and low swamping, when alternatives to the Family Wise Error Rate are considered. Multivariate outlier detection rules based on the False Discovery Rate and the False Discovery Exceedance criteria are proposed. The properties of these rules are evaluated through simulation. The rules are then applied to real data examples. The conclusion is that the proposed approach provides a sensible strategy in many situations of practical interest.

Introduction

With multivariate data, multiple outliers are revealed by their large distances from the robust fit provided by high-breakdown estimators of location and scatter (Hubert et al., 2008). An important issue is the occurrence of multiplicity problems when outlier detection is set up in a statistical testing framework. Multiplicity arises because the candidate outliers are not known in advance and all the observations are tested in sequence starting from the most remote one. Different error rates may be of interest when performing multiple tests. The multiplicity problem has not been considered thoroughly in the literature about outlier detection, even if there are notable exceptions like Becker and Gather (1999) and Davies and Gather (1993) who define outward testing procedures and use Sidak correction to guarantee that the level of swamping is below a threshold.

The goal of this paper is to show how carefully choosing the error rate to be controlled in multiple outlier detection can provide a reasonable compromise between good performance under the null hypothesis of no outliers and high power under contamination. In particular, we focus on multivariate outlier detection rules based on the False Discovery Rate (FDR) of Benjamini and Hochberg (1995), on the False Discovery eXceedance (FDX) of Lehmann and Romano (2005) and van der Laan et al. (2004), and we compare the power of the resulting outlier tests with those of alternative procedures attaining the same nominal size. We also evaluate the positive FDR (pFDR) of the procedures (Storey, 2002, Storey, 2003). We conclude that controlling these error rates, especially the FDR, can be a sensible strategy for outlier identification in many situations of practical interest.

The rest of the paper is as follows: in the remainder of this section we briefly review the error rates which are of interest in multiple testing. In Section 2 we set out our strategies for FDR and FDX control, and for pFDR estimation, in multivariate outlier identification. The merits of these strategies are illustrated with a simulation study in Section 3 and on two motivating examples in Section 4.

Let $y_i$ be a $v$-variate observation with mean vector $\mu$ and covariance matrix $\Sigma$. Our basic model explaining the genesis of $y_i$ is a two-component mixture model of the kind: $y_i|z_i\sim F_{z_i}$ for some unobserved $z_i\in \{0,1\}$. The clean observations arise from $F_0\sim N(\mu ,\Sigma )$, while the contaminated observations are those for which $z_i=1$, with $F_1$ arbitrary. Outlier detection is stated in terms of testing $n$ null hypotheses

$$H_{0i}:y_i\sim N(\mu ,\Sigma )\text{,}i=1,\dots ,n\text{.}$$

Each test is performed by computing the squared robust distance

$$d_i^2={(y_i-\tilde{\mu })}^{\prime }{\tilde{\Sigma }}^{-1}(y_i-\tilde{\mu })\text{,}$$

where $\tilde{\mu }$ and $\tilde{\Sigma }$ are high-breakdown estimators of $\mu$ and $\Sigma$. In this paper we take $\tilde{\mu }$ and $\tilde{\Sigma }$ to be the reweighted MCD (RMCD) estimators of Rousseeuw and Van Driessen (1999).

Suppose that there are $M_0$ clean observations and $M_1$ contaminated ones. $R$ is the number of observations declared to be outliers, i.e. those for which (1) is rejected. Table 1 summarizes the outcome of the outlier detection process. The values of $N_{0|1}$ and $N_{1|0}$ measure the amount of masking and swamping, respectively. Furthermore, the quantities in Table 1 are used to define error rates, which are deemed to be under control when they are bound, before the experiment, to be below a threshold $\alpha$.

Traditional methods in multiple testing involve control of the Family Wise Error Rate (FWER), defined as the probability of making one or more false rejections. There is a plethora of methods for FWER control, the simplest being the Bonferroni correction, which consists in performing each individual test at a level of $\alpha /n$. Another simple, but slightly more powerful, one-step procedure is Sidak correction, where each test is performed at a level of

$$\gamma =1-{(1-\alpha )}^{1/n}\text{.}$$

The observations selected after control of the FWER are all trusted to be outliers. The main drawback of FWER control is its low power. The consequences of FWER control may thus be close to those of masking.

A different approach is proposed by Benjamini and Hochberg (1995), who define the False Discovery Rate (FDR):

$$\mathrm{FDR}=E[\frac{N_{1|0}}{R}|R>0]Pr(R>0)\text{.}$$

The FDR is the expected proportion of erroneously rejected hypotheses, if any. The method developed by Benjamini and Hochberg (1995) (BH) is a stepwise procedure which proceeds by rejecting all tests corresponding to $p$-values below ${\rho }_i\alpha /n$, where ${\rho }_i$ is the rank of the $i$-th $p$-value. A very similar error rate, the positive FDR (pFDR), is defined by Storey, 2002, Storey, 2003 as

$$\mathrm{pFDR}=E[\frac{N_{1|0}}{R}|R>0]\text{,}$$

thus restricting to the cases in which there is at least one rejection. The pFDR has a nice Bayesian interpretation (Storey, 2003). It can be directly controlled or, as we do in this paper, it can be estimated to further evaluate the performance of any testing procedure. The pFDR is estimated by

$$\widehat{\mathrm{pFDR}}=\frac{\hat{a}{p}_{\left(r\right)}}{r(1-{(1-p_{\left(r\right)})}^n)}\text{,}$$

where $r>0$ denotes the observed value of $R,p_{\left(r\right)}$ is the largest $p$-value associated with rejected tests and $\hat{a}$ is an estimator of the number of true null hypotheses. In this paper we use the Schweder and Spjøtvoll (1982) estimator, and set $\hat{a}=2(n-{\tau }_{0.5})$, where ${\tau }_{0.5}$ denotes the count of $p$-values smaller than or equal to 0.5.

Both (4), (5) are based on an expectation, whereas the actual proportion of false discoveries may be larger than $\alpha$. Therefore, Lehmann and Romano (2005) and van der Laan et al. (2004) independently define the False Discovery eXceedance (FDX) as the probability of the false discovery proportion being above a threshold, that is,

$$\mathrm{FDX}=Pr(\frac{N_{1|0}}{max(R,1)}>c)\text{,}$$

where typically, and also in this paper, $c=0.1$. We control the FDX using the Lehmann and Romano (2005) (LR) procedure, which rejects all tests corresponding to $p$-values below $(\lfloor {\rho }_{i}c\rfloor +1)\alpha /(n+\lfloor {\rho }_{i}c\rfloor +1-{\rho }_i)$, where $\lfloor \cdot \rfloor$ is the integer part.

There is a plethora of other available methods, and error rates, for a review of which we refer to Farcomeni (2008). An important feature for our purposes is that the Bonferroni and Sidak corrections provide strong control of the FWER, which is bounded no matter the number and the configuration of outliers. Instead, the procedures based on the FDR and FDX ensure weak control of the FWER, which is then bounded only under the complete null hypothesis of no outliers

$$H_0:{\cap }_{i=1}^n{H}_{0i}\text{.}$$

The main consequence for outlier detection is that FDR (or FDX) control provides a balance between ignoring multiplicity, as in Hardin and Rocke (2005) or in Hubert et al. (2008), and strictly correcting for multiplicity through FWER control, as in Becker and Gather (1999) or in Cerioli et al. (2009). The improvement obtained by controlling (4) or (7) may be particularly advantageous when $n$ is high, or when many samples of moderate size need to be analyzed in sequence. In such instances the total number of hypotheses (1) to be tested will be large and the loss of power induced by strong FWER control will become more relevant.