The iteratively reweighted estimating equation in minimum distance problems

Abstract

The class of density based minimum distance estimators provide attractive alternatives to the maximum likelihood estimator because several members of this class have nice robustness properties while being first-order efficient under the assumed model. A helpful computational technique—similar to the iteratively reweighted least squares used in robust regression—is introduced which makes these estimators computationally much more feasible. This technique is much simpler than the Newton–Raphson (NR) method to implement. The loss suffered in the rate of convergence compared to the NR method can be made to vanish in some exponential family situations by a little modification in the weight function—in which case the performance is comparable to the NR method. For a large number of parameters the performance of this modified version is actually expected to be better than the NR method. In view of the widespread interest in density based robust procedures, this modification appears to be of great practical value.

Introduction

Minimum distance estimation forms an important subclass of statistical methodology. Originally, minimum distance functions were developed for goodness of fit purposes. The popular distances in the early literature were the Kolmogorov–Smirnov distance, the Cramér-von Mises distance, as well as weighted versions and other variants of these. The basic ingredient in this approach is the measurement of a distance between the data, summarized by the empirical distribution function, and the hypothesized probability distribution. During the last few decades statisticians have become increasingly aware of the potential of this approach in robust estimation. Much of the work in the minimum distance area was pioneered by Wolfowitz 1953, Wolfowitz 1954, Wolfowitz 1957 in the mid-1950s. There was a revival of this line of research in the early 1980s as evidenced by the works of Parr and Schucany (1980), Boos (1981), Parr and DeWet (1981), Heathcote and Silvapulle (1981), and others. Parr (1981) also provides an extensive bibliography of minimum distance estimation up to that point of time. Works of Wiens (1987), Hettmansperger et al. (1994), Özturk (1994), Özturk and Hettmansperger (1997), and Özturk et al. (1999) have further extended this line of research.

Many of the estimators proposed in the above papers have strong robustness properties under model misspecification. However their robustness is usually achieved at the cost of first order efficiency at the model. On the other hand, a second and a relatively more modern branch of minimum distance estimation, that based on density based distances (or divergences in general) has been shown to produce a large class of estimators which combine attractive robustness properties with full asymptotic efficiency. Beran (1977) appears to be the first to use a density based distance for the purpose of robust inference. He demonstrated that the minimum Hellinger distance estimators are simultaneously robust and first-order efficient. Other authors, such as Stather (1981), Tamura and Boos (1986), Simpson 1987, Simpson 1989 Donoho and Liu 1988a, Donoho and Liu 1988b, Eslinger and Woodward (1991), Basu and Harris (1994), Cao et al. (1995), Markatou (1996), Basu et al. (1997b), and Basu and Basu (1998) have further investigated related estimators. Lindsay (1994) introduced a class of minimum disparity estimators and illustrated the geometry behind their robustness. These ideas were extended to continuous models by Basu and Lindsay (1994). Also see Basu et al. (1997a) for a comprehensive review of minimum disparity inference.

The density based minimum distance estimators (or minimum disparity estimators in particular) provide attractive alternatives to the maximum likelihood estimator. However, the defining equations of the minimum disparity estimators are usually nonlinear and numerical methods have to be applied to solve them. The numerical difficulty increases greatly with the number of parameters. For example, to carry out the estimation of (μ,Σ) in a multidimensional normal model in d dimensions, there are p=d+d(d+1)/2=d(d+3)/2 unknown parameters. Each step of NR requires (p+1)(p+2)/2 numerical integrations and the inversion of a p dimensional Hessian matrix.

In this paper we consider a method closely related to iteratively reweighted least squares with the aim to reduce the numerical difficulty described in the previous paragraph. Our initial motivation came from the fact that the new method is vastly simpler to program and, in the d dimensional normal, requires (p+2) numerical integrations and no matrix inversion per step. Even for the case d=3, the number of parameters is 9 and so each NR step requires 55 numerical integrations and the inversion of a 9×9 matrix, while the new method requires only 11 numerical integrations per step. While the price one might expect to pay for this is a decrease from quadratic to linear convergence, our most striking finding was that by a careful (but very simple) selection of weights, we could make the method competitive in speed with NR even in the univariate model, where p=2. (A simple scalar adjustment makes the method quadratically convergent when the data exactly fit the model). Our theoretical calculations are substantiated by several numerical investigations. We believe this paper demonstrates generally applicable techniques for applying iterative reweighting algorithms in new problems and for making them more efficient. In particular we expect the algorithm described here to be of great practical use in view of the widespread interest in the minimum Hellinger distance and related methods.

The rest of the paper is organized as follows: In Section 2, we provide a brief review of minimum disparity estimation. The main contributions of the paper are presented in Section 3, where we first develop the iteratively reweighted estimating equation (IREE) algorithm in the spirit of iteratively reweighted least squares (IRLS) used in robust regression, and then demonstrate that a simple refinement can make the method comparable in performance to the NR algorithm, while keeping the implementation substantially simpler. Some further issues including a second order analysis, some discussion on the range of applicability of the method in small samples, and a weighted likelihood modification resulting from the IREE idea are discussed in Section 4. A small appendix presents a step by step implementation of the algorithm.