‘Nearly’ universally optimal designs for models with correlated observations

https://doi.org/10.1016/j.csda.2013.02.002Get rights and content

Abstract

The problem of determining optimal designs for least squares estimation is considered in the common linear regression model with correlated observations. The approach is based on the determination of ‘nearly’ universally optimal designs, even in the case where the universally optimal design does not exist. For this purpose, a new optimality criterion which reflects the distance between a given design and an ideal universally optimal design is introduced. A necessary condition for the optimality of a given design is established. Numerical methods for constructing these designs are proposed and applied for the determination of optimal designs in a number of specific instances. The results indicate that the new ‘nearly’ universally optimal designs have good efficiencies with respect to common optimality criteria.

Introduction

Consider the common linear regression model y(x)=θ1f1(x)++θmfm(x)+ε(x), where f1(x),,fm(x) are linearly independent continuous functions, ε(x) denotes a random error process or field, θ1,,θm are unknown parameters, and x is the explanatory variable, which varies in a compact design space XRd. We assume that N observations, say y1,,yN, can be taken at experimental conditions x1,,xN to estimate the parameters in the linear regression model (1). Suppose that ε(x) is a stochastic process with E[ε(x)]=0,E[ε(x)ε(x)]=K(x,x),x,xX. Throughout this paper, we call the function K(x,x) a covariance kernel. An important case appears when the error process is stationary and the covariance kernel is of the form K(x,x)=σ2ρ(xx). If ρ(0)=1, the function ρ() is called the correlation function, and, if ρ(t) as t0, the function ρ() is a singular covariance function. Regression models with correlated errors are often used in practice, for example, in analysis of spatial models (Fedorov, 1996, Müller, 2007), computer experiments (Bates et al., 1996), and nonlinear models of chemical processes (Dette et al., 2010, Ucinski and Atkinson, 2004).

If N observations, say Y=(y1,,yN)T, are available at experimental conditions x1,,xN, and the covariance kernel is known, then the vector of parameters can be estimated by the weighted least squares method, that is, by θˆ=(XTΣ1X)1XTΣ1Y, where X=(fi(xj))j=1,,Ni=1,,m is an N×m matrix and Σ=(K(xi,xj))i,j=1,,N is an N×N matrix. We assume that points x1,,xN are such that matrices Σ and XTΣ1X are invertible. Note that the estimator θˆ is the best unbiased linear estimator (BLUE) of θ, and its variance–covariance matrix is given by V ar(θˆ)=(XTΣ1X)1. If the correlation structure of the process is not known, one usually uses the ordinary least squares estimator θ̃=(XTX)1XTY, which has the covariance matrix V ar(θ̃)=(XTX)1XTΣX(XTX)1.

An exact experimental design {x1,,xN} is a collection of N points from the design space X, which defines the time points or experimental conditions where observations are taken. Optimal designs for weighted or ordinary least squares estimation minimize a functional of the covariance matrix of the weighted or ordinary least squares estimator, respectively, and numerous optimality criteria have been proposed in the literature to discriminate between competing designs (see  Pukelsheim, 2006).

Exact optimal designs for specific linear models with correlated observations have been investigated in Dette et al. (2008b), Kiseľák and Stehlík (2008), and Harman and Štulajter (2010). Because even in simple models exact optimal designs are difficult to find, most authors use asymptotic arguments to determine efficient designs for the estimation of the model parameters (see Sacks and Ylvisaker, 1966, Sacks and Ylvisaker, 1968, Bickel and Herzberg, 1979, or Zhigljavsky et al., 2010).

Sacks and Ylvisaker, 1966, Sacks and Ylvisaker, 1968 and Näther (1985, Chapter 4) assumed that the design points {x1,,xN} are generated by the quantiles of a distribution function; that is, xi=a((i1)/(N1)),i=1,,N, where the function a:[0,1]X is the inverse of a distribution function. Let ξN denote a normalized design supported at N points {x1,,xN} with the weight 1/N assigned to each point. Then the covariance matrix of the least squares estimator θ̃ given in (3) can be represented as V ar(θ̃)=D(ξN)=M1(ξN)B(ξN,ξN)M1(ξN), where the matrices M(ξ) and B(ξ,ν) are defined by M(ξ)=f(u)fT(u)ξ(du),B(ξ,ν)=K(u,v)f(u)fT(v)ξ(du)ν(dv), respectively (the integration is always taken over the set X), and f(u)=(f1(u),,fm(u))T denotes the vector of regression functions. We call any probability measure ξ on X an approximate design or simply a design; however, its interpretation in practice is different from the one given in Kiefer (1974), where, in the case of a discrete design ξ={x1,,xn;w1,,wn}, the weight wi means the relative proportion of observations performed at the point xi. In the case of correlated errors, only one realization of a stochastic process is usually observed, implying that no replication of design points is needed, and in practice design points are computed as quantiles of the cumulative distribution function defined by ξ; this rule applies independently of whether ξ is a discrete or continuous probability measure (or a mixture of the two). If some points in the collection of quantiles replicate (this can happen if N is small and the optimal design is discrete), we can replace the points which replicate by other points that are near to them. The definitions of the matrices M(ξ) and B(ξ,ξ) can be extended to an arbitrary design ξ, provided that the corresponding integrals exist. The matrix D(ξ)=M1(ξ)B(ξ,ξ)M1(ξ) is called the covariance matrix for the design ξ, and can be defined for any probability measure ξ supported on the design space X such that the matrices B(ξ,ξ) and M1(ξ) are well defined. This set will be denoted by Ξ. We assume that the design set X has enough points so that the set Ξ is non-empty; that is, there exists at least one design ξΞ.

Optimal designs for regression models with dependent data have been investigated mainly for the location scale model. The difficulties in a general development of the optimal design theory for correlated observations can be explained by the different structure of the covariance of the least squares estimator in model (1), which is of the form M1BM1. As a consequence, the corresponding design problems are in general not convex (except for the location scale model where m=1 and f1(u)1). Recently, Dette et al. (2011) derived universally optimal designs for regression models of arbitrary dimension if the corresponding regression functions are eigenfunctions of an integral operator defined by the covariance kernel of the error process. For example, the design with arcsine density is universally optimal for the polynomial model with logarithmic covariance kernel. On the other hand, there are many situations where this assumption is not satisfied, and in these cases there may not exist a universally optimal design.

The present paper is devoted to the numerical construction of ‘nearly’ universally optimal designs for regression models in such situations. This means that we consider model (1) with m>1 parameters in the case where a universally optimal design does not exist. In Section  2, we introduce a new optimality criterion which reflects the distance between a given design and an ideal universally optimal design. A necessary condition for the optimality of a given design is established in Section  3, and an algorithm for its numerical determination is proposed in Section  4. Finally, some illustrative examples are given in Section  5, where we calculate ‘nearly’ universally optimal designs for a quadratic regression model and a nonlinear model with various correlation functions. The results indicate that the new ‘nearly’ universally optimal designs have good efficiencies with respect to common optimality criteria.

Section snippets

A new optimality criterion

Throughout this paper we assume that the kernel K in (6) is continuous at all points (x,x)X×X except possibly the diagonal points (x,x). We also assume that K(x,x)0 for at least one pair (x,x) with xx. Singular kernels appear naturally if the approach in Bickel and Herzberg (1979) for the approximation of the covariance matrix in (3) is extended such that the variance of the observations also depends on the sample size (see Zhigljavsky et al. (2010) for details and Dette et al. (2011)

Necessary condition for optimality

In the case of correlated observations, optimality criteria are generally not convex; see Zhigljavsky et al. (2010). Therefore, standard optimality theorems do not give full characterizations of optimal designs. These results only provide necessary conditions for the optimality of a design. Theorem 1 below is not an exception, and it only gives a necessary condition for g-optimality.

Theorem 1

If the design ξ is g-optimal for the linear regression model   (1), thenφ(x,ξ)Φ(ξ)for all   xX,where the

An algorithm for construction of optimal designs

The fact that the optimality theorems only give a necessary condition for design optimality does not usually create additional problems for the algorithms of construction of designs. Numerical computation of optimal designs for the common linear regression model (1) with given correlation function can be performed by an extension of the multiplicative algorithm proposed by Dette et al. (2008c) for the case of non-correlated observations (see also  Yu, 2010, for some extensions). Note that the

Examples

In this section, we provide some numerical results which we have obtained applying the algorithm described in Section  4 for the calculation of g-optimal designs in several regression models. In the tables below we shall use the following notation for the D-efficiency, A-efficiency, and g-efficiency of a design  ξ: EffD(ξ)=(detD(ξD)detD(ξ))1/m,EffA(ξ)=Tr(D(ξA))Tr(D(ξ)), and Effg(ξ)=(gξgT(x)gξg(x)dxgξT(x)gξ(x)dx)1/2. Here, ξD is the D-optimal design, ξA is the A-optimal design, ξg is

Acknowledgments

The authors thank both referees for very valuable comments.

This work has been supported in part by the Collaborative Research Center “Statistical modeling of nonlinear dynamic processes” (SFB 823, Teilprojekt C2) of the German Research Foundation (DFG). The work of Pepelyshev was partly supported by the Russian Foundation of Basic Research, project 12-01-00747. Parts of this paper were written while H. Dette was visiting the Institute of Mathematical Sciences at the National University of

References (28)

  • H. Dette et al.

    Improving updating rules in multiplicative algorithms for computing D-optimal designs

    Computational Statistics and Data Analysis

    (2008)
  • V.V. Fedorov

    Design of spatial experiments: model fitting and prediction

  • J. Kiseľák et al.

    Equidistant and D-optimal designs for parameters of Ornstein–Uhlenbeck process

    Statistics & Probability Letters

    (2008)
  • A.C. Atkinson et al.

    Optimum experimental designs for properties of a compartmental model

    Biometrics

    (1993)
  • R.A. Bates et al.

    Experimental design and observation for large systems

    Journal of the Royal Statistical Society, Series B

    (1996)
  • P.J. Bickel et al.

    Robustness of design against autocorrelation in time I: asymptotic theory, optimality for location and linear regression

    Annals of Statistics

    (1979)
  • S. Biedermann et al.

    Maximin optimal designs for a compartmental model. MODA 7

    Advances in Model-oriented Design and Analysis

    (2004)
  • G.E.P. Box et al.

    Design of experiments in non-linear situations

    Biometrika

    (1959)
  • K. Chaloner et al.

    Bayesian experimental design: a review

    Statistical Science

    (1995)
  • H. Chernoff

    Locally optimal designs for estimating parameters

    Annals of Mathematical Statistics

    (1953)
  • H. Dette

    Designing experiments with respect to standardized optimality criteria

    Journal of the Royal Statistical Society, Series B

    (1997)
  • H. Dette et al.

    Optimal designs for dose-finding studies

    Journal of the American Statistical Association

    (2008)
  • H. Dette et al.

    Exact optimal designs for weighted least squares analysis with correlated errors

    Statistica Sinica

    (2008)
  • H. Dette et al.

    Optimality criteria for regression models based on predicted variance

    Biometrika

    (1999)
  • Cited by (2)

    • Special issue on algorithms for design of experiments

      2014, Computational Statistics and Data Analysis
    View full text