Application of imperialist competitive algorithm to find minimax and standardized maximin optimal designs

Abstract

Finding optimal designs for nonlinear models is complicated because the design criterion depends on the model parameters. If a plausible region for these parameters is available, a minimax optimal design may be used to remove this dependency by minimizing the maximum inefficiency that may arise due to misspecification in the parameters. Minimax optimal designs are often analytically intractable and are notoriously difficult to find, even numerically. A population-based evolutionary algorithm called imperialist competitive algorithm (ICA) is applied to find minimax or nearly minimax $D$-optimal designs for nonlinear models. The usefulness of the algorithm is also demonstrated by showing it can hybridize with a local search to find optimal designs under a more complicated criterion, such as standardized maximin optimality.

Introduction

A wide class of evolutionary algorithms has been increasingly used to solve hard optimization problems in engineering, bioinformatics, computer science and finance. Of particular interest is the class of nature-inspired algorithms motivated from the influence of biology and the life sciences. Examples of such algorithms frequently used in the statistics literature are simulated annealing (SA) and genetic algorithms (GA). There are more recent and potentially more powerful ones such as differential evolution (DE), particle swarm optimization (PSO), imperialist competitive algorithm (ICA) and cuckoo search (CS) that have not yet been well tested for solving statistical problems. Our interest in this paper is application of one such algorithm to construct optimal experimental designs. The usefulness of implementing such designs in terms of cost saving and accurate statistical inferences is detailed in Atkinson (1996). An introduction to this subfield of optimal experimental designs is available in Berger and Wong (2009) and real applications of optimal designs can be found in Berger and Wong (2005).

A common appeal of such algorithms is that they are mainly assumptions free, fast, easy to implement and are broadly applicable to different types of constrained or unconstrained optimization problems. Consequently, they have good potential to optimize complicated functions with many variables regardless whether the objective function is differentiable or not. A common feature among these algorithms is that there require tuning parameters and if they are well chosen, the algorithm finds the optimum very fast. If the tuning parameters are poorly chosen, the algorithm does not give satisfactory answers. These algorithms do not usually have a firm theoretical basis, such as proof of its convergence to the optimum. However, these algorithms have been used successfully in many applied fields to solve real, complicated and high dimensional optimization problems where traditional formulations or methods fail. Our view is that the lack of proof of convergence should not hinder their use in statistics; for some problems such as the ones we work with, there is a theory for verifying whether the generated design is optimum, and if it is not, theory is available to assess its proximity to the optimum without knowing the optimum. We next briefly review selected applications of such algorithms for finding optimal designs in the literature.

Simulated annealing was first proposed by Kirkpatrick et al. (1983) and Meyer and Nachtsheim (1988) appeared to be among the first to use SA and constructed exact $D$-optimal designs for both finite and continuous design spaces. Haines (1987) applied SA to construct exact $D$-, $I$-, and $G$-optimal designs for polynomial regression models. Atkinson (1992) discussed starting values for SA with a focus on optimal design construction and recommended segmenting the search to a maximum number of evaluations. Other applications of SA include Zhou (2008), who found exact minimax $D$-optimal design on discrete design spaces, and Wilmut and Zhou (2011), who constructed $D$-optimal minimax two-level fractional factorial designs using a sequential algorithm. Woods (2010) implemented SA to obtain exact optimal designs for binary models under the optimum-in-average criterion. This algorithm is one of the very few that can be shown to converge to the optimum.

GA was proposed by Holland and John (0000) and has been applied to search for exact optimal designs mainly for linear models. An early proponent is Montepiedra et al. (1998), who found exact optimal designs for polynomial models; others include Heredia-Langner et al. (2003), Drain et al. (2004) and Mandal et al. (2015). Hamada et al. (2001) used GA to find near-optimal Bayesian experimental designs for linear and nonlinear regression and dichotomous data. A most recent review on the application of GA to solve optimal design problems is given in Lin et al. (2015).

Particle swarm optimization (PSO) algorithm proposed by Eberhart and Kennedy (1995) has emerged to be a popular tool for solving real world optimization problems. In PSO terminology, each possible solution is called a “particle”. Similar to many evolutionary algorithms, PSO is initialized with a population of random particles, called a “swarm”. Each particle starts to fly through the problem space with its own “velocity” that is being updated in every iteration according to the particle’s best position and the best global position over the swarm. PSO has been used to find several types of optimal designs for different problems. For example, Qiu et al. (2014) applied PSO to find locally $D$- and $c$-optimal designs for the compartmental, logistic and double exponential models and comparing PSO performance with the differential evolution algorithm proposed by Storn and Price (1997). Wong et al. (2015) demonstrated the usefulness of using PSO to find various types of optimal designs by applying it to solve several types of optimal design problems for different mixture models defined over a regular or irregular simplex. In addition, Chen et al. (2014) modified PSO to find minimax optimal designs for the logistic and enzyme kinetic models. Such optimal designs are notoriously difficult to find because the design criterion is non-differentiable and involves two layers of optimization.

The main goal of this paper is to investigate the capability of the Imperialist Competitive Algorithm (ICA) for finding optimal designs. Our work appears to be the first to use ICA for a statistical application. ICA is a meta-heuristic evolutionary algorithm inspired from the socio-political process of humans and proposed in Atashpaz-Gargari and Lucas (2007). In this sense, it is different from the above mentioned nature-inspired algorithms, which are inspired by animal behavior. ICA has been successfully applied in engineering subfields such as industrial, civil, mechanical, electronic, petroleum and computer engineering; see Hosseini and Al Khaled (2014) for a review. Our interest in ICA is in part due to recent reports from the engineering literature that suggest ICA can outperform some widely used evolutionary nature-inspired algorithms including PSO. For example, Hamel et al. (2012) compared performances of ICA with PSO to optimize some famous test functions like the De Jong’s, Rastrigin’s and Hartmann’s functions in nondestructive Eddy-Current Testing (ECT) problems. They reported that when the objective function has five or fewer parameters, ICA and PSO techniques performed almost the same. However, when the number of parameters was increased, ICA found the solution faster than PSO and with more accuracy (see Table 3 of Hamel et al., 2012 for details).

In what is to follow, we focus on finding minimax type of optimal designs using ICA. These are hard design problem because they involve solving nested multi-level optimization problems, the optimality criterion is not differentiable and there is no algorithm that we know of that can be shown to converge to such optimal designs in a general nonlinear regression setup. We show how to modify ICA with a perturbed move, coupled with a local search procedure to find the optimal designs more effectively.

In the next section, we review the statistical setup and theory for finding optimal designs. We present details and implementation information for the ICA in Section  3. In Section  4 we demonstrate how ICA may be applied to find minimax optimal designs for the power logistic model. Section  5 modifies ICA to find standardized maximin $D$-optimal designs for the log-linear, exponential and enzyme kinetic models. Section  6 provides a discussion.