Order-based error for managing ensembles of surrogates in mesh adaptive direct search

Abstract

We investigate surrogate-assisted strategies for global derivative-free optimization using the mesh adaptive direct search (MADS) blackbox optimization algorithm. In particular, we build an ensemble of surrogate models to be used within the search step of MADS to perform global exploration, and examine different methods for selecting the best model for a given problem at hand. To do so, we introduce an order-based error tailored to surrogate-based search. We report computational experiments for ten analytical benchmark problems and three engineering design applications. Results demonstrate that different metrics may result in different model choices and that the use of order-based metrics improves performance.

Appendices

Greedy selection algorithm

This greedy algorithm is used in Sect. 2.2 to filter the set of projection candidates and in Sect. 3.1.2 to select a subset of the training points around which the radial basis functions will be centered. For the projection step, the inputs of Algorithm 2 are:

$$\begin{aligned} S_{in} \leftarrow&~ S_{Proj} \text { (Set of projection candidates),}\\ \mathbf {x}_0 \leftarrow&~ \mathbf {x}^S_t(\text { Non projected solution of}~(\hat{P})), \\ p_{out} \leftarrow&~ 100n. \end{aligned}$$

For the selection of the RBF kernels, the inputs of Algorithm 2 are:

$$\begin{aligned} S_{in} \leftarrow&~ \mathbf {X}(\text {Set of points previously evaluated}),\\ \mathbf {x}_0 \leftarrow&~ \mathbf {x}^*_t (\text {Incumbent solution of}~(P)), \\ p_{out} \leftarrow&~ {q^{\textit{RBF}} (\text {Number of radial basis functions}, q^{\textit{RBF}} = \min \{ p/2 , 10n \})}. \end{aligned}$$

In both cases, the goal of this algorithm is to select a set $S_{out}$ of $p_{out}$ points from the set $S_{in}\in {\mathbb {R}}^n$. The set $S_{out}$ must be spread as widely as possible in ${\mathbb {R}}^n$ while favoring points that are close to a target $\mathbf {x}_{0}$. Since these two goals can be conflicting, a trade-off parameter $\lambda$ is introduced. When many points that are close to $\mathbf {x}_{0}$ have been selected, it is possible that the goal of selecting even more points close to $\mathbf {x}_{0}$ will lead to selecting a point that is already selected. In that case, the trade-off coefficient $\lambda$ is decreased.

Description of the analytical problems

1.1 MAD6 [44]

The original formulation uses seven variables, but two linear equality constraints allow to express the problem with five variables as follows:

$$\begin{aligned} \underset{\mathbf {x}\in {\mathbb {R}}^{5}}{\min }&\,\, \underset{1\le i \le 163}{\max }f_i(\mathbf {x}) \\ \text {s.t.}&\left\{ \begin{array}{l} c_1(\mathbf {x}) =-x_1 + 0.4 \le 0 \\ c_2(\mathbf {x}) = x_1 - x_2 + 0.4 \le 0 \\ c_3(\mathbf {x}) = x_2 - x_3 + 0.4 \le 0 \\ c_4(\mathbf {x}) = x_3 - x_4 + 0.4 \le 0 \\ c_5(\mathbf {x}) = x_4 - x_5 + 0.4 \le 0 \\ c_6(\mathbf {x}) =-x_4+x_5 - 0.6 \le 0 \\ c_7(\mathbf {x}) = x_4 - 2.1 \le 0 \\ \end{array}\right. \end{aligned}$$

where, for $1\le i \le 163$,

$$\begin{aligned} f_i(\mathbf {x}) = \frac{1}{15}+\frac{2}{15} \left( \left( \sum _{k=1}^{5} \cos (2\pi x_k \sin v_i) \right) + \cos \big (2\pi (1+x_4) \sin v_i\big ) + \cos (7\pi \sin v_i) \right) \end{aligned}$$

and

$$\begin{aligned} v_i= & {} \frac{\pi }{180}\left( 8.5+\frac{i}{2}\right) .\\ \mathbf {x}_0= & {} \left[ \begin{array}{c} 0.5 \\ 1 \\ 1.5 \\ 2 \\ 2.5 \end{array}\right] , ~f(\mathbf {x}_0)=0.22052,~ \mathbf {x}^*= \left[ \begin{array}{c} 0.4 \\ 0.819839074 \\ 1.219839074 \\ 1.69398531 \\ 2.09398531 \end{array}\right] , ~f(\mathbf {x}^*)=0.101831 . \end{aligned}$$

Best value achieved in this work: $f=0.101831$.

1.2 CRESCENT [1, 10]

$$\begin{aligned} \underset{\mathbf {x}\in {\mathbb {R}}^{10}}{\min }&x_{10} \\ \text {s.t.}&\left\{ \begin{array}{l} c_1(\mathbf {x}) = \sum _{i=1}^{10}(x_i-1)^2 - 100 \le 0 \\ c_2(\mathbf {x}) = \sum _{i=1}^{10}(x_i+1)^2 - 100 \le 0 \end{array}\right. \\ \mathbf {x}_0= & {} [10 \; 0 \; 0 \; 0 \; 0 \; 0 \; 0 \; 0 \; 0 \; 0 ]^\top ,~f(\mathbf {x}_0)=0,\\ \mathbf {x}^*= & {} [1 \; 1 \; 1 \; 1 \; 1 \; 1 \; 1 \; 1 \; 1 \; -{}9 ]^\top ,~f(\mathbf {x}^*)=-9. \end{aligned}$$

Best value achieved in this work: $f = -9$.

1.3 SNAKE [10, 19]

$$\begin{aligned} \underset{\mathbf {x}\in {\mathbb {R}}^{2}}{\min }&\sqrt{ (x_1-20)^2 + (x_2-1)^2 } \\ \text {s.t.}&\left\{ \begin{array}{l} c_1(\mathbf {x}) = \sin x_1 -\frac{1}{10} - x_2 \le 0 \\ c_2(\mathbf {x}) = x_2 - \sin x_1 \le 0 \end{array}\right. \\ \mathbf {x}_0= & {} [0 \; -{}10]^\top , ~f(\mathbf {x}_0)=\infty \text{( }\mathbf {x}_0 \text{ is } \text{ infeasible) },\\ \mathbf {x}^*= & {} [20.02887 \; 0.92434]^\top ,~ f(\mathbf {x}^*)=0.08098094. \end{aligned}$$

Best value achieved in this work: $f=0.08098094$.¹

1.4 HS24 [35]

$$\begin{aligned} \underset{\mathbf {x}\in {\mathbb {R}}^{2}}{\min }&\frac{(x_1-3)^2-9}{27 \sqrt{3}} x_2^3 \\ \text {s.t.}&\left\{ \begin{array}{l} c_1(\mathbf {x}) = - \frac{x_1}{\sqrt{3}} + x_2 \le 0 \\ c_2(\mathbf {x}) = - x_1 - \sqrt{3} x_2 \le 0 \\ c_3(\mathbf {x}) =+ x_1 + \sqrt{3} x_2 - 6 \le 0 \\ x_1 , x_2 \ge 0 \end{array}\right. \\ \mathbf {x}_0= & {} [1 \; 0.5]^\top , ~f(\mathbf {x}_0)=-0.0133646, \\ \mathbf {x}^*= & {} [3 \; 1.7320508071 ]^\top , ~f(\mathbf {x}^*)=-1. \end{aligned}$$

Best value achieved in this work: $f=-1$.

1.5 HS36 [35]

$$\begin{aligned} \underset{\mathbf {x}\in {\mathbb {R}}^{3}}{\min }&-x_1 x_2 x_3 \\ \text {s.t.}&\left\{ \begin{array}{l} c_1(\mathbf {x}) = x_1 + 2 x_2 + 2 x_3 - 72 \le 0 \\ 0 \le x_1 \le 20 \\ 0 \le x_2 \le 11 \\ 0 \le x_3 \le 42 \end{array}\right. \\ \mathbf {x}_0= & {} [10 \; 10 \; 10]^\top , ~f(\mathbf {x}_0)=-1000,\\ \mathbf {x}^*= & {} [20 \; 11 \; 15]^\top , ~ f(\mathbf {x}^*)=-3300. \end{aligned}$$

Best value achieved in this work: $f=-3300$.

1.6 HS37 [35]

$$\begin{aligned} \underset{\mathbf {x}\in {\mathbb {R}}^{3}}{\min }&-x_1 x_2 x_3 \\ \text {s.t.}&\left\{ \begin{array}{l} c_1(\mathbf {x}) = x_1 + 2 x_2 + 2 x_3 - 72 \le 0 \\ c_2(\mathbf {x}) = -x_1 - 2 x_2 - 2 x_3 \le 0\\ 0\le x_1, x_2, x_3 \le 42 \\ \end{array}\right. \\ \mathbf {x}_0= & {} [10 \; 10 \; 10]^\top ,~f(\mathbf {x}_0)=-1000,\\ \mathbf {x}^*= & {} [24 \; 12 \; 12]^\top ,~f(\mathbf {x}^*)=-3456. \end{aligned}$$

Best value achieved in this work: $f=-3456$.

1.7 HS73 [35]

In the original problem, the value of $x_4$ is computed directly by eliminating one linear equality constraint ($x_4=1-x_1-x_2-x_3$). Constraint $c_3$ is added to handle the bound $x_4 \ge 0$.

$$\begin{aligned} \underset{\mathbf {x}\in {\mathbb {R}}^{3}}{\min }&24.55 x_1 +26.75 x_2 + 39 x_3 + 40.5 (1-x_1-x_2-x_3) \\ \text {s.t.}&\left\{ \begin{array}{lllll} c_1(\mathbf {x}) &{}= - 2.3 x_1 - 5.6 x_2 - 11.1 x_3 - 1.3 (1-x_1-x_2-x_3) + 5 \le 0 \\ c_2(\mathbf {x}) &{}= -12x_1-11.9x_2-41.8x_3-52.1(1-x_1-x_2-x_3)+21 \\ &{}\quad + 1.645 \sqrt{ 0.28 x_1^2 + 0.19 x_2^2 + 20.5 x_3^2 + 0.62 (1-x_1-x_2-x_3)^2} \le 0 \\ c_3(\mathbf {x}) &{}= x_1+x_2+x_3-1 \le 0 \\ &{}\quad 0 \le x_1, x_2, x_3 \le 1 \\ \end{array}\right. \\ \mathbf {x}_0= & {} \left[ \begin{array}{c} 0.3 \\ 0.3 \\ 0.3 \end{array}\right] , ~f(\mathbf {x}_0)=31.14,~ \mathbf {x}^*= \left[ \begin{array}{c} 0.6355215683 \\ 0 \\ 0.3127018814 \end{array}\right] ,~f(\mathbf {x}^*)=29.8944. \end{aligned}$$

Best value achieved in this work: $f=29.8944$.

1.8 HS101, 102 and 103 [35]

$$\begin{aligned} \underset{\mathbf {x}\in {\mathbb {R}}^{7}}{\min }&\frac{10 x_1 x_4^2 x_7^a }{x_2 x_6^3} + \frac{15 x_3 x_4}{x_1 x_2^2 x_5 \sqrt{x_7}} + \frac{20 x_2 x_6}{x_1^2 x_4 x_5^2} + \frac{25 x_1^2 x_2^2 \sqrt{x_5} x_7}{x_3 x_6^2} \\ \text {s.t.}&\left\{ \begin{array}{lrl} c_1(\mathbf {x}) = \frac{0.5 \sqrt{x_1} x_7}{x_3 \sqrt{x_6}} + \frac{0.7 x_1^3 x_2 x_6 \sqrt{x_7}}{x_3^2} + \frac{0.2 x_3 x_6^{2/3} \root 4 \of {x_7}}{x_2 \sqrt{x_4}} -1 \le 0 \\ c_2(\mathbf {x}) = \frac{2 x_1 x_5 \root 3 \of {x_7}}{x_3^{3/2} x_6} + \frac{0.1 x_2 x_5}{x_6\sqrt{x_3}\sqrt{x_7}} + \frac{x_2 \sqrt{x_3} x_5}{x_1} + \frac{0.65 x_3 x_5 x_7}{x_2^2 x_6} -1 \le 0 \\ c_3(\mathbf {x}) = \frac{2 x_1 x_5 \root 3 \of {x_7}}{x_3^{3/2} x_6} + \frac{0.1 x_2 x_5}{\sqrt{x_3} x_6 \sqrt{x_7}} + \frac{x_2 \sqrt{x_3} x_5}{x_1} + \frac{0.65 x_3 x_5 x_7}{x_2^2 x_6} -1 \le 0 \\ c_4(\mathbf {x}) = \frac{0.2 x_2 \sqrt{x_5} \root 3 \of {x_7}}{x_1^2 x_4} + \frac{0.3 \sqrt{x_1} x_2^2 x_3 \root 3 \of {x_4} \root 4 \of {x_7}}{x_5^{2/3}} + \frac{0.4 x_3 x_5 x_7^{3/4}}{x_1^3 x_2^2} + \frac{0.5 x_4 \sqrt{x_7}}{x_3^2} -1 \le 0 \\ 0.1 \le x_1, x_2, \ldots , x_6 \le 10 \\ 0.01 \le x_7 \le 10\\ \end{array}\right. \end{aligned}$$

The value of the parameter a varies between the three problems:

$$\begin{aligned} \left\{ \begin{array}{l} \hbox {HS101}: a = -1/4; \\ \hbox {HS102}: a = -1/8; \\ \hbox {HS103}: a = -1/2. \\ \end{array}\right. \end{aligned}$$

The starting point is $\mathbf {x}_0= [ 6 \; 6 \; 6 \; 6 \; 6 \; 6 \; 6 ]^\top$ with the value $f(\mathbf {x}_0)=+\infty$ ($\mathbf {x}_0$ is infeasible) and the best know solutions are:

$$\begin{aligned} \mathbf {x}^*_{\text {HS101}} = \left[ \begin{array}{c} 3.1921264708 \\ 0.7569354058 \\ 2.5119021207 \\ 5.6530753445 \\ 0.871294938 \\ 1.261887906 \\ 0.0558117342 \end{array}\right] , ~\mathbf {x}^*_{\text {HS102}} = \left[ \begin{array}{c} 3.3669180827 \\ 0.7679140551 \\ 2.797994961 \\ 4.1843330406 \\ 0.8791473174 \\ 1.0784651788 \\ 0.0328185121 \end{array}\right] , ~\mathbf {x}^*_{\text {HS103}} = \left[ \begin{array}{c} 2.2198703858 \\ 0.6133345902 \\ 2.105352648 \\ 4.5835873238 \\ 0.8698142544 \\ 1.1828253718 \\ 0.046094896 \end{array}\right] \end{aligned}$$

with

$$\begin{aligned} f(\mathbf {x}^*_{\text {HS101}})=1948.02,~ f(\mathbf {x}^*_{\text {HS102}})=1495.48,~ f(\mathbf {x}^*_{\text {HS103}})=3367.69. \end{aligned}$$

The best values achieved in this work are 2480.94, 1950.26, and 3367.69, for HS101, HS102, and HS103, respectively.

Data profiles

For a given tolerance $\tau \le 1$, the ratio of solved problems for solver s after i groups of $n+1$ evaluations is defined as:

$$\begin{aligned} r_{s,i}(\tau ) = \frac{1}{\rho _{\max }} \sum _{\rho =1}^{\rho _{\max }} \mathbb {1}\Big (\delta _{s,\rho ,i} \le \tau \Big ), \end{aligned}$$

where $\mathbb {1}$ is the indicator function. Note that if $\tau =1$, a problem is considered solved if a feasible solution has been found. For a given value of $\tau$, the data profile is the curve corresponding to the ratio of solved problems after i groups of $n+1$ evaluations [52]. For the set of analytical problems and for the two sets of MDO runs (simplified wing and aircraft range), we show the data profiles for $\tau \in \{10^{-3},10^{-5},10^{-7}\}$ (see Figs. 9, 10, 11). For the Lockwood problem runs, to which is allocated a much smaller budget of evaluations, we show the data profiles for $\tau \in \{10^{-1},10^{-2},10^{-3}\}$ (see Fig. 12). Note that for the medium discrepancy curve, the lower the curve value, the better the performance. However, for the data profiles, the higher the better.

Fig. 9

Data profiles for the ten analytical problems; $\tau = 10^{-3}$ (left), $10^{-5}$ (middle) and $10^{-7}$ (right)

Fig. 10

Data profiles for the 50 optimization runs of the simplified wing problem; $\tau = 10^{-3}$ (left), $10^{-5}$ (middle) and $10^{-7}$ (right)

Fig. 11

Data profiles for the 50 optimization runs of aircraft range problem; $\tau = 10^{-3}$ (left), $10^{-5}$ (middle) and $10^{-7}$ (right)

Fig. 12

Data profiles for the 50 optimization runs of Lockwood problem; $\tau = 10^{-1}$ (left), $10^{-2}$ (middle) and $10^{-3}$ (right)