A Non-intrusive Solution to the Ill-Conditioning Problem of the Gradient-Enhanced Gaussian Covariance Matrix for Gaussian Processes

Abstract

Gaussian processes (GPs) are used for numerous different applications, including uncertainty quantification and optimization. Ill-conditioning of the covariance matrix for GPs is common with the use of various kernels, including the Gaussian, rational quadratic, and Matérn kernels. A common approach to overcome this problem is to add a nugget along the diagonal of the covariance matrix. For GPs that are not constructed with gradients, it is straightforward to derive a nugget value that guarantees the condition number of the covariance matrix to be below a user-set threshold. However, for gradient-enhanced GPs, there are no existing practical bounds to select a nugget that guarantee that the condition number of the gradient-enhanced covariance matrix is below a user-set threshold. In this paper a novel approach is taken to bound the condition number of the covariance matrix for GPs that use the Gaussian kernel. This is achieved by using non-isotropic rescaling for the data and a modest nugget value. This non-intrusive method works for GPs applied to problems of any dimension and it allows all data points to be kept. The method is applied to a Bayesian optimizer using a gradient-enhanced GP to achieve deep convergence. Without this method, the high condition number constrains the hyperparameters for the GP and this is shown to impede the convergence of the optimizer. It is also demonstrated that applying this method to the rational quadratic and Matérn kernels alleviates the ill-conditioning of their gradient-enhanced covariance matrices. Implementation of the method is straightforward and clearly described in the paper.

Appendices

Proofs for Sect. 7

1.1 Proof of Proposition 6

The results from the two following lemmas prove that $v_a^*(\theta , d)$ is monotonically decreasing with respect to $\theta$ and monotonically increasing with respect to d.

Lemma 13

For $\theta > 0$ and $d \in {\mathbb {Z}}^+$, the function $v_a^*(\theta ;d)$ is monotonically decreasing with respect to $\theta$.

Proof

Consider the derivative of $v_a^*(\theta )$ with respect to $\theta$:

$$\begin{aligned} \frac{\partial v_a^*(\theta )}{\partial \theta }&= \frac{ \sqrt{1 + 8 \theta d} - (1 + 4 \theta d)}{4 \theta ^2 \sqrt{d} \sqrt{1 + 8 \theta d}}, \end{aligned}$$

(77)

where it is clear that the denominator is always positive for $d \in {\mathbb {Z}}^+$ and $\theta >0$. We begin by assuming that the numerator is negative and show that this holds for $d \in {\mathbb {Z}}^+$ and $\theta >0$:

$$\begin{aligned} \sqrt{1 + 8 \theta d} - (1 + 4 \theta d)&< 0 \\ \left( \sqrt{1 + 8 \theta d} \right) ^2&< \left( 1 + 4 \theta d \right) ^2 \\ -16 \theta ^2 d^2&< 0, \end{aligned}$$

where both sides of the inequality on the second line are positive prior to being squared for $d \in {\mathbb {Z}}^+$ and $\theta >0$. As such, the squaring operation does not change their respective signs or the direction of the inequality. Since the denominator of $\frac{\partial v_a^*(\theta )}{\partial \theta }$ is always positive and its numerator is negative, it follows that $v_a^*(\theta )$ is monotonically decreasing with respect to $\theta$, which completes the proof. $\square$

Lemma 14

The function $v_a^*(\theta ; d)$ is monotonically increasing with respect to d for $d \in {\mathbb {Z}}^+$ and $\theta > 0$.

Proof

We have $d \in {\mathbb {Z}}^+$ but for this analysis we consider d to be a continuous variable and we calculate the derivative of $v_a^*(d)$ with respect to d:

$$\begin{aligned} \frac{\partial v_a^*(d)}{\partial d}&= \frac{\sqrt{1 + 8 \theta d} - 1}{8 \theta d^{\frac{3}{2}} \sqrt{1 + 8 \theta d} }. \end{aligned}$$

(78)

Both the numerator and denominator are always positive for $\theta >0$ and $d>0$. Consequently, $v_a^*(d)$ is monotonically increasing for $d >0$. $\square$

The two following lemmas bound $v_a^*(\theta ;d)$ by using the result from Lemmas 13 and 14.

Lemma 15

For $\theta > 0$ and $d \in {\mathbb {Z}}^+$, the supremum of $v_a^*(\theta ;d)$ is $\sqrt{d}$ at $\theta = 0$.

Proof

As a consequence of Lemma 13, $v_a^*$ is maximized when $\theta$ is minimized. We thus evaluate $\lim _{\theta \rightarrow 0} v_a^*(\theta )$ using L’Hôpital’s rule:

$$\begin{aligned} \lim _{\theta \rightarrow 0} v_a^*(\theta ; d)&= \lim _{\theta \rightarrow 0} \frac{-1 + \sqrt{1 + 8 \theta d}}{4 \theta \sqrt{d}} \\&= \lim _{\theta \rightarrow 0} \frac{ 4d \left( 1 + 8 \theta d \right) ^{-\frac{1}{2}}}{4 \sqrt{d}} \\&= \sqrt{d}, \end{aligned}$$

which is the desired result. $\square$

Lemma 16

We have $v_a^*(\theta ; d) \le \frac{1}{\sqrt{2 \theta }}$ for $\theta > 0$ and $d \in {\mathbb {Z}}^+$.

Proof

From Lemma 14 we know that $v_a^*(\theta ; d)$ is monotonically increasing for $d >0$. As such, we can derive an upper bound on $v_a^*(\theta ; d)$ by considering the following limit:

$$\begin{aligned} u_{v_a^*}(\theta )&= \lim _{d \rightarrow \infty } v_a^*(\theta ; d) \\&= \lim _{d \rightarrow \infty } \frac{-\frac{1}{\sqrt{d}} + \sqrt{\frac{1}{d} + 8 \theta }}{4 \theta } \\&= \frac{1}{\sqrt{2 \theta }}, \end{aligned}$$

which completes the proof. $\square$

We have shown that $v_a^*(\theta ;d) \le \sqrt{d}$ and $v_a^*(\theta ;d) \le \frac{1}{\sqrt{2 \theta }}$. We therefore have $v_a^*(\theta ;d) \le \min \left( \sqrt{d}, \frac{1}{\sqrt{2 \theta }} \right)$, which completes the proof of Proposition 6.

1.2 Proof of Proposition 9

To identify the maximum of $u_{r_b}$ we begin by calculating its derivative:

$$\begin{aligned} \frac{\partial u_{r_b}(v_b)}{\partial v_b} = -2 \theta (n_x -1) \left[ 4 \theta ^2 \sqrt{d} v_b^3 + 2 \theta v_b^2 - 4 \theta \sqrt{d} v_b - 1 \right] e^{-\theta v_b^2}, \end{aligned}$$

(79)

which is zero if the following cubic is satisfied:

$$\begin{aligned} v_b^3 + \frac{1}{2 \theta \sqrt{d}} v_b^2 - \frac{1}{\theta } v_b - \frac{1}{4 \theta ^2 \sqrt{d}} = 0. \end{aligned}$$

(80)

A general cubic equation is $v^3 + a_2 v^2 + a_1 v + a_0 = 0$ where we have $a_2 = \frac{1}{2 \theta \sqrt{d}}$, $a_1 = - \frac{1}{\theta }$, and $a_0 = -\frac{1}{4 \theta ^2 \sqrt{d}}$. We can determine if the cubic equation has one or three real roots by checking the sign of the discriminant [10], which is given by

$$\begin{aligned} \varDelta&= \frac{4 a_2^3 a_0 - a_2^2 a_1^2 + 4 a_1^3 - 18a_2 a_1 a_0 + 27a_0^2}{108} \nonumber \\&= -\frac{64 \theta ^2 d^2 + 13 \theta d + 2}{1728 \, \theta ^5 d^2}, \end{aligned}$$

(81)

which is negative for $\theta > 0$ and $d \in {\mathbb {Z}}^+$. A negative discriminant indicates the cubic equation has three real roots [10]. Furthermore, for $d \in {\mathbb {Z}}^+$ and $\theta >0$, we have $a_2 >0$, $a_0 < 0$, which is Case III3 in the first table in [10]. Case III3 indicates that only one of the three roots is positive and the other two are negative. We use the cubic solution from [19] to calculate the only positive root of Eq. (80):

$$\begin{aligned} v_b^*&=\frac{\sqrt{12 d \theta +1}}{3 \theta \sqrt{d}} \cos \left( \frac{1}{3} \cos ^{-1} \left( \frac{ 9 d \theta -1}{(12 d \theta +1)^{\frac{3}{2}}} \right) \right) -\frac{1}{6 \theta \sqrt{d}}, \end{aligned}$$

which matches Eq. (44) and this completes the proof.

1.3 Proof of Lemma 2

Consider the following reformulation of $u_{r_b}(v_b)$:

$$\begin{aligned} u_{r_b}(v_b)&= 2\theta v_b (n_x - 1) \left( 1 + 2 \theta \sqrt{d} \, v_b \right) e^{-\theta v_b^2} \\&= \underbrace{ 2\theta v_b (n_x - 1) e^{-\theta v_b^2} }_{u_{r_b,1}(v_b)} + \underbrace{4\theta ^2 v_b^2 \sqrt{d} (n_x - 1) e^{-\theta v_b^2}}_{u_{r_b,2}(v_b)}. \end{aligned}$$

The positive root for the gradient of $u_{r_b,1}(v_b)$ is

$$\begin{aligned} \frac{\partial u_{r_b,1}(v_{b}) }{\partial v_{b}}&= 2 \theta (n_x - 1) \left[ 1 - 2 \theta v_{b}^2 \right] e^{-\theta v_{b}^2} = 0 \nonumber \\ v_{b1}^*(\theta )&= \frac{1}{ \sqrt{2 \theta }}, \end{aligned}$$

(82)

and analogously for $u_{r_b,2}(v_b)$ we find its unique positive critical point is $v_{b2}^*(\theta ) = \frac{1}{\sqrt{\theta }}$, which match Eqs. (45) and (46), respectively. It is straightforward to check that both $v_{b1}^*$ and $v_{b2}^*$ are the location of the maximum for $u_{r_b,1}(v_b)$ and $u_{r_b,2}(v_b)$, respectively. Both $\frac{\partial u_{r_b,1}(v_{b}) }{\partial v_{b}}$ and $\frac{\partial u_{r_b,2}(v_{b}) }{\partial v_{b}}$ are positive when $v <v_{b1}^*$, and both are negative when $v > v_{b2}^*$. Therefore, $\frac{\partial u_{r_b}(v_{b}) }{\partial v_{b}}$ can only be zero in the range $v_{b1}^*(\theta )= \frac{1}{\sqrt{2 \theta }}< v_b^*(\theta ) < \frac{1}{\sqrt{\theta }} = v_{b2}^*(\theta )$. We therefore have $\ell _{v_b^*}(\theta ) = v_{b1}^*(\theta ) = \frac{1}{\sqrt{2 \theta }}$ and $u_{v_b^*}(\theta ) = v_{b2}^*(\theta ) = \frac{1}{\sqrt{\theta }}$, which completes the proof of Lemma 2.

1.4 Proof of Lemma 4

We need to derive $\ell _{\theta _b^*}(v_{\text {min}})$ and $u_{\theta _b^*}(v_{\text {min}})$ and prove that $u_{{\hat{\theta }}_b} < \ell _{\theta _b^*}$ for $d \in {\mathbb {Z}}^+$ and $v_{\text {min}}> 0$. We begin by showing that $\theta _b^*(d)$ from Eq. (43) is monotonically increasing with respect to d:

$$\begin{aligned} \frac{\partial \theta _b^*}{\partial d}&= \frac{\sqrt{v_{\text {min}}^2 + 16d} - v_{\text {min}}}{8 d^{\frac{3}{2}} v_{\text {min}}\sqrt{v_{\text {min}}^2 +16d}}, \end{aligned}$$

(83)

which is always positive for $d \in {\mathbb {Z}}^+$ and $v_{\text {min}}> 0$. Lower and upper bounds on $\theta _b^*(d)$ are thus given by

$$\begin{aligned} \ell _{\theta _b^*}&= \theta _b^*(d=1) = \frac{1}{v_{\text {min}}^2} + \frac{ \sqrt{v_{\text {min}}^2 + 16} - v_{\text {min}}}{4 v_{\text {min}}^2} \end{aligned}$$

(84)

$$\begin{aligned} u_{\theta _b^*}&= \lim _{d \rightarrow \infty } \theta _b^*(d) \le \frac{2}{v_{\text {min}}^2}. \end{aligned}$$

(85)

Comparing $u_{{\hat{\theta }}_b}(v_{\text {min}}) = \frac{1}{v_{\text {min}}^2}$ from Eq. (48) and $\ell _{\theta _b^*}(v_{\text {min}})$ from Eq. (49) it is clear that $u_{{\hat{\theta }}_b}(v_{\text {min}}) < \ell _{\theta _b^*}(v_{\text {min}})$ for $v_{\text {min}}> 0$. We thus have ${\hat{\theta }}_b< u_{{\hat{\theta }}_b}< \ell _{\theta _b^*} \le \theta _b^* < u_{\theta _b^*}$, which completes the proof of Lemma 4.

1.5 Proof of Lemma 7

We start by considering the limit $\lim _{\theta \rightarrow 0} u_{r_a}$. From Proposition 6 we have $v_a^* \le \sqrt{d}$ and therefore $v_a(\theta ) = \min (v_{\text {min}}, \sqrt{d})$. Our limit of interest is given by

$$\begin{aligned} \lim _{\theta \rightarrow 0} u_{r_a}(\theta , v_a(\theta ))&= \lim _{\theta \rightarrow 0} (n_x -1) \left( 1 + 2 \theta \sqrt{d} v_a(\theta ) \right) e^{-\theta v_a^2(\theta )} \\&= n_x - 1, \end{aligned}$$

which is the desired result. We next consider the limit $\lim _{\theta \rightarrow 0} u_{r_b}$. From Lemma 2 we have $\frac{1}{\sqrt{2 \theta }} =\ell _{v_b^*}(\theta )< v_b^*(\theta ) < u_{v_b^*}(\theta )) = \frac{1}{\sqrt{\theta }}$. Therefore, we have $v_b^*(\theta \rightarrow 0) \rightarrow \infty$ and thus, $v_b(\theta \rightarrow 0) = v_b^*$. However, we also have the following relations:

$$\begin{aligned} \lim _{\theta \rightarrow 0} \theta \ell _{v_b^*}(\theta )&= \lim _{\theta \rightarrow 0} \sqrt{\frac{\theta }{2}} = 0 \\ \lim _{\theta \rightarrow 0} \theta u_{v_b^*}(\theta )&= \lim _{\theta \rightarrow 0} \sqrt{\theta } = 0, \end{aligned}$$

which implies that $\theta v_b^*(\theta ) = 0$ for $\theta \rightarrow 0$. It is straightforward to verify that $\frac{1}{2}< \theta \left( v_b^* \right) ^2 < 1$ when $\theta \rightarrow 0$. We now consider the limit of $2 \theta + u_{r_b}(\theta )$:

$$\begin{aligned} \lim _{\theta \rightarrow 0} \left( 2 \theta + u_{r_b}(\theta , v_b^*) \right)&= \lim _{\theta \rightarrow 0} 2 (\theta v_b^*) (n_x - 1) \left( 1 + 2 \sqrt{d} (\theta v_b^*) \right) e^{-\theta \left( v_b^*\right) ^2} \\&= 0, \end{aligned}$$

which is the desired result and this completes the proof of Lemma 7.

Proofs for Sect. 8

1.1 Proof of Proposition 11

From Corollary 1 and Lemma 10 it follows that with $v_{\text {min}}= 2 \sqrt{d}$ we only have $u_{\lambda _{\text {max}}}(\theta ) > n_x$ when $2 \theta + u_{r_b}(\theta ) > n_x$ and $\theta > \frac{1}{2}$. The next lemma considers the case when $2 \theta + u_{r_b}(\theta ; n_x) = n_x$.

Lemma 17

For $\theta > \frac{1}{2}$, $d \in {\mathbb {Z}}^+$, and $v_{\text {min}}= 2 \sqrt{d}$, we have $2 \theta + u_{r_b}(\theta ; n_x) = n_x$ when $n_x = {\hat{n}}_x > 0$, where

$$\begin{aligned} {\hat{n}}_x = \frac{2 \theta - g_b(\theta )}{1 - g_b(\theta )}, \end{aligned}$$

(86)

and $g_b(\theta )$ comes from Eq. (59).

Proof

For $v_{\text {min}}= 2 \sqrt{d}$ we have from Eq. (57) $\theta _b^* = \frac{1 + \sqrt{5}}{8d} < \frac{1}{2}$ since $d \in {\mathbb {Z}}^+$. As such, from Lemma 5, $v_b(\theta ) = v_{\text {min}}= 2 \sqrt{d}$ for $\theta \ge \frac{1}{2}$ and thus Eq. (59) is used:

$$\begin{aligned} 2 \theta + u_{r_b}(\theta ; {\hat{n}}_x)&= 2 \theta + ({\hat{n}}_x - 1) g_b \\&= {\hat{n}}_x, \end{aligned}$$

where Eq. (86) is recovered by isolating for ${\hat{n}}_x$. Both the numerator and denominator of Eq. (86) are positive for $\theta > \frac{1}{2}$ thanks to Lemma 9 and this completes the proof. $\square$

We now evaluate $1 - u_{r_a}(\theta ; {\hat{n}}_x)$ using Eqs. (58) and (86):

$$\begin{aligned} 1 - u_{r_a}(\theta ; {\hat{n}}_x)&= 1 - ({\hat{n}}_x - 1) g_a \nonumber \\&= \frac{1 - \left( 2 \theta (2 \sqrt{d} +1) -1 \right) (4d \theta + 1) e^{-4d \theta } }{1 - g_b} \nonumber \\&= \frac{1 - h_a(\theta ; d)}{1 - g_b}, \end{aligned}$$

(87)

where the denominator is always positive thanks to Lemma 9 and $h_a(\theta ; d) = \big (2 \theta (2 \sqrt{d} +1) -1 \big ) \left( 4d \theta +1\right) e^{-4 d \theta }$. The maximum of $h_a(\theta ; d)$ is considered in the following lemma.

Lemma 18

For $\theta > \frac{1}{2}$ and $d \in {\mathbb {Z}}^+$ we have $h_a(\theta , d) \le u_{h_a}(\theta ; d) < 1$, where

$$\begin{aligned} u_{h_a}(d) = (8 \theta d -2) ( 4 \theta d +1) e^{-4 \theta d}. \end{aligned}$$

(88)

Proof

We start by showing that $u_{h_a}(d) \ge h_a(\theta ;d)$:

$$\begin{aligned} (8 \theta d -2) ( 4 \theta d +1) e^{-4 \theta d}&\ge \left( 2 \theta (2 \sqrt{d} +1) -1 \right) \left( 4d \theta +1\right) e^{-4 d \theta } \nonumber \\ 2 \theta (4d - 2 \sqrt{d} -1)&\ge 1. \end{aligned}$$

(89)

For $\theta > \frac{1}{2}$ and $d \in {\mathbb {Z}}^+$ the LHS is minimized and equal to one with $\theta = \frac{1}{2}$ and $d=1$. We now calculate the maximum of $u_{h_a}(\theta , d)$ using $a = 4 \theta d$, where $a \ge 2$ since $\theta > \frac{1}{2}$ and $d \in {\mathbb {Z}}^+$. The maximum of $u_{h_a}(a)$ is at

$$\begin{aligned} \frac{\partial u_{h_a}}{\partial a}&= \frac{\partial \left( (a^2 -1) e^{-a} \right) }{\partial a} \nonumber \\&= -2 \left( a^2 -2a -1 \right) e^{-1} = 0 \nonumber \\ a&= 1 + \sqrt{2}, \end{aligned}$$

(90)

where we only keep the positive root of the quadratic and it straightforward to verify that this is the location of a maximum. The maximum of $u_{h_a}(a)$ is thus $u_{h_a}(a = 1 + \sqrt{2}) \approx 0.432$, which completes the proof. $\square$

Since $h_a(\theta , d) < 1$ for $\theta > \frac{1}{2}$ and $d \in {\mathbb {Z}}^+$ it follows from Eq. (87) that $1 - u_{r_a}(\theta ) >0$ when $u_{\lambda _{\text {max}}}(\theta ) \ge n_x$, which completes the proof of Proposition 11.

1.2 Proof of Proposition 12

We follow the same approach used for the proof of Proposition 11 in Sect. 13.1 and begin by evaluating $2 \theta - u_{r_b}(\theta ; {\hat{n}}_x)$ using Eqs. (59) and (86):

$$\begin{aligned} 2 \theta - u_{r_b}(\theta ; {\hat{n}}_x)&= 2 \theta - ({\hat{n}}_x -1) g_b \nonumber \\&= \frac{ 2 \theta }{1-g_b} \left[ 1 - 2 \sqrt{d} (4 \theta -1) \left( 4 \theta d + 1 \right) e^{-4d \theta } \right] \nonumber \\&= \frac{ 2 \theta }{1-g_b} \left( 1 - h_b(\theta , d) \right) , \end{aligned}$$

(91)

where $h_b(\theta , d) = 2 \sqrt{d} (4 \theta -1) \left( 4 \theta d + 1 \right) e^{-4d \theta }$ and the denominator is positive as a result of Lemma 9. In order to have $2 \theta - u_{r_b}(\theta ; {\hat{n}}_x) > 0$ for $\theta > \frac{1}{2}$ we need to prove that $h_b(\theta , d) < 1$ for $d \in {\mathbb {Z}}^+$ and $\theta > \frac{1}{2}$. We begin by showing that $h_b < u_{h_a}$ for $d \in {\mathbb {Z}}^+$ and $\theta > \frac{1}{2}$, where $u_{h_a}$ comes from Eq. (88):

$$\begin{aligned} (8 \theta d -2) ( 4 \theta d +1) e^{-4 \theta d}&\ge 2 \sqrt{d} \left( 4 \theta -1 \right) \left( 4d \theta + 1\right) e^{-4 d \theta } \\ (8 \theta \sqrt{d} +2)(\sqrt{d} -1)&> 0, \end{aligned}$$

which is satisfied for $\theta > 0$ and $d \in {\mathbb {Z}}^+$. From Lemma 18 we have $u_{h_a}(\theta , d) \le 1$ for $d \in {\mathbb {Z}}^+$ and $\theta > \frac{1}{2}$. Therefore, $h_b< u_{h_a} < 1$ for $d \in {\mathbb {Z}}^+$ and $\theta > \frac{1}{2}$. Since $h_b < 1$, it follows from Eq. (91) that $2 \theta - u_{r_b}(\theta ; n_x) > 0$ when $u_{\lambda _{\text {max}}}(\theta ) \ge n_x$, which completes the proof of Proposition 12.

Proofs for Sect. 9

1.1 Proof of Proposition 13

Equation (63) is derived by setting $\theta = \frac{1}{2}$ in $2 \theta - u_{r_b}(\theta , v_b(\theta )) = 0$ along with the constraint $v_{\text {min}}\ge \sqrt{2}$. The solution to Eq. (63) thus satisfies all the criteria from Lemma 11 to ensure $\ell _{\lambda _{\text {min}}}> 0 \, \forall \, \theta > \frac{1}{2}$, $(d, n_x) \in \left( {\mathbb {Z}}^+, {\mathbb {Z}}^+ \setminus 1\right)$. In order for the solution to Eqs. (61) and (63) to be the same, we must show that the solution to Eq. (63) is the smallest one that ensures $\ell _{\lambda _{\text {min}}}(\theta ; d, n_x)> 0 \, \forall \, \theta > \frac{1}{2}$, $(d, n_x) \in \left( {\mathbb {Z}}^+, {\mathbb {Z}}^+ \setminus 1\right)$.

Equation (63) is simply $u_{r_b}(\theta , v) = 1$ with $v = v_{\text {req}}> \sqrt{2}$ and $\theta = \frac{1}{2}$. The function $u_{r_b}(\theta , v)$ evaluates to zero for $v = 0$ and $v \rightarrow \infty$ and from Proposition 9 we know it is a quasiconcave function with respect to v. Therefore, if there is a value of $v>0$ such that $u_{r_b}(\theta , v) > 1$, then there are two values of v that satisfy $u_{r_b}(\theta , v) = 1$, one smaller and one greater than $\sqrt{2}$ and $v_b^*$. With $d=1$, $n_x=2$, and $v = \sqrt{2}$ we have $u_{r_b} \approx 1.26$. Since $u_{r_b}$ is monotonically increasing with respect to both d and $n_x$, it follows that there are two values of v that satisfy $u_{r_b}(\theta , v) = 1$ for $d \in {\mathbb {Z}}^+$ and $n_x \in {\mathbb {Z}}^+ \setminus 1$. However, only the larger solution is greater than $v_b^*$ and ensures $u_{r_b}(\theta , v_b) = 1$ since for the smaller solution we have $v_b = v_b^*$ and thus $u_{r_b}(\theta , v_b) > 1$. Therefore, the requirement in Eq. (63) that $v_{\text {min}}\ge v_{\text {req}}> \sqrt{2}$ ensures that the larger solution of $u_{r_b}(\theta , v) = 1$ is found. With $v_{\text {min}}= v_{\text {req}}$ we thus have the smallest value of $v_{\text {min}}$ that ensures $u_{r_b}(\theta , v_b) = 1$ and thus $\ell _{\lambda _{\text {min}}}(\theta ) = 0$ at $\theta = \frac{1}{2}$. The next two lemmas prove that with $v_{\text {min}}\ge v_{\text {req}}$, the bound $\ell _{\lambda _{\text {min}}}(\theta )$ is monotonically increasing for $\theta > \frac{1}{2}$.

Lemma 19

The solution to Eq. (63) ensures that $\ell _{\lambda _{\text {min}}}(\theta )$ is monotonically increasing for $\theta \ge \frac{1}{2}$ and $(d, n_x) \in \left( {\mathbb {Z}}^+, {\mathbb {Z}}^+ \setminus 1\right) \setminus (1,2)$.

Proof

The solution to Eq. (63) for $(d, n_x) = (1,3)$ and $(d,n_x) = (2,2)$ is $v_{\text {req}}= 2.347$ and $v_{\text {req}}= 2.031$, respectively. The LHS of Eq. (63) is monotonically increasing with respect to d and $n_x$ but monotonically decreasing with respect to $v_{\text {req}}$ since $v_{\text {req}}> v_b^*$. Therefore, since the RHS of Eq. (63) is constant, $v_{\text {req}}$ must be monotonically increasing with respect to d and $n_x$. As such $v_{\text {req}}> 2$ for $(d, n_x) \in \left( {\mathbb {Z}}^+, {\mathbb {Z}}^+ \setminus 1\right) \setminus (1,2)$ and Lemma 12 indicates this is a sufficient condition to have $\ell _{\lambda _{\text {min}}}(\theta )$ be monotonically increasing for $\theta \ge \frac{1}{2}$, which completes the proof. $\square$

Lemma 20

The solution to Eq. (63) ensures that $\ell _{\lambda _{\text {min}}}(\theta )$ is monotonically increasing for $\theta \ge \frac{1}{2}$ and $(d,n_x) = (1,2)$.

Proof

A sufficient condition for $\ell _{\lambda _{\text {min}}}(\theta )$ to be monotonically increasing for $\theta > \frac{1}{2}$ is that both $1 - u_{r_a}(\theta )$ and $2 \theta - u_{r_b}(\theta )$ are themselves monotonically increasing for $\theta > \frac{1}{2}$. The solution to Eq. (63) with $(d, n_x) = (1, 2)$ is $v_{\text {req}}= 1.797$. From Eq. (37) we have $\theta _a^*(v = 1.797, d=1) = 0.031$, which indicates $1 - u_{r_a}(\theta )$ is monotonically increasing for $\theta > \frac{1}{2}$. We now differentiate $2 \theta - u_{r_b}(\theta ; d=1, n_x=2)$:

$$\begin{aligned} \frac{\partial \left( 2 \theta - u_{r_b}(\theta ) \right) }{\partial \theta } = 2 +2 v_{\text {min}}\left( 2 v_{\text {min}}^3 \theta ^2 + v_{\text {min}}(v_{\text {min}}- 4) \theta -1 \right) e^{-\theta v_{\text {min}}^2}. \end{aligned}$$

(92)

For $v_{\text {min}}= v_{\text {req}}= 1.797$ we have $\frac{\partial \left( 2 \theta - u_{r_b}(\theta ) \right) }{\partial \theta } = 0$ only when $\theta = 0.224$. Therefore, $2 \theta - u_{r_b}(\theta ; d=1, n_x=2)$ is monotonically increasing for $\theta > 0.224$, which completes the proof. $\square$

The solution to Eq. (63) is the smallest one that ensures $\ell _{\lambda _{\text {min}}}(\theta ) = 0$ for $\theta = \frac{1}{2}$ and it is also sufficiently large to ensure $\ell _{\lambda _{\text {min}}}(\theta )$ is monotonically increasing for $\theta > \frac{1}{2}$. Therefore, the solution to Eq. (63) is the solution to Eq. (61), which completes the proof.

1.2 Proof of Proposition 14

We take the natural logarithm of both sides of Eq. (63):

$$\begin{aligned} \ln \left( (n_x -1) v \left( 1 + \sqrt{d} v \right) e^{-\frac{v^2}{2}} \right) = \ln (1), \end{aligned}$$

and isolate $v^2$ to obtain

$$\begin{aligned} v^2 = 2 \ln (n_x - 1) + 2 \ln (v) + 2 \ln (1 + \sqrt{d} v). \end{aligned}$$

(93)

The two following lemmas provide upper bounds on the two non-polynomial terms in Eq. (93) that contain v.

Lemma 21

For $v \ge 2$ and $d \in {\mathbb {Z}}^+$ we have

$$\begin{aligned} \ln (v) + \ln \left( \frac{1 + 2 \sqrt{d}}{2} \right) \ge \ln \left( 1 + \sqrt{d} v \right) . \end{aligned}$$

(94)

Proof

We start by finding the smallest value of $c_1$ such that the following inequality holds for $v > 2$ and $d \in {\mathbb {Z}}^+$:

$$\begin{aligned} \ln (c_1 \sqrt{d} v)&\ge \ln (1 + \sqrt{d} v) \end{aligned}$$

(95)

$$\begin{aligned} c_1&= \max _{v \ge 2} \frac{1 + \sqrt{d} v}{\sqrt{d} v} \nonumber \\&= \frac{1 + 2 \sqrt{d}}{2 \sqrt{d}}, \end{aligned}$$

(96)

where the minimum value of v, i.e. $v=2$, was used since $\frac{1 + \sqrt{d} v}{\sqrt{d} v} = \frac{1}{\sqrt{d} v} + 1$ is monotonically decreasing with respect to v. Equation (94) is recovered by substituting the value of $c_1$ from Eq. (96) into Eq. (95), which completes the proof. $\square$

Lemma 22

For $v>0$ we have $\frac{v}{e} \ge \ln (v)$.

Proof

We have $\max \frac{\ln (v)}{v} = \frac{1}{e}$ at $v = e$. Therefore, $\frac{v}{e} \ge \ln (v) \, \forall \, v >0$, which completes the proof. $\square$

We now use Lemmas 21 and 22 to convert Eq. (93) into an algebraic equation:

$$\begin{aligned} v^2&> 2 \ln (v) + 2 \ln (n_x - 1) + 2 \left[ \ln (v) + \ln \left( \frac{1 + 2 \sqrt{d}}{2} \right) \right] \nonumber \\&> \frac{4}{e} v + 2 \ln \left( \frac{(n_x - 1) \left( 1 + 2 \sqrt{d} \right) }{2} \right) , \end{aligned}$$

(97)

which is a quadratic relation that can be satisfied with the positive root of the quadratic equation:

$$\begin{aligned} {\tilde{v}}_{\text {req}}(d, n_x)&= \frac{ 2 + \sqrt{ 4+ 2 e^2 \ln \left( \frac{(n_x - 1) \left( 1 + 2 \sqrt{d} \right) }{2} \right) }}{e}, \end{aligned}$$

(98)

which matches Eq. (64). Lemma 21 considered the case with $v\ge 2$ but ${\tilde{v}}_{\text {req}}(n_x=2, d=1) = 1.899$. Nonetheless, $v_{\text {req}}(n_x=2, d=1) = 1.814$ and as such, ${\tilde{v}}_{\text {req}}\ge v_{\text {req}}\, \forall (d, n_x) \in \left( {\mathbb {Z}}^+, {\mathbb {Z}}^+ \setminus 1\right)$. From Proposition 13 it follows that $v_{\text {min}}\ge v_{\text {req}}$ ensures that $\ell _{\lambda _{\text {min}}}$ is positive and monotonically increasing for $\theta > \frac{1}{2}$, which satisfies the constraint in Eq. (61) and this completes the proof of Proposition 14.

1.3 Proof of Proposition 15

We consider the maximum of $u_{\lambda _{\text {max}}} (\theta )$ for $0 < \theta \le \frac{1}{2}$ using Eq. (36):

$$\begin{aligned} \max _{0< \theta \le \frac{1}{2}} u_{\lambda _{\text {max}}}&= \max _{0< \theta \le \frac{1}{2}} \left( 1 + u_{r_a}^*, 2 \theta + u_{r_b}^* \right) \nonumber \\&\le \max _{0 < \theta \le \frac{1}{2}} \left( 1 + u_{r_a}^*, 1 + u_{r_b}^* \right) . \end{aligned}$$

(99)

To prove that $u_{\lambda _{\text {max}}} = 1 + u_{r_a}^*$ for $d \in {\mathbb {Z}}^+$, $n_x \in {\mathbb {Z}}^+ \setminus 1$, $0< \theta \le \frac{1}{2}$, and $v_{\text {req}}\le v_{\text {min}}\le 2 \sqrt{d}$ it is thus sufficient to show that the following relation is satisfied:

$$\begin{aligned} u_{r_b}^*(v_{\text {min}})&< u_{r_a}^*(v_{\text {min}}) \nonumber \\ 2(n_x -1) \frac{ 4 \sqrt{d} + \sqrt{v_{\text {min}}^2 + 16d}}{v_{\text {min}}^2} e^{- \frac{4 \sqrt{d} + \sqrt{v_{\text {min}}^2 + 16d} - v_{\text {min}}}{4 \sqrt{d}}}&< (n_x -1) \frac{2 \sqrt{d}}{v_{\text {min}}} e^{1 - \frac{v_{\text {min}}}{2 \sqrt{d}}} \nonumber \\ \frac{\sqrt{d} v_{\text {min}}}{4 \sqrt{d} + \sqrt{v_{\text {min}}^2 + 16d}} e^{\left( \frac{v_{\text {min}}+\sqrt{v_{\text {min}}^2 + 16d}}{4 \sqrt{d}} \right) }&> 1, \end{aligned}$$

(100)

where Eqs. (42) and (51) were used for $u_{r_a}^*$ and $u_{r_b}^*$, respectively. To determine when Eq. (100) is satisfied the trend of the LHS of Eq. (100) with respect to $v_{\text {min}}$ and d is required and this is considered in the two following lemmas.

Lemma 23

The LHS of Eq. (100), which is denoted as $g(v_{\text {min}}, d)$, is monotonically increasing with respect to $v_{\text {min}}$ for $v_{\text {min}}> 0$ and $d \in {\mathbb {Z}}^+$.

Proof

The derivative of $g(v_{\text {min}}, d)$ with respect to $v_{\text {min}}$ is

$$\begin{aligned} \frac{\partial g}{\partial v_{\text {min}}} = \frac{ \left( v_{\text {min}}+ 4 \sqrt{d} \right) \sqrt{v_{\text {min}}^2 + 16d} + 4 \sqrt{d} v_{\text {min}}+ 16 d + v_{\text {min}}^2}{4 \left( \sqrt{v_{\text {min}}^2 + 16d} + 4 \sqrt{d} \right) ^2} e^{ \left( \frac{v_{\text {min}}+\sqrt{v_{\text {min}}^2 + 16d}}{4 \sqrt{d}} \right) }, \nonumber \\ \end{aligned}$$

(101)

which is always positive for $v_{\text {r}}> 0$ and $d \in {\mathbb {Z}}^+$ and this completes the proof. $\square$

Lemma 24

The LHS of Eq. (100), which is denoted as $g(v_{\text {min}}, d)$, is monotonically decreasing with respect to d for $v_{\text {r}}> 0$ and $d \in {\mathbb {Z}}^+$.

Proof

The partial derivative of $g(v_{\text {r}}, d)$ with respect to d is

$$\begin{aligned} \frac{\partial g}{\partial d}&= - \frac{ v_{\text {r}}\left( v_{\text {r}}+ \sqrt{v_{\text {r}}^2 + 16d} - 4 \sqrt{d} \right) }{8d \left( \sqrt{v_{\text {r}}^2 + 16d} + 4 \sqrt{d} \right) } e^{ \left( \frac{v_{\text {min}}+\sqrt{v_{\text {min}}^2 + 16d}}{4 \sqrt{d}} \right) }, \end{aligned}$$

(102)

which is always negative since $\sqrt{v_{\text {r}}^2 + 16d} - 4 \sqrt{d} > 0$ for $v_{\text {r}}>0$ and $d \in {\mathbb {Z}}^+$. $\square$

We are interested in proving that $u_{r_a}^*(v_{\text {min}}) > u_{r_b}^*(v_{\text {min}})$ for $v_{\text {min}}\ge v_{\text {req}}$. In order to do that the following lemma solves for $v_{\text {r}}$, which is the solution to $u_{r_a}^*(v_{\text {r}}) = u_{r_b}^*(v_{\text {r}})$. It is then proven that $v_{\text {req}}> v_{\text {r}}$, which ensures $u_{r_a}^*(v_{\text {req}}) > u_{r_b}^*(v_{\text {req}})$.

Fig. 14

Comparing $v_{\text {req}}$ and $v_r$, which are the roots to Eqs. (63) and (100), respectively

Lemma 25

There is a unique solution $v_{\text {r}}$ to the equation $u_{r_a}^*(v_{\text {r}}) = u_{r_b}^*(v_{\text {r}})$, where $0 < v_{\text {r}}\le u_{v_{\text {r}}} = \frac{8}{e}$.

Proof

From Lemma 23 we know that the LHS of Eq. (100), which is denoted as $g(v_{\text {min}}, d)$, increases monotonically with respect to $v_{\text {min}}$. As such, there is a unique $v_{\text {r}}(d)$ that satisfies $g(v_{\text {r}}) = 1$, which is equivalent to $u_{r_a}^*(v_{\text {r}}) = u_{r_b}^*(v_{\text {r}})$. We have $g(0) = 0$ and we therefore require $v_{\text {r}}> 0$ to have $g(v_{\text {r}}) = 1$. From Lemma 24 we know that g(d) is monotonically decreasing with respect to d. As such, when d increases, the LHS decreases, and thus $v_{\text {r}}$ must increase to ensure $g(v_{\text {r}}) = 1$. This can be seen in Fig. 14. Since $v_{\text {r}}(d)$ increases with respect to d, we consider the following limit:

$$\begin{aligned} \lim _{d \rightarrow \infty } g(v_{\text {r}}; d)&= \lim _{d \rightarrow \infty } \left( \frac{v_{\text {r}}}{4 + \sqrt{\frac{v_{\text {r}}^2}{d} + 16}} \right) e^{\left( \frac{\frac{v_{\text {r}}}{\sqrt{d}} +\sqrt{\frac{v_{\text {r}}^2}{\sqrt{d}} + 16}}{4} \right) } \nonumber \\&= \frac{v_{\text {r}}}{8} e. \end{aligned}$$

(103)

For $d \in {\mathbb {Z}}^+$ it follows that there is a unique value of $v_{\text {r}}$ such that $u_{r_a}^*(v_{\text {r}}) = u_{r_b}^*(v_{\text {r}})$, where $v_{\text {r}}\in \left( 0, \frac{8}{e} \right]$, which completes the proof. $\square$

As explained in the proof of Lemma 19, $v_{\text {req}}(d, n_x)$ increases monotonically with respect to d and $n_x$ and is thus minimized with $n_x = 2$. We thus want to show that $v_{\text {req}}(d, n_x=2) >v_{\text {r}}(d) \, \forall d \in {\mathbb {Z}}^+$ and this will also hold for all $n_x \in {\mathbb {Z}}^+ \setminus 1$. Figure 14 shows that for $1 \le d \le 100$ we have $v_{\text {req}}(d, n_x=2) > v_{\text {r}}$. For $d = 100$ we have $v_{\text {req}}(d, n_x=2) > u_{v_{\text {r}}}$ and this holds for all $d> 100$ since $v_{\text {req}}(d)$ is monotonically increasing with respect to d. We thus have $v_{\text {req}}(d, n_x) > v_{\text {r}}(d) \, \forall \, (d, n_x) \in \left( {\mathbb {Z}}^+, {\mathbb {Z}}^+ \setminus 1\right)$ and it follows from Eq. (100) that $u_{r_a}^*(v_{\text {min}}) > u_{r_b}^*(v_{\text {min}})$ when $v_{\text {min}}\ge v_{\text {req}}$, which completes the proof of Proposition 15.