Heavy traffic queue length scaling in switches with reconfiguration delay

Abstract

The Adaptive MaxWeight policy achieves optimal throughput for switches with nonzero reconfiguration delay and has been shown to have good delay performance in simulation. In this paper, we analyze the queue length behavior of a switch with nonzero reconfiguration delay operating under the Adaptive MaxWeight. We first show that the Adaptive MaxWeight policy exhibits a weak state space collapse behavior in steady state, which can be viewed as an inheritance of a similar property of the MaxWeight policy in a switch with zero reconfiguration delay. The weak state space collapse result is then utilized to obtain an asymptotically tight bound on an expression involving the steady-state queue length and the probability of reconfiguration for the Adaptive MaxWeight policy in the heavy traffic regime. We then derive the relation between the expected schedule duration and the steady-state queue length through Lyapunov drift analysis and characterize bounds for the expected steady-state queue length. While the resulting queue length bounds are not asymptotically tight, they suggest an approximate queue length scaling behavior, which approaches the optimal scaling with respect to the traffic load and the reconfiguration delay when the hysteresis function of the Adaptive MaxWeight policy approaches a linear function.

Appendices

Appendices

Proof of Proposition 1

Proof

For ease of notation, we drop the superscript $(\epsilon )$ in the following derivation. For each state $\mathbf {X} = (\mathbf {q}, \mathbf {s}, r)$, we define the Lyapunov function $Z(\mathbf {X}) = \max \{ \Vert \mathbf {q}_{\perp }\Vert - \theta \Vert \mathbf {q}_{\parallel } \Vert , 0 \}$. We then apply Lemma 1 with the Lyapunov function Z to obtain the result. Note that the selection of the Lyapunov function is such that Z is a nonnegative function. Since $\Vert \mathbf {q}_{\perp }\Vert - \theta \Vert \mathbf {q}_{\parallel } \Vert \le Z(\mathbf {X})$ for any state $\mathbf {X} = (\mathbf {q}, \mathbf {s}, r)$, the statement of Proposition 1 follows from a bound on ${\mathbb {E}}[Z(\bar{\mathbf {X}})]$.

We first verify Condition C.2 for $Z(\mathbf {X})$. Since $|\max \{a, 0\} - \max \{b, 0\}| \le |a - b|$ for any $a, b \in {\mathbb {R}}$, we have

$$\begin{aligned} |\Delta ^{\mathrm{T}} Z(\mathbf {X})|&\le \Big | \Big (\Vert \mathbf {q}_{\perp }(t+T)\Vert - \theta \Vert \mathbf {q}_{\parallel }(t+T)\Vert \Big ) - \Big (\Vert \mathbf {q}_{\perp }(t)\Vert - \theta \Vert \mathbf {q}_{\parallel }(t)\Vert \Big ) \Big | \nonumber \\&\le \Big | \Vert \mathbf {q}_{\perp }(t+T)\Vert - \Vert \mathbf {q}_{\perp }(t)\Vert \Big | + \theta \Big | \Vert \mathbf {q}_{\parallel }(t+T)\Vert - \Vert \mathbf {q}_{\parallel }(t)\Vert \Big | \nonumber \\&\le \Vert \mathbf {q}_{\perp }(t+T) - \mathbf {q}_{\perp }(t)\Vert + \theta \Vert \mathbf {q}_{\parallel }(t+T) - \mathbf {q}_{\parallel }(t)\Vert \nonumber \\&\le (1+\theta ) \left\| \mathbf {q}(t+T) - \mathbf {q}(t) \right\| \nonumber \\&\le (1+\theta ) na_{\max }T \triangleq D. \end{aligned}$$

(43)

Here, we use the fact that $\mathbf {q}_{\perp }$ is a projection onto $\mathcal {K}^{\circ } = \{\mathbf {x} \in {\mathbb {R}}^{n^2}: \langle \mathbf {x}, \mathbf {y} \rangle \le 0, \forall \mathbf {y} \in \mathcal {K} \}$, the polar cone of $\mathcal {K}$. Since the projection onto a cone is nonexpansive, we have $\Vert \mathbf {x}_{\perp } - \mathbf {y}_{\perp } \Vert \le \Vert \mathbf {x} - \mathbf {y}\Vert$ and $\Vert \mathbf {x}_{\parallel } - \mathbf {y}_{\parallel } \Vert \le \Vert \mathbf {x} - \mathbf {y}\Vert$, $\forall \mathbf {x}, \mathbf {y}$.

To verify condition C.1, we need to bound the expected T-step drift for $Z(\mathbf {X})$. For ease of notation, we denote ${\mathbb {E}}[\ \cdot \ |\mathbf {X}(t) = \mathbf {X}]$ as ${\mathbb {E}}_{\mathbf {X}}[\ \cdot \ ]$.

Since (43) provides an upper bound on the magnitude of the T-step drift, we know that if $\mathbf {X}(t)$ satisfies $Z(\mathbf {X}) > D$, then $Z(\mathbf {X}(t+T))$ is positive, and we may drop the $\max \{\cdot , 0\}$ expression in the definition of $Z(\mathbf {X})$ in this case. In other words, for all $\mathbf {X}$ such that $Z(\mathbf {X}) > D$, we may write the expected T-step drift as

$$\begin{aligned} {\mathbb {E}}_{\mathbf {X}}\Big [\Delta ^{\mathrm{T}}Z(\mathbf {X}) \Big ] = {\mathbb {E}}_{\mathbf {X}} \Big [ \Big (\Vert \mathbf {q}_{\perp }(t+T)\Vert -\Vert \mathbf {q}_{\perp }(t)\Vert \Big ) - \theta \Big (\Vert \mathbf {q}_{\parallel }(t+T)\Vert -\Vert \mathbf {q}_{\parallel }(t)\Vert \Big ) \Big ]. \end{aligned}$$

(44)

Therefore, we need only to consider the T-step expected drift of $\Vert \mathbf {q}_{\perp }\Vert$ and $\Vert \mathbf {q}_{\parallel }\Vert$.

We first consider the drift of $\Vert \mathbf {q}_{\perp }\Vert$. The derivation follows along the lines of [15], where the relation in [15, Lemma 4] is used: Let $V(\mathbf {X}) = ||\mathbf {q}||^2$, $V_{\parallel }(\mathbf {X}) = ||\mathbf {q}_{\parallel }||^2$, and $\Delta V$, $\Delta V_{\parallel }$ denote the one-step drift of V, $V_{\parallel }$, respectively. Then

$$\begin{aligned} \Vert \mathbf {q}_{\perp }(t+1)\Vert - \Vert \mathbf {q}_{\perp }(t)\Vert \le \frac{1}{2\Vert \mathbf {q}_{\perp }(t)\Vert } \big (\Delta V(\mathbf {X}(t)) - \Delta V_{\parallel }(\mathbf {X}(t)) \big ). \end{aligned}$$

(45)

The inequality could be derived as follows:

$$\begin{aligned} \Vert \mathbf {q}_{\perp }(t+1)\Vert - \Vert \mathbf {q}_{\perp }(t)\Vert = \sqrt{\Vert \mathbf {q}_{\perp }(t+1)\Vert ^2} - \sqrt{\Vert \mathbf {q}_{\perp }(t)\Vert ^2} \le \frac{\Vert \mathbf {q}_{\perp }(t+1)\Vert ^2 - \Vert \mathbf {q}_{\perp }(t)\Vert ^2 }{2\Vert \mathbf {q}_{\perp }(t)\Vert }, \end{aligned}$$

where the inequality follows from the concavity of the square root function: Since $f(x) = \sqrt{x}, x > 0$ is concave, we have $f(y) - f(x) \le (y - x) f'(x) = \frac{y-x}{2\sqrt{x}}$. Setting $x = \Vert \mathbf {q}_{\perp }(t)\Vert ^2$ and $y = \Vert \mathbf {q}_{\perp }(t+1)\Vert ^2$ gives the inequality. Then, with the orthogonality between $\mathbf {q}_{\perp }$ and $\mathbf {q}_{\parallel }$, we have $\Vert \mathbf {q}_{\perp }\Vert ^2 = \Vert \mathbf {q}\Vert ^2 - \Vert \mathbf {q}_{\parallel }\Vert ^2$, and (45) follows by applying the relation for $\mathbf {q}_{\perp }(t+1)$ and $\mathbf {q}_{\perp }(t)$ and then rearranging the terms.

With (45), we have the following inequality for the T-step drift of $\Vert \mathbf {q}_{\perp }(t)\Vert$:

$$\begin{aligned} {\mathbb {E}}_{\mathbf {X}} \Big [ \Vert \mathbf {q}_{\perp }(t+T)\Vert -\Vert \mathbf {q}_{\perp }(t)\Vert \Big ]&= {\mathbb {E}}_{\mathbf {X}} \bigg [ \sum _{\tau =t}^{t+T-1} \big ( \Vert \mathbf {q}_{\perp }(\tau +1)\Vert - \Vert \mathbf {q}_{\perp }(\tau ) \Vert \big ) \bigg ] \nonumber \\&\le {\mathbb {E}}_{\mathbf {X}} \bigg [ \sum _{\tau =t}^{t+T-1} \frac{\Delta V(\mathbf {X}(\tau )) - \Delta V_{\parallel }(\mathbf {X}(\tau ))}{2\Vert \mathbf {q}_{\perp }(\tau )\Vert } \bigg ] \nonumber \\&= {\mathbb {E}}_{\mathbf {X}} \bigg [ \sum _{\tau =t}^{t+T-1} {\mathbb {E}}\Big [ \frac{\Delta V(\mathbf {X}(\tau )) - \Delta V_{\parallel }(\mathbf {X}(\tau ))}{2\Vert \mathbf {q}_{\perp }(\tau )\Vert } \Big | \mathbf {X}(\tau ) \Big ] \bigg ]. \end{aligned}$$

(46)

We now derive bounds for $\Delta V$ and $\Delta V_{\parallel }$:

$$\begin{aligned}&{\mathbb {E}}\Big [ \Delta V(\mathbf {X}(\tau )) \Big | \mathbf {X}(\tau ) \Big ] \nonumber \\&\quad = {\mathbb {E}}\Big [ \Vert \mathbf {q}(\tau +1)\Vert ^2 - \Vert \mathbf {q}(\tau )\Vert ^2 \Big | \mathbf {X}(\tau ) \Big ] \\&\quad = {\mathbb {E}}\Big [ \Vert \mathbf {q}(\tau ) + \mathbf {a}(\tau ) - \mathbf {s}(\tau )\mathbb {1}_{\{r(\tau ) = 0\}} \Vert ^2 + \Vert \mathbf {u}(\tau ) \Vert ^2 \nonumber \\&\quad \quad + 2\Big \langle \mathbf {q}(\tau +1) - \mathbf {u}(\tau ), \mathbf {u}(\tau ) \Big \rangle - \Vert \mathbf {q}(\tau )\Vert ^2 \Big | \mathbf {X}(\tau ) \Big ] \\&\quad \le {\mathbb {E}}\Big [ \Vert \mathbf {q}(\tau ) + \mathbf {a}(\tau ) - \mathbf {s}(\tau )\mathbb {1}_{\{r(\tau ) = 0\}} \Vert ^2 - \Vert \mathbf {q}(\tau )\Vert ^2 \Big | \mathbf {X}(\tau ) \Big ] \\&\quad = \sum _{i,j} {\mathbb {E}}\Big [ a_{ij}^2(\tau ) + s_{ij}(\tau )\mathbb {1}_{\{r(\tau ) = 0\}} - 2 a_{ij}(\tau ) s_{ij}(\tau ) \mathbb {1}_{\{r(\tau ) = 0\}} \Big | \mathbf {X}(\tau ) \Big ] \nonumber \\&\quad \quad + {\mathbb {E}}\Big [ 2 \Big \langle \mathbf {q}(\tau ), \varvec{\lambda } - \mathbf {s}(\tau )\mathbb {1}_{\{r(\tau ) = 0\}} \Big \rangle \Big | \mathbf {X}(\tau ) \Big ] \\&\quad {\mathop {\le }\limits ^{(a)}} \sum _{ij} (\lambda _{ij}^2 + \sigma _{ij}^2) + n + 2 \Big \langle \mathbf {q}(\tau ), (1-\epsilon )\varvec{\nu } - \mathbf {s}(\tau ) \Big \rangle + 2 \Big \langle \mathbf {q}(\tau ), \mathbf {s}(\tau ) \Big \rangle \mathbb {1}_{\{r(\tau )> 0\}} \\&\quad = \Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n - 2\epsilon \Big \langle \mathbf {q}(\tau ), \varvec{\nu } \Big \rangle + 2 \Big \langle \mathbf {q}(\tau ), \varvec{\nu } - \mathbf {s}(\tau ) \Big \rangle + 2 \Big \langle \mathbf {q}(\tau ), \mathbf {s}(\tau ) \Big \rangle \mathbb {1}_{\{r(\tau ) > 0\}}, \end{aligned}$$

where (a) follows from ${\mathbb {E}}[ a_{ij}^2] = \lambda _{ij}^2 + \sigma _{ij}^2$, $a_{ij}(t)s_{ij}(t) \ge 0$ for all i, j, and $\sum _{ij} s_{ij}(t) = 1$ for all t.

Suppose g is the sublinear hysteresis function for Adaptive MaxWeight, then by the sublinearity, there exists a constant $K_{\theta }$ such that $g(x) < \frac{\theta }{\alpha } x$ for any $x > K_{\theta }$, where $\alpha = \frac{8\Vert \varvec{\nu }\Vert }{\nu _{\min }}$. Hence, by the definition of Adaptive MaxWeight, we have, for any $\mathbf {X}(\tau )$ such that $\langle \mathbf {q}(\tau ), \mathbf {s}^*(\tau ) \rangle > K_{\theta }$,

$$\begin{aligned} \Big \langle \mathbf {q}(\tau ), \varvec{\nu }-\mathbf {s}(\tau ) \Big \rangle&= \Big \langle \mathbf {q}(\tau ), \varvec{\nu }-\mathbf {s}^*(\tau ) \Big \rangle + \Big \langle \mathbf {q}(\tau ), \mathbf {s}^*(\tau )-\mathbf {s}(\tau ) \Big \rangle \\&\le \Big \langle \mathbf {q}(\tau ), \varvec{\nu }-\mathbf {s}^*(\tau ) \Big \rangle + g\left( \Big \langle \mathbf {q}(\tau ), \mathbf {s}^*(\tau ) \Big \rangle \right) \\&\le \Big \langle \mathbf {q}(\tau ), \varvec{\nu }-\mathbf {s}^*(\tau ) \Big \rangle + \frac{\theta }{\alpha } \Big \langle \mathbf {q}(\tau ), \mathbf {s}^*(\tau ) \Big \rangle \\&= \Big (1- \frac{\theta }{\alpha } \Big ) \Big \langle \mathbf {q}(\tau ), \varvec{\nu }-\mathbf {s}^*(\tau ) \Big \rangle + \frac{\theta }{\alpha } \Big \langle \mathbf {q}(\tau ), \varvec{\nu } \Big \rangle . \end{aligned}$$

From [15, Claim 2], we have $\Big \langle \mathbf {q}(\tau ), \varvec{\nu }-\mathbf {s}^*(\tau ) \Big \rangle \le -\nu _{\min } \Vert \mathbf {q}_{\perp }(\tau )\Vert$. Therefore,

$$\begin{aligned} {\mathbb {E}}\Big [ \Delta V(\mathbf {X}(\tau )) \Big | \mathbf {X}(\tau ) \Big ]&\le \Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n - 2\epsilon \Big \langle \mathbf {q}(\tau ), \varvec{\nu } \Big \rangle -2 \Big (1 - \frac{\theta }{\alpha }\Big ) \nu _{\min }\Vert \mathbf {q}_{\perp }(\tau )\Vert \nonumber \\&\quad \quad + 2 \frac{\theta }{\alpha } \Big \langle \mathbf {q}(\tau ), \varvec{\nu } \Big \rangle + 2 \Big \langle \mathbf {q}(\tau ), \mathbf {s}(\tau ) \Big \rangle \mathbb {1}_{\{r(\tau ) > 0\}}. \end{aligned}$$

(47)

For $\Delta V_{\parallel }$, we have

$$\begin{aligned}&{\mathbb {E}}\Big [ \Delta V_{\parallel }(\mathbf {X}(\tau )) \Big | \mathbf {X}(\tau ) \Big ] = {\mathbb {E}}\Big [ \Vert \mathbf {q}_{\parallel }(\tau +1)\Vert ^2 - \Vert \mathbf {q}_{\parallel }(\tau )\Vert ^2 \Big | \mathbf {X}(\tau ) \Big ] \nonumber \\&\quad = {\mathbb {E}}\Big [ \Big \langle \mathbf {q}_{\parallel }(\tau +1) + \mathbf {q}_{\parallel }(\tau ), \mathbf {q}_{\parallel }(\tau +1) - \mathbf {q}_{\parallel }(\tau ) \Big \rangle \Big | \mathbf {X}(\tau ) \Big ] \nonumber \\&\quad = {\mathbb {E}}\Big [ \Vert \mathbf {q}_{\parallel }(\tau +1) - \mathbf {q}_{\parallel }(\tau )\Vert ^2 + 2\Big \langle \mathbf {q}_{\parallel }(\tau ), \mathbf {q}_{\parallel }(\tau +1) - \mathbf {q}_{\parallel }(\tau ) \Big \rangle \Big | \mathbf {X}(\tau ) \Big ] \nonumber \\&\quad \ge 2{\mathbb {E}}\Big [ \Big \langle \mathbf {q}_{\parallel }(\tau ), \mathbf {q}_{\parallel }(\tau +1) - \mathbf {q}_{\parallel }(\tau ) \Big \rangle \Big | \mathbf {X}(\tau ) \Big ] \nonumber \\&\quad = 2{\mathbb {E}}\Big [ \Big \langle \mathbf {q}_{\parallel }(\tau ), \mathbf {q}(\tau +1) - \mathbf {q}(\tau ) \Big \rangle - \Big \langle \mathbf {q}_{\parallel }(\tau ), \mathbf {q}_{\perp }(\tau +1) - \mathbf {q}_{\perp }(\tau ) \Big \rangle \Big | \mathbf {X}(\tau ) \Big ] \nonumber \\&\quad {\mathop {\ge }\limits ^{(b)}} 2{\mathbb {E}}\Big [ \Big \langle \mathbf {q}_{\parallel }(\tau ), \mathbf {a}(\tau ) - \mathbf {s}(\tau ) \mathbb {1}_{\{r(\tau ) = 0\}} + \mathbf {u}(\tau ) \Big \rangle \Big | \mathbf {X}(\tau ) \Big ] \nonumber \\&\quad \ge 2\Big \langle \mathbf {q}_{\parallel }(\tau ), \varvec{\lambda } \Big \rangle -2 \Big \langle \mathbf {q}_{\parallel }(\tau ), \mathbf {s}(\tau ) \mathbb {1}_{\{r(\tau ) = 0\}} \Big \rangle \nonumber \\&\quad = -2\epsilon \Big \langle \mathbf {q}_{\parallel }(\tau ), \varvec{\nu } \Big \rangle + 2 \Big \langle \mathbf {q}_{\parallel }(\tau ), \varvec{\nu } - \mathbf {s}(\tau ) \Big \rangle + 2 \Big \langle \mathbf {q}_{\parallel }(\tau ), \mathbf {s}(\tau ) \Big \rangle \mathbb {1}_{\{r(\tau )> 0\}} \nonumber \\&\quad = -2\epsilon \Big \langle \mathbf {q}_{\parallel }(\tau ), \varvec{\nu } \Big \rangle + 2 \Big \langle \mathbf {q}_{\parallel }(\tau ), \mathbf {s}(\tau ) \Big \rangle \mathbb {1}_{\{r(\tau ) > 0\}}. \end{aligned}$$

(48)

For (b), we use the following properties of the projection onto cone $\mathcal {K}$: For $\mathbf {q} \in {\mathbb {R}}^{n^2}$, $\langle \mathbf {q}_{\parallel }, \mathbf {q}_{\perp } \rangle = 0$, and $\mathbf {q}_{\perp } \in \mathcal {K}^{\circ }$. Therefore $\langle \mathbf {q}_{\parallel }(t), \mathbf {q}_{\perp }(t) \rangle = 0$, and $\langle \mathbf {q}_{\parallel }(t), \mathbf {q}_{\perp }(t+1) \rangle \le 0$.

Applying (47) and (48) in (46), we obtain

$$\begin{aligned}&{\mathbb {E}}_{\mathbf {X}} \Big [ \Vert \mathbf {q}_{\perp }(t+T)\Vert - \Vert \mathbf {q}_{\perp }(t)\Vert \Big ] \nonumber \\&\quad \le {\mathbb {E}}_{\mathbf {X}} \bigg [ \sum _{\tau =t}^{t+T-1} \Big ( \frac{\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n}{2\Vert \mathbf {q}_{\perp }(\tau )\Vert } - \epsilon \Big \langle \frac{\mathbf {q}_{\perp }(\tau )}{\Vert \mathbf {q}_{\perp }(\tau )\Vert }, \varvec{\nu } \Big \rangle - \big (1-\frac{\theta }{\alpha }\big ) \nu _{\min } + \frac{\theta \big \langle \mathbf {q}(\tau ), \varvec{\nu } \big \rangle }{\alpha \Vert \mathbf {q}_{\perp }(\tau )\Vert } \nonumber \\&\quad \quad + \Big \langle \frac{\mathbf {q}_{\perp }(\tau )}{\Vert \mathbf {q}_{\perp }(\tau )\Vert }, \mathbf {s}(\tau ) \Big \rangle \mathbb {1}_{\{r(\tau )> 0\}} \Big ) \bigg ] \nonumber \\&\quad \le {\mathbb {E}}_{\mathbf {X}} \bigg [ T \bigg ( \frac{\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n}{\min \limits _{\tau \in [t, t+T]}2\Vert \mathbf {q}_{\perp }(\tau )\Vert } + \epsilon \Vert \varvec{\nu }\Vert - \big (1-\theta \big ) \nu _{\min } + \frac{1 + \theta }{\alpha } \Vert \varvec{\nu }\Vert \bigg ) + \sqrt{n}\sum _{\tau =t}^{t+T-1}\mathbb {1}_{\{r(\tau ) > 0\}} \bigg ] , \end{aligned}$$

(49)

where we have used the fact that $\Vert \mathbf {q}_{\perp }\Vert \ge \theta \Vert \mathbf {q}_{\parallel }\Vert$ implies $\theta \Vert \mathbf {q}\Vert \le \theta (\Vert \mathbf {q}_{\parallel }\Vert + \Vert \mathbf {q}_{\perp }\Vert ) \le (1+\theta ) \Vert \mathbf {q}_{\perp }\Vert$, and $\Vert \mathbf {s}(\tau )\Vert \le \sqrt{n}$ for any schedule $\mathbf {s}(\tau ) \in \mathcal {S}$.

On the other hand, the drift of $\Vert \mathbf {q}_{\parallel }\Vert$ can be obtained following (48):

$$\begin{aligned} {\mathbb {E}}_{\mathbf {X}} \Big [ \Vert \mathbf {q}_{\parallel }(t+T)\Vert - \Vert \mathbf {q}_{\parallel }(t)\Vert \Big ]&= {\mathbb {E}}_{\mathbf {X}} \bigg [ \sum _{\tau =t}^{t+T-1} {\mathbb {E}}\Big [ \Vert \mathbf {q}_{\parallel }(\tau +1)\Vert - \Vert \mathbf {q}_{\parallel }(\tau )\Vert \Big | \mathbf {X}(\tau ) \Big ] \bigg ] \nonumber \\&\ge {\mathbb {E}}_{\mathbf {X}} \bigg [ \sum _{\tau =t}^{t+T-1} {\mathbb {E}}\Big [ \frac{\Vert \mathbf {q}_{\parallel }(\tau +1)\Vert ^2 - \Vert \mathbf {q}_{\parallel }(\tau )\Vert ^2}{\Vert \mathbf {q}_{\parallel }(\tau +1)\Vert +\Vert \mathbf {q}_{\parallel }(\tau )\Vert } \Big | \mathbf {X}(\tau ) \Big ] \bigg ] \nonumber \\&\ge {\mathbb {E}}_{\mathbf {X}} \bigg [ \sum _{\tau =t}^{t+T-1} {\mathbb {E}}\Big [ \frac{-2\epsilon \big \langle \mathbf {q}_{\parallel }(\tau ), \varvec{\nu } \big \rangle }{\Vert \mathbf {q}_{\parallel }(\tau +1)\Vert +\Vert \mathbf {q}_{\parallel }(\tau )\Vert } \Big | \mathbf {X}(\tau ) \Big ] \bigg ] \nonumber \\&\ge {\mathbb {E}}_{\mathbf {X}} \bigg [ \sum _{\tau =t}^{t+T-1} \frac{-2\epsilon \big \langle \mathbf {q}_{\parallel }(\tau ), \varvec{\nu } \big \rangle }{\Vert \mathbf {q}_{\parallel }(\tau )\Vert } \bigg ] \ge -2T \epsilon \Vert \varvec{\nu }\Vert , \end{aligned}$$

(50)

where the last inequality follows from the Cauchy–Schwartz inequality.

Now, applying (49) and (50) in (44), we obtain

$$\begin{aligned} {\mathbb {E}}_{\mathbf {X}} \Big [ \Delta ^{\mathrm{T}} Z(\mathbf {X}) \Big ]&\le {\mathbb {E}}_{\mathbf {X}} \bigg [ T \bigg ( \frac{\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n}{\min \limits _{\tau \in [t, t+T]}2\Vert \mathbf {q}_{\perp }(\tau )\Vert } + (1+2\theta )\epsilon \Vert \varvec{\nu }\Vert - \big (1-\theta \big ) \nu _{\min } + \frac{1 + \theta }{\alpha } \Vert \varvec{\nu }\Vert \bigg ) \nonumber \\&\quad + \sqrt{n}\sum _{\tau =t}^{t+T-1}\mathbb {1}_{\{r(\tau ) > 0\}} \bigg ]. \end{aligned}$$

(51)

From [19, Lemma 1], we know that for any fixed $T>0$, if $W^*(t) > g^{-1}\Big ( nT(a_{\max }+1) \Big ) + nT$, then at most one reconfiguration could occur within $[t, t+T]$, which gives $\sum _{\tau =t}^{t+T-1} \mathbb {1}_{r(\tau ) > 0} \le \Delta _r$.

Select $T = \frac{8 \sqrt{n} \Delta _r}{\nu _{\min }}$, then set $D = \frac{3}{2}na_{\max }T = \frac{12 n^{3/2} a_{\max } \Delta _r}{\nu _{\min }}$. Then, $\forall \mathbf {X}$ such that $Z(\mathbf {X}) > \kappa = \Big \{ D, na_{\max }T + \frac{4(\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n)}{\nu _{\min }} , nK_{\theta },$ $ng^{-1}\Big ( nT(a_{\max }+1) \Big ) + n^2T \Big \}$, and $\forall \epsilon$ such that $0 < \epsilon \le \frac{\nu _{\min }}{16\Vert \varvec{\nu }\Vert }$, we have ${\mathbb {E}}_{\mathbf {X}} \Big [ \Delta ^{\mathrm{T}} Z(\mathbf {X}) \Big ] \le -\frac{(1-\theta )\nu _{\min }}{4} \le -\frac{\nu _{\min }}{8}$.

Hence, by Lemma 1, we have $\forall \epsilon$ such that $0 < \epsilon \le \frac{\nu _{\min }}{16\Vert \varvec{\nu }\Vert }$,

$$\begin{aligned} {\mathbb {E}}\left[ \Vert \bar{\mathbf {q}}_{\perp } \Vert - \theta \Vert \bar{\mathbf {q}}_{\parallel } \Vert \right] \le {\mathbb {E}}\left[ Z(\bar{\mathbf {X}}) \right] \le \kappa + \frac{16 D^2}{\nu _{\min }}. \end{aligned}$$

Letting $M_{\theta } = \kappa + \frac{16 D^2}{\nu _{\min }}$, we then have the result. $\square$

Proof of Proposition 2

Proof

We will derive the lower bound by bounding the expected queue length sum at each input port, in particular, ${\mathbb {E}}\big [ \sum _{j} \bar{q}_{ij}(t) \big ]$ for each input port i.

The queue length dynamics of the coupled queue $\phi _i(t)$ are given by

$$\begin{aligned} \phi _i(t+1)&= \big [ \phi _i(t) + b_i(t) - \mathbb {1}_{\{r(t) = 0\}} \big ]^+ \\&= \phi _i(t) + b_i(t) - \mathbb {1}_{\{r(t) = 0\}} + v_i(t), \end{aligned}$$

where $v_i(t)$ is the unused service and satisfies $\phi _i(t+1)v_i(t) = 0$.

We may show by induction that ${\mathbb {E}}\big [ \sum _{j} \bar{q}_{ij}(t) \big ] \ge {\mathbb {E}}\big [ \bar{\phi }_i(t) \big ]$. It then remains to lower bound ${\mathbb {E}}\big [ \bar{\phi }_i(t) \big ]$. We consider the expected drift of $\big ( \bar{\phi }_i(t) \big )^2$ as follows:

$$\begin{aligned}&{\mathbb {E}}\big [ \big ( \bar{\phi }_i(t+1) \big )^2 - \big ( \bar{\phi }_i(t) \big )^2 \big ] \\&\quad = {\mathbb {E}}\big [ \big ( \bar{\phi }_i(t) + b_i(t) - \mathbb {1}_{\{r(t)=0\}} \big )^2 - \big ( v_i(t) \big )^2 - \big ( \bar{\phi }_i(t) \big )^2 \big ] \\&\quad = {\mathbb {E}}\big [ 2\bar{\phi }_i(t) \big ( b_i(t) - \mathbb {1}_{\{r(t)=0\}} \big ) + \big ( b_i(t) - \mathbb {1}_{\{r(t)=0\}} \big )^2 - \big ( v_i(t) \big )^2 \big ] \\&\quad = {\mathbb {E}}\big [ 2\bar{\phi }_i(t) \big ( (1-\epsilon ) - (1-\mathbb {1}_{\{r(t)>0\}}) \big ) + \big ( b_i(t) - (1-\epsilon ) + (\mathbb {1}_{\{r(t)>0\}} - \epsilon ) \big )^2 - v_i(t) \big ] \\&\quad = -2\epsilon {\mathbb {E}}\big [\bar{\phi }_i(t)\big ] + 2 {\mathbb {E}}\big [\bar{\phi }_i(t)\big ] {\mathbb {E}}\big [\mathbb {1}_{\{r(t)>0\}}\big ] + \mathrm {Var}(b_i(t)) + {\mathbb {E}}\big [ (\mathbb {1}_{\{r(t)>0\}} - \epsilon )^2 \big ] - {\mathbb {E}}\big [ v_i(t) \big ], \\ \end{aligned}$$

where the last equality follows from the independence between the queue length process $\phi _i(t)$ and the schedule reconfiguration decision. We have

$$\begin{aligned} 2 (\epsilon - p) {\mathbb {E}}\big [\bar{\phi }_i(t)\big ] = \sum _{j} \sigma _{ij}^2 + p -2p\epsilon + \epsilon ^2 - {\mathbb {E}}\big [ v_i(t) \big ]. \end{aligned}$$

Considering the drift of $\bar{\phi }_i(t)$, we can derive ${\mathbb {E}}\big [ v_i(t) \big ]$ as follows:

$$\begin{aligned}&{\mathbb {E}}\big [ \bar{\phi }_t(t+1) - \bar{\phi }_i(t) \big ] = (1-\epsilon ) - (1 - {\mathbb {E}}\big [ \mathbb {1}_{\{r(t)>0\}} \big ]) + {\mathbb {E}}\big [ v_i(t) \big ] = 0 \\&\quad \Rightarrow {\mathbb {E}}\big [ v_i(t) \big ] = \epsilon - {\mathbb {E}}\big [ \mathbb {1}_{\{r(t)>0\}} \big ] = \epsilon - p. \end{aligned}$$

We thus have

$$\begin{aligned} {\mathbb {E}}\big [\bar{\phi }_i(t)\big ] = \frac{\displaystyle \sum _{j} \sigma _{ij}^2}{2 (\epsilon - p)} - \frac{(1-\epsilon )(\epsilon -2p)}{2 (\epsilon - p) }. \end{aligned}$$

Using ${\mathbb {E}}\big [ \sum _{j} \bar{q}_{ij}(t) \big ] \ge {\mathbb {E}}\big [ \bar{\phi }_i(t) \big ]$, and summing over each input port, we obtain

$$\begin{aligned} {\mathbb {E}}\bigg [\sum _{ij} \bar{q}_{ij}(t)\bigg ] \ge {\mathbb {E}}\bigg [ \sum _{i} \bar{\phi }_i(t)\bigg ] \ge \frac{\displaystyle \sum _{ij} \sigma _{ij}^2}{2 (\epsilon - p)} - \frac{(1-\epsilon )(\epsilon -2p)}{2 (\epsilon - p) }. \end{aligned}$$

$\square$

Proof of Proposition 3

Proof

Let $t_k, k = 1, \ldots$, denote the $k\mathrm{th}$ schedule reconfiguration time. By the assumption that the system state $\mathbf {X}(t) = \big ( \mathbf {q}(t), \mathbf {s}(t), r(t) \big )$ converges in distribution to a steady-state random vector $\bar{\mathbf {X}} = \big ( \bar{\mathbf {q}}, \bar{\mathbf {s}}, \bar{r} \big )$, we may consider a renewal-reward process associated with $\mathbf {X}(t)$ where $t_k$ are the arrival epochs, while the reward function is the sum of queue lengths given by $R(t) = \sum _{ij} q_{ij}(t)$.

From renewal-reward theory, we have

$$\begin{aligned} {\mathbb {E}}\bigg [ \sum _{ij} q_{ij}(t) \bigg ] = \lim _{T \rightarrow \infty } \frac{1}{T} {\mathbb {E}}\bigg [ \sum _{t=0}^{T} q_{ij}(t) \bigg ] = \frac{\displaystyle {\mathbb {E}}\bigg [ \sum _{t=t_k}^{t_{k+1}-1} \sum _{ij} q_{ij}(t) \bigg ] }{ {\mathbb {E}}[ t_{k+1} - t_k ] }. \end{aligned}$$

For any (i, j) such that $S_{ij}(t_k) = 0$, we have $q_{ij}(t) = q_{ij}(t_k) + \sum _{\tau = t_k}^{t-1} a_{ij}(\tau ) \ge \sum _{\tau = t_k}^{t-1} a_{ij}(\tau )$, while for any (i, j) such that $S_{ij}(t_k) = 1$, we have $q_{ij}(t) \ge 0$.

$$\begin{aligned} {\mathbb {E}}\bigg [ \sum _{t=t_k}^{t_{k+1}-1} \sum _{ij} q_{ij}(t) \bigg ]&= {\mathbb {E}}\bigg [ \sum _{t=t_k}^{t_{k+1}-1} {\mathbb {E}}\Big [ \sum _{ij} q_{ij}(t) \Big | \mathbf {q}(t_k) \Big ] \bigg ] \\&\ge {\mathbb {E}}\bigg [ \sum _{t=t_k}^{t_{k+1}-1} \sum _{(i,j): S_{ij}(t_k) = 0} \lambda _{ij}(t-t_k) \bigg ] \\&\ge {\mathbb {E}}\bigg [ \sum _{t=t_k}^{t_{k+1}-1} \Big ( t-t_{k} \Big ) \Big ( n(1-\epsilon ) - \max _{\mathbf {S}} \langle \mathbf {S}, \varLambda \rangle \Big ) \bigg ] \\&= {\mathbb {E}}\bigg [ \frac{(t_{k+1} - t_k)^2}{2} \bigg ] (n - \bar{\alpha }) (1-\epsilon ). \end{aligned}$$

We then have a lower bound on the expected queue length sum as follows:

$$\begin{aligned} {\mathbb {E}}\bigg [ \sum _{ij} q_{ij}(t) \bigg ]&= \frac{{\mathbb {E}}[(t_{k+1}-t_k)^2] }{2 {\mathbb {E}}[t_{k+1}-t_k]} (n - \bar{\alpha }) (1-\epsilon ) \\&\ge \frac{{\mathbb {E}}[t_{k+1}-t_k]}{2} (n - \bar{\alpha }) (1-\epsilon ) \\&= \frac{\Delta _r}{2p}(n - \bar{\alpha }) (1-\epsilon ). \end{aligned}$$

$\square$

Proof of Proposition 4

Proof

Given the period T, we consider the Markov chain $\mathbf {X}(t)$ being sampled at times $t_k = kT, k = 0, 1, \ldots$. Since the Fixed Frame MaxWeight policy stabilizes the system if $T > \frac{\Delta _r}{\epsilon }$, we know that $\mathbf {X}(t)$ converges to a steady-state distribution, and so does $\mathbf {X}(t_k)$. Let $\bar{\mathbf {X}} = (\bar{\mathbf {q}}, \bar{\mathbf {s}}, \bar{r})$ and $\hat{\mathbf {X}} = (\hat{\mathbf {q}}, \hat{\mathbf {s}}, \hat{r})$ denote the steady-state distribution of $\mathbf {X}(t)$ and $\mathbf {X}(t_k)$, respectively. By the assumption on the maximum arrival, we immediately have that ${\mathbb {E}}[ \sum _{ij} \bar{q}_{ij} ] \le {\mathbb {E}}[ \sum _{ij} \hat{q}_{ij} ] + n^2a_{\max }T$. It then remains to bound ${\mathbb {E}}[ \sum _{ij} \hat{q}_{ij} ]$ following the similar procedures in [15]:

• 1. Derive an upper bound on ${\mathbb {E}}[\Vert \mathbf {q}_{\perp }(t_k)\Vert ^2]$.

• 2. Derive the queue length upper bound which depends on ${\mathbb {E}}[\Vert \mathbf {q}_{\perp }(t_k)\Vert ^2]$.

Consider the Lyapunov function $Z(\mathbf {X}) = \Vert \mathbf {q}_{\perp }\Vert$. By the assumption on the maximum arrival, we have

$$\begin{aligned} | \Delta Z(\mathbf {X}) |&= \Big | \Vert \mathbf {q}_{\perp }(t_{k+1})\Vert - \Vert \mathbf {q}(t_k)\Vert \Big | \le \Vert \mathbf {q}_{\perp }(t_{k+1}) - \mathbf {q}(t_k)\Vert = \sqrt{\sum _{ij} |q_{ij}(t_{k+1} - q_{ij}(t_k)|^2} \nonumber \\&\le na_{\max }T. \end{aligned}$$

(52)

For the expected drift at steady state, we have

$$\begin{aligned} {\mathbb {E}}\Big [ \Vert \mathbf {q}_{\perp }(t_{k+1})\Vert - \Vert \mathbf {q}(t_k)\Vert \Big ] \le {\mathbb {E}}\bigg [ \sum _{\tau =t_k}^{t_{k+1}-1} {\mathbb {E}}\Big [ \frac{\Delta V (\mathbf {X}(\tau )) - \Delta V_{\parallel }(\mathbf {X}(\tau ))}{2\Vert \mathbf {q}_{\perp }\Vert } \Big | \mathbf {X}(\tau ) \Big ] \bigg ], \end{aligned}$$

where $\Delta V(\mathbf {X})$ and $\Delta V_{\parallel }(\mathbf {X})$ are the drift of Lyapunov functions $V(\mathbf {X}) = \Vert \mathbf {q}\Vert ^2$ and $V_{\parallel }(\mathbf {X}) = \Vert \mathbf {q}_{\parallel }\Vert ^2$, respectively. Now, for each $\tau \in [t_k, t_{k+1}-1]$, we have

$$\begin{aligned} {\mathbb {E}}\Big [ \Delta V(\mathbf {X}(\tau )) \Big | \mathbf {X}(\tau ) \Big ]&\le \Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n - 2\epsilon \langle \mathbf {q}(\tau ), \varvec{\nu } \rangle + 2 \langle \mathbf {q}(\tau ), \varvec{\nu } - \mathbf {S}^*(\tau ) \rangle \\&\quad + 2 \langle \mathbf {q}(\tau ), \mathbf {S}^*(\tau ) - \mathbf {S}(\tau ) \rangle + 2\langle \mathbf {q}(\tau ), \mathbf {s}(\tau ) \rangle \mathbb {1}_{\{r(\tau )>0\}} \\&\le \Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n - 2\epsilon \langle \mathbf {q}(\tau ), \varvec{\nu } \rangle - 2 \nu _{\min } \Vert \mathbf {q}_{\perp }(\tau )\Vert \\&\quad + 2n(a_{\max }+1)\tau + 2\langle \mathbf {q}(\tau ), \mathbf {s}(\tau ) \rangle \mathbb {1}_{\{r(\tau )>0\}} \end{aligned}$$

and

$$\begin{aligned} {\mathbb {E}}\Big [ \Delta V_{\parallel }(\mathbf {X}(\tau )) \Big | \mathbf {X}(\tau ) \Big ] \ge -2\epsilon \langle \mathbf {q}_{\parallel }(\tau ), \varvec{\nu } \rangle + 2\langle \mathbf {q}_{\parallel }(\tau ), \mathbf {s}(\tau ) \rangle . \end{aligned}$$

We then have the expected drift of $Z(\mathbf {X})$ at steady state given by

$$\begin{aligned}&{\mathbb {E}}\Big [ \Vert \mathbf {q}_{\perp }(t_{k+1})\Vert - \Vert \mathbf {q}(t_k)\Vert \Big ] \\&\quad \le {\mathbb {E}}\bigg [ \sum _{\tau =t_k}^{t_{k+1}-1} \Big ( \frac{\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n}{2\Vert \mathbf {q}(\tau )\Vert } - \epsilon \left\langle \frac{\mathbf {q}_{\perp }(\tau )}{\Vert \mathbf {q}_{\perp }(\tau )\Vert }, \varvec{\nu } \right\rangle - \nu _{\min } + \frac{n(a_{\max }+1)\tau }{\Vert \mathbf {q}(\tau )\Vert } \Big ) \nonumber \\&\quad \quad + 2\left\langle \frac{\mathbf {q}_{\perp }(\tau )}{\Vert \mathbf {q}_{\perp }(\tau )\Vert }, \mathbf {s}(\tau ) \right\rangle \mathbb {1}_{\{r(\tau )>0\}} \Big ) \bigg ] \\&\quad \le T \Big ( \frac{\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n + n(a_{\max }+1)T}{2(\Vert \mathbf {q}(t_k)\Vert - nT)} + \epsilon \Vert \mathbf {\nu }\Vert - \nu _{\min } \Big ) + \sqrt{n}\Delta _r \\&\quad \le T \Big ( \frac{\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n + n(a_{\max }+1)T}{2(\Vert \mathbf {q}(t_k)\Vert - nT)} - \frac{\nu _{\min }}{4} \Big ), \end{aligned}$$

where the last inequality follows from $\epsilon \le \frac{\nu _{\min }}{4\Vert \varvec{\nu }\Vert }$ and $T > \frac{\Delta _r}{\epsilon } \ge \frac{4\Vert \varvec{\nu }\Vert \Delta _r}{\nu _{\min }} \ge \frac{4\sqrt{n}\Delta _r}{\nu _{\min }}$.

Let $\kappa = \frac{2(\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + n)}{\nu _{\min }} + nT(\frac{2(a_{\max }+1)}{\nu _{\min }} + 1)$ . We have that $\Vert \mathbf {q}(t_k)\Vert > \kappa$ implies

$$\begin{aligned} {\mathbb {E}}\Big [ \Vert \mathbf {q}_{\perp }(t_{k+1})\Vert - \Vert \mathbf {q}(t_k)\Vert \Big ] \le -\frac{\nu _{\min }}{4}T, \end{aligned}$$

then using [15, Lemma 3] with $D = na_{\max }T$ and $\eta = \frac{\nu _{\min }T}{4}$, we have

$$\begin{aligned} {\mathbb {E}}\Big [ \Vert \mathbf {q}_{\perp }(t_k)\Vert ^2 \Big ]&\le 4 \kappa ^2 + 32D^2 \Big ( 1+\frac{D}{\eta } \Big )^2 \le \Big ( 2\kappa + 4\sqrt{2}D(1+\frac{D}{\eta }) \Big )^2 \\&\le T^2 \Big ( \frac{ 4na_{\max } + 4(\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + 2n) + 16\sqrt{2} n^2a_{\max }^2 }{\nu _{\min }} + 2n + 4\sqrt{2} na_{\max } \Big )^2. \end{aligned}$$

Letting $M = \frac{ 4na_{\max } + 4(\Vert \varvec{\lambda }\Vert ^2 + \Vert \varvec{\sigma }\Vert ^2 + 2n) + 16\sqrt{2} n^2a_{\max }^2 }{\nu _{\min }} + 2n + 4\sqrt{2} na_{\max }$, we have ${\mathbb {E}}\Big [ \Vert \mathbf {q}_{\perp }(t_k)\Vert ^2 \Big ] \le T^2M^2$.

Consider the Lyapunov function $W(\mathbf {X}) = \sum _{i} \Big ( \sum _{j} q_{ij} \Big )^2 + \sum _{i} \Big ( \sum _{i} q_{ij} \Big )^2 - \frac{1}{n} \Big ( \sum _{ij} q_{ij} \Big )^2$, and set the corresponding Lyapunov drift at steady state to zero, ${\mathbb {E}}\Big [ W(\mathbf {X}(t_{k+1})) - W(\mathbf {X}(t_{k})) \Big ] = 0$. We then have $T_1 = T_2 + T_3 + T_4$, where

$$\begin{aligned} T_1&= 2 {\mathbb {E}}\Big [ \sum _{i} \Big ( \sum _{j} q_{ij}(t_k) \Big ) \Big ( \sum _{j} \sum _{\tau = t_k}^{t_{k+1}-1} (s_{ij}(\tau ) \mathbb {1}_{r(\tau )=0} - a_{ij}(\tau )) \Big ) \nonumber \\&\quad + \sum _{j} \Big ( \sum _{i} q_{ij}(t_k) \Big ) \Big ( \sum _{i} \sum _{\tau = t_k}^{t_{k+1}-1} (s_{ij}(\tau ) \mathbb {1}_{r(\tau )=0} - a_{ij}(\tau )) \Big ) \\&\quad - \frac{1}{n} \Big ( \sum _{ij} q_{ij}(t_k) \Big ) \Big ( \sum _{ij} \sum _{\tau = t_k}^{t_{k+1}-1} (s_{ij}(\tau ) \mathbb {1}_{r(\tau )=0} - a_{ij}(\tau )) \Big ) \Big ] \\&= 2 (\epsilon T - \Delta _r) {\mathbb {E}}\Big [ \sum _{ij} q_{ij}(t_k) \Big ], \\ T_2&= {\mathbb {E}}\Big [ \sum _{i} \Big ( \sum _{j} \sum _{\tau = t_k}^{t_{k+1}-1} ( a_{ij}(\tau ) - s_{ij}(\tau ) \mathbb {1}_{\{r(\tau )=0\}} ) \Big )^2 \nonumber \\&\quad + \sum _{j} \Big ( \sum _{i} \sum _{\tau = t_k}^{t_{k+1}-1} ( a_{ij}(\tau ) - s_{ij}(\tau ) \mathbb {1}_{\{r(\tau )=0\}} ) \Big )^2 \\&\quad - \frac{1}{n} \Big ( \sum _{ij} \sum _{\tau = t_k}^{t_{k+1}-1} ( a_{ij}(\tau ) - s_{ij}(\tau ) \mathbb {1}_{\{r(\tau )=0\}} ) \Big )^2 \Big ] \\&= \Big (2-\frac{1}{n} \Big )T \sum _{ij} \sigma _{ij}^2 + n (\epsilon T - \Delta _r)^2, \\ T_3&= {\mathbb {E}}\Big [ -\sum _{i} \Big ( \sum _{j} \sum _{\tau = t_k}^{t_{k+1}-1} u_{ij}(\tau ) \Big )^2 -\sum _{j} \Big ( \sum _{i} \sum _{\tau = t_k}^{t_{k+1}-1} u_{ij}(\tau ) \Big )^2 + \frac{1}{n} \Big ( \sum _{ij} \sum _{\tau = t_k}^{t_{k+1}-1} u_{ij}(\tau ) \Big )^2 \Big ] \\&\le nT(\epsilon T - \Delta _r), \\ T_4&= 2{\mathbb {E}}\Big [ \sum _{i} \Big ( \sum _{j} q_{ij}(t_{k+1}) \Big ) \Big (\sum _{j} \sum _{\tau = t_k}^{t_{k+1}-1} u_{ij}(\tau ) \Big ) + \sum _{j} \Big ( \sum _{i} q_{ij}(t_{k+1}) \Big ) \Big (\sum _{i} \sum _{\tau = t_k}^{t_{k+1}-1} u_{ij}(\tau ) \Big ) \\&\quad - \frac{1}{n} \Big ( \sum _{ij} q_{ij}(t_{k+1}) \Big ) \Big (\sum _{ij} \sum _{\tau = t_k}^{t_{k+1}-1} u_{ij}(\tau ) \Big ) \Big ] \\&\le 2na_{\max }T {\mathbb {E}}\Big [ \sum _{ij} \sum _{\tau = t_k}^{t_{k+1}-1} u_{ij}(\tau ) \Big ] + 4n \sum _{\tau = t_k}^{t_{k+1}-1} {\mathbb {E}}\Big [ \sum _{ij} u_{ij}(\tau ) \Big ] \sqrt{{\mathbb {E}}[ \Vert \mathbf {q}_{\perp }(t_k+\tau )\Vert ^2]} \\&\le 2n^2 (\epsilon T - \Delta _r) T (a_{\max }+ 2M). \end{aligned}$$

Hence, we have the upper bound

$$\begin{aligned} {\mathbb {E}}\Big [ \sum _{ij} \bar{q}_{ij} \Big ]&\le \Big (1 - \frac{1}{2n} \Big ) \frac{T}{\epsilon T - \Delta _r} \Vert \varvec{\sigma }\Vert ^2 + \frac{n((1+\epsilon ) T - \Delta _r)}{2} + n^2 T(a_{\max } + 2M) \\&\le \left( 1-\frac{1}{2n}\right) \frac{T}{\epsilon T - \Delta _r} \Vert \varvec{\tilde{\sigma }} \Vert ^2 + T \Big ( \frac{n (1+\epsilon )}{2} + n^2(a_{\max } + 2M) \Big ). \end{aligned}$$

$\square$