Distributed Oracle for Estimating Global Network Delay with Known Error Bounds

Abstract

Partially synchronous models are often assumed for designing distributed protocols because they capture realistic timing assumptions, such as the asynchronous and synchronous periods that the system can experience. In some of these models, protocols need to estimate network delays. Some protocols fix the global message delay bound for all executions, which leads to sub-optimal solutions in terms of latency, because this bound must be chosen conservatively. And other protocols employ delay estimation mechanisms that only give an upper bound on the delay without quantifying the estimation error. The performance of these protocols depends on how close their estimations are in relation to the actual network delay. For instance, some Byzantine consensus protocols use timeouts based on this estimation. We formalize this problem as the Global Delay Bound Estimation ($\textsf{GDBE}$) and address it by introducing a distributed oracle that enriches partial synchronous models. This oracle produces estimates of the channel delays that allow processes to derive an efficient global bounded estimate. Oracles and global bounded estimates, provide a framework that facilitates the design of protocols for partially synchronous models and the analysis of their time complexity. We formalize the properties of the oracle and the proposed framework and show that it can be implemented in the presence of crash failures. In contrast, we prove that $\textsf{GDBE}$ cannot be solved in the Byzantine failure model, and show how to circumvent this impossibility using an extra assumption. Finally, we show how to use our framework to implement a view synchronizer thus obtaining an efficient solution for Byzantine consensus.

A Implementing Consensus with \(\textsf{GDBE}\)

1.1 A.1 Synchronizer

1.2 B.2 Correctness Proof

We assume that each process has access to a complete and accurate with constant $\mathcal {C}$ $\varDelta$-estimator. Let $\varDelta _i^v$ be the output of the $\varDelta$-estimator at process $p_i$ when it is called to enter view v. Let $t^\varDelta \ge \textsf{GST}$ be the time after which for every correct process $p_i$, the $\varDelta$-estimator is complete and accurate property, i.e. $\forall t \ge t^{\varDelta }. \forall v' \le v. \varDelta \le \varDelta _i^v \le \mathcal {C}\cdot \varDelta$ for some fixed constant $\mathcal {C} \ge 0$.

Let $\varDelta _\textrm{max}^v = max\{\varDelta _i^v \} \cdot 2$ and $\varDelta _\textrm{min}^v = min\{\varDelta _i^v \} \cdot 2$ be the maximum and the minimum output of the $\varDelta$-estimator at the time it is called for entering view v.

We rewrite the lemmas related to the function F(v) in [2]. As in [2], the local view of a process $p_i$ at time t, denoted $\textsf{LV}_i(t)$, is the latest view entered by $p_i$ at or before t, or 0 if $p_i$ has not entered any views by then. Thus, $\textsf{GV}(t) = max\{\textsf{LV}_i(t) |p_i$ is correct$\}$. We say that a process $p_i$ attempts to advance from a view $v \ge 0$ at time t if at this time $p_i$ executes the code in either line 3 or line 5, and $\textsf{LV}_i(t) = v$. The global view at time t, denoted $\textsf{GV}(t)$, be the maximum view entered by a correct process at or before t, or 0 if no view was entered by a correct process.

Definition 6

Synchronizer properties are the following:

• 1. $\forall i ,v,v'.(E_i(v) \text { and } E_i(v') \text { are defined }) \wedge v<v' \Rightarrow E_i(v) < E_i(v')$

• 2. There is a view $\mathcal {V}$ where synchronization starts and $E_\textrm{first}(\mathcal {V}) \ge \textsf{GST}$

• 3. $\forall i.\forall v \ge \mathcal {V}. p_i \text { is correct } \Rightarrow p_i \text { enters } v$

• 4. $\forall v \ge \mathcal {V}.E_\textrm{last}(v) \le E_\textrm{first}+ 2\varDelta$

• 5. $\forall v \ge \mathcal {V}.E_\textrm{first}(v+1) \ge E_\textrm{first}(v)+\varDelta _\textrm{max}^v$

Definition 7

Synchronizer latency bounds are the following:

• A. $\forall v \ge \mathcal {V}\ . \ E_\textrm{last}(v+1) \le E_\textrm{first}(v) +\varDelta _\textrm{max}^v+ \varDelta$

• B. $S_\textrm{first}\ge \textsf{GST}\wedge t^{\varDelta } \Rightarrow \mathcal {V}= 1 \wedge E_\textrm{last}(1) \le S_\textrm{last}+ \varDelta$

• C. $\varDelta$-estimator is complete an accurate $\wedge \ S_{f+1} \le \textsf{GST}+ \rho \Rightarrow \mathcal {V}= \textsf{GV}(\textsf{GST}+\rho )+1 \wedge E_\textrm{last}(\mathcal {V}) \le \textsf{GST}+ \rho + \varDelta _\textrm{max}^{\mathcal {V}-1} \le \textsf{GST}+ \rho + \mathcal {C} \cdot \varDelta$

Lemma 22

If a correct process enters a view $v > 0$ and $E_\textrm{first}(v) \ge \textsf{GST}$, then for all $v' > v$, no correct process attempts to advance from $v'-1$ before $E_\textrm{first}(v) + \varDelta _\textrm{min}^v$.

Proof

Proof by contradiction. Assume that there $\exists$ a time $t' < E_\textrm{first}+ \varDelta _\textrm{min}^v$ and a correct process $p_i$ that attempts to advance from $v'-1 > v-1$ at $t'$. Let us consider the time in which process $p_i$ executes the code in line 5 (since the case in which line 3 is executed is not possible) and $\textsf{LV}_i(t') = 0 = v'-1 > v-1 \ge 0$. We have that $E_\textrm{first}(v'-1) \le E_i(v'-1)$.

Since $p_i.\mathsf {timer\_view}$ is not enabled at $t'$, $p_i$ must have entered $v'-1$ at least $\varDelta _\textrm{min}^v$ before $t'$ according to its local clock, then $E_i(v'-1) \le t'-\varDelta _\textrm{min}^v$. Since $v'-1 \ge v$, by Corollary 20 in [2], it is true that $E_\textrm{first}(v'-1) \ge E_\textrm{first}(v) \ge \textsf{GST}$. Therefore, given that after $\textsf{GST}$ all processes clocks run a the same rate as real time, we have

$$E_\textrm{first}(v) \le E_\textrm{first}(v'-1) \le E_i(v'-1) \le t'-\varDelta _\textrm{min}^v $$

Hence, $t' \ge E_\textrm{first}(v'-1) + \varDelta _\textrm{min}^v$, which contradicts our assumption. Therefore no correct process attempts to advance from $v'-1$ before $E_\textrm{first}(v) + \varDelta _\textrm{min}^v$.    $\Box$

Corollary 23

Assume a correct process enters a view $v>0$ and $E_\textrm{first}(v) \ge \textsf{GST}$. For all views $v'> v$ if there exists a correct process that enters $v'$, then $E_\textrm{first}(v') > E_\textrm{first}(v) + \varDelta ^v_{\textsf{min}}$.

Proof

Since a correct process enters a view $v'>0$, by Lemma 16 in [2], there exists a time $t < E_\textrm{first}(v')$ at which some correct process attempts to advance from $v'-1$. By Lemma 22, we get $t \ge E_\textrm{first}(v) + \varDelta _\textrm{min}^v$ as required.    $\Box$

Corollary 24

Consider a view v and assume that v is entered by a correct process. If $E_\textrm{first}(v) \ge \textsf{GST}$, then a correct process cannot send a $\texttt{WISH}(v')$ with $v'>v$ earlier than $E_\textrm{first}(v) + \varDelta _\textrm{min}^v$.

Proof

Assume a correct process sends a $\texttt{WISH}(v')$ with $v' > v$ at time $t'$. By Lemma 15 in [2], there $\exists s \le t'$ such that some correct process $p_i$ attempts to advance from $v'-1 > v-1$ at s. By Lemma 22, $s \ge E_\textrm{first}(v) + \varDelta _\textrm{min}^v$, which implies that $t'\le s \le E_\textrm{first}(v) + \varDelta _\textrm{min}^v$.    $\Box$

Lemma 28

For all v if some correct process enters v and

• (i) $E_\textrm{first}(v) \ge \textsf{GST}$,

• (ii) $\textsf{postGST}(E_\textrm{first}(v))$ holds, and

• (iii) $\varDelta$-estimator is complete

then all correct process enter v and $E_\textrm{last}(v) \le E_\textrm{first}(v) + 2 \varDelta$

Proof

Since $E_\textrm{first}(v) \ge \textsf{GST}$ and $\varDelta _\textrm{min}^v > 2 \cdot \varDelta$ for every $v' > v$, by Corollary 24, we have

• (iv) no correct process sends $\texttt{WISH}(v')$ with $v'>v$ until after $E_\textrm{first}(v)+2\varDelta$

The rest of the proof is as in [2].    $\Box$

Corollary 29

For all views v, if a correct process enters v, $E_\textrm{first}(v) > \textsf{GST}+ \rho$ and the $\varDelta$-estimator is complete, then all correct processes enter v and $E_\textrm{last}(v) \le E_\textrm{first}(v) + 2\varDelta$.

Lemma 34

Assume a correct process enters a view v, $E_\textrm{first}(v) \ge \textsf{GST}$, the $\varDelta$-estimator is complete, and $\textsf{postGST}(E_\textrm{first}(v))$ holds. Then all correct processes enter the view $v+1$ and $E_\textrm{last}(v+1) \le E_\textrm{last}(v) + \varDelta _\textrm{max}^v + \varDelta$.

Proof

Let $T = E_\textrm{last}(v) + \varDelta _\textrm{max}^v$. Assume that some correct process enters view $v+1$ before T, then by Lemma 28, all correct processes enter view $v+1$ and

$$ E_\textrm{last}(v+1) \le E_\textrm{first}(v+1)+ 2\varDelta \le T+2\varDelta = E_\textrm{last}(v)+\varDelta _\textrm{max}^v + \varDelta $$

as required.

Now assume that no correct process enters $v+1$ before T. We have $T > E_\textrm{first}(v) \ge \textsf{GST}$. By Lemmas 33 and 18 in [2], eventually some correct process enters $v+1$, so by Corollary 24, $T \ge \textsf{GST}$ implies that no correct process can send $\texttt{WISH}(v')$ for any $v' > v+1$ earlier than $T+\varDelta _\textrm{min}^{v+1}$. Then $\texttt{WISH}(v') >$ $T + \varDelta _\textrm{min}^{v+1}$. Thus, given that $\texttt{WISH}(v') >$ T and $\varDelta _\textrm{min}^{v+1} > 2\varDelta$ we get

$$\begin{aligned} \text {no correct process sends }\texttt{WISH}(v') \text { with } v'>v+1 \text { before } T+2\varDelta \end{aligned}$$

(23)

By Lemma 28, all correct process enter v. Let $p_i$ be a correct process that enters v at $E_{i}(v) \le \textsf{GST}$, at this time $p_i$ starts $p_i.\mathsf {timer\_view}$ for the duration of $\varDelta _i^v$. Note that $\varDelta _\textrm{min}^v \le \varDelta _i^v \le \varDelta _\textrm{max}^v$. Since by this time all clocks run a the same speed as real time, $p_i.\mathsf {timer\_view}$ cannot last more than $E_i(v) + \varDelta _i^v \le E_i(v) + \varDelta _\textrm{max}^v \le E_\textrm{last}(v) + \varDelta _\textrm{max}^v$. Let $s_i$ be the time at which $p_i.\mathsf {timer\_view}$ either expires or is stopped prematurely by executing the code in line 17; then $E_i(v) < s_i \le E_\textrm{last}(v)+\varDelta _\textrm{max}^v$ and therefore

$$\begin{aligned} s_i \le E_\textrm{last}(v) + \varDelta _\textrm{max}^v = (E_\textrm{last}(v) + \varDelta _\textrm{max}^v - \varDelta ) + \varDelta = T + \varDelta \end{aligned}$$

(24)

From here the proof follows exactly as in Lemma 34 in [2].    $\Box$

Corollary 35

For all views v, if a correct process enters v, $E_\textrm{first}(v) > \textsf{GST}+ \rho$, and the $\varDelta$-estimator is complete, then all correct processes enter the view $v+1$ and $E_\textrm{last}(v+1) \le E_\textrm{last}(v)+\varDelta _\textrm{max}^v+\varDelta$.

Theorem 36

FastSync satisfies properties 1–5 in Definition 6 for $d = 2\varDelta$.

Proof

Let $t^{\varDelta }$ be the time after which the $\varDelta$-estimator is complete. Property 1 is trivially satisfied. Let $\mathcal {V}$ be the first view such that a correct process enters $\mathcal {V}$, $E_\textrm{first}(\mathcal {V}) \le t^\varDelta > \textsf{GST}+ \rho$, the view $\mathcal {V}$ satisfies Property 2. Such a view exists because of the existence of a time in which the $\varDelta$-estimator is correct and Lemma 33 in [2] (global view keeps increasing). Since $E_\textrm{first}(\mathcal {V}) \ge t^\varDelta > \textsf{GST}$, the view $\mathcal {V}$ satisfies Property 2. By Lemmas 18 and 33 in [2], a correct process enters every view $v\ge \mathcal {V}$. By Corollary 20 in [2]

$$\begin{aligned} E_\textrm{first}(v) \ge E_\textrm{first}(\mathcal {V}) > \textsf{GST} \end{aligned}$$

(26)

Since the $\varDelta$-estimator eventually satisfies completeness, $\varDelta ^v_{\textsf{min}} > 2\varDelta$ starting at some view v. Thus, by Corollary 29, all correct processes enter v and $E_\textrm{last}(v) \le E_\textrm{first}(v) + 2 \varDelta$ which validates Properties 3 and 4. To prove 5, fix a view $v \ge \mathcal {V}$. By (26), $E_\textrm{first}(v) > \textsf{GST}$, and therefore, by Corollary 23, we get $E_\textrm{first}(v+1) \ge E_\textrm{first}(v) + \varDelta ^v_{\textsf{max}}$ which implies Property 5.    $\Box$

Theorem 38

Let $\mathcal {V}= \textsf{GV}(\textsf{GST}+\rho )+1$ and $d = 2\varDelta$. Assume that $S_{f+1} \le \textsf{GST}+ \rho$ and $\varDelta$ estimator is complete. Then FastSync satisfies properties 1–5 in Definition 6 and latency properties A–C in Definition 7.

Proof

Property 1 is satisfied trivially. Let $W = \textsf{GST}+\rho$ and $\mathcal {V}= \textsf{GV}(W)+1$. By Lemmas 18 and 33 from [2], some correct process enters $\mathcal {V}$. By Lemma 19 in [2], $\textsf{GV}(E_\textrm{first}(\mathcal {V})) = \mathcal {V}$. Since $\textsf{GV}$ is non-decreasing and $\mathcal {V}> \textsf{GV}(W)$, we have $E_\textrm{first}(\mathcal {V}) > \textsf{GST}+ \rho \ge \textsf{GST}$. Hence Property 2 holds. By Lemmas 18 and 33 from [2], some correct process enters every view $v\ge \mathcal {V}$. By Corollary 20 in [2], $v\ge \mathcal {V}$ implies that

$$\begin{aligned} E_\textrm{first}(v) \ge E_\textrm{first}(\mathcal {V}) \ge W \end{aligned}$$

(28)

Since completeness is satisfied, $\forall v \ge \mathcal {V}\ . \ \varDelta _\textrm{min}^v \ge 2\varDelta$. Thus, by Corollary 29, all correct processes enter v and $E_\textrm{last}(v) \le E_\textrm{first}(v) + 2\varDelta$ which validates Properties 3 and 4.

To prove Properties 5 and A, fix a view $v \ge \mathcal {V}$. By (28), $E_\textrm{first}(v) \ge \textsf{GST}$, and therefore, by Corollary 23 we get $E_\textrm{first}(v+1) \ge E_\textrm{first}(v) + \varDelta _\textrm{min}^v$, which implies Property 5. Since (28), $E_\textrm{first}(v) \ge W$, by Lemma 26 in [2], $\textsf{postGST}(E_\textrm{first}(v))$ holds. We also have $\varDelta _\textrm{min}^v$, $\varDelta _\textrm{min}^\mathcal {V}\ge 2\varDelta$. Thus, by Corollary 35, $E_\textrm{last}(v+1) \le E_\textrm{last}(v) + \varDelta _\textrm{max}^v + \varDelta$, and therefore, Property A holds. To prove Property C, we consider two cases:

Case 1: $\textsf{GV}(W) = 0$. Hence $\mathcal {V}= 1$. Let

$$t_1 = max(S_{f+1},W)$$

$$t_2 = max(S_\textrm{last}, W)$$

$$T = max(min(S_{f+1}, S_\textrm{last}-\varDelta ),W)$$

Since $S_{f+1} \le W$ and $min(W,S_\textrm{last}-\varDelta ) \le W$, the above can be re-written as follows:

$$t_1 = max(S_{f+1},W) = W$$

$$t_2 = max(S_\textrm{last}, W)$$

$$T = W$$

Then $\textsf{GV}(T) = 0$. Since $\textsf{GV}$ is non decreasing, $E_\textrm{first}(1) \ge \textsf{GST}$. Thus by Corollary 24, no correct process can send $\texttt{WISH}(v)$ for any $v > 1$ earlier than $T+\varDelta _\textrm{min}^1 > T+2\varDelta$. Since by Lemma 26 in [2] $\textsf{postGST}(W)$ holds, by Lemma 31 in [2], $E_\textrm{last}(\mathcal {V}) \le min(t_1+2\varDelta , t_2+\varDelta )\le t_1+2\varDelta = \textsf{GST}+ \rho + 2\varDelta$. Since $\varDelta _\textrm{max}^0 > 0$ we have

$$E_\textrm{last}(\mathcal {V}) \le \textsf{GST}+ \rho + \varDelta _\textrm{max}^{\textsf{GV}(W)} + 2\varDelta $$

which implies the upper bound stated on Property C.

Case 2: $\textsf{GV}(W) > 0$. Let $T = W+\varDelta _\textrm{max}^{\textsf{GV}(W)} +\varDelta$. Suppose first that some correct process enters $\textsf{GV}(W)+1$ before T. By Lemma 19 in [2], $\textsf{GV}(E_\textrm{first}(\textsf{GV}(W)+1)) = \textsf{GV}(W)+1$. Since $\textsf{GV}$ is non-decreasing, we have $E_\textrm{first}(\textsf{GV}(W)+1) > W$. Thus by Corollary 29, all correct processes enter $\mathcal {V}$ by $\textsf{GST}+ \rho + \varDelta _\textrm{max}^{\textsf{GV}(W)} + 3\varDelta$ as needed. Suppose now that no correct process enters $\mathcal {V}$ before T, so that $E_\textrm{first}(\mathcal {V}) \ge T \ge \textsf{GST}$. Then, by Corollary 24,

$$\begin{aligned} \text {no correct process can send }\texttt{WISH}(v) \text { for any } v> \mathcal {V}\text { earlier that } T+ \varDelta _\textrm{min}^\mathcal {V}> T +2 \varDelta \end{aligned}$$

(29)

From Lemma 26 in [2], $\textsf{postGST}(W)$ and therefore by Lemma 32 in [2], all correct processes send $\texttt{WISH}(\mathcal {V})$ to all processes no later than $T+\varDelta$. Since (29) holds, by Lemma 30 in [2], all correct processes enter $\mathcal {V}$, and $E_\textrm{last}(\mathcal {V}) \le T + 2\varDelta = \textsf{GST}+\rho +\varDelta _\textrm{max}^{\textsf{GV}(W)} + 3\varDelta$ as needed. Since the $\varDelta$-estimator is accurate with constant $\mathcal {C}$

$$E_\textrm{last}(\mathcal {V}) \le T + 2\varDelta = \textsf{GST}+\rho +(\mathcal {C}+3)\varDelta $$

is true too.    $\Box$