Flow-aware explicit congestion notification for datacenter networks

Abstract

Explicit congestion notification (ECN) has been widely adopted by recent proposals to build up high-throughput and low-latency datacenter network transport. In these ECN-based proposals, when the queue length of a switch exceeds a pre-defined threshold, the switch would mark all arriving packets with ECN to explicitly notify their senders to slow down the rates. Such a design enables the network to eliminate congestions quickly. However, it marks packets without considering the flow state, which may overkill flows, especially those only send a few packets, thus resulting in significant throughput loss and long flow completion times. In this paper, we propose a novel flow-aware ECN marking approach (FECN), which can improve the throughput and flow completion time by taking flow states into consideration. By selectively marking packets respecting to their flow rates, FECN enables the network to precisely slow down the high-speed flows to avoid congestions without killing low-speed short flows. Moreover, FECN does not require switches to maintain per-flow state, which yields low overhead and thus makes FECN to be easily implemented and deployed in commodity switches. Simulations show that FECN can shorten the flow completion time by up to 44.7% and reduce the throughput loss by up to 40.3%, compared with prior flow-agnostic ECN marking approach.

Appendix

Before introducing our fluid model, we first briefly review FECN at switch side and the DCTCP [8] algorithm at the source side. Then, like [37, 38], we model N greedy flows at a single bottleneck with capacity C. We assume that all flows have the same RTTs.

Switch side In FECN, the switch makes ECN marking decisions based on flows’ sending rate. The sending rate of each flow is calculated at the source side and is carried in the headers of sending packets. The switch read the sending rate carried in the packet headers and estimates the average sending rate of all flows traversing the same outport, which is symbolized as aveRate. For simplicity, we assume that aveRate can accurately estimate the average sending rate which is not true. However, this approximation does not change the fundamental nature of marking packets from flows with relatively high sending rate, and we are able to capture FECN dynamics. Then aveRate can be modeled as follows:

$$\begin{aligned} aveRate(t) = \frac{\sum _{i=1}^N r_i(t)}{N} \end{aligned}$$

(8)

where N is the number of flows traversing the same outport at the switch, $r_i(t)$ is the sending rate of the ith flow at time t. In our realization, we use congestion window w(t) as the calculated value of sending rate r(t) to reduce the overhead of source side. So the model of aveRate(t) can be replaced as follows:

$$\begin{aligned} aveRate(t) = \frac{\sum _{i=1}^N w_i(t)}{N} \end{aligned}$$

(9)

The marking probability of packets from ith flow can be modeled as follows:

$$p_{i} \left( t \right) = \left\{ {\begin{array}{ll} 0 & {q(t) < Kmin} \\ 0 & {Kmin \le q(t) < Kmax,w_{i} (t) < aveRate(t)} \\ {f(q(t))} & {Kmin \le q(t) < Kmax,w_{i} (t) \ge aveRate(t)} \\ 1 & {q(t) \ge Kmax} \\ \end{array} } \right.$$

(10)

$$\begin{aligned} f(q(t)) = Pmin+\frac{(q(t)-Kmin)(Pmax-Pmin)}{Kmax-Kmin} \end{aligned}$$

(11)

where q(t) is the instantaneous queue length at time t.

Source side The DCTCP source maintains an estimation of the fraction of its ECN marked packets. This estimate, $\alpha$, is updated once for per RTT (or per window of data) as follows:

$$\begin{aligned} \alpha = (1-g)\alpha +gF \end{aligned}$$

(12)

where F is the fraction of packets that were marked in the most recent RTT, and g is a fixed parameter. DCTCP uses $\alpha$ to reduce its window size in response to marked ACKs as follows:

$$\begin{aligned} w = (1-\frac{\alpha }{2})w \end{aligned}$$

(13)

Note that, DCTCP reduce its window size at most per RTT. For each ACK, DCTCP increases its window size like TCP as follows:

$$\begin{aligned} w = w + \frac{1}{w} \end{aligned}$$

(14)

State equations We choose w and $\alpha$ at each source and q(t) at the switch as the state variables of the system. The evolution of state variables can discribe the dynamics of the system. The evoluation of q(t) can be modeled as follows:

$$\begin{aligned} \frac{dq(t)}{dt} = \frac{\sum _{i=1}^N w_i(t)}{R(t)} - C \end{aligned}$$

(15)

where R(t) is the RTT at time t, $w_i(t)$ is the window size of ith flow, and $\frac{\sum _{i=1}^N w_i(t)}{R(t)}$ is the average total input rate. R(t) can be modeled as follows:

$$\begin{aligned} R(t) = d+\frac{q(t)}{C} \end{aligned}$$

(16)

where d is the propagation delay (assumed to be equal for all flows), and q(t) / C is the queueing delay at the switch. The evolution of $\alpha _i(t)$ at source i can be modeled as follows:

$$\begin{aligned} \begin{aligned} \frac{d\alpha _i(t)}{dt}&= \frac{\alpha _i(t)-\alpha _i(t-R(t))}{R(t)}\\&= \frac{(1-g)\alpha _i(t-R(t))+gF_i(t)-\alpha _i(t-R(t))}{R(t)}\\&= \frac{g}{R(t)}\bigg (F_i(t)-\alpha _i(t-R(t))\bigg ) \end{aligned} \end{aligned}.$$

(17)

The expected value of ECN marked packets is $p_i(t-R(t))w_i(t-R(t))$ in the last RTT, and the fraction of marked packets can be estimated as $F_i(t)=p_i(t-R(t))w_i(t-R(t))/w_i(t-R(t))=p_i(t-R(t)).$ Then we can obtain

$$\begin{aligned} \frac{d\alpha _i(t)}{dt} = \frac{g}{R(t)}\bigg (p_i(t-R(t))-\alpha _i(t-R(t))\bigg ) \end{aligned}$$

(18)

The evolution of $w_i(t)$ can be modeled as follows:

$$\begin{aligned} \frac{dw_i(t)}{dt}= & {} \frac{1}{R(t)}p_{noECN}+\bigg (\frac{1}{R(t)}-\frac{w_i(t) \alpha (t)}{2R(t)}\bigg )p_{hasECN}\end{aligned}$$

(19)

$$\begin{aligned} p_{noECN}= & {} (1-p_i(t-R(t)))^{w_i(t-R(t))} \end{aligned}$$

(20)

$$\begin{aligned} p_{hasECN}= & {} 1-(1-p_i(t-R(t)))^{w_i(t-R(t))} \end{aligned}$$

(21)

where $p_{noECN}$ is the probability of no ECN marked packets in the last RTT and $p_{hasECN}$ is the probability of receiving ECN marked packets in the last RTT. Combining (19), (20) and (21), we can obtain

$$\begin{aligned} \frac{dw_i(t)}{dt} = \frac{1}{R(t)} - \frac{\alpha _i(t)w_i(t)}{2R(t)}\bigg (1-(1-p_i(t-R(t)))^{w_i(t-R(t))}\bigg ) \end{aligned}$$

(22)

This equation models the additive-increase and multiplicative-decrease behavior of DCTCP.