Elsevier

Computer Networks

Volume 52, Issue 6, 24 April 2008, Pages 1193-1219
Computer Networks

Jetmax: Scalable max–min congestion control for high-speed heterogeneous networks

https://doi.org/10.1016/j.comnet.2007.11.021Get rights and content

Abstract

Recent surge of interest towards congestion control that relies on single-link feedback (e.g., XCP, RCP, MaxNet, EMKC, VCP), suggests that such systems may offer certain benefits over traditional models of additive packet loss. Besides topology-independent stability and faster convergence to efficiency/fairness, it was recently shown that any stable single-link system with a symmetric Jacobian tolerates arbitrary fixed, as well as time-varying, feedback delays. Although delay-independence is an appealing characteristic, the EMKC system developed in exhibits undesirable equilibrium properties and slow convergence behavior. To overcome these drawbacks, we propose a new method called JetMax and show that it admits a low-overhead implementation inside routers (three additions per packet), overshoot-free transient and steady state, tunable link utilization, and delay-insensitive flow dynamics. The proposed framework also provides capacity-independent convergence time, where fairness and utilization are reached in the same number of RTT steps for a link of any bandwidth. Given a 1 mb/s, 10 gb/s, or googol (10100) bps link, the method converges to within 1% of the stationary state in six RTTs. We finish the paper by comparing JetMax’s performance to that of existing methods in ns2 simulations and discussing its Linux implementation.

Introduction

In the light of TCP’s scalability issues in high-speed networks [9], explicit-feedback congestion control has gained renewed interest in the last several years [16], [22], [32], [33]. Sometimes referred to as Active Queue Management (AQM) congestion control, these algorithms rely on routers to provide congestion feedback in the form of changes to the congestion window [16], packet loss [33], single-bit congestion indication [12], [18], [26], queuing delay [14], [29], or link prices [17], [20], [23]. This information helps end-flows converge their sending rates to some social optimum and achieve a certain optimization objective.

Unlike some of the largely ineffective AQM aimed at improving the performance of TCP [4], properly designed explicit congestion control promises to provide scalability to arbitrary bandwidth (i.e., terabits and petabits per second1), tunable link utilization, low delay, zero loss, oscillation-free steady state, and exponential convergence to fairness/efficiency, all of which suggests that such algorithms, once deployed in the Internet, may remain in service for many years to come. Note that the purpose of this paper is not to settle the debate of whether or when explicit congestion control will be adopted by the Internet, but to explore the various properties of existing AQM methods, propose a new controller we call JetMax, and compare its ns2 and Linux performance with that of the existing methods.

The first half of the paper deals with understanding delayed stability and convergence performance of several recently proposed AQM approaches: eXplicit Control Protocol (XCP) [16], Rate Control Protocol (RCP) [7], Exponential Max–min Kelly Control (EMKC) [33], and a hybrid method suggested in [33] that combines EMKC with Adaptive Virtual Queue (AVQ) [20], [19]. We find from this study that both XCP and RCP are sensitive to RTT estimation and is prone to instability even in single-link topologies where the average RTT dl estimated by the router is significantly different from the maximum RTT Dmax of end-flows. Although this issue can be overcome by utilizing Dmax instead of dl, additional problems may occur under time-varying delays. Moreover, XCP may become unstable in certain multi-link networks when the flows receive feedback on different time scales (i.e., under heterogeneous delay). The root of this problem lies in the oscillatory switching between the bottlenecks (i.e., changes in the bottleneck link) and inability of each XCP flow to permanently decide its most-congested resource in the presence of delayed feedback. This phenomenon in turn arises from the discontinuous nature and non-monotonic transient properties of the feedback function used in the control equation of XCP. Discontinuity of feedback follows from XCP’s algorithm for selecting the most-congested link along its path, while non-monotonicity is caused by the oscillatory nature of the controller when the feedback delays of competing flows are heterogeneous.

To further understand the reasons for XCP’s instability in multi-link networks, we analyze the problem of bottleneck oscillation in more depth and show that only consistent (i.e., agreed upon by every flow) bottleneck assignment allows one to reduce stability analysis of max–min protocols in multi-link networks to that of the single-link case studied in prior work [7], [16], [29], [33]. In all other cases, max–min methods require a much more complicated analysis not available within the current framework of congestion control. We additionally observe that feedback that remains monotonic when a flow changes its most-congested resource allows the protocol to achieve a consistent bottleneck assignment and thus remain stable. This partially explains EMKC’s stability in multi-link networks observed in simulations.

Although EMKC remains stable in multi-link topologies, we find that its transient and equilibrium properties (such as linear convergence to fairness and steady-state packet loss) are potential drawbacks for its use in practice. The problem of EMKC’s equilibrium packet loss can be overcome using EMKC-AVQ; however, the resulting method exhibits undesirable oscillations and transient overshoot of link’s capacity. Combined with a large number of flows, transient overshoot leads to long-lasting packet loss and non-negligible increase in queuing delay, both of which are highly undesirable.

Our conclusion from the first half of the paper is that any new designs of max–min AQM congestion control should decouple feedback delay from control equations and converge to stationarity monotonically. Thus, the second part of the paper designs a new method we call JetMax that satisfies these criteria while offering additional features:

  • Capacity-independent convergence time. The algorithm reaches fairness and efficiency in the same number of RTT steps regardless of link’s capacity.

  • Zero packet loss. Loss-free operation is ensured both in the transient and stationary state.

  • Tunable link utilization. Each router can be independently configured to control its steady-state link utilization.

  • RTT-independent max–min fairness. Resource allocation is max–min fair regardless of end-user delays.

  • Global multi-link stability under consistent bottleneck assignment for all types of delay. Flows converge to the equilibrium and maintain their steady-state rates in generic networks regardless of any fluctuation in the RTT as long as end-users can correctly choose their bottleneck links (see below for more).

  • Low overhead. The AQM algorithm requires only three additions per arriving packet and no per-flow state information inside routers.

We finish the paper by repeating the same ns2 simulations that earlier highlighted the limitations of existing methods and demonstrate that JetMax outperforms its predecessors using a number of metrics such as multi-link stability, convergence rate, transient overshoot, and steady-state rate allocation. We also show that JetMax can be easily integrated into the Linux router kernel and present the results of Linux experiments with JetMax running over 1 gb/s links, both in single- and multi-bottleneck topologies.

The rest of the paper is organized as follows. We review the existing explicit congestion control algorithms in Section 2 and identify their problems in Section 3. We then highlight the importance of studying multi-link stability of max–min systems in Section 4. Following that, we introduce JetMax in Section 5 and discuss its implementation issues in Section 6. We then demonstrate JetMax’s performance through ns2 simulations and Linux experiments in Sections 7 Simulations, 8 Linux performance, respectively. We conclude the paper in Section 9.

Section snippets

Background

We start by describing the notation used throughout the paper. Assume N users in the network whose rates at time t are given by {xr(t)}r=1N. Following notation in [15], we denote the RTT of each flow by Dr(t) and the forward/backward delays of user r to/from link l by Dr,l(t) and Dr,l(t), respectively. The aggregate arrival rate of all users at link l is written as yl(t)=rlxr(t), where rl is the set of flows r passing through link l. Similarly, notation lr refers to the set of links l

Understanding existing methods

This section discusses the desired properties of future congestion control and examines whether the existing methods satisfy these requirements. We focus on such issues as flow dynamics under heterogeneous (both time-invariant and time-varying) feedback delay, stability in multi-link scenarios, convergence behavior, and overshoot properties in transient and equilibrium states.

Max–min bottleneck assignment

This section highlights the importance of analyzing discontinuous stability of max–min congestion control and explains some of the phenomena observed in the previous section.

JetMax

In this section, we present JetMax and provide an analytical study of its properties. The next section discusses implementation and performance details of this protocol.

Estimating number of flows

The first issue encountered by a JetMax router l is how to estimate the current number of responsive flows Nl(n). Dynamic tracking of active flow population Nl(n) has been actively studied in ATM networks [1], [11], [28]. Specifically, as suggested in [1], [11], Nl(n) can be simply approximated by the ratio between wl(n) and gl(n). However, similar to RCP discussed in Section 3.8, this method may result in significant transient overshoot of the bottleneck link when new flows join the system.

Behavior in T2, T3, T4, and T5

We first repeat the ns2 simulations that earlier presented stability and equilibrium problems to existing methods and then examine how JetMax handles additional scenarios. Simulation code used in this paper is available in [13].

Performance of JetMax in T2,T3,T4, and T5 is shown in Fig. 13, in which the protocol demonstrates monotonic convergence, max–min allocation of bottleneck resource in the equilibrium, effective handling of bottleneck selection, and loss-free operation in both the

Linux performance

We finish the paper by examining performance and implementation overhead of JetMax in Linux software routers. The main goal of this study is to advance beyond 10 mb/s cases studied in the literature [32] and achieve true gigabit speeds where AQM algorithms would have the most impact in practice. For the experiments reported in this paper, we use two Linux routers shown in Fig. 20a, where R1 is a single Pentium 4 running at 3.4 GHz and R2 is a dual-Xeon box running at 3 GHz. Propagation delays of

Conclusion

This paper examined several max–min AQM congestion controllers and found that all of them exhibited undesirable properties under certain criteria. A bigger problem, however, discovered in this work was the susceptibility of XCP and potentially other max–min systems with non-monotonic feedback to oscillation between bottlenecks and unstable behavior in multi-router topologies. We proposed a new method JetMax that was able to overcome the identified issues with existing methods and admitted

Yueping Zhang received the B.S. degree in computer science from Beijing University of Aeronautics and Astronautics, Beijing, China, in 2001. He is currently working toward the Ph.D. degree in computer engineering at Texas A&M University, College Station, USA. His research interests include congestion control, delayed stability analysis, active queue management (AQM), router buffer sizing, and peer-to-peer networks.

References (33)

  • J. Aweya et al.

    Scalable and provably stable explicit rate computation scheme for flow control in communication networks

    Int. J. Commun. Syst.

    (2001)
  • D. Bertsekas et al.

    Data Networks

    (1992)
  • F. Blanchini et al.

    Robust rate control for integrated services packet networks

    IEEE/ACM Trans. Netw.

    (2002)
  • M. Christiansen, K. Jeffay, D. Ott, F.D. Smith, Tuning RED for Web traffic, in: Proceedings of the ACM SIGCOMM, August...
  • S. Deb, R. Srikant, Rate-based versus queue-based models of congestion control, in: Proceedings of the ACM SIGMETRICS,...
  • R. DeCarlo et al.

    Perspectives and results on the stability and stabilizability of hybrid systems

    Proc. IEEE

    (2000)
  • N. Dukkipati, M. Kobayashi, R. Zhang-Shen, N. McKeown, Processor sharing flows in the internet, in: Proceedings of the...
  • A. Falk, D. Katabi, Specification for the Explicit Control Protocol (XCP), USC/ISI, Technical Report, October...
  • S. Floyd, High-speed TCP for Large Congestion Windows, IETF RFC 3649, December...
  • S. Floyd et al.

    Internet research needs better models

    ACM SIGCOMM Comput. Commun. Rev.

    (2003)
  • C. Fulton, S.-Q. Li, C.S. Lim, An ABR feedback control scheme with tracking, in: Proceedings of the IEEE INFOCOM, April...
  • C.V. Hollot, V. Misra, D. Towsley, W.-B. Gong, On designing improved controllers for AQM routers supporting TCP flows,...
  • JetMax@TAMU. [Online]....
  • C. Jin, D. Wei, S.H. Low, FAST TCP: motivation, architecture, algorithms, performance, in: Proceedings of the IEEE...
  • R. Johari et al.

    End-to-end congestion control for the internet: delays and stability

    IEEE/ACM Trans. Netw.

    (2001)
  • D. Katabi, M. Handley, C. Rohrs, Congestion control for high bandwidth delay product networks, in: Proceedings of the...
  • Cited by (0)

    Yueping Zhang received the B.S. degree in computer science from Beijing University of Aeronautics and Astronautics, Beijing, China, in 2001. He is currently working toward the Ph.D. degree in computer engineering at Texas A&M University, College Station, USA. His research interests include congestion control, delayed stability analysis, active queue management (AQM), router buffer sizing, and peer-to-peer networks.

    Derek Leonard received the B.A. degree (with distinction) in computer science and mathematics from Hendrix College, Conway, Arkansas, in 2002. Since 2002, he has been a Ph.D. student in the Department of Computer Science at Texas A&M University, College Station, TX. His research interests include peer-to-peer networks, optimization-based graph construction, and large-scale measurement studies of the Internet.

    Dmitri Loguinov received the B.S. degree (with honors) in computer science from Moscow State University, Russia, in 1995 and the Ph.D. degree in computer science from the City University of New York, New York, in 2002.

    Between 2002 and 2007, he was an Assistant Professor in the Department of Computer Science at Texas A&M University, College Station. He is currently a tenured Associate Professor and Director of the Internet Research Lab (IRL) in the same department. His research interests include peer-to-peer networks, video streaming, congestion control, Internet measurement and modeling.

    An earlier version of this paper has appeared in IEEE INFOCOM 2006.

    View full text