Microscopic examination of TCP flows over transatlantic links

Abstract

Much of the recent research and development in the area of high-speed TCP is focused on the steady state behavior of TCP flows. However, our experience with the first research only transatlantic 2.5 Gbps Lambda link clearly demonstrates the need to focus on the initial stages of TCP. The work we present here examines the behavior of TCP flows at microscopic level over high-bandwidth long delay networks. This examination has led us to study the influence of the minute properties of the underlying network on bursty protocols such as TCP at these very high speeds combined with high latency. In this paper we briefly describe the requirements for such an extreme network environment to support high-speed TCP flows. We also present results collected using transatlantic links at iGrid2002 where we tuned various host parameters and used modified TCP stacks.

Introduction

Grid applications in general can be demanding in terms of bandwidth requirements.

A typical large scale scientific experiment involves at least two parties: a data producer and a data consumer. Many of the large High Energy Physics (HEP) experiments coming online in the near future such as LHC [16], D∅ [12], CDF [11] and BaBar [10] are all excellent examples of this model of computing. In these large experiments the data is mostly produced at a few locations (the data producers) and then many researchers analyze this data at their home institutes and universities (the data consumers). Often the researchers are geographically separated by large distances. The throughput requirements for these experiments are high.

For example, the European Data Grid (EDG) roughly estimates the peak bandwidth of the network traffic that will flow over it in the year 2005 to be 8000 Mbps from a single project (D∅), and that this link will have to be shared among several different HEP projects, all wishing to disseminate their data. Not only does this data need to be collected from the experiment but a large part of it needs to be transferred to various locations over the network. Network architectures are evolving to meet this unprecedented demand. Herein we present research on the scalability of protocols to allow these kinds of applications to reach the required high-bandwidths on new network architectures. One way to provide this bandwidth is by provisioning end-to-end paths, called Lambdas [5], up to several Gbps. In the fall of 2001 SURFnet [18] provisioned a 2.5 Gbps Lambda between Amsterdam and Chicago to be used only for research. Initial tests showed that only increasing the speed of links, switches and routers in the path was not sufficient to obtain a throughput at or near the available bandwidth. We conducted extensive experiments on this transatlantic link both to understand how transport protocols behave and what additional requirements high-speed flows, such as TCP, impose on optical networks. One architectural shift in our high-speed networking experiments was to minimize the number of routers (devices which process packets at layer 3 and above), and instead delegate packet forwarding to switching devices at layer 2 or below. Initial throughput measurements of a single TCP stream over such an extreme network infrastructure (i.e. the SURFnet Lambda) showed surprisingly poor results. This led us to further examination of the dynamics of TCP at the microscopic level to better understand its behavior. The primary motivation of this work is the demand from HEP community to obtain maximum throughput over long distance links using a single or utmost a few TCP streams. Currently in the HEP community there are several projects underway to try to over come these limitations. However, these projects focus primarily on increasing some of the default parameters of TCP (SSThreash [7], etc.). Simply increasing network capacity does not always improve end-to-end performance. The exclusive availability of the SURFnet Lambda for research has allowed us to investigate this problem.

The performance issues of TCP/IP for large data transfers over high-bandwidth long-latency path is a well known problem [6]. The problem is to discover the bottleneck of a TCP flow (the slowest link in a chain of networks) between two PCs connected using a long-latency high-bandwidth path. There are several issues related to this problem: network characteristics (router, switches, slow links), the implementation of the TCP stack and specific parameters passed to the TCP algorithm by the hosts. In Section 2 we examine the characteristics of the equipment and how this influences TCP and what requirements TCP imposes on the network. Section 3 briefly discusses some of the problems with the TCP algorithm on high-bandwidth long delay paths. In Section 4 we discuss the effect of host and operating system parameter tuning on performance. Section 5 shows test results from different modifications to the TCP/IP algorithms and particular those of HSTCP [4] implemented by the Net100 [17] project.

In the following sections we broadly classify the various stages of a TCP session into: bandwidth discovery phase (aka slow start), steady state [4], and congestion avoidance. In this work we focus mostly on the initial phase of a TCP flow, the bandwidth discovery, as we believe that this phase most influences the bandwidth obtained using TCP.