Elsevier

Computer Communications

Volume 30, Issue 9, 30 June 2007, Pages 1980-1993
Computer Communications

Reorder buffer-occupancy density and its application for measurement and evaluation of packet reordering

https://doi.org/10.1016/j.comcom.2007.03.001Get rights and content

Abstract

Increasing internal parallelism within nodes due to increase in links speeds, and external parallelism among the links due to QoS, ad hoc routing in wireless, etc., point to a significant increase in packet reordering. Application performance also degrades considerably due to out-of-order arrivals. The concept of “Reorder Buffer-occupancy Density” (RBD) is defined, which provides the statistics of occupancy of a buffer used to recover from reordering. RBD also captures and measures reordering effectively, helps relate causes of reordering to observations, and allows the development of models for packet reordering. A formal representation and analysis of reordering is presented along with the derivation of RBD models for basic reordering patterns, such as independent, overlapping and embedded reordering. Measurements associate most of the packet reordering occurrences in the Internet to such patterns. The RBD models for these patterns are used to build a buffer-occupancy model with a scenario associated with traffic splitting or bandwidth aggregation as an example. Such models can aid in determining the resource requirements for mitigating effects due to reordering, as well as performance characteristics of underlying network. Developed models are verified using measurements on emulated networks.

Introduction

Out-of-order arrival of packets is an increasingly common phenomenon on the Internet. The causes for out-of sequence delivery of packets include, but are not limited to the following:

  • (1)

    Packet striping at layer 2 and 3. When an earlier packet is placed in a longer queue and later packet in a shorter queue, the packets may arrive out of order [1], [2].

  • (2)

    Retransmissions. When a packet is lost, due to retransmission the packet may arrive out-of-sequence. Examples include layer 2 retransmissions on wireless links [3] and retransmissions in TCP.

  • (3)

    Load sharing. When packets from a same flow are processed by multiple network processors in the routers due to load sharing and load balancing, it can introduce a significant amount of reordering [4].

  • (4)

    Priority scheduling in routers. For example, per DiffServ, a priority flow may obey the negotiated bandwidth constraints. If the flow exceeds these constraints, the nonconforming packets are either dropped or given a lower priority. In the latter case, the packets will be placed in different queues resulting in out-of-order delivery [2], [5].

  • (5)

    Route fluttering. Fluttering refers to rapidly oscillating routing, which for example may be due to dynamic load splitting among the links. In such cases, different packets of the same stream take different routes leading to different delays [6].

Measurements have confirmed that packet reordering is a common occurrence on the Internet [1], [2], [7], [8]. The increase of network link speeds and the resulting increase in parallelism required to handle the packets within routers point to a significant increase in packet reordering in future high-speed networks. As the packet time to end-to-end delay ratios decrease, the amount of reordering experienced by flows will also increase [9], [10]. Even in low-bandwidth networks, instances of high packet reordering have been reported that caused problems such as wasting a noticeable fraction of ISP’s bandwidth [7].

Reordering can have a significant impact on end-to-end performance of applications, and lead to inefficient use of resources, both within the network and at the end stations. In the case of TCP, the impact of reordering depends on whether data packets are reordered (forward path reordering) or acknowledgements are reordered (reverse path reordering). Out of order delivery of packets in the forward path may be perceived as loss of packets at the receiver, resulting in reduced congestion window and increased retransmissions [1], [11], [12] that further degrade the performance. Reverse path reordering results in a loss of TCP’s self-clocking property, leading to transmission of bursts of packets and possible congestion. In delay sensitive applications based on UDP, for example VoIP, an out-of-order packet that arrives after the elapse of playback time is as good as lost, thereby, decreasing the perceived quality of voice. Reordering can also increase buffer requirements at the receiver as well as processing related latencies. Even a small percentage of reordered packets in a backbone link have been observed to cause a significant degradation of the throughput [13]. Manifestation of reordering as a spurious congestion indication for example can have a detrimental impact on performance of many protocols such as DCCP (datagram congestion protocol), SCTP (stream control transmission protocol), and even on security related features. Packet reordering may also impact the performance of the anti-replay sliding window in IPSec where, for example, an early arrival of packet with large sequence number may cause a lot of useful but late sequence numbers to be dropped [14].

Mitigating impact of out-of-order delivery using end-node solutions and reducing out-of-order packet delivery using network-centric solutions have thus received increased attention. Approaches for mitigating the impact of out-of-order packet delivery on TCP performance include adjusting ‘dupthresh’ parameter, i.e., the number of duplicate ACKs to be allowed before classifying a following unacknowledged packet as lost [11], [15]. However, certain intermediate nodes using network processors take a more aggressive approach, which attempts to eliminate reordering due to parallelism within the routers [16]. Input buffering and output buffering are two such methods [17]. In input buffering, the individual streams are identified and the packets of a given stream are placed in same queue, thereby, preventing reordering. As such a solution requires monitoring traffic characteristics such as per-flow state at the router, and solving an NP-complete mapping problem in run-time, a fully optimal mapping is not achievable [4]. NP completeness of the problem also implies that the solution becomes increasingly difficult as the number of parallel flows in a link increases, which is expected as link speeds increase. Alternatively, in output buffering, the packets are stored at an output queue of the router and they are then released in order. The drawbacks that limit its usage include problems related to certain flow-based applications and a very large buffer that may be required depending on the packet processing time and the link speed [16]. A packet taking longer to process can hold a large number of subsequent packets in the flow in the output reordering buffer, causing it to overflow [16]. While both input buffering and output buffering methods are capable of significantly reducing the amount of reordering introduced by nodes, they do not scale well with the increase of packet rates and number of flows per link. With future networks, there will be fat-pipes with multiple short flows [18]. The number of lookup entries in routers is increasing exponentially [19], which is also an indication of increase in number of parallel flows per port. Therefore, identification of individual flows at input or output buffers adds to delay of the packets significantly, and increases the waiting time of the packets within the nodes [20]. Moreover, with greater number of flows, per-packet scheduling will become more attractive, and achieving high packet throughput will require parallel paths within routers. However, neither input buffering nor output buffering can compensate reordering caused by multiple paths. All these trends point to a significant increase in packet reordering.

Careful redesign of protocols, at application and transport levels, may help in recovery from reordering at end host in case of sporadic packet reordering. However, many studies have shown that reordering is not a sporadic event and is likely to increase in the future. Most protocols assume that phenomenon such as corruption, packet loss and reordering are infrequent events, and when it is not the case, the protocols can suffer in performance, correctness or both [3], [4]. Packet reordering has thus been recognized as a significant impediment to the performance, and new mitigation schemes are in discussion at Internet Engineering Task Force (IETF) [21] for TCP in particular. Given that packet reordering is not a sporadic event, and in fact it is increasing with network speeds and increased parallelism, generalized solutions may tax the resources at the end nodes significantly and/or prevent application requirements from being met. In the case of non-traditional networks, especially for wireless networks involving small form factor devices, with possible resource limitations, reordering is an unwanted effect resulting from ad hoc routing and handoff schemes [22]. Communication over such networks is not restricted to TCP/UDP transport protocols with underlying IP, and the ongoing proposals for mitigation may not be applicable. Moreover, with increasing parallelism it is envisioned that (partial or complete) recovery from out-of-order arrivals will be attempted more at the intermediate nodes and the transport/application specific information may not be available/useful.

Argument can be made that end applications can take appropriate action such as packet buffering to recover from reordering. In such a case one of the key requirements is to have enough buffer resources allocated at end nodes for recovery from packet reordering. The quantitative estimate of packet reordering may help in optimal resource provisioning such as buffer allocation at the end nodes. Moreover, packet reordering metric has the potential to be useful for characterizing the end-to-end performance of a network. This end-to-end characterization of network using reorder metrics may help application developers predict the performance of their applications/protocols before the deployment over the network. Reordering metric may be used to adjust some protocol operating parameters such as ‘dupthresh’ for TCP protocol to enhance its performance in presence of packet reordering. There are lots of studies that have highlighted presence of packet reordering in networks [1], [2], [8], [10], [12]. However, in the absence of any standards, many of these experiments have used different metrics without any consensus. For example, [10] uses ‘Monotonic increase’ and ‘TCP-like’ metrics for measuring reordering. Alternatively, [12] uses ‘reordering rate’ to measure degree of packet reordering. In order to compare results from different studies and experiments there is a need to define common metrics for packet reordering. All the above mentioned possible applications highlight the need for a standard metric to quantitatively capture packet reordering.

Quantification of the amount of packet reordering present in a sequence of packets is a first step for understanding and dealing with reordering. The percentage of out-of-order packets has been used as a metric for characterizing reordering [3], [6], [13]. This metric is vague and has no uniform definition. It does not provide sufficient information for characterizing different packet sequences. For example, consider the two packet sequences (1, 3, 4, 2, 5) and (1, 4, 3, 2, 5). It is possible to interpret the out-of-orderliness of packets differently in this case resulting in different percentages of reordering, for example,

  • (i)

    Packets 2, 3 and 4 are out-of-order in both cases,

  • (ii)

    Only packet 2 is out-of-order in the first, while packets 3 and 4 are out-of-order in the second,

  • (iii)

    Packets 3 and 4 are out-of-order in both the sequences,

  • (iv)

    Packets 2, 3 and 4 are out-of-order in first sequence, while packets 4 and 2 are out-of-order in the second sequence.

The need for a precise definition is evident from this example. If buffers are available to store packets 3 and 4 while waiting for packet 2, it is possible to recover from the reordering. However, there may be cases where the application requirement is such that arrival of packet 2 after this delay renders it useless. While one can argue that a good packet reordering measurement scheme should capture such effects, a counter argument can also be made that packet reordering should be measured strictly with respect to the order of delivery and should be application independent. The need for an application independent measure arises due to the fact that reordering is not a problem only of concern with respect to end-to-end connections. In fact, many modern router and switch designs attempt to minimize the reordering they inevitably introduce. To evaluate effectiveness of such strategies in routers, to evaluate reordering introduced by different network segments (e.g., ISPs), etc., measures capable of quantifying reordering in packet streams are needed. Here a stream could mean an aggregate of flows (correspond to different applications, different source–destination pairs, etc.) passing through the network segment/node of interest. An application dependent measure is not applicable in this case. A framework for metrics presented in [23] states that “The metrics must be useful to users and providers in understanding the performance they experience or provide”. Packet reordering is one of many undesirable attributes introduced by a network. Others include packet loss and duplication. As causes, recovery mechanisms, and impacts of these anomalies are different from each other, it is desirable to measure and quantify reordering, losses and duplicates orthogonal to each other as much as possible.

Several metrics are under consideration by the IETF for packet reordering [24], [25]. The metrics presented in [25] are classified into sample metrics and receiver assessment metrics. Sample metrics are defined on a sample of sequence numbers and percentage reordering can be an example of such a metric. However, these metrics do not differentiate sequences as explained earlier and as such are not useful for applications and monitoring tools. The receiver assessment metrics, e.g., n-reordering and reorder extent, consider only late packets as out-of-order. Reorder Density (RD) and Reorder Buffer-occupancy Density (RBD) are two proposed metrics for packet reordering [24]. RD characterizes a sequence of packets based on earliness and lateness of packets relative to each other [26]. RBD, in contrast to other metrics, characterizes reordering of packets from the point of view of its impact on the application, or the resources required to recover from reordering. A comparison of different metrics can be found in [27].

This paper investigates packet reordering and its impact from the point of view of resources required for recovery from reordering. Reorder Buffer-occupancy Density (RBD) is defined as the normalized histogram of the occupancy of a buffer that allows the recovery from out-of-order delivery of packets [24], [28]. Estimates of RBD are useful for efficient allocation of resources, (a) at end-nodes, either at the application or transport levels, for recovery from reordering, and (b) at routers where output buffering is used to prevent reordering from being introduced at a router node. Furthermore, RBD can be used as a metric for measurement of reordering introduced by networks or subnets. In case of measurements, the buffer can be considered as a hypothetical buffer used for the purpose of evaluation, while an actual physical buffer is needed for recovery from reordering. A threshold on buffer size is used to declare a packet as lost, which keeps measurement complexity within bounds and makes the metric robust against cases such as a burst of losses and/or duplicates. At times the early arrivals are buffered in anticipation of a packet that probably is not reordered but lost. To accommodate such cases, a variant of RBD is also presented in [24] that makes it completely orthogonal to packet losses.

Though the future high-speed networks will have to deal with the problem of out-of-sequence packets proactively, scant attention has been paid so far towards understanding the nature of reordering that occurs in networks. Information on the nature of reordering that occurs in a given end-to-end path may be used to enhance the ways that the protocols deal with this problem. Measurement and characterization of reordering in packet sequences and models that provide insight into this problem can lead to the development of scalable techniques to deal with reordering and perhaps even to develop tools that diagnose network problems. RBD is not only a metric for capturing the amount and extent of reordering of packets, but also a tool that allows the allocation of buffer resources at nodes to deal with packet reordering. We present an analysis that relates packet reordering to RBD. Packet reordering is specified using reorder events. Interactions among events result in independent reordering in which reordering of one packet does not interfere with that of another, embedded reordering where one event is embedded within another, and overlapped reordering in which two events overlap. RBD is derived for subsequences containing independent, overlapped and embedded reorder events. The analysis is then extended to cover sequences with multiple reorder events. The analysis presented for RBD is also directly applicable to resource allocation for playout buffers, a recovery mechanism implemented at the application level.

The remainder of the paper is organized as follows: Section 2 defines the Reorder-Buffer-occupancy density and illustrates its computation. Section 3 presents a formal representation for reordering, introduces the basic and prevalent patterns of reordering observed in the Internet, and derives RBDs for such multiple reorderings. In Section 4, an analysis of reordering is presented using a formal characterization of reordering for a load-balancing scenario. The RBD model developed for this scenario is validated using measurements on an emulated network. Section 5 presents two schemes for evaluating a loss-orthogonal version of RBD, for cases where reordering needs to be evaluated independent of packet losses. Conclusions are presented in Section 6.

Section snippets

Reorder buffer-occupancy density (RBD)

The Reorder Buffer-occupancy Density (RBD) is defined and explained next. Without loss of generality, consider a sequence of packets (1, 2, …, N) transmitted from a source node to a destination. RBD is also applicable for cases where the sequence number is incremented in a different manner, such as with TCP, or where the sequence of packets may be identified based on timestamps.

Formal representation and analysis of packet reordering

A formal representation of packet reordering is important to characterize and understand the reordering phenomena. Understanding packet reordering using an atomic measure such as percentage reordering does not permit such characterization. A formal representation of reordering is presented next along with expressions for RBD for different possible scenarios of multiple reordering.

Without loss of generality, consider a sequence of packets (1, 2, …, N) transmitted over a network. A receive_index

RBD for a simple load-balancing scenario

We now illustrate the application of the methodology presented in Section 3 for deriving the RBD to evaluate packet reordering and its impact under a load-balancing scenario. The analytical results for this scenario are verified against results from an emulated network with the corresponding load-balancing scenarios.

Consider the case where a certain number of packets in a flow undergo significantly higher delay due to packets being routed via a longer path Y instead of path of choice X. Such a

Measurements of reordering over the Internet

In this section, we explain a refinement to use RBD as a metric for reordering that is orthogonal to packet losses. Consider a sequence without reordering, e.g., (1, 2, 3, 4, 5, 6, 7, 8, 9, 10). Since there is no reordering, RBD has a single component of value 1 for buffer-occupancy equal to zero. However, if there is no reordering in a sequence but a single packet is lost, e.g., (1, 2, 3, 4, 5, 6, 7, 9, 10, 11), then the RBD will have components for B = 1, 2, 3 corresponding to the arrivals 9, 10 and 11 as 8 is

Conclusions

This paper investigated the problem of packet reordering. Reorder Buffer-occupancy Density (RBD) was defined, which can be used both as a metric for capturing reordering in a stream, and also as a measure of resources needed to recover from such reordering. A formal analysis was presented relating reordering introduced by a simple load balancing scenario to the corresponding RBD. The existing metrics to measure reordering are vague or insufficient to characterize packet reordering, and RBD

Acknowledgements

This research was supported in part by NSF ITR Grant No. 0121546, Agilent Laboratories, and Ixia University Partners Program.

Nischal M. Piratla is a Senior Research Scientist at Deutsche Telekom Laboratories. He is also a co-founder and CTO of Qiro GmbH in Berlin, Germany. He received his Ph.D. (2005) and MS (1999) in Electrical Engineering from Colorado State University, and B.Tech (with Distinction) from JNTU College of Engineering, Hyderabad, India. His industry terms in R&D were at Avaya Inc. (2000–2002), Seagate Technology (1999–2000) and CMC (India) Ltd. (1996–1997). His areas of interest include Context-Aware

References (34)

  • J.C.R. Bennett et al.

    Packet reordering is not pathological network behavior

    IEEE/ACM Trans. Networking

    (1999)
  • G. Iannaccone, S. Jaiswal and C. Diot, Packet Reordering Inside the Sprint Backbone, Tech. Report, TR01-ATL-062917,...
  • J. Bellardo, S. Savage, Measuring Packet Reordering, Proc. Internet Measurements Workshop, November 2002, pp....
  • L. Kencl et al.

    Adaptive load sharing for network processors

    Proc. IEEE INFOCOM, 2002

    (2002)
  • S. Bohacek, J. Hespanha, J. Lee, C. Lim, K. Obraczka, TCP-PR: TCP for persistent packet reordering, Proc. IEEE 23rd...
  • V. Paxson, Measurements and Analysis of End-to-End Internet Dynamics, Ph.D. dissertation, Computer Science Department,...
  • D. Loguinov, H. Radha, Measurement study of low-bitrate internet video streaming, Proc. 1st ACM SIGCOMM Workshop on...
  • S. Jaiswal, G. Iannaccone, C. Diot, J. Kurose, D.Towsley, Measurement and classification of out-of-sequence packets in...
  • A.A. Bare, Measurement and Analysis of Packet Reordering using Reorder Density, Masters Thesis, Department of Computer...
  • L. Gharai, C. Perkins, T. Lehman, Packet reordering, high speed networks and transport protocol performance, Proc. IEEE...
  • E. Blanton et al.

    On making TCP more robust to packet reordering

    ACM Comput. Comm. Rev.

    (2002)
  • X. Luo, R. Chang, Novel Approaches to End-to-end Packet Reordering Measurement, Proc. ACM/USENIX Conf. Internet...
  • M. Laor et al.

    The effect of packet reordering in a backbone link on application throughput

    IEEE Network, September/October 2002

    (2002)
  • F. Zhao, Y. Shin, S.F. Wu, H. Johnson, A. Nilsson, RBWA: an efficient random-bit Window-based authentication protocol,...
  • M. Zhang, B. Karp, S. Floyd, L. Peterson, RR-TCP: a reordering-robust TCP with DSACK, Proc. of the Eleventh IEEE...
  • H. Liu, A trace driven study of packet level parallelism, Proc. IEEE Int. Conf. on Communications (ICC 2002), vol,. 4,...
  • H. Liu, A trace driven study of packet level parallelism, Proc. International Conference on Communications (ICC),...
  • Cited by (21)

    • Building packet length covert channel over mobile VoIP traffics

      2018, Journal of Network and Computer Applications
    • Stabilization of stochastic delay systems via a disordered controller

      2017, Applied Mathematics and Computation
      Citation Excerpt :

      However, the motivation of disordering problem also comes from the data transmitted through the shared communication networks. It is an inevitable phenomenon that the transmitted data arriving at the destination is usually out of order [43–45]. In other words, the transmission of data packets cannot always satisfy the “first send first arrive” principle.

    • Real-time guaranteed cost control of MIMO networked control systems with packet disordering

      2011, Journal of Process Control
      Citation Excerpt :

      In this case, obviously, the controllers obtained hardly stabilize the systems and even may make the systems unstable in the practical application of NCSs. Packet disordering, out of order, is an increasingly common phenomenon on the Internet [14]. It is really nice that increasing research interests have recently been paid to the study of packet disordering of NCSs [15–17,19].

    • Packet Reordering Metrics to Enable Performance Comparison in IP-Networks

      2020, Journal of Computer Networks and Communications
    • A novel class of robust covert channels using out-of-order packets

      2017, IEEE Transactions on Dependable and Secure Computing
    View all citing articles on Scopus

    Nischal M. Piratla is a Senior Research Scientist at Deutsche Telekom Laboratories. He is also a co-founder and CTO of Qiro GmbH in Berlin, Germany. He received his Ph.D. (2005) and MS (1999) in Electrical Engineering from Colorado State University, and B.Tech (with Distinction) from JNTU College of Engineering, Hyderabad, India. His industry terms in R&D were at Avaya Inc. (2000–2002), Seagate Technology (1999–2000) and CMC (India) Ltd. (1996–1997). His areas of interest include Context-Aware Services, Recommender Systems, HCI and Network Measurements.

    Anura P. Jayasumana is a Professor of Electrical and Computer Engineering, and Computer Science at Colorado State University. He received his Ph.D. (1985) and M.S. (1982) in Electrical Engineering from Michigan State University, and B.Sc. in Electronics and Telecommunication Engineering, with first class honors, from University of Sri Lanka, Moratuwa (1978). Awards he has won include award for best student in electrical engineering at Univ. of Sri Lanka, Moratuwa (1978), College of Engineering Outstanding Academic Achievement Awards at Michigan State University (1982,1983), Outstanding Faculty of the Year Award from Mountain States Council of the American Electronics Association (1990), and Engineering Deans Council Award for Academic Excellence from the Colorado State University in 1998. His research interests include network protocols, sensor networks, performance evaluation, and VLSI testing. He is a member of Phi Kappa Phi.

    Abhijit Bare received his B.E. degree from University of Mumbai, India in Computer Engineering and M.S. degree from Colorado State University, Fort Collins, CO in Computer Science. His work revolved around research in measurement and analysis of packet reordering and other areas such as distributed computing. Currently he is working with Agilent Technologies Inc., Colorado.

    Tarun Banka is a Ph.D. candidate in the Department of Electrical and Computer Engineering at the Colorado State University (CSU). He received his M.S. degree in Computer Science from the CSU and B.E. degree in Computer Science and Engineering from the Punjab Engineering College, India. His research interests include sensor networks, overlay networks, application-aware protocols, network performance modeling, and heterogeneous computing.

    View full text