Skip to main content
SpringerLink
Log in

Impact of protocol overheads on network throughput over high-speed interconnects: measurement, analysis, and improvement

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Although extremely high-speed interconnects are available today, the traditional protocol stacks such as TCP/IP and UDP/IP are not able to utilize the maximum network bandwidth due to inherent overheads in the protocol stacks. Such overheads are a big obstacle for high-performance computing applications to exploit high-speed interconnects in cluster environments. To address this issue, many researchers have been presenting analyses of protocol overheads and suggesting a number of optimization approaches to harness the TCP/IP suite over high-speed interconnects. However, to the best of our knowledge, there is no study that analyzes and optimizes the protocol overheads thoroughly in an integrated manner. In this paper, we exploit a set of protocol optimization mechanisms in an integrated manner while dealing with the full spectrum of the protocol layers from the transport layer to the physical layer. To evaluate the impact of each protocol overhead, we apply the optimization mechanisms one by one and perform detailed analyses at each step. The thorough overhead measurements and analyses reveal the dependencies between protocol overheads. With our comprehensive optimizations, we show that UDP/IP can utilize more than 95% of the maximum network throughput a Myrinet-based experimental system can provide.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Anderson D, Chase J, Gadde S, Gallatin A, Yocum K, Feeley M (1998) Cheating the I/O bottleneck: network storage with trapeze/myrinet. In: Proceedings of the 1998 USENIX technical conference, June 1998

  2. Anderson TE, Culler DE, Patterson DA, NOW Team (1995) A case for networks of workstations: NOW. IEEE Micro 15(1):54–64

    Article  Google Scholar 

  3. Arnould E, Bitz F, Cooper E, Kung HT, Sansom R, Steenkiste P (1989) The design of nectar: a network backplane for heterogeneous multicomputers. In: Third international conference on architectural support for programming languages and operating systems, April 1989, pp 205–216

  4. Balaji P, Narravula S, Vaidyanathan K, Krishnamoorthy S, Wu J, Panda DK (2004) Sockets direct protocol over InfiniBand in clusters: is it beneficial? IEEE international symposium on performance analysis of systems and software (ISPASS 04), March 2004

  5. Balaji P, Shah HV, Panda DK (2004) Sockets vs RDMA interface over 10-Gigabit networks: an in-depth analysis of the memory traffic bottleneck. In: Workshop on remote direct memory access (rdma): applications, implementations, and technologies (RAIT 2004), September 2004

  6. Banks D, Prudence M (1993) A high-performance network architecture for a PA-RISC workstation. IEEE J Sel Areas Commun 11(2):191–202

    Article  Google Scholar 

  7. Barak A, Gilderman I, Metrik I (1999) Performance of the communication layers of TCP/IP with the myrinet gigabit LAN. Comput Commun 22:989–997

    Article  Google Scholar 

  8. Boden NJ, Cohen D, Felderman RE, Kulawik AE, Seitz CL, Seizovic JN, Su W-K (1995) Myrinet—a gigabit-per-second local-area network. IEEE Micro 15(1):29–36

    Article  Google Scholar 

  9. Braden R, Borman D, Partridge C (1988) Computing the Internet checksum. In: IETF networking working group request for comments: 1071, September 1988

  10. Brustoloni JC, Steenkiste P (1997) Copy emulation in checksummed, multiple-packet communication. In: IEEE Infocom ’97, April 1997

  11. Chase J, Gallatin A, Yocum K (2001) End-system optimizations for high-speed TCP. IEEE Commun Mag 39(4):68–75

    Article  Google Scholar 

  12. Chu HJ (1996) Zero-copy TCP in solaris. In: 1996 Winter USENIX, 1996

  13. Clark D, Jacobson V, Romkey J, Salwen H (1989) An analysis of TCP processing overhead. IEEE Commun Mag 27(6):23–29

    Article  Google Scholar 

  14. Dalton C, Watson G, Banks D, Calamvokis C, Edwards A, Lumley J (1993) Afterburner. IEEE Netw 7(4):36–43

    Article  Google Scholar 

  15. Druschel P, Peterson LL (1993) Fbufs: a high-bandwidth cross-domain transfer facility. In: Proceedings of 14th ACM SOSP, 1993, pp 189–202

  16. Dunning D, Regnier G, McAlpine G, Cameron D, Shubert B, Berry AM, Gronke E, Dodd C (1998) The virtual interface architecture. IEEE Micro 8:66–76

    Article  Google Scholar 

  17. von Eicken T, Basu A, Buch V, Vogels W (1995) U-Net: a user-level network interface for parallel and distributed computing. In: Proceedings of 15th ACM SOSP, December 1995, pp 40–53

  18. Feng W-C, Hurwitz J, Newman H, Ravot S, Cottrell RL, Martin O, Coccetti F, Jin C, Wei X, Low S (2003) Optimizing 10-gigabit Ethernet for networks of workstations, clusters, and grids: a case study. In: Proceedings of SC2003: high-performance networking and computing conference, November 2003

  19. Foster I, Kesselman C (1999) The grid: blueprint for a new computing infrastructure. Morgan Kaufmann, San Francisco

    Google Scholar 

  20. Gallatin A, Chase J, Yocum K (1999) Trapeze/IP: TCP/IP at near-gigabit speeds. In: Proceedings of 1999 USENIX technical conference (freenix track), June 1999

  21. Gigabit Ethernet Alliance (1997) IEEE 802.3z. The Emerging Gigabit Ethernet Standard

  22. 10 Gigabit Ethernet Alliance (2002) Introduction to TCP/IP offload engine (TOE), Version 1.0, April 2002

  23. Halstead DM, Bode B, Turner D, Lewis V (1999) Giga-plant scalable cluster. In: Proceedings of the USENIX extreme Linux technical conference, June 1999, pp 10–15

  24. IEEE (2002) IEEE Std 802.3ae-2002, Media access control (MAC) Parameters, physical layers, and management parameters for 10 Gbps operation, August 2002

  25. InfiniBand Trade Association. http://www.infinibandta.org

  26. Jin H-W, Balaji P, Yoo C, Choi J-Y, Panda DK (2005) Exploiting NIC architectural support for enhancing IP based protocols on high performance networks. J Parallel Distrib Comput 65(11):1348–1365

    Article  Google Scholar 

  27. Jin H-W, Yoo C, Choi J-Y (2003) Firmware-level latency analysis on a gigabit network. J Supercomput 26(1):59–75

    Article  MATH  Google Scholar 

  28. Kay J, Pasquale J (1996) Profiling and reducing processing overheads in TCP/IP. IEEE/ACM Trans Netw 4(6):817–828

    Article  Google Scholar 

  29. Kurmann C, Muller M, Rauch F, Stricker T (2000) Speculative defragmentation a technique to improve the communication software efficiency for gigabit ethernet. In: Proceedings of 9th IEEE symposium on high performance distributed computing (HPDC), August 2000

  30. Makineni S, Iyer R (2004) Architectural characterization of TCP/IP packet processing on the Pentium M microprocessor. In: Proceedings of 10th international symposium on high performance computer architecture (HPCA-10), February 2004, pp 152–161

  31. Massalin H, Pu C (1989) Threads and input/output in the synthesis kernel. In: Proceedings of ACM symposium on operating systems principles (SOSP), 1989, pp 191–201

  32. McKenney P, Dove K (1992) Efficient demultiplexing of incoming TCP packets. In: Proceedings of ACM SIGCOMM ’92, August 1992

  33. Mogul J (1991) Network locality at the scale of pro cessors. In: Proceedings of ACM SIGCOMM ’91, September 1991

  34. Myricom Inc (1996) Myrinet user’s guide. http://www.myri.com

  35. Myricom Inc (1999) LANai 4. http://www.myri.com, February 1999

  36. Myricom Inc (2000) The GM message passing system. http://www.myri.com, January 2000

  37. Myricom Inc (2001) PCI64 programmer’s documentation. http://www.myri.com, May 2001

  38. Pakin S, Lauria M, Chien A (1995) High performance messaging on workstations: Illinois fast messages (FM) for Myrinet. In: Proceedings of SC95, 1995

  39. Partridge C, Pink S (1993) A faster UDP. IEEE/ACM Trans Netw 1(4):429–440

    Article  Google Scholar 

  40. Prylli L, Tourancheau B (1998) BIP: a new protocol designed for high performance networking on myrinet. In: Proceedings of the international parallel processing symposium workshop on personal computer based networks of workstations, 1998

  41. Romanow A, Bailey S (2003) An overview of RDMA over IP. In: Proceedings on first international workshop on protocols for fast long-distance networks (PFLDnet2003), February 2003

  42. Shah H, Minturn D, Foong A, McAlpine G, Madukkarumukumana R, Regnier G (2001) CSP: a novel system architecture for scalable Internet and communication services. In: Proceedings of USENIX symposium on internet technologies and systems (USITS01), 2001

  43. Shivam P, Wyckoff P, Panda DK (2002) Can user level protocols take advantage of Multi-CPU NICs? In: Proceedings of international parallel and distributed processing symposium (IPDPS ’02), April 2002

  44. Smith J, Traw C (1993)Giving applications access to Gb/s networking. IEEE Netw 7(4):44–52

    Article  Google Scholar 

  45. USNA (1984) TTCP: a test of TCP and UDP performance, December 1984

  46. Willmann P, Kim H-Y, Rixner S, Pai V (2005) An efficient programmable 10 gigabit ethernet network interface card. In: Proceedings of HPCA-11, February 2005

  47. Wolman A, Voelker G, Thekkath C (1994) Latency analysis of TCP on an ATM network. In: 1994 Winter USENIX, January 1994

  48. Woodside CM, Montealegre JR (1989) The effect of buffering strategies on protocol execution performance. IEEE Trans Commun 37(6): 545–554

    Article  Google Scholar 

  49. Yoo C, Jin H-W, Kwon S-C (2001) Asynchronous UDP. IEICE Trans Commun E84-B(12):3243–3251

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Konkuk University, 1 Hwayang-dong, Gwangjin-gu, Seoul 143-701, Korea

    Hyun-Wook Jin

  2. Department of Computer Science and Engineering, Korea University, 1, 5Ka, Anam-Dong, Sungbuk-Ku, Seoul, 136-701, Korea

    Chuck Yoo

Authors
  1. Hyun-Wook Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chuck Yoo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hyun-Wook Jin.

Additional information

This work was supported by the Faculty Research Fund of Konkuk University in 2006.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jin, HW., Yoo, C. Impact of protocol overheads on network throughput over high-speed interconnects: measurement, analysis, and improvement. J Supercomput 41, 17–40 (2007). https://doi.org/10.1007/s11227-007-0101-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-007-0101-0

Keywords

Search

Navigation