Abstract
In this paper we present an in-depth description of the Quadrics interconnection network (QsNET) and an experimental performance evaluation on a 64-node AlphaServer cluster. We explore several performance dimensions and scaling properties of the network by using a collection of benchmarks, based on different traffic patterns. Experiments with permutation patterns and uniform traffic are conducted to illustrate the basic characteristics of the interconnect under conditions commonly created by parallel scientific applications. Moreover, the behavior of the QsNET under I/O traffic, and the influence of the placement of the I/O servers are analyzed. The effects of using dedicated I/O nodes or shared I/O nodes are also exposed. In addition, we evaluate how background I/O traffic interferes with other parallel applications running concurrently. The experimental results indicate that the QsNET provides excellent performance in most cases, with excellent contention resolution mechanisms. Some important guidelines for applications and I/O servers mapping on large-scale clusters are also given.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
N.J. Boden, D. Cohen, R.E. Felderman, A.E. Kulawick, C.L. Seitz, J.N. Seizovic and W.-K. Su, Myrinet: A gigabit-per-second local area network, IEEE Micro (1) (1995) 29–36.
H. Chen and P. Wyckoff, Simulation studies of Gigabit Ethernet versus Myrinet using real application cores, in: Proceedings of CANPC'00, Workshop of High-Performance Computer Architecture, Toulouse, France, January 2000.
W.J. Dally and C.L. Seitz, Deadlock-free message routing in multi-processor interconnection networks, IEEE Transactions on Computers C-36(5) (1987) 547–553.
J. Duato, S. Yalamanchili and L. Ni, Interconnection Networks: An Engineering Approach (IEEE Computer Society Press, 1997).
A. Griest, W. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, W. Saphir, T. Skjellum and M. Snir, MPI-2: Extending the message passing interface, in: 2nd International Euro-Par Conference, Lyon, France, Vol. 1, Lecture Notes in Computer Science, Vol. 1123 (Springer, 1996) pp. 128–135.
S. Heller, Congestion-free routing on the CM-5 data router, in: 1st International Workshop, PCRCW'94, Seattle, Washington, USA, Lecture Notes in Computer Science, Vol. 853, eds. K. Bolding and L. Snyder (Springer, 1994) pp. 176–184.
H. Hellwagner, The SCI standard and applications of SCI, in: SCI: Scalable Coherent Interface, Lecture Notes in Computer Science, Vol. 1291, eds. H. Hellwagner and A. Reinfeld (Springer, 1999) pp. 95–116.
InfiniBand Specification 1.0a. InfiniBand Trade Association (June 2001).
D. Kerbyson, H. Alme, A. Hoisie, F. Petrini, H. Wasserman and M. Gittings, Predictive performance and scalability modeling of a largescale application, in: Supercomputing 2001, Denver, CO, November 2001.
C.E. Leiserson, Fat-trees: Universal networks for hardware efficient supercomputing, IEEE Transactions on Computers C-34(10) (1985) 892–901.
L.M. Ni, Y. Gui and S. Moore, Performance evaluation of switch-based wormhole networks, IEEE Transactions on Parallel and Distributed Systems 8(5) (1997) 462–474.
F. Petrini, Communication performance of wormhole interconnection networks, Ph.D. thesis, Università degli Studi di Pisa, Dipartimento di Informatica, February 1997.
F. Petrini, W. Chun Feng, A. Hoisie, S. Coll and E. Frachtenberg, The quadrics network: High performance clustering technology, IEEE Micro 22(1) (January–February 2002) 46–57.
F. Petrini, S. Coll, E. Frachtenberg and A. Hoisie, Hardware-and software-based collective communication on the Quadrics network, in: IEEE International Symposium on Network Computing and Applications 2001 (NCA 2001), Boston, MA, October 2001.
F. Petrini, A. Hoisie, W. Chun Feng and R. Graham, Performance evaluation of the quadrics interconnection network, in: Workshop on Communication Architecture for Clusters (CAC'01), San Francisco, CA, April 2001.
F. Petrini and M. Vanneschi, k-ary n-trees: High performance networks for massively parallel architectures, in: Proceedings of the 11th International Parallel Processing Symposium, IPPS'97, Geneva, Switzerland, April 1997, pp. 87–93.
F. Petrini and M. Vanneschi, Performance analysis of wormhole routed k-ary n-trees, International Journal on Foundations of Computer Science 9(2) (June 1998) 157–177.
G.F. Pfister and V.A. Norton, Hot-spot contention and combining in multistage interconnection networks, IEEE Transactions on Computers C-34(10) (1985) 943–948.
Quadrics Supercomputers World Ltd., Elan Programming Manual (January 1999).
Quadrics Supercomputers World Ltd., Elan Reference Manual (January 1999).
Quadrics Supercomputers World Ltd., Elite Reference Manual (November 1999).
R. Seifert, Gigabit Ethernet: Technology and Applications for High Speed LANs (Addison-Wesley, 1998).
M. Snir, S. Otto, S. Huss-Lederman, D. Walker and J. Dongarra, MPI–The Complete Reference, The MPI Core, Vol. 1 (TheMIT Press, 1998).
D. Tolmie, T.M. Boorman, A. DuBois, D. DuBois, W. Feng and I. Philp, From HiPPI-800 to HiPPI-6400: A changing of the guard and gateway to the future, in: Proceedings of the 6th International Conference on Parallel Interconnects (PI'99), October 1999.
W. Vogels, D. Follett, J. Hsieh, D. Lifka and D. Stern, Tree-saturation control in the AC3 velocity cluster, in: Hot Interconnects 8 (Stanford University, Palo Alto CA, 2000).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Petrini, F., Frachtenberg, E., Hoisie, A. et al. Performance Evaluation of the Quadrics Interconnection Network. Cluster Computing 6, 125–142 (2003). https://doi.org/10.1023/A:1022852505633
Issue Date:
DOI: https://doi.org/10.1023/A:1022852505633