Abstract
The performance of interconnection networks is a challenging issue for High-Performance Computing (HPC) systems, which becomes even more important when the number of interconnected endnodes grows. In that sense, Dragonfly interconnection patterns are a very popular option to configure the network topology, especially for large systems, as they are able to achieve a high scalability relying on high-radix switches. This kind of hierarchical topologies has two levels of interconnection (i.e., connections within the element of a group and connections among groups) and each one can be interconnected using different patterns. However, regardless of the Dragonfly interconnection pattern, the Head-of-Line (HoL) blocking effect derived from congestion situations may jeopardize the Dragonfly performance. This paper analyzes the dynamics of congestion in different Dragonfly fully-connected interconnection patterns. Also, we describe a queuing scheme called Hierarchical Two-Level Queuing (H2LQ), designed specially to reduce HoL blocking in any fully-connected Dragonfly network that uses minimal-path routing. Finally, we present experiment results which show that this scheme significantly boost Dragonfly performance, regardless the interconnection pattern, especially when congestion arises, while requiring fewer network resources than other techniques oriented to deal with the effects of congestion.
Similar content being viewed by others
Notes
Lossless networks are those where packet discarding is not allowed. Note that lossless networks are the common option for HPC systems, the InfiniBand technology being the most significant example of HPC-based network technology.
We assume that this value is the \(number\_of\_nodes \times link\_bandwidth\).
Note that in real HPC clusters it is typical that a queuing-based scheduler is in charge of mapping jobs to the available processing nodes, so that several applications can be run at the same time.
References
The Graph 500 List. www.graph500.org
VEF traces: An easy way to model MPI traffic in network simulators. http://www.i3a.uclm.es/VEFtraces/
Alversons B, Froese E, Kaplan L, Roweth D (2012) Cray XC Series Network. Tech. rep. Cray Inc,
Anderson T, Owicki S, Saxe J, Thacker C (1993) High-Speed Switch Scheduling for Local-Area Networks. ACM Transactions on Computer Systems 11(4):319–352
Andujar FJ, Villar JA, Sanchez JL, Alfaro FJ, Escudero-Sahuquillo J (2015) VEF Traces: A Framework for Modelling MPI Traffic in Interconnection Network Simulators. In: Cluster Computing (CLUSTER), 2015 IEEE International Conference on, pp. 841–848. doi:10.1109/CLUSTER.2015.141
Arimilli B, Arimilli R, Chung V, Clark S, Denzel W, Drerup B, Hoefler T, Joyner J, Lewis J, Li J, Ni N, Rajamony R (2010) The PERCS High-Performance Interconnect. In: High Performance Interconnects (HOTI), 2010 IEEE 18th Annual Symposium on, pp. 75–82. doi:10.1109/HOTI.2010.16
Association IT (2007) InfiniBand Architecture Specification. http://www.infinibandta.org
Camarero C, Vallejo E, Beivide R (2014) Topological Characterization of Hamming and Dragonfly Networks and Its Implications on Routing. ACM Trans. Archit. Code Optim. 11(4):39:1–39:25. doi:10.1145/2677038
Dally W (1992) Virtual-Channel Flow Control. IEEE Trans. on Parallel and Distributed Systems 3(2):194–205
Dally W, Carvey P, Dennison L (1998) Architecture of the Avici terabit switch/router. In: 6th Hot Interconnects, pp. 41–50
Dally WJ, Towles B (2003) Principles and Practices of Interconnection Networks. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA
Escudero-Sahuquillo J, García PJ, Quiles FJ, Flich J, Duato J (2011) OBQA: Smart and cost-efficient queue scheme for Head-of-Line blocking elimination in fat-trees. J. Parallel Distrib. Comput. 71(11):1460–1472
Escudero-Sahuquillo J, Garcia PJ, Quiles FJ, Reinemo SA, Skeie T, Lysne O, Duato J (2014) A New Proposal to Deal with Congestion in InfiniBand-based Fat-trees. J. Parallel Distrib. Comput. 74(1):1802–1819
Escudero-Sahuquillo J, Gran E, Garcia-Garcia P, Flich J, Skeie T, Lysne O, Quiles F, Duato J (2014) Efficient and Cost-Effective Hybrid Congestion Control for HPC Interconnection Networks. Parallel and Distributed Systems, IEEE Transactions on PP(99):1–1. doi:10.1109/TPDS.2014.2307851
Garcia P, Quiles F, Flich J, Duato J, Johnson I, Naven F (2006) Efficient, Scalable Congestion Management for Interconnection Networks. Micro, IEEE 26(5):52–66
Gomez C, Gilabert F, Gomez M, Lopez P, Duato J (2007) Deterministic versus Adaptive Routing in Fat-Trees. In: Workshop CAC in conjunction with the IPDPS, p. 235
Guay WL, Bogdanski B, Reinemo SA, Lysne O, Skeie T (2011) vFtree - A Fat-Tree Routing Algorithm Using Virtual Lanes to Alleviate Congestion. In: Proc. of IPDPS, pp. 197–208
Hastings E, Rincon-Cruz D, Spehlmann M, Meyers S, Xu A, Bunde DP, Leung VJ (2015) Comparing Global Link Arrangements for Dragonfly Networks. In: Cluster Computing (CLUSTER), 2015 IEEE International Conference on, pp. 361–370. doi:10.1109/CLUSTER.2015.57
Jurczyk M, Schwederski T (1996) Phenomenon of Higher Order Head-of-Line Blocking in Multistage Interconnection Networks under Nonuniform Traffic Patterns. IEICE Transactions on Information and Systems E79–D(8):1124–1129
Karol MJ, Hluchyj MG, Morgan SP (1987) Input versus output queuing on a space-division packet switch. IEEE Transactions on Communications. COM–35:1347–1356
Katevenis M, Serpanos D, Spyridakis E (1998) Credit-flow-controlled ATM for MP interconnection: The ATLAS I single-chip ATM switch. In: High-Performance Computer Architecture, 1998. Proceedings., 1998 Fourth International Symposium on, pp. 47–56
Kim J, Dally WJ, Scott S, Abts D (2008) Technology-Driven, Highly-Scalable Dragonfly Topology. SIGARCH Comput. Archit. News 36(3):77–88
Nachiondo T, Flich J, Duato J (2010) Buffer Management Strategies to Reduce HoL Blocking. Parallel and Distributed Systems, IEEE Transactions on 21(6):739–753. doi:10.1109/TPDS.2009.63
Olesinski W, Eberle H, Gura N (2009) Scalable alternatives to virtual output queueing. In: Proc. IEEE ICC, pp. 1–6
OpenSim Ltd: OMNeT++ Discrete Event Simulator. http://omnetpp.org/
Peir JK, Lee YH (1993) Look-ahead routing switches for multistage interconnection networks. Journal of Parallel and Distributed Computing 19(1):1–10. doi:10.1006/jpdc.1993.1085
Penaranda R, Gomez C, Gomez M, Lopez P, Duato J (2012) A New Family of Hybrid Topologies for Large-Scale Interconnection Networks. In: Network Computing and Applications (NCA), 2012 11th IEEE International Symposium on, pp. 220–227
Pfister G, Gusat M, Denzel W, Craddock D, Ni N, Rooney W, Engbersen T, Luijten R, Krishnamurthy R, Duato J (2005) Solving Hot Spot Contention Using InfiniBand Architecture Congestion Control. In: Proc. of Int. Workshop HPI-DC
Phillips JC, Braun R, Wang W, Gumbart J, Tajkhorshid E, Villa E, Chipot C, Skeel RD, Kalé L, Schulten K (2005) Scalable molecular dynamics with NAMD. Journal of Computational Chemistry 26(16):1781–1802. doi:10.1002/jcc.20289
Pinkston TM, Duato J (2006) Appendix E. In: Elsevier (ed.) Computer Architecture: A Quantitative Approach. Morgan Kaufmann Publishers
Tamir Y, Frazier G (1992) Dynamically-Allocated Multi-Queue Buffers for VLSI Communication Switches. IEEE Trans. on Computers
Yebenes P, Escudero-Sahuquillo J, Garcia P, Quiles F (2013) Towards Modeling Interconnection Networks of Exascale Systems with OMNet++. In: Parallel, Distributed and Network-Based Processing. doi:10.1109/PDP.2013.36
Yebenes Segura P, Escudero-Sahuquillo J, Gomez Requena C, Garcia P, Quiles F, Duato J (2013) BBQ: A Straightforward Queuing Scheme to Reduce HoL-Blocking in High-Performance Hybrid Networks. In: Euro-Par 2013 Parallel Processing, vol. 8097, pp. 699–712
Zahavi E, Johnson G, Kerbyson DJ, Lang M (2010) Optimized \(\text{ InfiniBand }^{{\rm TM}}\) fat-tree routing for shift all-to-all communication patterns. Journal of CCPE 22(2):217–231
Author information
Authors and Affiliations
Corresponding author
Additional information
This work has been jointly supported by the Spanish MINECO and European Commission (FEDER funds) under the projects TIN2012-38341-C04 and TIN2015-66972-C5-2-R, and the FPI grant BES-2013-063681, and by Junta de Comunidades de Castilla- La Mancha under the project PEII-2014-028-P. Jesus Escudero-Sahuquillo has been funded by the Spanish MINECO under the postdoctoral grant FPDI-2013-18787 until November 2015 and, from that date, he has been funded by the University of Castilla-La Mancha (UCLM) and the European Commission (FSE funds), with a contract for accessing the Spanish System of Science, Technology and Innovation, for the implementation of the UCLM research program (UCLM resolution date: 31/07/2014).
Rights and permissions
About this article
Cite this article
Yébenes, P., Escudero-Sahuquillo, J., García, P.J. et al. Straightforward solutions to reduce HoL blocking in different Dragonfly fully-connected interconnection patterns. J Supercomput 72, 4497–4519 (2016). https://doi.org/10.1007/s11227-016-1756-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-016-1756-1