Skip to main content
Log in

Straightforward solutions to reduce HoL blocking in different Dragonfly fully-connected interconnection patterns

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

The performance of interconnection networks is a challenging issue for High-Performance Computing (HPC) systems, which becomes even more important when the number of interconnected endnodes grows. In that sense, Dragonfly interconnection patterns are a very popular option to configure the network topology, especially for large systems, as they are able to achieve a high scalability relying on high-radix switches. This kind of hierarchical topologies has two levels of interconnection (i.e., connections within the element of a group and connections among groups) and each one can be interconnected using different patterns. However, regardless of the Dragonfly interconnection pattern, the Head-of-Line (HoL) blocking effect derived from congestion situations may jeopardize the Dragonfly performance. This paper analyzes the dynamics of congestion in different Dragonfly fully-connected interconnection patterns. Also, we describe a queuing scheme called Hierarchical Two-Level Queuing (H2LQ), designed specially to reduce HoL blocking in any fully-connected Dragonfly network that uses minimal-path routing. Finally, we present experiment results which show that this scheme significantly boost Dragonfly performance, regardless the interconnection pattern, especially when congestion arises, while requiring fewer network resources than other techniques oriented to deal with the effects of congestion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. Lossless networks are those where packet discarding is not allowed. Note that lossless networks are the common option for HPC systems, the InfiniBand technology being the most significant example of HPC-based network technology.

  2. Note that the switch model that we use is based on an IQ-switch architecture where each input buffer is divided into several logical queues sharing a single read port, as described in [4] and [31].

  3. We assume that this value is the \(number\_of\_nodes \times link\_bandwidth\).

  4. Note that in real HPC clusters it is typical that a queuing-based scheduler is in charge of mapping jobs to the available processing nodes, so that several applications can be run at the same time.

References

  1. The Graph 500 List. www.graph500.org

  2. VEF traces: An easy way to model MPI traffic in network simulators. http://www.i3a.uclm.es/VEFtraces/

  3. Alversons B, Froese E, Kaplan L, Roweth D (2012) Cray XC Series Network. Tech. rep. Cray Inc,

  4. Anderson T, Owicki S, Saxe J, Thacker C (1993) High-Speed Switch Scheduling for Local-Area Networks. ACM Transactions on Computer Systems 11(4):319–352

    Article  Google Scholar 

  5. Andujar FJ, Villar JA, Sanchez JL, Alfaro FJ, Escudero-Sahuquillo J (2015) VEF Traces: A Framework for Modelling MPI Traffic in Interconnection Network Simulators. In: Cluster Computing (CLUSTER), 2015 IEEE International Conference on, pp. 841–848. doi:10.1109/CLUSTER.2015.141

  6. Arimilli B, Arimilli R, Chung V, Clark S, Denzel W, Drerup B, Hoefler T, Joyner J, Lewis J, Li J, Ni N, Rajamony R (2010) The PERCS High-Performance Interconnect. In: High Performance Interconnects (HOTI), 2010 IEEE 18th Annual Symposium on, pp. 75–82. doi:10.1109/HOTI.2010.16

  7. Association IT (2007) InfiniBand Architecture Specification. http://www.infinibandta.org

  8. Camarero C, Vallejo E, Beivide R (2014) Topological Characterization of Hamming and Dragonfly Networks and Its Implications on Routing. ACM Trans. Archit. Code Optim. 11(4):39:1–39:25. doi:10.1145/2677038

    Article  Google Scholar 

  9. Dally W (1992) Virtual-Channel Flow Control. IEEE Trans. on Parallel and Distributed Systems 3(2):194–205

    Article  Google Scholar 

  10. Dally W, Carvey P, Dennison L (1998) Architecture of the Avici terabit switch/router. In: 6th Hot Interconnects, pp. 41–50

  11. Dally WJ, Towles B (2003) Principles and Practices of Interconnection Networks. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA

    Google Scholar 

  12. Escudero-Sahuquillo J, García PJ, Quiles FJ, Flich J, Duato J (2011) OBQA: Smart and cost-efficient queue scheme for Head-of-Line blocking elimination in fat-trees. J. Parallel Distrib. Comput. 71(11):1460–1472

    Article  Google Scholar 

  13. Escudero-Sahuquillo J, Garcia PJ, Quiles FJ, Reinemo SA, Skeie T, Lysne O, Duato J (2014) A New Proposal to Deal with Congestion in InfiniBand-based Fat-trees. J. Parallel Distrib. Comput. 74(1):1802–1819

    Article  Google Scholar 

  14. Escudero-Sahuquillo J, Gran E, Garcia-Garcia P, Flich J, Skeie T, Lysne O, Quiles F, Duato J (2014) Efficient and Cost-Effective Hybrid Congestion Control for HPC Interconnection Networks. Parallel and Distributed Systems, IEEE Transactions on PP(99):1–1. doi:10.1109/TPDS.2014.2307851

    Google Scholar 

  15. Garcia P, Quiles F, Flich J, Duato J, Johnson I, Naven F (2006) Efficient, Scalable Congestion Management for Interconnection Networks. Micro, IEEE 26(5):52–66

    Article  Google Scholar 

  16. Gomez C, Gilabert F, Gomez M, Lopez P, Duato J (2007) Deterministic versus Adaptive Routing in Fat-Trees. In: Workshop CAC in conjunction with the IPDPS, p. 235

  17. Guay WL, Bogdanski B, Reinemo SA, Lysne O, Skeie T (2011) vFtree - A Fat-Tree Routing Algorithm Using Virtual Lanes to Alleviate Congestion. In: Proc. of IPDPS, pp. 197–208

  18. Hastings E, Rincon-Cruz D, Spehlmann M, Meyers S, Xu A, Bunde DP, Leung VJ (2015) Comparing Global Link Arrangements for Dragonfly Networks. In: Cluster Computing (CLUSTER), 2015 IEEE International Conference on, pp. 361–370. doi:10.1109/CLUSTER.2015.57

  19. Jurczyk M, Schwederski T (1996) Phenomenon of Higher Order Head-of-Line Blocking in Multistage Interconnection Networks under Nonuniform Traffic Patterns. IEICE Transactions on Information and Systems E79–D(8):1124–1129

    Google Scholar 

  20. Karol MJ, Hluchyj MG, Morgan SP (1987) Input versus output queuing on a space-division packet switch. IEEE Transactions on Communications. COM–35:1347–1356

    Article  Google Scholar 

  21. Katevenis M, Serpanos D, Spyridakis E (1998) Credit-flow-controlled ATM for MP interconnection: The ATLAS I single-chip ATM switch. In: High-Performance Computer Architecture, 1998. Proceedings., 1998 Fourth International Symposium on, pp. 47–56

  22. Kim J, Dally WJ, Scott S, Abts D (2008) Technology-Driven, Highly-Scalable Dragonfly Topology. SIGARCH Comput. Archit. News 36(3):77–88

    Article  Google Scholar 

  23. Nachiondo T, Flich J, Duato J (2010) Buffer Management Strategies to Reduce HoL Blocking. Parallel and Distributed Systems, IEEE Transactions on 21(6):739–753. doi:10.1109/TPDS.2009.63

    Article  Google Scholar 

  24. Olesinski W, Eberle H, Gura N (2009) Scalable alternatives to virtual output queueing. In: Proc. IEEE ICC, pp. 1–6

  25. OpenSim Ltd: OMNeT++ Discrete Event Simulator. http://omnetpp.org/

  26. Peir JK, Lee YH (1993) Look-ahead routing switches for multistage interconnection networks. Journal of Parallel and Distributed Computing 19(1):1–10. doi:10.1006/jpdc.1993.1085

    Article  Google Scholar 

  27. Penaranda R, Gomez C, Gomez M, Lopez P, Duato J (2012) A New Family of Hybrid Topologies for Large-Scale Interconnection Networks. In: Network Computing and Applications (NCA), 2012 11th IEEE International Symposium on, pp. 220–227

  28. Pfister G, Gusat M, Denzel W, Craddock D, Ni N, Rooney W, Engbersen T, Luijten R, Krishnamurthy R, Duato J (2005) Solving Hot Spot Contention Using InfiniBand Architecture Congestion Control. In: Proc. of Int. Workshop HPI-DC

  29. Phillips JC, Braun R, Wang W, Gumbart J, Tajkhorshid E, Villa E, Chipot C, Skeel RD, Kalé L, Schulten K (2005) Scalable molecular dynamics with NAMD. Journal of Computational Chemistry 26(16):1781–1802. doi:10.1002/jcc.20289

    Article  Google Scholar 

  30. Pinkston TM, Duato J (2006) Appendix E. In: Elsevier (ed.) Computer Architecture: A Quantitative Approach. Morgan Kaufmann Publishers

  31. Tamir Y, Frazier G (1992) Dynamically-Allocated Multi-Queue Buffers for VLSI Communication Switches. IEEE Trans. on Computers

  32. Yebenes P, Escudero-Sahuquillo J, Garcia P, Quiles F (2013) Towards Modeling Interconnection Networks of Exascale Systems with OMNet++. In: Parallel, Distributed and Network-Based Processing. doi:10.1109/PDP.2013.36

  33. Yebenes Segura P, Escudero-Sahuquillo J, Gomez Requena C, Garcia P, Quiles F, Duato J (2013) BBQ: A Straightforward Queuing Scheme to Reduce HoL-Blocking in High-Performance Hybrid Networks. In: Euro-Par 2013 Parallel Processing, vol. 8097, pp. 699–712

  34. Zahavi E, Johnson G, Kerbyson DJ, Lang M (2010) Optimized \(\text{ InfiniBand }^{{\rm TM}}\) fat-tree routing for shift all-to-all communication patterns. Journal of CCPE 22(2):217–231

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Pedro Yébenes.

Additional information

This work has been jointly supported by the Spanish MINECO and European Commission (FEDER funds) under the projects TIN2012-38341-C04 and TIN2015-66972-C5-2-R, and the FPI grant BES-2013-063681, and by Junta de Comunidades de Castilla- La Mancha under the project PEII-2014-028-P. Jesus Escudero-Sahuquillo has been funded by the Spanish MINECO under the postdoctoral grant FPDI-2013-18787 until November 2015 and, from that date, he has been funded by the University of Castilla-La Mancha (UCLM) and the European Commission (FSE funds), with a contract for accessing the Spanish System of Science, Technology and Innovation, for the implementation of the UCLM research program (UCLM resolution date: 31/07/2014).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yébenes, P., Escudero-Sahuquillo, J., García, P.J. et al. Straightforward solutions to reduce HoL blocking in different Dragonfly fully-connected interconnection patterns. J Supercomput 72, 4497–4519 (2016). https://doi.org/10.1007/s11227-016-1756-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-016-1756-1

Keywords

Navigation