survey

Interconnection Networks in Petascale Computer Systems: A Survey

Authors:

Radivoje Vasiljević,

Milo Tomašević,

Veljko Milutinović,

Mateo ValeroAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 49, Issue 3

Article No.: 44, Pages 1 - 24

https://doi.org/10.1145/2983387

Published: 16 September 2016 Publication History

Abstract

This article provides background information about interconnection networks, an analysis of previous developments, and an overview of the state of the art. The main contribution of this article is to highlight the importance of the interpolation and extrapolation of technological changes and physical constraints in order to predict the optimum future interconnection network. The technological changes are related to three of the most important attributes of interconnection networks: topology, routing, and flow-control algorithms. On the other hand, the physical constraints, that is, port counts, number of communication nodes, and communication speed, determine the realistic properties of the network. We present the state-of-the-art technology for the most commonly used interconnection networks and some background related to often-used network topologies. The interconnection networks of the best-performing petascale parallel computers from past and present Top500 lists are analyzed. The lessons learned from this analysis indicate that computer networks need better performance in future exascale computers. Such an approach leads to the conclusion that a high-radix topology with optical connections for longer links is set to become the optimum interconnect for a number of relevant application domains.

References

[1]

D. Abts, A. Bataineh, S. Scott, G. Faanes, J. Schwarzmeier, E. Lundberg, T. Johnson, M. Bye, and G. Schwoerer. 2007. The cray blackwidow: A highly scalable vector multiprocessor. In Proceedings of ACM/IEEE Conference on Supercomputing. 1--12.

Digital Library

[2]

Y. Ajima, S. Sumimoto, and T. Shimizu. 2009. Tofu: A 6D mesh/torus interconnect for exascale computers. Computer 42, 11, 36--40.

Digital Library

[3]

S. R. Alam, J. A. Kuehn, R. F. Barrett, J. M. Larkin, M. R. Fahey, R. Sankaran, and P. H. Worley. 2007. Cray XT4: An early evaluation for petascale scientific simulation. In Proceedings of ACM/IEEE Conference on Supercomputing, 1--12.

Digital Library

[4]

R. Alverson, D. Roweth, and L. Kaplan. 2010. The gemini system interconnect. In Proceedings of the IEEE 18th Annual Symposium on High Performance Interconnects. 83--87.

Digital Library

[5]

B. W. Arden and H. Lee. 1982. A regular network for multicomputer systems. IEEE Trans. Comput. C-31, 1, 60--69.

Digital Library

[6]

B. Arimilli, R. Arimilli, V. Chung, S. Clark, W. Denzel, B. Drerup, T. Hoefler, J. Joyner, J. Lewis, J. Li, N. Ni, and R. Rajamony. 2010. The PERCS high-performance interconnect. In Proceedings of the 18th IEEE Symposium on High Performance Interconnects. 75--82.

Digital Library

[7]

K. J. Barker, K. Davis, A. Hoisie, D. J. Kerbyson, M. Lang, S. Pakin, and J. C. Sancho. 2008. Entering the petaflop era: The architecture and performance of roadrunner. International Conference for High Performance Computing, Networking, Storage and Analysis, 1--11.

Digital Library

[8]

A. F. Benner, M. Ignatowski, J. A. Kash, D. M. Kuchta, and M. B. Ritter. 2005. Exploitation of optical interconnects in future server architectures. IBM J. Res. Dev. 49, 4--5, 755--775.

Digital Library

[9]

R. Brightwell, W. Camp, B. Cole, E. DeBenedictis, R. Leland, J. Tomkins, and A. B. MacCabe. 2005a. Architectural specification for massively parallel computers: An experience and measurement-based approach: Research articles. Concurr. Comput. Pract. Exper. 17, 10, 1271--1316.

Digital Library

[10]

R. Brightwell, K. Pedretti, and K. D. Underwood. 2005b. Initial performance evaluation of the Cray SeaStar interconnect. Proceedings 13th Symposium on High Performance Interconnects, 51--57.

Digital Library

[11]

T. Buh, R. Trobec, and A. Ciglič. 2014. Adaptive network-traffic balancing on multi-core software networking devices. Comput. Netw. 69, 19--34.

Digital Library

[12]

J. M. Camara, M. Moreto, E. Vallejo, R. Beivide, J. Miguel-Alonso, C. Martinez, and J. Navaridas. 2010. Twisted torus topologies for enhanced interconnection networks. IEEE Trans. Parallel Distrib. Syst. 21, 12, 1765--1778.

Digital Library

[13]

P. Charles, C. Grothoff, V. Saraswat, C. Donawa, A. Kielstra, K. Ebcioglu, C. von Praun, and V. Sarkar. 2005. X10: An object-oriented approach to non-uniform cluster computing. SIGPLAN Not., 40, 10, 519--538.

Digital Library

[14]

C. Clos. 1953. A study of non-blocking switching networks. Bell Syst. Technol. J. 32, 406--424.

[15]

P. W. Coteus, J. U. Knickerbocker, C. H. Lam, and Y. A. Vlasov. 2011. Technologies for exascale systems. IBM J. Res. Dev. 55, 5, 581--592.

Digital Library

[16]

W. J. Dally and C. L. Seitz. 1987. Deadlock-free message routing in multiprocessor interconnection networks. IEEE Trans. Comput. C-36, 5, 547--553.

Digital Library

[17]

W. J. Dally and B. Towles. 2004. Principles And Practices of Interconnection Networks. Morgan Kaufmann.

Digital Library

[18]

J. J. Dongarra and M. A. Heroux. 2013. Toward a New Metric for Ranking High Performance Computing Systems. Sandia National Laboratories.

[19]

J. J. Dongarra, P. Luszczek, and A. Petitet. 2003. The LINPACK benchmark: Past, present and future. Concurr. Comput. Pract. Exper. 15, 9, 803--820.

[20]

J. Duato, S. Yalamanchili, and L. Ni. 2002. Interconnection Networks. Morgan Kaufmann.

Digital Library

[21]

M. J. Flynn, O. Mencer, V. Milutinovic, G. Rakocevic, P. Stenstrom, R. Trobec, and M. Valero. 2013. Moving from petaflops to petadata. Commun. ACM, 56, 5, 39--42.

Digital Library

[22]

J. Friedman. 2008. New views of the structure of the universe. The IPSI BgD Transactions Advanced Research, 4, 5--6.

[23]

P. Fuentes, E. Vallejo, C. Camarero, R. Beivide, and M. Valero. 2015. Throughput unfairness in dragonfly networks under realistic traffic patterns. In Proceedings of the IEEE International Conference on Cluster Computing. 801--808.

Digital Library

[24]

M. García, E. Vallejo, R. Beivide, M. Odriozola, C. Camarero, M. Valero, G. Rodríguez, J. Labarta, and C. Minkenberg. 2012. On-the-fly adaptive routing in high-radix hierarchical networks. In Proceedings of the International Conference on Parallel Processing. 279--288.

Digital Library

[25]

A. Grama, A. Gupta, V. Karypis, and V. Kumar. 2003. Introduction to Parallel Computing, 2nd ed. Pearson Education Limited, Essex, England.

[26]

T. Hoefler, T. Schneider, and A. Lumsdaine. 2008. Multistage switches are not crossbars: Effects of static routing in high-performance networks. In Proceedings of the IEEE International Conference on Cluster Computing, 116--125.

[27]

S. V. Jeffrey, R. A. Sadaf, H. D. Thomas, Jr., R. F. Mark, C. R. Philip, and H. W. Patrick. 2006. Early evaluation of the cray XT3. In Proceedings of the 20th IEEE International Parallel & Distributed Processing Symposium. 1--10.

Digital Library

[28]

D. J. Kerbyson and P. W. Jones. 2005. A performance model of the parallel ocean program. International J. High Perform. Comput. Appl. 19, 3, 261--276.

Digital Library

[29]

E. J. Kim, G. M. Link, K. H. Yum, N. Vijaykrishnan, M. Kandemir, M. J. Irwin, and C. R. Das. 2005a. A holistic approach to designing energy-efficient cluster interconnects. IEEE Trans. Comput. 54, 660--671.

Digital Library

[30]

J. Kim, W. J. Dally, S. Scott, and D. Abts. 2008. Technology-driven, highly-scalable dragonfly topology. 35th International Symposium on Computer Architecture, 77--88.

Digital Library

[31]

J. Kim, W. J. Dally, B. Towles, and A. K. Gupta. 2005b. Microarchitecture of a high radix router. In Proceedings 32nd International Symposium on Computer Architecture, 420--431.

Digital Library

[32]

C. E. Kozyrakis, S. Perissakis, D. Patterson, T. Anderson, K. Asanovic, N. Cardwell, R. Fromm, J. Golbus, B. Gribstad, K. Keeton, R. Thomas, N. Treuhaft, and K. Yelick. 1997. Scalable processors in the billion-transistor era: IRAM. Computer, 30, 9, 75--78.

Digital Library

[33]

J. Laudon and D. Lenoski. 1997. The SGI Origin: A ccNUMA highly scalable server. SIGARCH Comput. Archit. News, 25, 2, 241--251.

Digital Library

[34]

W. Lawry, C. Wilson, A. B. Maccabe, and R. Brightwell. 2002. COMB: A portable benchmark suite for assessing MPI overlap. In Proceedings of the IEEE International Conference on Cluster Computing (ICCC’02).472--475.

Digital Library

[35]

C. E. Leiserson. 1985. Fat-trees - universal networks for hardware-efficient supercomputing. IEEE Trans. Comput. 34, 10, 892--901.

Digital Library

[36]

P. Luszczek, J. J. Dongarra, D. Koester, R. Rabenseifner, B. Lucas, J. Kepner, J. McCalpin, D. Bailey, and D. Takahashi. 2005. Introduction to the HPC Challenge Benchmark Suite. Electronic Book.

[37]

V. Marjanović, J. Labarta, E. Ayguadé, and M. Valero. 2010. Overlapping communication and computation by using a hybrid MPI/SMPSs approach. In Proceedings of the 24th ACM International Conference on Supercomputing. 5--16.

Digital Library

[38]

C. Martínez, E. Vallejo, R. Beivide, C. Izu, and M. Moretó. 2006. Dense gaussian networks: suitable topologies for on-chip multiprocessors. Int. J. Parallel Program. 34, 3, 193--211.

Digital Library

[39]

Mellanox. 2013. Mellanox company site. Sunnyvale, California. Retrieved from http://www.mellanox.com.

[40]

NNSA. 2013. Advanced Simulation & Computing. National Nuclear Security Administration, USA. Retrieved from http://www.nnsa.energy.gov/asc.

[41]

M. Nüssle, H. Fröning, S. Kapferer, and U. Brüning. 2013. Accelerate communication, not computation&excl; In High-Performance Computing Using FPGAs, 507--542.

[42]

R. Peñaranda, C. Gómez, M. E. Gómez, P. López, and J. Duato. 2016. The k-ary n-direct s-indirect family of topologies for large-scale interconnection networks. J. Supercomput, 72, 1035--1062.

Digital Library

[43]

S. Scott, D. Abts, J. Kim, and W. J. Dally. 2006. The blackwidow high-radix clos network. In Proceedings of the 33rd International Symposium on Computer Architecture, 16--28.

Digital Library

[44]

G. Shainer, T. Liu, J. Liberman, J. Layton, O. Celebioglu, S. A. Schultz, J. Mora, D. Cownie, and V. Holst. 2009. LS-DYNA productivity and power-aware simulations in cluster environments. In Proceedings of the 7th European LS-DYNA Conference.

[45]

E. Stafford, J. L. Bosque, C. Martinez, F. Vallejo, R. Beivide, and C. Camarero. 2010. A first approach to king topologies for on-chip networks. In Proceedings of the 16th International Euro-Par Conference on Parallel Processing: Part II, 428--439.

Digital Library

[46]

V. Subotic, J. C. Sancho, J. Labarta, and M. Valero. 2010. A simulation framework to automatically analyze the communication-computation overlap in scientific applications. In Proceedings of the IEEE International Conference on Cluster Computing. 275--283.

Digital Library

[47]

M. A. Taubenblatt. 2012. Optical interconnects for high-performance computing. J. Lightwave Technol. 30, 448--458.

[48]

TheBlueGene/LTeam. 2002. An overview of the bluegene/l supercomputer. In Proceedings of the ACM/IEEE 2002 Conference on Supercomputing. 1--22.

Digital Library

[49]

TheBlueGene/PTeam. 2008. Overview of the IBM blue gene/p project. IBM J. Res. Dev. 52, 1.2, 199--220.

Digital Library

[50]

Top500. 2015. Top500 supercomputers site. Retrieved from http://www.top500.org.

[51]

R. Trobec. 2000. Two-dimensional regular d-meshes. Parallel Comput. 26, 13--14, 1945--1953.

Digital Library

[52]

R. Trobec. 2009. Evaluation of d-mesh interconnect for SoC. Parallel Processing Workshops, 2009. ICPPW’09. International Conference on. 507--512.

Digital Library

[53]

R. Trobec, U. Borštnik, and D. Janežič. 2009. Communication performance of d-meshes in molecular dynamics simulation. J. Math. Chem. 45, 2, 503--512.

[54]

C. Vaughan, M. Rajan, R. Barrett, D. Doerfler, and K. Pedretti. 2011. Investigating the impact of the cielo cray XE6 architecture on scientific application codes. IEEE International Symposium on Parallel and Distributed Processing, 1831--1837.

Digital Library

[55]

P. H. Worley, R. F. Barrett, and J. A. Kuehn. 2009. Early evaluation of the cray XT5. Cray User Group Conference. New York, NY.

Cited By

Korolija NŠtrbac-Savić S(2024)Merging control-flow and dataflow architectures on a single chipJournal of Computer and Forensic Sciences10.5937/jcfs3-493923:1(33-44)Online publication date: 2024
https://doi.org/10.5937/jcfs3-49392
Yan FDeng XYuan CYan BXie C(2024)On the Performance Investigation of a Recursive Fast Optical Switch-Based High Performance Computing Network ArchitectureIEEE/ACM Transactions on Networking10.1109/TNET.2023.330265032:1(777-790)Online publication date: Feb-2024
https://doi.org/10.1109/TNET.2023.3302650
Zeng YFeng GChen ZLu YXiao N(2024)ATM: Area-based Partition and Topology-aware Mapping for Large-scale SNN Simulation2024 IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA)10.1109/ISPA63168.2024.00251(1841-1848)Online publication date: 30-Oct-2024
https://doi.org/10.1109/ISPA63168.2024.00251
Show More Cited By

Index Terms

Interconnection Networks in Petascale Computer Systems: A Survey
1. Networks
  1. Network properties
    1. Network structure
  2. Network protocols

Recommendations

A multilayer nanophotonic interconnection network for on-chip many-core communications
DAC '10: Proceedings of the 47th Design Automation Conference

Multi-core chips or chip multiprocessors (CMPs) are becoming the de facto architecture for scaling up performance and taking advantage of the increasing transistor count on the chip within reasonable power consumption levels. The projected increase in ...
Cable-geometric error-prone approach for low-latency interconnection networks
CCGrid '17: Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Interconnection network is a main concern in the architecture design of highly parallel systems such as high-density data centers and supercomputers that reach millions of endpoints, e.g., 10M cores for Sunway TaihuLight system. As the number of ...
Pin Limitations and Partitioning of VLSI Interconnection Networks

Multiple processor interconnection networks can be characterized as having N' inputs and N' outputs, each being B' bits wide. A major implementation constraint of large networks in the VLSI environment is the number of pins available on a chip, Np. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 49, Issue 3

September 2017

658 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/2988524

Editor:
Sartaj Sahni
Department of Computer and Information Science and Engineering/University of Florida/Gainesville, FL

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 September 2016

Accepted: 01 July 2016

Revised: 01 July 2016

Received: 01 November 2015

Published in CSUR Volume 49, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey
Research
Refereed

Funding Sources

Slovenian Research Agency

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
825
Total Downloads

Downloads (Last 12 months)48
Downloads (Last 6 weeks)4

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Korolija NŠtrbac-Savić S(2024)Merging control-flow and dataflow architectures on a single chipJournal of Computer and Forensic Sciences10.5937/jcfs3-493923:1(33-44)Online publication date: 2024
https://doi.org/10.5937/jcfs3-49392
Yan FDeng XYuan CYan BXie C(2024)On the Performance Investigation of a Recursive Fast Optical Switch-Based High Performance Computing Network ArchitectureIEEE/ACM Transactions on Networking10.1109/TNET.2023.330265032:1(777-790)Online publication date: Feb-2024
https://doi.org/10.1109/TNET.2023.3302650
Zeng YFeng GChen ZLu YXiao N(2024)ATM: Area-based Partition and Topology-aware Mapping for Large-scale SNN Simulation2024 IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA)10.1109/ISPA63168.2024.00251(1841-1848)Online publication date: 30-Oct-2024
https://doi.org/10.1109/ISPA63168.2024.00251
Dong QZhao J(2024)The diameter of rectangular twisted torusTheoretical Computer Science10.1016/j.tcs.2024.1146101003(114610)Online publication date: Jul-2024
https://doi.org/10.1016/j.tcs.2024.114610
Korolija N(2024)Drawbacks of Programming Dataflow Architectures and Methods to Overcome ThemApplied Artificial Intelligence 2: Medicine, Biology, Chemistry, Financial, Games, Engineering10.1007/978-3-031-60840-7_9(57-70)Online publication date: 25-May-2024
https://doi.org/10.1007/978-3-031-60840-7_9
Yang JFahad AMukhtar MAnees MShahzad AIqbal Z(2023)Complexity Analysis of Benes Network and Its Derived Classes via Information Functional Based EntropiesSymmetry10.3390/sym1503076115:3(761)Online publication date: 20-Mar-2023
https://doi.org/10.3390/sym15030761
Lu PLai MChang J(2022)A Survey of High-Performance Interconnection Networks in High-Performance Computer SystemsElectronics10.3390/electronics1109136911:9(1369)Online publication date: 25-Apr-2022
https://doi.org/10.3390/electronics11091369
Kotlar MPunt MMilutinović V(2022)Energy efficient implementation of tensor operations using dataflow paradigm for machine learning10.1016/bs.adcom.2021.11.011(151-199)Online publication date: 2022
https://doi.org/10.1016/bs.adcom.2021.11.011
Mohtavipour SShahhoseini H(2022)An analytically derived vectorized model for application graph mapping in interconnection networksJournal of Ambient Intelligence and Humanized Computing10.1007/s12652-021-03637-414:7(8899-8911)Online publication date: 24-Jan-2022
https://doi.org/10.1007/s12652-021-03637-4
Milutinović VKotlar MRatković IKorolija NDjordjevic MYoshimoto KValero M(2021)The Ultimate Data Flow for Ultimate Super Computers-on-a-ChipHandbook of Research on Methodologies and Applications of Supercomputing10.4018/978-1-7998-7156-9.ch021(312-318)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-7156-9.ch021
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents