Skip to main content
SpringerLink
Log in

Detailed and clock-driven simulation for HPC interconnection network

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Performance and energy consumption of high performance computing (HPC) interconnection networks have a great significance in the whole supercomputer, and building up HPC interconnection network simulation platform is very important for the research on HPC software and hardware technologies. To effectively evaluate the performance and energy consumption of HPC interconnection networks, this article designs and implements a detailed and clock-driven HPC interconnection network simulation platform, called HPC-NetSim. HPC-NetSim uses applicationdriven workloads and inherits the characteristics of the detailed and flexible cycle-accurate network simulator. Besides, it offers a large set of configurable network parameters in terms of topology and routing, and supports router’s on/off states.We compare the simulated execution time with the real execution time of Tianhe-2 subsystem and the mean error is only 2.7%. In addition, we simulate the network behaviors with different network structures and low-power modes. The results are also consistent with the theoretical analyses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Dongarra J J, Meuer HW, Strohmaier E. TOP500 supercomputer sites. Supercomputer, 1997, 13: 89–111

    Google Scholar 

  2. Pang Z B, Xie M, Zhang J, Zheng Y, Wang G B, Dong D Z, Suo G. The TH Express high performance interconnect networks. Frontiers of Computer Science, 2014, 8(3): 357–366

    Article  MathSciNet  Google Scholar 

  3. Raponi P G, Petrini F, Walkup R, Checconi F. Characterization of the communication patterns of scientific applications on Blue Gene/P. In: Proceedings of 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum (IPDPSW). 2011: 1017–1024

    Google Scholar 

  4. Kogge P M. Architectural challenges at the exascale frontier (invited talk). Simulating the Future: Using One Million Cores and Beyond, 2008

    Google Scholar 

  5. Abts D, Marty MR, Wells PM, Klausler P, Liu H. Energy proportional datacenter networks. In: Proceedings of the 37th Annual International Symposium on Computer Architecture. 2010, 338–347

    Google Scholar 

  6. Shalf J, Dosanjh S, Morrison J. Exascale Computing Technology Challenges. In: Palma JMLM, Daydé M, Marques O, Lopes J C, eds. High Performance Computing for Computational Science lCVECPAR 2010. Berkeley, CA: Springer Berlin Heidelberg, 2011, 1–25

    Chapter  Google Scholar 

  7. Alonso M, Coll S, Martinez J M, Santonja V, Duato J. Dynamic power saving in fat-tree interconnection networks using on/off links. In: Pro ceedings of the 20th International. IEEE Parallel and Distributed Processing Symposium. 2006

    Google Scholar 

  8. Raghunathan V, Srivastava M B, Gupta R K. A survey of techniques for energy ecient on-chip communication. In: Proceedings of the 40th Annual Design Automation Conference. 2003, 900–905

    Google Scholar 

  9. Deveci M, Rajamanickam S, Leung V J, Pedretti K, Olivier S L, Bunde D P, Çatalyürek U V, Devine K. Exploiting geometricpartitioning in task mapping for parallel computers. In: Proceedings of the 28th International IEEE Parallel and Distributed Processing Symposium. 2014, 27–36

    Google Scholar 

  10. Zhang P, Gao Y, Fierson J, Deng Y F. Eigenanalysis-based task mapping on parallel computers with cellular networks. Mathematics of Computation, 2014, 83(288): 1727–1756

    Article  MathSciNet  MATH  Google Scholar 

  11. Jiang N, Balfour J, Becker D U, Towies B, Dally W J, Michelogiannakis G, Kim J. A detailed and flexible cycle-accurate Network-on- Chip simulator. In: Proceedings of 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 2013, 86–96

    Chapter  Google Scholar 

  12. Agarwal N, Krishna T, Peh L S, Jha N K. GARNET: A detailed onchipnetwork model inside a full-system simulator. In: Proceedings of 2009 IEEE International Symposium on Performance Analysis of Systems and Software. 2009, 33–42

    Chapter  Google Scholar 

  13. Zhai J D, Chen W G, Zheng W M. PHANTOM: predicting performance of parallel applications on large-scale parallel machines using a single node. In: Proceedings of the 15th ACM SIGPLAN Symposium on Principles of Parellel Programming. 2010, 305–314

    Google Scholar 

  14. Denzel WE, Li J, Walker P, Jin Y. A framework for end-to-end simulation of high-performance computing systems. Simulation, 2010, 86(5–6): 331–350

    Article  Google Scholar 

  15. Wilke J J, Kenny J P. Using discrete event simulation for programming model exploration at extreme-scale: macroscale components for the structural simulation toolkit (SST). Sandia Report SAND2015-1027, Sandia National Laboratories, 2015

    Google Scholar 

  16. Binkert N, Beckmann B, Black G, Reinhardt S T, Saidi A, Basu A, Hestness J, Hower D R, Krishna T, Sardashti S, Sen R, Sewell K, Shoaib M, Vaish N, Hill M D, Wood D A. The gem5 simulator. ACMSIGARCH Computer Architecture News, 2011, 39(2): 1–7

    Article  Google Scholar 

  17. Peno B, Wagner A, Tuxen M, Rüngeler I. MPI-NeTSim: a network simulation module for MPI. In: Proceedings of the 15th IEEE International Conference on Parallel and Distributed Systems (ICPADS). 2009, 464–471

    Google Scholar 

  18. Zheng G, Kakulapati G, Kale L V. BigSim: a parallel simulator for performance prediction of extremely large parallel machines. In: Proceedings of the 18th International IEEE Parallel and Distributed Processing Symposium. 2004

    Google Scholar 

  19. Dally WJ, Towles B P. Principles and Practices of Interconnection Networks. San Francisco, CA: Elsevier, 2004

    Google Scholar 

  20. Culler D, Karp R, Patterson D, Sahay A, Schauser K E, Santos E, Subramonian R, von Eicken T. LogP: towards a realistic model of parallel computation. In: Proceedings of the 4th ACMSIGPLAN Symposium on Principles and Practice of Parallel Programming. 1993, 1–12

    Google Scholar 

  21. Alexandrov A, Ionescu MF, Schauser K E, Scheiman C. LogGP: incorporating long messages into the LogP modela—one step closer towards a realisticmodel for parallel computation. In: Proceedings of the 7th Annual ACM symposium on Parallel Algorithms and Architectures. 1995, 95–105

    Google Scholar 

  22. Moritz C A, Frank M I. LoGPC: Modeling network contention in message-passing programs. In: Proceedings of the 1998 ACM SIGMETRICS Joint International Conference on Measurement and Modeling of Computer Systems. 1998, 254–263

    Google Scholar 

  23. Chen W G, Zhai J D, Zhang J, Zheng W M. LogGPO: an accurate communication model for performance prediction of MPI programs. Science in China Series F: Information Sciences, 2009, 52(10): 1785–1791

    Article  MATH  Google Scholar 

  24. Liao X K, Xiao L Q, Yang C Q, Lu Y T. MilkyWay-2 supercomputer: system and application. Frontiers of Computer Science, 2014, 8(3): 345–356

    Article  MathSciNet  Google Scholar 

  25. Kelton W D, Law A M. Simulation Modeling and Analysis. Boston: McGraw Hill, 2000

    MATH  Google Scholar 

  26. Varga A. The OMNeT++ discrete event simulation system. In: Proceedings of the European Simulation Multiconference. 2001

    Google Scholar 

  27. Gropp W. MPICH2: a new start for MPI implementations. In: Kranzlmüller D, Volkert J, Kacsuk P, Dongarra J eds. Recent Advances in Parallel Virtual Machine and Message Passing Interface.Springer Berlin Heidelberg, 2002: 7

  28. Gabriel E, Fagg G E, Bosilca G, Angskun T, Dongrra J J, Squyres J M, Sahay V, Kambadur P, Barrett B, Lumsdaine A, Castain R H, Daniel D J, Graham R L, Woodall T S. Open MPI: goals, concept, and design of a next generation MPI implementation. In: Kranzlmüller D, Kacsuk P, Dongarra J, eds. Recent Advances in Parallel Virtual Machine and Message Passing Interface. Springer Berlin Heidelberg, 2004: 97–104

    Chapter  Google Scholar 

  29. Kim M S, Son D M, Ko Y B, Kim Y H. A simulation study of the PLC-MAC performance using network simulator-2. In: Proceedings of 2008 IEEE International Symposium on Power Line Communications and Its Applications. 2008, 99–104

    Google Scholar 

  30. Vetter J S, Mueller F. Communication characteristics of large-scale scientific applications for contemporary cluster architectures. Journal of Parallel and Distributed Computing, 2003, 63(9): 853–865

    Article  MATH  Google Scholar 

  31. Becker D, Wolf F, Frings W, Geimer M, Wylie B J N, Mohr B. Automatic trace-based performance analysis of metacomputing applications. In: Proceedings of 2007 IEEE International Parallel and Distributed Processing Symposium. 2007, 1–10

    Chapter  Google Scholar 

  32. Nagel W E, Arnold A, Weber M, Hoppe H S, Solchenbach K. VAMPIR: visualization and analysis of MPI resources. MD5 SHA512, 1996

    Google Scholar 

  33. Mohr B, Wolf F. KOJAK—a tool set for automatic performance analysis of parallel programs. In: Kosch H, Böszörményi L, Hellwagner H, eds. Euro-Par 2003 Parallel Processing. Springer Berlin Heidelberg, 2003: 1301–1304

    Chapter  Google Scholar 

  34. Shende S S, Malony A D. The Tau parallel performance system. International Journal of High Performance Computing Applications, 2006, 20(2): 287–311

    Article  Google Scholar 

  35. O’Carroll F, Tezuka H, Hori A, Ishikawa Y. The design and implementation of zero copy MPI using commodity hardware with a high performance network. In: Proceedings of the 12th ACM International Conference on Supercomputing. 1998, 243–250

    Chapter  Google Scholar 

  36. PadovanoM. System and method for accessing a storage area network as network attached storage. US Patent, 6606690, 2003-08-12

  37. Hamada T, Nakasato N. InfiniBand Trade Association, InfiniBand architecture specification: release 1.0. In: Proceedings of 2005 International Conference on Field Programmable Logic and Applications. 2005

    Google Scholar 

  38. Xie M, Lu Y, Wang K F, Liu L, Cao H J, Yang X J. Tianhe-1A interconnect and message-passing services. IEEE Micro, 2011 (1): 8–20

    Google Scholar 

  39. Wu J, Liao X K, Dong D Z, Wang L, Li C L. HVCRouter: energy ecient networkon-chip router with heterogeneous virtual channels. In: Wang G J, Zomaya A, Perez G M, Li K L, EDS. Algorithms and Architectures for Parallel Processing. Springer International Publishing, 2015: 199–213

    Chapter  Google Scholar 

  40. Ma S, Jerger N E, Wang Z Y. DBAR: an effcient routing algorithm to support multiple concurrent applications in networks-on-chip. In: Proceedings of the 38th Annual International Symposium on Computer Architecture (ISCA). 2011, 413–424

    Google Scholar 

  41. Chen J, Zhou W, Ben C. Supremum of idle routers on 2D-mesh with dimension-order routing. Journal of Computational Information Systems, 2014, 10(22): 9897–9906

    Google Scholar 

  42. Lusk E, Huss S, Saphir B, Snir M. MPI: a message-passing interface standard. 2009

    Google Scholar 

  43. Li J, Huang W, Lefurgy C, Zhang L X, Denzel W E, Treumann R R, Wang K. Power shifting in thrifty interconnection network. In: Proceedings of the 17th IEEE International Symposium on High Performance Computer Architecture (HPCA). 2011, 156–167

    Google Scholar 

  44. Liao X K. MilkyWay-2: back to the world Top 1. Frontiers of Computer Science, 2014, 8(3): 343–344

    Article  MathSciNet  Google Scholar 

  45. Bailey D H, Barszcz E, Barton J T, Browning D S, Carter R L, Dagum L, Fatoohi R A, Frederickson P O, Lasinski T A, Schreiber R S, Simon H D, Venkatakrishnan V, Weeratunga S K. The NAS parallel benchmarks. International Journal of High Performance Computing Applications, 1991, 5(3): 63–73

    Article  Google Scholar 

  46. Initiative A S C. The ASCI sweep3d benchmark code. 1995

    Google Scholar 

  47. Velho P, Legrand A. Accuracy study and improvement of network simulation in the SimGrid framework. In: Proceedings of the 2nd International Conference on Simulation Tools and Techniques. 2009, 13

    Google Scholar 

  48. Tabe T B, Stout Q F. The use of the MPI communication library in the NAS parallel benchmarks. Ann Arbor, 1999(1001): 48109

    Google Scholar 

  49. Matsutani H, Koibuchi M, Wang D, Amano H. Run-time power gating of on-chip routers using look-ahead routing. In: Proceedings of the 2008 Asia and South Pacific Design Automation Conference. 2008, 55–60

    Chapter  Google Scholar 

  50. Mihic K, Simunic T, De Micheli G. Reliability and power management of integrated systems. In: Proceedings of 2004 IEEE Euromicro Symposium on Digital System Design. 2004, 5–11

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of High Performance Computing, School of Computer, National University of Defense Technology, Changsha, 410073, China

    Wenhao Zhou, Juan Chen, Chen Cui, Qian Wang & Yuhua Tang

  2. Science and Technology on Parallel and Distributed Processing Laboratory, National University of Defense Technology, Changsha, 410073, China

    Dezun Dong

Authors
  1. Wenhao Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chen Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dezun Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yuhua Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Chen.

Additional information

Wenhao Zhou received the BS and MS degrees in the School of Computer, National University of Defense Technology, China in 2013 and 2015. His research interests focus on energy-aware HPC interconnection networks and parallel software framework.

Juan Chen received the PhD degree in Computer Department, National University of Defense Technology (NUDT), China in 2007. She is now an associate professor in Key Laboratory of High Performance Computing at NUDT. Her research interests focus on supercomputer systems, energy-aware interconnection network design, and parallel software framework.

Chen Cui received the BS degree in the School of Electronics Engineering and Computer Science at Peking University, China in 2015, and now he is a MS student at National University of Defense Technology, China. His research interests focus on the large scale parallel numerical simulation and parallel software framework.

Qian Wang received the BS degree in the School of Computer at National University of Defense Technology (NUDT), China in 2011, and now is a PhD student at NUDT. Her research interests focus on the large scale parallel numerical simulation and parallel software framework.

Dezun Dong received the BS, MS, and PhD degrees from the National University of Defense Technology (NUDT), China in 2002, 2004 and 2010, respectively. He is an associate professor in the Collage of Computer, NUDT. His research interests range across high performance computer systems, high speed interconnect networks, wireless networks, and distributed computing algorithms. Currently, he focuses on performance evaluation of high-performance interconnection networks for supercomputers and data centers. He is a member of the ACM, IEEE, and CCF.

Yuhua Tang received her BS and MS degrees in Computer Department from National University of Defense Technology (NUDT), China in 1983 and 1986, respectively. She is now a professor in National Laboratory for Paralleling and Distributed Processing at NUDT. Her research interests include supercomputer architecture and core router’s design.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, W., Chen, J., Cui, C. et al. Detailed and clock-driven simulation for HPC interconnection network. Front. Comput. Sci. 10, 797–811 (2016). https://doi.org/10.1007/s11704-016-5035-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-016-5035-3

Keywords

Search

Navigation