Skip to main content
Log in

Wormhole optical network: a new architecture to solve long diameter problem in exascale computer

  • Regular Paper
  • Published:
CCF Transactions on High Performance Computing Aims and scope Submit manuscript

Abstract

The exascale computer will be built in the near future thanks to rapid innovations in semiconductor logic, memory, architectures, interconnections and other essential technologies. It is difficult to design an interconnection network that combines high performance with low power consumption. Therefore, building an interconnection network with high cost performance plays a critical role in building such a large scale system. Currently, torus-interconnected network like 6D-Torus possesses suitable properties for the petascale computer. However, the diameter within the torus-interconnected network is too long to achieve efficient global communication in the exascale computer. In addition, a direct connection method is not adaptive to the diverse characteristics of traffic. Here, we propose an architecture called Wormhole Optical Network (WON) for the exascale computer which is based on optical circuit switching. WON was designed to fully integrate into the electrical network of 6D-Torus. WON allows for the use of three novel technologies, namely the dynamic topology with optical links, algorithm of cross dimension order routing, and strategy of flow control for deadlock-free. We evaluated WON using both a prototype system and a simulator for the exascale computer. Our analysis shows that compared to the traditional electrical architecture, WON architecture reduced the time of data communication by 14–29% on exascale, a result obtained for a wide selection of diverse applications. Through enabling an SDN controller to adjust topology, WON maintains high utilization of optical links for inter-process communication from diverse applications. Further, we quantified WON’s flexibility of job deployment for mitigating hotspot traffic. We show that WON reduced latency by 20–35% in the large-range deployment and improved throughput by 30% in the long-distance deployment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Abts, D., Marty, M.R., Wells, P.M., Klausler, P., Liu, H.: Energy proportional datacenter networks. ACM SIGARCH Comput. Arch. News 38(3), 338–347 (2010)

    Article  Google Scholar 

  • Ahn, J., Fiorentino, M., Beausoleil, R.G., Binkert, N., Davis, A., Fattal, D., Jouppi, N.P., Mclaren, M., Santori, C.M., Schreiber, R.S.: Devices and architectures for photonic chip-scale integration. Appl. Phys. A 95(4), 989–997 (2009)

    Article  Google Scholar 

  • Ajima, Y., Inoue, T., Hiramoto, S., Uno, S., Sumimoto, S., Miura, K., Shida, N., Kawashima, T., Okamoto, T., Moriyama, O., et al.: Tofu interconnect 2: system-on-chip integration of high-performance interconnect. In: International Supercomputing Conference, pp. 498–507. Springer, Berlin (2014)

  • Ajima, Y., Inoue, T., Hiramoto, S., Takagi, Y., Shimizu, T.: The tofu interconnect. IEEE Micro 32(1), 21–31 (2012)

    Article  Google Scholar 

  • Arimilli, B., Arimilli, R., Chung, V., Clark, S., Denzel, W., Drerup, B., Hoefler, T., Joyner, J., Lewis, J., Li, J., et al.: The percs high-performance interconnect. In: High Performance Interconnects (HOTI), 2010 IEEE 18th Annual Symposium on, pp. 75–82, IEEE (2010)

  • Barker, K.J., Benner, A., Hoare, R., Hoisie, A., Jones, A.K., Kerbyson, D.K., Li, D., Melhem, R., Rajamony, R., Schenfeld, E., et al.: On the feasibility of optical circuit switching for high performance computing systems. In: Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p. 16, IEEE Computer Society (2005)

  • Batten, C., Joshi, A., Stojanov, V., Asanovi, K.: Designing Chip-Level Nanophotonic Interconnection Networks. Springer, New York (2013)

    Book  Google Scholar 

  • Beausoleil, R.G., Ahn, J., Spillane, S.M., Vantrease, D., Xu, Q., Binkert, N., Davis, A., Fattal, D., Fiorentino, M., Jouppi, N.P.: A nanophotonic interconnect for high-performance many-core computation. In: IEEE International Conference on Group IV Photonics, pp. 182–189 (2008)

  • Benson, T., Anand, A., Akella, A., Zhang, M.: Understanding data center traffic characteristics. ACM SIGCOMM Comput. Commun. Rev. 40(1), 92–99 (2010)

    Article  Google Scholar 

  • Binkert, N., Davis, A., Jouppi, N.P., Mclaren, M., Muralimanohar, N., Schreiber, R., Ahn, J.H.: The role of optics in future high radix switch design. ACM SIGARCH Comput. Arch. News 39(3), 437–448 (2011)

    Article  Google Scholar 

  • NERSC. Characterization of the DOE mini-apps (2019). https://portal.nersc.gov/project/CAL/doe-miniapps.htm

  • Chen, L., Chen, K., Zhu, Z., Yu, M., Porter, G., Qiao, C., Zhong, S.: Enabling wide-spread communications on optical fabric with megaswitch. In: 14th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2017, Boston, MA, USA, March 27-29, 2017, pp. 577–593 (2017)

  • Chen, K., Wen, X., Ma, X., Chen, Y.: Wavecube: a scalable, fault-tolerant, high-performance optical data center architecture. In: Computer Communications, pp. 1903–1911 (2015)

  • Chen, K., Singlay, A., Singhz, A., Ramachandranz, K., Xuz, L., Zhangz, Y., Wen, X., Chen, Y.: Osa: An optical switching architecture for data center networks with unprecedented flexibility. IEEE/ACM Trans. Netw. 22(2), 498–511 (2014)

    Article  Google Scholar 

  • Dally, W.J., Seitz, C.L.: The torus routing chip. Distrib. Comput. 1(4), 187–196 (1986)

    Article  Google Scholar 

  • Dally, W., Towles, B.: Principles and Practices of Interconnection Networks. Morgan Kaufmann, Burlington (2004)

    Google Scholar 

  • Deveci, M., Rajamanickam, S., Leung, V.J., Pedretti, K., Olivier, S.L., Bunde, D.P., Devine, K.: Exploiting geometric partitioning in task mapping for parallel computers. In: IEEE International Parallel and Distributed Processing Symposium, pp. 27–36 (2014)

  • Domke, J., Hoefler, T.: Scheduling-aware routing for supercomputers, Nov. 2016. In: Accepted at The International Conference for High Performance Computing, Networking, Storage and Analysis (SC’16) (2016)

  • Faanes, G., Bataineh, A., Roweth, D., Court, T., Froese, E., Alverson, B., Johnson, T., Kopnick, J., Higgins, M., Reinhard, J.: Cray cascade:a scalable hpc system based on a dragonfly network. In: High PERFORMANCE Computing, Networking, Storage and Analysis, pp. 1–9 (2012)

  • Fan, Z., Cao, Z., Su, Y., Liu, X., Wang, Z., Liu, X., Zang, D., An, X.: Hinetsim: a parallel simulator for large-scale hierarchical direct networks. In: IFIP International Conference on Network and Parallel Computing, pp. 120–131, Springer, Berlin (2014)

  • Farrington, N., Porter, G., Radhakrishnan, S., Bazzaz, H.H., Subramanya, V., Fainman, Y., Papen, G., Vahdat, A.: Helios: a hybrid electrical/optical switch architecture for modular data centers. ACM SIGCOMM Comput. Commun. Rev. 40(4), 339–350 (2010)

    Article  Google Scholar 

  • Ghobadi, M., Mahajan, R., Phanishayee, A., Devanur, N., Kulkarni, J., Ranade, G., Blanche, P.A., Rastegarfar, H., Glick, M., Kilper, D.: Projector: Agile reconfigurable data center interconnect. In: Conference on ACM SIGCOMM 2016 Conference, pp. 216–229 (2016)

  • Glick, M.: The role of integrated photonics in datacenter networks. In: Society of Photo-Optical Instrumentation Engineers (2017)

  • Guo, Z., Yang, Y.: Multicast fat-tree data center networks with bounded link oversubscription. In: INFOCOM, 2013 Proceedings IEEE, pp. 350–354, IEEE (2013)

  • Haring, R.: The blue gene/q compute chip. In: Hot Chips 23 Symposium (HCS), 2011 IEEE, pp. 1–20, IEEE (2011)

  • Jain, N., Bhatele, A., Howell, L.H., Bhme, D., Karlin, I., Len, E.A., Mubarak, M., Wolfe, N., Gamblin, T., Leininger, M.L.: Predicting the performance impact of different fat-tree configurations. In: International Conference for High PERFORMANCE Computing, Networking, Storage and Analysis, pp. 1–13 (2017)

  • Jain, N., Bhatele, A., Ni, X., Gamblin, T., Kale, L.V.: Partitioning low-diameter networks to eliminate inter-job interference. In: Parallel and Distributed Processing Symposium, pp. 439–448 (2017)

  • Kandula, S., Padhye, J., Bahl, P.: Flyways to de-congest data center networks. HotNets (2009)

  • Li, P., Beard, J., Buhler, J.: Deadlock-free buffer configuration for stream computing. In: International Workshop on Programming Models and Applications for Multicores and Manycores (2015)

  • Li, P., Buhler, J.: Polyhedral constraints for bounded-memory execution of synchronized filtering dataflow. In: Data-Flow Execution MODELS for Extreme Scale Computing, pp. 29–37 (2014)

  • Moon, B., Jagadish, H.V., Faloutsos, C., Saltz, J.H.: Analysis of the clustering properties of the hilbert space-filling curve. IEEE Trans. Knowl. Data Eng. 13(1), 124–141 (2001)

    Article  Google Scholar 

  • Morris, M.S., Thorne, K.S., Yurtsever, U.: Wormholes, time machines, and the weak energy condition. Phys. Rev. Lett. 61(13), 1446 (1988)

    Article  Google Scholar 

  • Pan, Y., Kumar, P., Kim, J., Memik, G., Zhang, Y., Choudhary, A.: Firefly: illuminating future network-on-chip with nanophotonics. SIGARCH Comput. Arch. News 37, 429–440 (2009)

    Article  Google Scholar 

  • Polatis Inc, Polatis 7000n Data Sheet, 2016. Rev. 7000n.012016.001 (2016)

  • Rumley, S., Calhoun, D.M., Rodrigues, A., Hammond, S.: Toward transparent optical networking in exascale computers. In: European Conference on Optical Communication, pp. 1–3 (2015)

  • Rumley, S., Glick, M., Hammond, S.D., Rodrigues, A., Bergman, K.: Design methodology for optimizing optical interconnection networks in high performance systems. In: International Conference on High Performance Computing, pp. 454–471 (2015)

  • Saha, S., Deogun, J.S., Xu, L.: Hyscale: a hybrid optical network based scalable, switch-centric architecture for data centers. In: Communications (ICC), 2012 IEEE International Conference on, pp. 2934–2938. IEEE (2012)

  • Takizawa, S., Endo, T., Matsuoka, S.: Locality aware mpi communication on a commodity opto-electronic hybrid network. In: Parallel and Distributed Processing, 2008. IPDPS 2008. IEEE International Symposium on, pp. 1–8, IEEE (2008)

  • Totoni, E., Jain, N., Kale, L.V.: Power management of extreme-scale networks with on/off links in runtime systems. ACM Trans. Parallel Comput. 1(2), 16 (2015)

    Article  Google Scholar 

  • Toyoshima, T.: Icc: An interconnect controller for the tofu interconnect architecture. In: Hot Chips, p. 22 (2010)

  • Vantrease, D., Schreiber, R., Monchiero, M., Mclaren, M., Jouppi, N.P., Fiorentino, M., Davis, A., Binkert, N., Beausoleil, R.G., Ahn, J.H.: Corona: system implications of emerging nanophotonic technology. In: International Symposium on Computer Architecture, pp. 153–164 (2008)

  • Wang, G., Andersen, D.G., Kaminsky, M., Papagiannaki, K., Ng, T., Kozuch, M., Ryan, M.: c-through: Part-time optics in data centers. In: ACM SIGCOMM Computer Communication Review, vol. 40, pp. 327–338. ACM (2010)

  • Wen, K., Samadi, P., Rumley, S., Chen, C.P., Shen, Y., Bahadori, M., Bergman, K., Wilke, J.: Flexfly: Enabling a reconfigurable dragonfly through silicon photonics. In: High Performance Computing, Networking, Storage and Analysis, SC16: International Conference for, p. 15 (2017)

  • Yu, X., Gu, H., Wang, K., Wu, G.: Enhanced fat tree-an optical/electrical hybrid interconnection for data center. In: Asia Communications and Photonics Conference, pp. AS4A–4, Optical Society of America (2012)

Download references

Acknowledgements

This work is supported in part by National Key R&D Program of China 2016YFB0200204, National Natural Science Foundation of China 61702484, and National Program on Key Research Project 2016YFB0200300.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ninghui Sun.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shao, E., Wang, Z., Yuan, G. et al. Wormhole optical network: a new architecture to solve long diameter problem in exascale computer. CCF Trans. HPC 1, 73–91 (2019). https://doi.org/10.1007/s42514-019-00006-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42514-019-00006-8

Keywords

Navigation