Dennis Abts
Michael R. Marty
ABSTRACT
Numerous studies have shown that datacenter computers rarely operate at full utilization, leading to a number of proposals for creating servers that are energy proportional with respect to the computation that they are performing.
In this paper, we show that as servers themselves become more energy proportional, the datacenter network can become a significant fraction (up to 50%) of cluster power. In this paper we propose several ways to design a high-performance datacenter network whose power consumption is more proportional to the amount of traffic it is moving -- that is, we propose energy proportional datacenter networks.
We first show that a flattened butterfly topology itself is inherently more power efficient than the other commonly proposed topology for high-performance datacenter networks. We then exploit the characteristics of modern plesiochronous links to adjust their power and performance envelopes dynamically. Using a network simulator, driven by both synthetic workloads and production datacenter traces, we characterize and understand design tradeoffs, and demonstrate an 85% reduction in power --- which approaches the ideal energy-proportionality of the network.
Our results also demonstrate two challenges for the designers of future network switches: 1) We show that there is a significant power advantage to having independent control of each unidirectional channel comprising a network link, since many traffic patterns show very asymmetric use, and 2) system designers should work to optimize the high-speed channel designs to be more energy efficient by choosing optimal data rate and equalization technology. Given these assumptions, we demonstrate that energy proportional datacenter communication is indeed possible.
- Jung Ho Ahn, Nathan Binkert, Al Davis, Moray McLaren, and Robert S. Schreiber. HyperX: topology, routing, and packaging of efficient large-scale networks. In SC '09: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, pages 1--11. ACM, 2009. Google Scholar
Digital Library
- Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat. A scalable, commodity data center network architecture. In SIGCOMM '08: Proceedings of the ACM SIGCOMM 2008 Conference on Data Communication, 2008. Google ScholarDigital Library
- Luiz André Barroso and Urs Hölzle. The case for energy-proportional computing. Computer, 40(12):33--37, 2007. Google ScholarDigital Library
- Luiz André Barroso and Urs Hölzle. The Datacenter as a Computer: An Introduction to Design of Warehouse-scale Machines. Morgan Claypool, 2009. Google ScholarDigital Library
- Borthakur, Dhruba. The Hadoop distributed file system: Architecture and design. http://hadoop.apache.org/common/docs/r0.18.0/hdfs_design.pdf.Google Scholar
- Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google file system. In 19th ACM Symposium on Operating System Principles, 2003. Google ScholarDigital Library
- Albert Greenberg, James R. Hamilton, Navendu Jain, Srikanth Kandula, Changhoon Kim, Parantap Lahiri, David A. Maltz, Parveen Patel, and Sudipta Sengupta. Vl2: a scalable and flexible data center network. In SIGCOMM '09: Proceedings of the ACM SIGCOMM 2009 Conference on Data Communication, pages 51--62, 2009. Google ScholarDigital Library
- Chamara Gunaratne, Kenneth Christensen, Bruce Nordman, and Stephen Suen. Reducing the energy consumption of ethernet with Adaptive Link Rate (ALR). IEEE Trans. Comput., 57:448--461, April 2008. Google ScholarDigital Library
- Chuanxiong Guo, Guohan Lu, Dan Li, Haitao Wu, Xuan Zhang, Yunfeng Shi, Chen Tian, Yongguang Zhang, and Songwu Lu. BCube: A high performance, server-centric network architecture for modular data centers. In SIGCOMM '09: Proceedings of the ACM SIGCOMM 2009 Conference on Data Communication, pages 63--74, 2009. Google ScholarDigital Library
- H. Hatamkhani and Yang C-K.K. A study of the optimal data rate for minimum power of I/Os. In IEEE Transactions on Circuits and Systems, pages 1230--1234, 2006.Google Scholar
- Brandon Heller, Srini Seetharaman, Priya Mahadevan, Yiannis Yiakoumis, Puneet Sharma, Sujata Banerjee, and Nick McKeown. Elastictree: Saving energy in data cneter networks. In Proceedings of the 7th USENIX Symposium on Networked System Design and Implementation (NSDI), pages 249--264. ACM, 2010. Google ScholarDigital Library
- Google Inc. Efficient computingâ step 2: efficient datacenters. http://www.google.com/corporate/green/datacenters/step2.html.Google Scholar
- Nan Jiang, John Kim, and William J. Dally. Indirect adaptive routing on large scale interconnection networks. In ISCA '09: Proceedings of the 36th annual International Symposium on Computer Architecture, pages 220--231, 2009. Google ScholarDigital Library
- John Kim, William J. Dally, and Dennis Abts. Adaptive routing in high-radix clos network. In SC '06: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, page 92, 2006. Google ScholarDigital Library
- John Kim, William J. Dally, and Dennis Abts. Flattened butterfly: a cost-efficient topology for high-radix networks. In ISCA '07: Proceedings of the 34th annual International Symposium on Computer Architecture, pages 126--137, 2007. Google ScholarDigital Library
- John Kim, William J. Dally, Brian Towles, and Amit K. Gupta. Microarchitecture of a high-radix router. In ISCA '05: Proceedings of the 32nd Annual International Symposium on Computer Architecture, pages 420--431, 2005. Google ScholarDigital Library
- Charles E. Leiserson. Fat-trees: universal networks for hardware-efficient supercomputing. IEEE Trans. Comput., 34(10):892--901, 1985. Google ScholarDigital Library
- Jian Li, Lixin Zhang, Charles Lefurgy, Richard Treumann, and Wolfgang E. Denzel. Thrifty interconnection network for hpc systems. In ICS '09: Proceedings of the 23rd International Conference on Supercomputing, pages 505--506, 2009. Google ScholarDigital Library
- Priya Mahadevan, Puneet Sharma, Sujata Banerjee, and Parthasarathy Ranganathan. Energy aware network operations. In INFOCOM'09: Proceedings of the 28th IEEE International Conference on Computer Communications Workshops, pages 25--30, 2009. Google ScholarDigital Library
- Radhika Niranjan Mysore, Andreas Pamboris, Nathan Farrington, Nelson Huang, Pardis Miri, Sivasankar Radhakrishnan, Vikram Subramanya, and Amin Vahdat. Portland: a scalable fault-tolerant layer 2 data center network fabric. SIGCOMM Comput. Commun. Rev., 39(4):39--50, 2009. Google ScholarDigital Library
- Rambus Corporation. http://www.rambus.com/us/patents/innovations/detail/low_power_multi_gbps.html.Google Scholar
- S. Scott and G. Thorson. The Cray T3E Network: Adaptive Routing in a High Performance 3D Torus. In Hot Interconnects 4, Aug. 1996.Google Scholar
- Steve Scott, Dennis Abts, John Kim, and William J. Dally. The blackwidow high-radix clos network. In ISCA '06: Proceedings of the 33rd annual International Symposium on Computer Architecture, pages 16--28, 2006. Google ScholarDigital Library
- Sun Microsystems. Lustre file system. http://www.sun.com/software/products/lustre/.Google Scholar
- U.S. Department of Energy. Average retail price of electricity. http://www.eia.doe.gov/cneaf/electricity/epm/table5_6_a.html.Google Scholar
- U.S. Environmental Protection Agency. Report to congress on server and datacenter energy efficiency. Public Law 109-431, August 2, 2007.Google Scholar
Index Terms
- Energy proportional datacenter networks
Recommendations
Towards predictable datacenter networks
SIGCOMM '11: Proceedings of the ACM SIGCOMM 2011 conferenceThe shared nature of the network in today's multi-tenant datacenters implies that network performance for tenants can vary significantly. This applies to both production datacenters and cloud environments. Network performance variability hurts ...
Energy proportional datacenter networks
ISCA '10Numerous studies have shown that datacenter computers rarely operate at full utilization, leading to a number of proposals for creating servers that are energy proportional with respect to the computation that they are performing.
In this paper, we show ...
Green spine switch management for datacenter networks
Energy consumption for datacenter has grown significantly and the trend is still growing due to the increasing popularity of cloud computing. Datacenter networks (DCNs), however, are starting to consume a greater portion of overall energy in comparison ...
Comments