Abstract
As an important part of the new infrastructure, the cloud data center is developing rapidly, and its energy consumption problem is becoming more and more prominent. Therefore, research on energy-saving technologies for cloud data centers has attracted widespread attention from academia and industry. Some studies have reviewed energy-saving optimization methods and technologies for data centers, but recently, many state-of-the-art optimization methods of energy consumption and energy-saving technologies have sprung out, which are still worth analyzing and discussing. Depending on the in-depth investigation and analysis of related research status, this article firstly focuses on analyzing and discussing the energy-saving technologies of the two components: IT equipment and cooling systems, both of which bring about the largest energy consumption in cloud data centers. As for IT equipment, its energy-saving technologies mainly include the energy saving of servers, storage systems, and network systems. While as for cooling systems, airflow organization in the computer room, thermal-aware scheduling technology, and other new energy-saving technologies are involved. Secondly, on the basis of analyzing the energy-saving technologies of the two major components, a new optimization scheme of energy consumption for the jointing computing system and cooling system is explained. Throughout this work, various energy-saving strategies and technologies have been described and compared. Finally, the future trends and development directions of energy saving for data centers are further promoted, such as integral optimization of energy consumption jointing multiple components, energy saving using artificial intelligence methods, energy saving based on novel hardware equipment, hybrid cooling energy saving, and comprehensive energy conservation with various energy technologies.








Similar content being viewed by others
References
White paper (2020) IDC:2025 China will have the world's largest data circle. http://www.d1net.com/uploadfile/2019/0214/20190214023650515.pdf. Accessed 28 June 2020.
Belkhir L, Elmeligi A (2018) Assessing ICT global emissions footprint: Trends to 2040 & recommendations. J Clean Prod 177:448–463. https://doi.org/10.1016/j.jclepro.2017.12.239
Data Center Cooling Working Group of Chinese Refrigeration Society (2018) China Data Center Annual Research Report on Cooling Technology Development. China Construction Industry
Ren C, Wang D, Urgaonkar B, Sivasubramaniam A (2012). Carbon-aware energy capacity planning for datacenters. In: 2012 IEEE 20th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, pp 391–400. doi: https://doi.org/10.1109/MASCOTS.2012.51
Johnson P, Marker T (2009) Data centre energy efficiency product profile. Equipment energy efficiency committee (E3) of the Australian Government Department of the Environment, Water, Heritage and the Arts (DEWHA), Tech. Rep, 212.
Orgerie AC, Assuncao MD, Lefevre L (2014) A survey on techniques for improving the energy efficiency of large-scale distributed systems. ACM Comput Surv (CSUR) 46(4):1–31. https://doi.org/10.1145/2532637
Beloglazov A, Buyya R, Lee YC, Zomaya A (2010) A taxonomy and survey of energy-efficient data centers and cloud computing systems. Adv Comput 82:47–111. https://doi.org/10.1016/B978-0-12-385512-1.00003-7
Salimian L, Safi F (2013) Survey of energy efficient data centers in cloud computing In: Proceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing. pp 369–374. doi: https://doi.org/10.1109/UCC.2013.76
Kheirabadi AC, Groulx D (2016) Cooling of server electronics: A design review of existing technology. Appl Therm Eng. https://doi.org/10.1016/j.applthermaleng.2016.03.056
Ni J, Bai X (2017) A review of air conditioning energy performance in data centers. Renew Sustain Energy Rev 67:625–640. https://doi.org/10.1016/j.rser.2016.09.050
Wan J, Gui X, Kasahara S, Zhang Y, Zhang R (2018) Air flow measurement and management for improving cooling and energy efficiency in raised-floor data centers: A survey. IEEE Access 6:48867–48901. https://doi.org/10.1109/ACCESS.2018.2866840
Li X, Jiang XH, Wu CH, Ke-Jiang YE (2015) Research on Thermal Management Methods for Green Data Centers. J Computer, (10):72–92. doi: https://doi.org/10.11897/SP.J.1016.2015.01976
Nadjahi C, Louahlia H, Lemasson S (2018) A review of thermal management and innovative cooling strategies for data center. Sustain Comput: Inform Syst 19:14–28. https://doi.org/10.1016/j.suscom.2018.05.002
Mastelic T, Oleksiak A, Claussen H, Brandic I, Pierson JM, Vasilakos AV (2015) Cloud computing: survey on energy efficiency. ACM Comput Surv 47(2):36. https://doi.org/10.1145/2656204
Malla S, Christensen K (2019) A Survey on Power Management Techniques for Oversubscription of Multi-Tenant Data Centers. ACM Comput Surv 52(1):1–31. https://doi.org/10.1145/3291049
Zhang W, Wen Y, Wong YW, Toh KC, Chen CH (2016) Towards Joint Optimization Over ICT and Cooling Systems in Data Centre: A Survey. IEEE Commun Surv Tutorials 18(3):1596–1616. https://doi.org/10.1109/COMST.2016.2545109
Kim KH, Beloglazov A, Buyya R (2011) Power-aware provisioning of virtual machines for real-time Cloud services. Concurr Comput: Practice Exp 23(13):1491–1505. https://doi.org/10.1002/cpe.1712
Wu CM, Chang RS, Chan HY (2014) A green energy-efficient scheduling algorithm using the DVFS technique for cloud datacenters. Futur Gener Comput Syst 37:141–147. https://doi.org/10.1016/j.future.2013.06.009
Gu L, Zeng D, Barnawi A, Guo S, Stojmenovic I (2014) Optimal task placement with QoS constraints in geo-distributed data centers using DVFS. IEEE Trans Comput 64(7):2049–2059. https://doi.org/10.1109/TC.2014.2349510
Tang Z, Qi L, Cheng Z, Li K, Khan SU, Li K (2016) An energy-efficient task scheduling algorithm in DVFS-enabled cloud environment. J Grid Comput 14(1):55–74. https://doi.org/10.1007/s10723-015-9334-y
Ge R, Feng X, Feng W, Cameron K W (2007) Cpu miser: A performance-directed, run-time system for power-aware clusters In: 2007 International Conference on Parallel Processing (ICPP 2007). IEEE, pp 18–18. doi: https://doi.org/10.1109/ICPP.2007.29
Shuja J, Gani A, Shamshirband S, Ahmad RW, Bilal K (2016) Sustainable Cloud Data Centers: A survey of enabling techniques and technologies. Renew Sustain Energy Rev. https://doi.org/10.1016/j.rser.2016.04.034
Berral J L, Goiri Í, Nou R, Julià F, Guitart J, Gavaldà R, Torres J (2010) Towards energy-aware scheduling in data centers using machine learning. In: Proceedings of the 1st International Conference on Energy-Efficient Computing and Networking. pp 215–224. doi: https://doi.org/10.1145/1791314.1791349
Duy TVT, Sato Y, Inoguchi Y (2010) Performance evaluation of a green scheduling algorithm for energy savings in cloud computing. In: 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW). IEEE, 1-8. doi: https://doi.org/10.1109/IPDPSW.2010.5470908
Beloglazov A, Abawajy J, Buyya R (2012) Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing. Futur Gener Comput Syst 28(5):755–768. https://doi.org/10.1016/j.future.2011.04.017
Meisner D, Gold BT, Wenisch TF (2009) PowerNap: eliminating server idle power. ACM SIGARCH Comput Architec News 37(1):205–216. https://doi.org/10.1145/2528521.1508269
Rong H, Zhang H, Xiao S, Li C, Hu C (2016) Optimizing energy consumption for data centers. Renew Sustain Energy Rev 58:674–691. https://doi.org/10.1016/j.rser.2015.12.283
Zhang L, Zhuang Y, Zhu W (2013) Constraint programming based virtual cloud resources allocation model. Int J Hybrid Inform Technol 6(6):333–344. https://doi.org/10.14257/ijhit.2013.6.6.30
Jiankang D, Hongbo W, Shiduan C (2015) Energy-performance tradeoffs in IaaS cloud with virtual machine scheduling. China Commun 12(002):155–166. https://doi.org/10.1109/CC.2015.7084410
Song W, Xiao Z, Chen Q, Luo H (2014) Adaptive resource provisioning for the cloud using online bin packing. IEEE Trans Comput 63(11):2647–2660. https://doi.org/10.1109/TC.2013.148
Chen M, Zhang H, Su Y Y, Wang X, Yoshihira K (2011) Effective VM sizing in virtualized data centers. In: 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops. IEEE, pp 594-601. doi:https://doi.org/10.1109/INM.2011.5990564
Xu M, Tian W, Buyya R (2017) A survey on load balancing algorithms for virtual machines placement in cloud computing. Concurr Comput: Practice Exper 29(12):e4123. https://doi.org/10.1002/cpe.4123
Ding Y, Qin X, Liang L, Wang T (2015) Energy efficient scheduling of virtual machines in cloud with deadline constraint. Futur Gener Comput Syst 50:62–74. https://doi.org/10.1016/j.future.2015.02.001
Barthwal V, Rauthan M, Verma R (2019) Virtual Machines Placement Using Predicted Utilization of Physical Machine in Cloud Datacenter. In: International Conference on Advances in Engineering Science Management & Technology (ICAESMT)-2019, Uttaranchal University, Dehradun, India.
Khoshkholghi MA, Derahman MN, Abdullah A, Subramaniam S, Othman M (2017) Energy-efficient algorithms for dynamic virtual machine consolidation in cloud data centers. IEEE Access 5:10709–10722. https://doi.org/10.1109/ACCESS.2017.2711043
Shaw R, Howley E, Barrett E (2017) An advanced reinforcement learning approach for energy-aware virtual machine consolidation in cloud data centers. In: 2017 12th International Conference for Internet Technology and Secured Transactions (ICITST). IEEE, pp. 61–66. doi: https://doi.org/10.23919/ICITST.2017.8356347
Liu Y, Sun X, Wei W, Jing W (2018) Enhancing energy-efficient and QoS dynamic virtual machine consolidation method in cloud environment. IEEE Access 6:31224–31235. https://doi.org/10.1109/ACCESS.2018.2835670
Richard E, Jim G (2020) DeepMind AI Reduces Google Data Centre Cooling Bill by 40%. https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40. Accessed 10 July 2020.
Yu W, Li X, Yang H, Huang B (2017) A multi-objective metaheuristics study on solving constrained relay node deployment problem in WSNS. Intell Autom Soft Comput. https://doi.org/10.1080/10798587.2017.1294873
Mansouri N, Zade BMH, Javidi MM (2019) Hybrid task scheduling strategy for cloud computing by modified particle swarm optimization and fuzzy theory. Comput Ind Eng 130:597–633. https://doi.org/10.1016/j.cie.2019.03.006
Amini Motlagh A, Movaghar A, Rahmani AM (2020) Task scheduling mechanisms in cloud computing: A systematic review. Int J Commun Syst 33(6):e4302. https://doi.org/10.1002/dac.4302
Ragmani A, Elomri A, Abghour N, Moussaid K, Rida M (2019) FACO: A hybrid fuzzy ant colony optimization algorithm for virtual machine scheduling in high-performance cloud computing. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-019-01631-5
Kruekaew B, Kimpan W (2020) Enhancing of Artificial Bee Colony Algorithm for Virtual Machine Scheduling and Load Balancing Problem in Cloud Computing. Int J Comput Intell Syst 13(1):496–510. https://doi.org/10.2991/ijcis.d.200410.002
Vila S, Guirado F, Lerida JL, Cores F (2019) Energy-saving scheduling on IaaS HPC cloud environments based on a multi-objective genetic algorithm. J Supercomput 75(3):1483–1495. https://doi.org/10.1007/s11227-018-2668-z
Chhabra A, Singh G, Kahlon KS (2020) QoS-Aware energy-efficient task scheduling on HPC cloud infrastructures using swarm-intelligence meta-heuristics. CMC-Comput Mater Continua 64(2):813–834
O’Connor M, Chatterjee N, Lee D, Wilson J, Agrawal A, Keckler SW, Dally WJ (2017) Fine-grained DRAM: energy-efficient DRAM for extreme bandwidth systems. In: 2017 50th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 41-54. doi:https://doi.org/10.1145/3123939.3124545
Li F, Das S, Syamala M, Narasayya V R (2016) Accelerating relational databases by leveraging remote memory and RDMA. In: Proceedings of the 2016 International Conference on Management of Data. pp 355–370. doi: https://doi.org/10.1145/2882903.2882949
Novakovic S, Daglis A, Bugnion E, Falsafi B, Grot B (2016) The case for RackOut: Scalable data serving using rack-scale systems. In: Proceedings of the Seventh ACM Symposium on Cloud Computing. pp 182–195. doi: https://doi.org/10.1145/2987550.2987577
Barthels C, Loesing S, Alonso G, Kossmann D (2015) Rack-scale in-memory join processing using RDMA. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. pp 1463–1475. doi: https://doi.org/10.1145/2723372.2750547
Nitu V, Teabe B, Tchana A, Isci C, Hagimont D (2018) Welcome to zombieland: practical and energy-efficient memory disaggregation in a datacentre. In: The Thirteenth EuroSys Conference. doi: https://doi.org/10.1145/3190508.3190537
Mann V, Kumar A, Dutta P, Kalyanaraman S (2011) VMFlow: Leveraging VM mobility to reduce network power costs in data centers. In: International Conference on Research in Networking. Springer, Berlin, Heidelberg, pp 198–211.
Masdari M, Nabavi SS, Ahmadi V (2016) An overview of virtual machine placement schemes in cloud computing. J Netw Comput Appl 66:106–127. https://doi.org/10.1016/j.jnca.2016.01.011
Zhang, Yan, Ansari, Nirwan (2012) HERO: Hierarchical energy optimization for data center networks. In: IEEE International Conference on Communications. iEEE.
Zhang Y, Ansari N (2015) HERO: hierarchical energy optimization for data center networks. IEEE Syst J 9(2):406–415. https://doi.org/10.1109/JSYST.2013.2285606
Zhou L, Bhuyan LN, Ramakrishnan KK (2019) DREAM: Distributed energy-aware traffic management for data center networks. In: Proceedings of the Tenth ACM International Conference on Future Energy Systems. pp 273–284. doi: https://doi.org/10.1145/3307772.3328291
Zhang H, Shao S, Xu H, Zou H, Tian C (2014) Free cooling of data centers: A review. Renew Sustain Energy Rev 35:171–182. https://doi.org/10.1016/j.rser.2014.04.017
Nagarathinam S, Fakhim B, Behnia M, Armfield S (2013) A comparison of parametric and multivariable optimization techniques in a raised-floor data center. J Electron Pack, 135(3). doi: https://doi.org/10.1115/1.4023214
Mulay V, Agonafer D, Irwin G, Patell D (2009) Effective thermal management of data centers using efficient cabinet designs. In: International Electronic Packaging Technical Conference and Exhibition. 43604: 993-999. doi: https://doi.org/10.1115/InterPACK2009-89351
Patankar SV, Karki KC (2004) Distribution of cooling airflow in a raised-floor data center. ASHRAE Trans 110:629–634
Karki KC, Patankar SV (2006) Airflow distribution through perforated tiles in raised-floor data centers. Build Environ 41(6):734–744. https://doi.org/10.1016/j.buildenv.2005.03.005
Zhuravlev S, Saez JC, Blagodurov S, Fedorova A, Prieto M (2012) Survey of energy-cognizant scheduling techniques. IEEE Trans Parallel Distrib Syst 24(7):1447–1464. https://doi.org/10.1109/TPDS.2012.20
Wang L, Khan SU, Dayal J (2012) Thermal aware workload placement with task-temperature profiles in a data center. Journal of Supercomputing 61(3):780–803. https://doi.org/10.1007/s11227-011-0635-z
Sun H, Stolf P, Pierson JM (2017) Spatio-temporal thermal-aware scheduling for homogeneous high-performance computing datacenters. Future Gener Comput Syst 71(jun):157–170. https://doi.org/10.1016/j.future.2017.02.005
MirhoseiniNejad SM, Moazamigoodarzi H, Badawy G, Down DG (2020) Joint data center cooling and workload management: A thermal-aware approach. Futur Gener Comput Syst 104:174–186. https://doi.org/10.1016/j.future.2019.10.040
Tuma P E (2010) The merits of open bath immersion cooling of datacom equipment. In: 2010 26th Annual IEEE Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM). IEEE, pp 123–131.https://doi.org/10.1109/STHERM.2010.5444305
Yan Z B, Duan F, Wong T N, et al. (2010) Large area spray cooling by inclined nozzles for electronic board. In: Electronics Packaging Technology Conference. IEEE. doi: https://doi.org/10.1109/EPTC.2010.5702609
Zimmermann S, Meijer I, Tiwari MK, Paredes S, Michel B, Poulikakos D (2012) Aquasar: A hot water cooled data center with direct energy reuse. Energy 43(1):237–245. https://doi.org/10.1016/j.energy.2012.04.037
Lee YJ, Singh PK, Lee PS (2015) Fluid flow and heat transfer investigations on enhanced microchannel heat sink using oblique fins with parametric study. Int J Heat Mass Transf 81:325–336. https://doi.org/10.1016/j.ijheatmasstransfer.2014.10.018
Dede EM, Liu Y (2013) Experimental and numerical investigation of a multi-pass branching microchannel heat sink. Appl Therm Eng 55(1–2):51–60. https://doi.org/10.1016/j.applthermaleng.2013.02.038
Ebrahimi K, Jones GF, Fleischer AS (2014) A review of data center cooling technology, operating conditions and the corresponding low-grade waste heat recovery opportunities. Renew Sustain Energy Rev 31:622–638. https://doi.org/10.1016/j.rser.2013.12.007
Liu J, Goraczko M, James S, Belady C, Lu J, Whitehouse K (2011) The data furnace: Heating up with cloud computing. HotCloud.
Marcinichen JB, Olivier JA, Thome JR (2012) On-chip two-phase cooling of datacenters: Cooling system and energy recovery evaluation. Appl Therm Eng 41:36–51. https://doi.org/10.1016/j.applthermaleng.2011.12.008
Atwood D, Miner JG (2008) Reducing data center cost with an air economizer. Intel Corporation, White Paper
Miller R. (2009) Microsoft's chiller-less data center. Data center knowledge. https://www.datacenterknowledge.com/archives/2009/09/24/microsofts-chiller-less-data-center
Miller R. (2009) Google's chiller-less data center. Data center knowledge. https://www.datacenterknowledge.com/archives/2009/07/15/googles-chiller-less-data-center/
Yin H, Zhu Y, Wang YL, Gao Y (2011) Effects of rapamycin on cell growth and apoptosis of pancreatic carcinoma SW1990 cells. Tumor 31(1):49–52
Bao L, Wang J, Kang L (2012) The applied effect analysis of heat exchanger installed in a typical communication base station in Beijing of China. Energy Proc 14:620–625. https://doi.org/10.1016/j.egypro.2011.12.985
Cho J, Kim Y (2016) Improving energy efficiency of dedicated cooling system and its contribution towards meeting an energy-optimized data center. Appl Energy 165:967–982. https://doi.org/10.1016/j.apenergy.2015.12.099
Li X, Garraghan P, Jiang X, Wu Z, Xu J (2017) Holistic virtual machine scheduling in cloud datacenters towards minimizing total energy. IEEE Trans Parallel Distrib Syst 29(6):1317–1331. https://doi.org/10.1109/TPDS.2017.2688445
Feng H, Deng Y, Li J (2021) A global-energy-aware virtual machine placement strategy for cloud data centers. J Syst Architect 116:102048. https://doi.org/10.1016/j.sysarc.2021.102048
Ilager S, Ramamohanarao K, Buyya R (2019) ETAS: Energy and thermal-aware dynamic virtual machine consolidation in cloud data center with proactive hotspot mitigation. Concurr Comput: Practice Exp 31(17):e5221. https://doi.org/10.1002/cpe.5221
Arroba P, Risco-Martín JL, Moya JM, Ayala JL (2018) Heuristics and metaheuristics for dynamic management of computing and cooling energy in cloud data centers. Softw: Practice Exp 48(10):1775–1804. https://doi.org/10.1002/spe.2603
Gill SS, Garraghan P, Stankovski V et al (2019) Holistic resource management for sustainable and reliable cloud computing: An innovative solution to global challenge. J Syst Softw 155:104–129. https://doi.org/10.1016/j.jss.2019.05.025
Zheng K, Zheng W, Li L, Wang X (2017) PowerNetS: Coordinating Data Center Network With Servers and Cooling for Power Optimization. IEEE Trans Netw Serv Manag 14(3):1–1. https://doi.org/10.1109/TNSM.2017.2711567
Wan J, Gui X, Zhang R, Fu L (2017) Joint cooling and server control in data centers: A cross-layer framework for holistic energy minimization. IEEE Syst J 12(3):2461–2472. https://doi.org/10.1109/JSYST.2017.2700863
Wu Qiang, Deng Qingyuan, Ganesh L, et al. (2016) Dynamo: Facebook's data center-wide power management system. In: Proceedings of the ACM/IEEE 43rd Annual International Symposium on Computer Architecture. Seoul, South Korea: IEEE, pp 469–480. doi: https://doi.org/10.1145/3007787.3001187
Gao J (2020) Machine learning applications for data center optimization. Google, http://research.google.com/pubs/pub42542.html. Accessed 26 July 2020.
Mastelic T, Brandic I (2015) Recent trends in energy-efficient cloud computing. IEEE Cloud Comput 2(1):40–47. https://doi.org/10.1109/MCC.2015.15
Pang W, Wang C, Ahuja N, et al. (2017) An advanced energy efficient rack server design, In: 2017 16th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm). IEEE. doi: https://doi.org/10.1109/ITHERM.2017.7992569
CERN Accelerating science, Data Centre. (2020) http://information-technology.web.cern.ch/about/computer-centre. [2020]. Accessed 20 July 2020.
Acknowledgements
The authors are grateful to the anonymous reviewers for their valuable comments and suggestions. This work is supported by National Natural Science Foundation of China (Grant Nos. 62072187, 61872084), Major Program and of Guangdong Basic and Applied Research (2019B030302002), Guangzhou Science and Technology Program key projects (Grant Nos. 202007040002, 201907010001).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Cheng, H., Liu, B., Lin, W. et al. A survey of energy-saving technologies in cloud data centers. J Supercomput 77, 13385–13420 (2021). https://doi.org/10.1007/s11227-021-03805-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-021-03805-5