Abstract
Supply chain is a hard business area, where you need to have a perfect balance between demand and supply, day in and day out, by an intricate system that sits underneath it all. Achieving such a system by meeting the ambitious targets of the agreement on climate change can only be achieved through an effective combination of energy efficiency and renewable energy integration. The uncertainty and variability of renewable energy generation can pose challenges for grid operators and can requires additional actions to balance the system. Significant researches which aim is to improve energy efficiency of data center indicates that operating reserves could be procured from many complex and costly techniques. In this paper, we investigate the problem from scheduling of workloads in a data center in order to minimize its energy consumption budget, minimize the conventional grid dependence, and maximize the renewable energy provided to data center, by the ability to temporarily delay or degrade service, with a modified supply-following algorithm. This algorithm attempts to align power consumption with the amount of wind power available, while minimizing the time by which jobs exceed their deadlines. Modification of the algorithm has been performed in the direction of big data processing (wind trace, workload requests, prices, …), servers management. This modification is performed by jobs classification into predefined classes using the classification and regression trees algorithm. New hybrid architecture that manages the Meter Data Management Repository MDM/R was introduced using MapReduce programming model for ETL process and Massive Parallel Processing Database for requests which strongly influences the accuracy and the speediness of the scheduler.














Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
ABB Data Centers (2015) Adapting renewable energy to the data center. ABB article available from: https://library.e.abb.com/public/7786cac8c2874692b3164cc695e573fe/Adapting%2520Renewable%2520Energy%2520to%2520Data%2520Center.pdf
Abdul-Qawy ASH, Srinivasulu T (2018) SEES: a scalable and energy-efficient scheme for green IoT-based heterogeneous wireless nodes. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0758-7
Aissi H, Bazgan C, Vanderpooten D (2005) Complexity of the min–max and min–max regret assignment problems. Oper Res Lett 33(6):634–640
Aksanli B, Venkatesh J, Zhang L, Rosing T (2011) Utilizing green energy prediction to schedule mixed batch and service jobs in data centers. In: Proceedings of the 4th workshop on power-aware computing and systems. ACM, p 5
Al Mamoori S, Rami D, Jaekel A (2018) Energy-efficient anycast scheduling and resource allocation in optical grids. J Ambient Intell Humaniz Comput 9(1):73–83
Amur H, Cipar J, Gupta V, Ganger GR, Kozuch MA, Schwan K (2010) Robust and flexible power-proportional storage. In: Proceedings of the 1st ACM symposium on cloud computing. ACM, pp 217–228
Bajaber F, Awan I (2010) Energy efficient clustering protocol to enhance lifetime of wireless sensor network. J Ambient Intell Humaniz Comput 1(4):239–248
Bayard J, French C, Bowers S (2011) Scientific and statistical database management’. In: Proceedings of the 23rd international conference SSDBM 2011, Portland, OR, USA, July 2011
Black JW, de Bedout J, Tyagi R (2008) Incorporating demand resources into optimal dispatch. In: Energy 2030 conference, 2008. ENERGY 2008. IEEE, pp 1–8
Burkhard WA, Varaiya PP (1971) Complexity problems in real time languages. Inf Sci 3(1):87–100
Chen Y, Alspaugh S, Borthakur D, Katz R (2012) Energy efficiency for large-scale mapreduce workloads with significant interactive analysis. In: Proceedings of the 7th ACM European conference on computer systems. ACM, pp 43–56
Chen S, Zhou Z, Liu F, Li Z, Ren S (2018) CloudHeat: an efficient online market mechanism for datacenter heat harvesting. ACM Trans Model Perform Eval Comput Syst (TOMPECS) 3(3):11
CPU-WORLD (2018) http://www.cpu-world.com/Compare/978/Intel_Atom_330_vs_Intel_Xeon_E5-2609_v2.html. Accessed 23 Dec 2017
Deng W, Liu F, Jin H, Wu C (2013a) SmartDPSS: cost-minimizing multi-source power supply for datacenters with arbitrary demand. In: 2013 IEEE 33rd international conference on distributed computing systems (ICDCS). IEEE, pp 420–429
Deng W, Liu F, Jin H, Wu C, Liu X (2013b) Multigreen: cost-minimizing multi-source datacenter power supply with online control. In: Proceedings of the fourth international conference on future energy systems. ACM, pp 149–160
Deng W, Liu F, Jin H, Li B, Li D (2014a) Harnessing renewable energy in cloud datacenters: opportunities and challenges. IEEE Netw 28(1):48–55
Deng W, Liu F, Jin H, Liao X, Liu H (2014b) Reliability-aware server consolidation for balancing energy-lifetime tradeoff in virtualized cloud datacenters. Int J Commun Syst 27(4):623–642
Duin CW, Volgenant A (2012) On weighting two criteria with a parameter in combinatorial optimization problems. Eur J Oper Res 221(1):1–6
El-Guindy A, Han D, Althoff M (2016) Formal analysis of drum-boiler units to maximize the load-following capabilities of power plants. IEEE Trans Power Syst 31(6):4691–4702
Energy Information Administration (EIA)-Electricity (2018) International Energy Outlook. http://www.eia.doe.gov. Accessed 18 Jan 2018
Fisher ML, Jaikumar R, Van Wassenhove LN (1986) A multiplier adjustment method for the generalized assignment problem. Manag Sci 32(9):1095–1103
Goiri Í, Beauchea R, Le K, Nguyen TD, Haque ME, Guitart J, Bianchini R (2011) Greenslot: scheduling energy consumption in green datacenters. In: 2011 international conference for high performance computing, networking, storage and analysis (SC). IEEE, pp 1–11
Goiri Í, Le K, Nguyen TD, Guitart J, Torres J, Bianchini R (2012) GreenHadoop: leveraging green energy in data-processing frameworks. In: Proceedings of the 7th ACM European conference on computer systems. ACM, pp 57–70
Idoudi H, Mabrouk O, Minet P, Saidane LA (2018) Cluster-based scheduling for cognitive radio sensor networks. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0670-6
Jackson D, Snell Q, Clement M (2001) Core algorithms of the Maui scheduler. In: Workshop on job scheduling strategies for parallel processing. Springer, Berlin, pp 87–102
Javaid N, Ahmad Z, Sher A, Wadud Z, Khan ZA, Ahmed SH (2018) Fair energy management with void hole avoidance in intelligent heterogeneous underwater WSNs. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0765-8
Kaushik RT, Bhandarkar M (2010) Greenhdfs: towards an energy-conserving, storage-efficient, hybrid hadoop compute cluster. In: Proceedings of the USENIX annual technical conference, vol 109, p 34
Kearney KT, Torelli F (2011) The SLA model. In: Service level agreements for cloud computing. Springer, New York, pp 43–67
Khan SU, Ardil C (2009) A weighted sum technique for the joint optimization of performance and power consumption in data centers. Int J Electr Comput Syst Eng 3(1):35–40
Kiani AK, Ansari N (2017) On the fundamental energy trade-offs of geographical load balancing. IEEE Commun Mag 55(5):170–175
Kiani SL, Anjum A, Antonopoulos N, Knappmeyer M (2014) Context-aware service utilisation in the clouds and energy conservation. J Ambient Intell Humaniz Comput 5(1):111–131
Klabjan D, Nemhauser GL, Tovey C (1998) The complexity of cover inequality separation 1. Oper Res Lett 23(1–2):35–40
Kolici V, Herrero A, Xhafa F (2014) On the performance of oracle grid engine queuing system for computing intensive applications. JIPS 10(4):491–502
Kosar T, Balman M (2009) A new paradigm: data-aware scheduling in grid computing. Future Gen Comput Syst 25(4):406–413
Krioukov A, Goebel C, Alspaugh S, Chen Y, Culler DE, Katz RH (2011) Integrating renewable energy using data analytics systems: challenges and opportunities. IEEE Data Eng Bull 34(1):3–11
Kumar V, Kumar A (2018) Improved network lifetime and avoidance of uneven energy consumption using load factor. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0857-5
Lang W, Patel JM (2010) Energy management for mapreduce clusters. Proc VLDB Endow 3(1–2):129–139
Lang W, Patel JM, Shankar S (2010) Wimpy node clusters: what about non-wimpy workloads? In: Proceedings of the sixth international workshop on data management on new hardware. ACM, pp 47–55
Larusic J, Punnen AP (2011) The balanced traveling salesman problem. Comput Oper Res 38(5):868–875
Le K, Bianchini R, Martonosi M, Nguyen TD (2009) Cost-and energy-aware load distribution across data centers. In: Proceedings of hotpower, pp 1–5
Le K, Bianchini R, Nguyen TD, Bilgir O, Martonosi M (2010) Capping the brown energy consumption of internet services at low cost. In: Green computing conference, 2010 international. IEEE, pp 3–14
Liu F, Zhou Z, Jin H, Li B, Li B, Jiang H (2014) On arbitrating the power-performance tradeoff in SaaS clouds. IEEE Trans Parallel Distrib Syst 25(10):2648–2658
Liu Z, Lin M, Wierman A, Low S, Andrew LL (2015) Greening geographical load balancing. IEEE ACM Trans Netw 23(2):657–671
Nace D, Doan LN, Klopfenstein O, Bashllari A (2008) Max–min fairness in multi-commodity flows. Comput Oper Res 35(2):557–573
Niu Z, He B, Liu F (2018) Joulemr: towards cost-effective and green-aware data processing frameworks. IEEE Trans Big Data 4(2):258–272
NREL (2017) National Renewable Energy Laboratory. Wind integration datasets. https://www.nrel.gov/grid/wind-integration-data.html. Accessed 08 Dec 2017
OGS/GE (2013) Oracle open grid scheduler/grid engine http://gridscheduler.sourceforge.net/. Accessed 08 Dec 2017
Özbakir L, Baykasoğlu A, Tapkan P (2010) Bees algorithm for generalized assignment problem. Appl Math Comput 215(11):3782–3795
Pettypool MD, Karathanos P (2004) An equity check. Eur J Oper Res 157(2):465–470
Qureshi K, Shah SMH, Manuel P (2011) Empirical performance evaluation of schedulers for cluster of workstations. Clust Comput 14(2):101–113
Reuther A, Byun C, Arcand W, Bestor D, Bergeron B, Hubbell M, Kepner J (2018) Scalable system scheduling for HPC and big data. J Parallel Distrib Comput 111:76–92
Saha S (2012) Data center design: architecture and energy consumption. The University of Nebraska-Lincoln, Lincoln
Serafini P (1987) Some considerations about computational complexity for multi objective combinatorial problems. In: Recent advances and historical development of vector optimization. Springer, Berlin, pp 222–232
Sijie C, Chen-Ching L (2017) From demand response to transactive energy: state of the art. J Mod Power Syst Clean Energy 5(1):10–19
SPEC (2008) Standard Performance Evaluation Corporation. https://www.spec.org/power_ssj2008/. Accessed 23 Dec 2017
Staples G (2006) TORQUE resource manager. In: Proceedings of the 2006 ACM/IEEE conference on supercomputing. ACM, p 8
Su S, Zhao S (2017) A hierarchical hybrid of genetic algorithm and particle swarm optimization for distributed clustering in large-scale wireless sensor networks. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0619-9
Sultanpure KA, Gupta A, Reddy LSS (2018) An efficient cloud scheduling algorithm for the conservation of energy through broadcasting. Int J Electr Comput Eng (IJECE) 8(1):179–188
Turner L, Punnen AP, Aneja YP, Hamacher HW (2011) On generalized balanced optimization problems. Math Methods Oper Res 73(1):19–27
Tyagi SKS, Zhou Y, Lin T, Marahatta A, Shi J (2018) Realization of a computational efficient BBU cluster for cloud RAN. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0995-9
Urgaonkar R, Urgaonkar B, Neely MJ, Sivasubramaniam A (2011) Optimal power cost management using stored energy in data centers. In: Proceedings of the ACM SIGMETRICS joint international conference on measurement and modeling of computer systems. ACM, pp 221–232
US EPA (2007) EPA report on server and data center energy efficiency. In: ENERGY STAR program
Vakilinia S (2018) Energy efficient temporal load aware resource allocation in cloud computing datacenters. J Cloud Comput 7(1):2
Veni T, Bhanu S, Mary S (2013) Dynamic energy management in cloud data centers: a survey. Int J Cloud Comput Serv Archit (IJCCSA) 3(4):13
Wagner F (2016) Surplus from and storage of electricity generated by intermittent sources. Eur Phys J Plus 131(12):445
Wu Z, Yang K, Yang J, Cao Y, Gan Y (2018) Energy-efficiency-oriented scheduling in smart manufacturing. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-1022-x
Xu C, Wang K, Li P, Xia R, Guo S, Guo M (2018) Renewable energy-aware big data analytics in geodistributed data centers with reinforcement learning. IEEE Trans Netw Sci Eng. https://doi.org/10.1109/TNSE.2018.2813333
Yang J, Liu F (2017) Greedy discrete particle swarm optimization based routing protocol for cluster-based wireless sensor networks. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0515-3
Yang L, He M, Vittal V, Zhang J (2016) Stochastic optimization-based economic dispatch and interruptible load management with increased wind penetration. IEEE Trans Smart Grid 7(2):730–739
Yoo AB, Jette MA, Grondona M (2003) Slurm: simple linux utility for resource management. In: Workshop on job scheduling strategies for parallel processing. Springer, Berlin, pp 44–60
Zhang Y, Wang Y, Wang X (2011) Greenware: greening cloud-scale data centers to maximize the use of renewable energy. In: ACM/IFIP/USENIX international conference on distributed systems platforms and open distributed processing. Springer, Berlin, pp 143–164
Zhou Z, Liu F, Li B, Li B, Jin H, Zou R, Liu Z (2014) Fuel cell generation in geo-distributed cloud services: a quantitative study. In: 2014 IEEE 34th international conference on distributed computing systems (ICDCS). IEEE, pp 52–61
Zhou Z, Liu F, Zou R, Liu J, Xu H, Jin H (2016a) Carbon-aware online control of geo-distributed cloud services. IEEE Trans Parallel Distrib Syst 27(9):2506–2519
Zhou Z, Liu F, Li Z (2016b) Bilateral electricity trade between smart grids and green datacenters: pricing models and performance evaluation. IEEE J Sel Areas Commun 34(12):3993–4007
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mehenni, A., Alimazighi, Z., Bouktir, T. et al. An optimal big data processing for smart grid based on hybrid MDM/R architecture to strengthening RE integration and EE in datacenter. J Ambient Intell Human Comput 10, 3709–3722 (2019). https://doi.org/10.1007/s12652-018-1097-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-018-1097-4