Skip to main content

Advertisement

SpringerLink
Log in

An optimal big data processing for smart grid based on hybrid MDM/R architecture to strengthening RE integration and EE in datacenter

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Supply chain is a hard business area, where you need to have a perfect balance between demand and supply, day in and day out, by an intricate system that sits underneath it all. Achieving such a system by meeting the ambitious targets of the agreement on climate change can only be achieved through an effective combination of energy efficiency and renewable energy integration. The uncertainty and variability of renewable energy generation can pose challenges for grid operators and can requires additional actions to balance the system. Significant researches which aim is to improve energy efficiency of data center indicates that operating reserves could be procured from many complex and costly techniques. In this paper, we investigate the problem from scheduling of workloads in a data center in order to minimize its energy consumption budget, minimize the conventional grid dependence, and maximize the renewable energy provided to data center, by the ability to temporarily delay or degrade service, with a modified supply-following algorithm. This algorithm attempts to align power consumption with the amount of wind power available, while minimizing the time by which jobs exceed their deadlines. Modification of the algorithm has been performed in the direction of big data processing (wind trace, workload requests, prices, …), servers management. This modification is performed by jobs classification into predefined classes using the classification and regression trees algorithm. New hybrid architecture that manages the Meter Data Management Repository MDM/R was introduced using MapReduce programming model for ETL process and Massive Parallel Processing Database for requests which strongly influences the accuracy and the speediness of the scheduler.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • ABB Data Centers (2015) Adapting renewable energy to the data center. ABB article available from: https://library.e.abb.com/public/7786cac8c2874692b3164cc695e573fe/Adapting%2520Renewable%2520Energy%2520to%2520Data%2520Center.pdf

  • Abdul-Qawy ASH, Srinivasulu T (2018) SEES: a scalable and energy-efficient scheme for green IoT-based heterogeneous wireless nodes. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0758-7

    Article  Google Scholar 

  • Aissi H, Bazgan C, Vanderpooten D (2005) Complexity of the min–max and min–max regret assignment problems. Oper Res Lett 33(6):634–640

    MathSciNet  MATH  Google Scholar 

  • Aksanli B, Venkatesh J, Zhang L, Rosing T (2011) Utilizing green energy prediction to schedule mixed batch and service jobs in data centers. In: Proceedings of the 4th workshop on power-aware computing and systems. ACM, p 5

  • Al Mamoori S, Rami D, Jaekel A (2018) Energy-efficient anycast scheduling and resource allocation in optical grids. J Ambient Intell Humaniz Comput 9(1):73–83

    Google Scholar 

  • Amur H, Cipar J, Gupta V, Ganger GR, Kozuch MA, Schwan K (2010) Robust and flexible power-proportional storage. In: Proceedings of the 1st ACM symposium on cloud computing. ACM, pp 217–228

  • Bajaber F, Awan I (2010) Energy efficient clustering protocol to enhance lifetime of wireless sensor network. J Ambient Intell Humaniz Comput 1(4):239–248

    Google Scholar 

  • Bayard J, French C, Bowers S (2011) Scientific and statistical database management’. In: Proceedings of the 23rd international conference SSDBM 2011, Portland, OR, USA, July 2011

  • Black JW, de Bedout J, Tyagi R (2008) Incorporating demand resources into optimal dispatch. In: Energy 2030 conference, 2008. ENERGY 2008. IEEE, pp 1–8

  • Burkhard WA, Varaiya PP (1971) Complexity problems in real time languages. Inf Sci 3(1):87–100

    MathSciNet  MATH  Google Scholar 

  • Chen Y, Alspaugh S, Borthakur D, Katz R (2012) Energy efficiency for large-scale mapreduce workloads with significant interactive analysis. In: Proceedings of the 7th ACM European conference on computer systems. ACM, pp 43–56

  • Chen S, Zhou Z, Liu F, Li Z, Ren S (2018) CloudHeat: an efficient online market mechanism for datacenter heat harvesting. ACM Trans Model Perform Eval Comput Syst (TOMPECS) 3(3):11

    Google Scholar 

  • CPU-WORLD (2018) http://www.cpu-world.com/Compare/978/Intel_Atom_330_vs_Intel_Xeon_E5-2609_v2.html. Accessed 23 Dec 2017

  • Deng W, Liu F, Jin H, Wu C (2013a) SmartDPSS: cost-minimizing multi-source power supply for datacenters with arbitrary demand. In: 2013 IEEE 33rd international conference on distributed computing systems (ICDCS). IEEE, pp 420–429

  • Deng W, Liu F, Jin H, Wu C, Liu X (2013b) Multigreen: cost-minimizing multi-source datacenter power supply with online control. In: Proceedings of the fourth international conference on future energy systems. ACM, pp 149–160

  • Deng W, Liu F, Jin H, Li B, Li D (2014a) Harnessing renewable energy in cloud datacenters: opportunities and challenges. IEEE Netw 28(1):48–55

    Google Scholar 

  • Deng W, Liu F, Jin H, Liao X, Liu H (2014b) Reliability-aware server consolidation for balancing energy-lifetime tradeoff in virtualized cloud datacenters. Int J Commun Syst 27(4):623–642

    Google Scholar 

  • Duin CW, Volgenant A (2012) On weighting two criteria with a parameter in combinatorial optimization problems. Eur J Oper Res 221(1):1–6

    MathSciNet  MATH  Google Scholar 

  • El-Guindy A, Han D, Althoff M (2016) Formal analysis of drum-boiler units to maximize the load-following capabilities of power plants. IEEE Trans Power Syst 31(6):4691–4702

    Google Scholar 

  • Energy Information Administration (EIA)-Electricity (2018) International Energy Outlook. http://www.eia.doe.gov. Accessed 18 Jan 2018

  • Fisher ML, Jaikumar R, Van Wassenhove LN (1986) A multiplier adjustment method for the generalized assignment problem. Manag Sci 32(9):1095–1103

    MATH  Google Scholar 

  • Goiri Í, Beauchea R, Le K, Nguyen TD, Haque ME, Guitart J, Bianchini R (2011) Greenslot: scheduling energy consumption in green datacenters. In: 2011 international conference for high performance computing, networking, storage and analysis (SC). IEEE, pp 1–11

  • Goiri Í, Le K, Nguyen TD, Guitart J, Torres J, Bianchini R (2012) GreenHadoop: leveraging green energy in data-processing frameworks. In: Proceedings of the 7th ACM European conference on computer systems. ACM, pp 57–70

  • Idoudi H, Mabrouk O, Minet P, Saidane LA (2018) Cluster-based scheduling for cognitive radio sensor networks. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0670-6

    Article  Google Scholar 

  • Jackson D, Snell Q, Clement M (2001) Core algorithms of the Maui scheduler. In: Workshop on job scheduling strategies for parallel processing. Springer, Berlin, pp 87–102

  • Javaid N, Ahmad Z, Sher A, Wadud Z, Khan ZA, Ahmed SH (2018) Fair energy management with void hole avoidance in intelligent heterogeneous underwater WSNs. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0765-8

    Article  Google Scholar 

  • Kaushik RT, Bhandarkar M (2010) Greenhdfs: towards an energy-conserving, storage-efficient, hybrid hadoop compute cluster. In: Proceedings of the USENIX annual technical conference, vol 109, p 34

  • Kearney KT, Torelli F (2011) The SLA model. In: Service level agreements for cloud computing. Springer, New York, pp 43–67

  • Khan SU, Ardil C (2009) A weighted sum technique for the joint optimization of performance and power consumption in data centers. Int J Electr Comput Syst Eng 3(1):35–40

    Google Scholar 

  • Kiani AK, Ansari N (2017) On the fundamental energy trade-offs of geographical load balancing. IEEE Commun Mag 55(5):170–175

    Google Scholar 

  • Kiani SL, Anjum A, Antonopoulos N, Knappmeyer M (2014) Context-aware service utilisation in the clouds and energy conservation. J Ambient Intell Humaniz Comput 5(1):111–131

    Google Scholar 

  • Klabjan D, Nemhauser GL, Tovey C (1998) The complexity of cover inequality separation 1. Oper Res Lett 23(1–2):35–40

    MathSciNet  MATH  Google Scholar 

  • Kolici V, Herrero A, Xhafa F (2014) On the performance of oracle grid engine queuing system for computing intensive applications. JIPS 10(4):491–502

    Google Scholar 

  • Kosar T, Balman M (2009) A new paradigm: data-aware scheduling in grid computing. Future Gen Comput Syst 25(4):406–413

    Google Scholar 

  • Krioukov A, Goebel C, Alspaugh S, Chen Y, Culler DE, Katz RH (2011) Integrating renewable energy using data analytics systems: challenges and opportunities. IEEE Data Eng Bull 34(1):3–11

    Google Scholar 

  • Kumar V, Kumar A (2018) Improved network lifetime and avoidance of uneven energy consumption using load factor. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0857-5

    Article  Google Scholar 

  • Lang W, Patel JM (2010) Energy management for mapreduce clusters. Proc VLDB Endow 3(1–2):129–139

    Google Scholar 

  • Lang W, Patel JM, Shankar S (2010) Wimpy node clusters: what about non-wimpy workloads? In: Proceedings of the sixth international workshop on data management on new hardware. ACM, pp 47–55

  • Larusic J, Punnen AP (2011) The balanced traveling salesman problem. Comput Oper Res 38(5):868–875

    MathSciNet  MATH  Google Scholar 

  • Le K, Bianchini R, Martonosi M, Nguyen TD (2009) Cost-and energy-aware load distribution across data centers. In: Proceedings of hotpower, pp 1–5

  • Le K, Bianchini R, Nguyen TD, Bilgir O, Martonosi M (2010) Capping the brown energy consumption of internet services at low cost. In: Green computing conference, 2010 international. IEEE, pp 3–14

  • Liu F, Zhou Z, Jin H, Li B, Li B, Jiang H (2014) On arbitrating the power-performance tradeoff in SaaS clouds. IEEE Trans Parallel Distrib Syst 25(10):2648–2658

    Google Scholar 

  • Liu Z, Lin M, Wierman A, Low S, Andrew LL (2015) Greening geographical load balancing. IEEE ACM Trans Netw 23(2):657–671

    Google Scholar 

  • Nace D, Doan LN, Klopfenstein O, Bashllari A (2008) Max–min fairness in multi-commodity flows. Comput Oper Res 35(2):557–573

    MathSciNet  MATH  Google Scholar 

  • Niu Z, He B, Liu F (2018) Joulemr: towards cost-effective and green-aware data processing frameworks. IEEE Trans Big Data 4(2):258–272

    Google Scholar 

  • NREL (2017) National Renewable Energy Laboratory. Wind integration datasets. https://www.nrel.gov/grid/wind-integration-data.html. Accessed 08 Dec 2017

  • OGS/GE (2013) Oracle open grid scheduler/grid engine http://gridscheduler.sourceforge.net/. Accessed 08 Dec 2017

  • Özbakir L, Baykasoğlu A, Tapkan P (2010) Bees algorithm for generalized assignment problem. Appl Math Comput 215(11):3782–3795

    MathSciNet  MATH  Google Scholar 

  • Pettypool MD, Karathanos P (2004) An equity check. Eur J Oper Res 157(2):465–470

    MATH  Google Scholar 

  • Qureshi K, Shah SMH, Manuel P (2011) Empirical performance evaluation of schedulers for cluster of workstations. Clust Comput 14(2):101–113

    Google Scholar 

  • Reuther A, Byun C, Arcand W, Bestor D, Bergeron B, Hubbell M, Kepner J (2018) Scalable system scheduling for HPC and big data. J Parallel Distrib Comput 111:76–92

    Google Scholar 

  • Saha S (2012) Data center design: architecture and energy consumption. The University of Nebraska-Lincoln, Lincoln

    Google Scholar 

  • Serafini P (1987) Some considerations about computational complexity for multi objective combinatorial problems. In: Recent advances and historical development of vector optimization. Springer, Berlin, pp 222–232

  • Sijie C, Chen-Ching L (2017) From demand response to transactive energy: state of the art. J Mod Power Syst Clean Energy 5(1):10–19

    Google Scholar 

  • SPEC (2008) Standard Performance Evaluation Corporation. https://www.spec.org/power_ssj2008/. Accessed 23 Dec 2017

  • Staples G (2006) TORQUE resource manager. In: Proceedings of the 2006 ACM/IEEE conference on supercomputing. ACM, p 8

  • Su S, Zhao S (2017) A hierarchical hybrid of genetic algorithm and particle swarm optimization for distributed clustering in large-scale wireless sensor networks. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0619-9

    Article  Google Scholar 

  • Sultanpure KA, Gupta A, Reddy LSS (2018) An efficient cloud scheduling algorithm for the conservation of energy through broadcasting. Int J Electr Comput Eng (IJECE) 8(1):179–188

    Google Scholar 

  • Turner L, Punnen AP, Aneja YP, Hamacher HW (2011) On generalized balanced optimization problems. Math Methods Oper Res 73(1):19–27

    MathSciNet  MATH  Google Scholar 

  • Tyagi SKS, Zhou Y, Lin T, Marahatta A, Shi J (2018) Realization of a computational efficient BBU cluster for cloud RAN. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0995-9

    Article  Google Scholar 

  • Urgaonkar R, Urgaonkar B, Neely MJ, Sivasubramaniam A (2011) Optimal power cost management using stored energy in data centers. In: Proceedings of the ACM SIGMETRICS joint international conference on measurement and modeling of computer systems. ACM, pp 221–232

  • US EPA (2007) EPA report on server and data center energy efficiency. In: ENERGY STAR program

  • Vakilinia S (2018) Energy efficient temporal load aware resource allocation in cloud computing datacenters. J Cloud Comput 7(1):2

    Google Scholar 

  • Veni T, Bhanu S, Mary S (2013) Dynamic energy management in cloud data centers: a survey. Int J Cloud Comput Serv Archit (IJCCSA) 3(4):13

    Google Scholar 

  • Wagner F (2016) Surplus from and storage of electricity generated by intermittent sources. Eur Phys J Plus 131(12):445

    Google Scholar 

  • Wu Z, Yang K, Yang J, Cao Y, Gan Y (2018) Energy-efficiency-oriented scheduling in smart manufacturing. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-1022-x

    Article  Google Scholar 

  • Xu C, Wang K, Li P, Xia R, Guo S, Guo M (2018) Renewable energy-aware big data analytics in geodistributed data centers with reinforcement learning. IEEE Trans Netw Sci Eng. https://doi.org/10.1109/TNSE.2018.2813333

    Article  Google Scholar 

  • Yang J, Liu F (2017) Greedy discrete particle swarm optimization based routing protocol for cluster-based wireless sensor networks. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0515-3

    Article  Google Scholar 

  • Yang L, He M, Vittal V, Zhang J (2016) Stochastic optimization-based economic dispatch and interruptible load management with increased wind penetration. IEEE Trans Smart Grid 7(2):730–739

    Google Scholar 

  • Yoo AB, Jette MA, Grondona M (2003) Slurm: simple linux utility for resource management. In: Workshop on job scheduling strategies for parallel processing. Springer, Berlin, pp 44–60

  • Zhang Y, Wang Y, Wang X (2011) Greenware: greening cloud-scale data centers to maximize the use of renewable energy. In: ACM/IFIP/USENIX international conference on distributed systems platforms and open distributed processing. Springer, Berlin, pp 143–164

  • Zhou Z, Liu F, Li B, Li B, Jin H, Zou R, Liu Z (2014) Fuel cell generation in geo-distributed cloud services: a quantitative study. In: 2014 IEEE 34th international conference on distributed computing systems (ICDCS). IEEE, pp 52–61

  • Zhou Z, Liu F, Zou R, Liu J, Xu H, Jin H (2016a) Carbon-aware online control of geo-distributed cloud services. IEEE Trans Parallel Distrib Syst 27(9):2506–2519

    Google Scholar 

  • Zhou Z, Liu F, Li Z (2016b) Bilateral electricity trade between smart grids and green datacenters: pricing models and performance evaluation. IEEE J Sel Areas Commun 34(12):3993–4007

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Université Sciences Technologies Houari Boumediene, Bab Ezzouar, Algiers, Algeria

    Abdeslam Mehenni, Zaia Alimazighi & Mohamed Ahmed-Nacer

  2. Université Ferhat Abbas Sétif 1, Sétif, Algeria

    Tarek Bouktir

Authors
  1. Abdeslam Mehenni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zaia Alimazighi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tarek Bouktir
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohamed Ahmed-Nacer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abdeslam Mehenni.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mehenni, A., Alimazighi, Z., Bouktir, T. et al. An optimal big data processing for smart grid based on hybrid MDM/R architecture to strengthening RE integration and EE in datacenter. J Ambient Intell Human Comput 10, 3709–3722 (2019). https://doi.org/10.1007/s12652-018-1097-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-018-1097-4

Keywords

Search

Navigation