Skip to main content

Advertisement

SpringerLink
Log in

Energy-aware processing of big data in homogeneous cluster

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The size of data centers is becoming larger to deal with the exponential data growth, and the energy consumption challenges the services providers and the environment. Various data placement strategies were developed to reduce the energy consumption of processing big data on the level of storage system, but they were typically developed for specific applications and storage medium. This paper proposes an energy-aware algorithm EABD of processing big data in homogeneous cluster with general data storage. We show that a variation of this optimization can be reduced to set cover problem, and a heuristic algorithm is proposed to reduce the energy consumption by selecting proper nodes and assigning balanced workload to each selected node. This algorithm will not be influenced by the data placement strategies and storage medium. Simulation results show that our algorithm significantly reduces energy consumption in different situations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Delforge, P.: Americas Data Centers Consuming and Wasting Growing Amounts of Energy. February 06, (2015). Available: https://www.nrdc.org/resources/americas-data-centers-consuming-and-wasting-growing-amounts-energy

  2. Sehgal, P., Tarasov, V., Zadok, E.: Optimizing energy and performance for server-class file system workloads. Trans. Storage 6(3), 10 (2010)

    Article  Google Scholar 

  3. Dayarathna, M., Wen, Y., Fan, R.: Data center energy consumption modeling: a survey. IEEE Commun. Surv. Tutor. 18(1), 732–794 (2016)

    Article  Google Scholar 

  4. Ho, C.C., Chen, H.W., Chang, Y.H., Chang, Y.M.: Energy-aware data placement strategy for SSD-assisted streaming video servers. In: Proceedings of Non-Volatile Memory Systems and Applications Symposium (NVMSA). August 20–21 (2014), pp. 1–6

  5. Zhang, L., Deng, Y., Zhu, W., Zhou, J., Wang, F.: Skewly replicating hot data to construct a power-efficient storage cluster. J. Netw. Comput. Appl. 50, 168–179 (2015)

    Article  Google Scholar 

  6. Chou, J., Kim, J., Rotem, D.: Energy-aware scheduling in disk storage systems. In: Proceedings of the 31st International Conference on Distributed Computing Systems. June 20–24 (2011), pp. 423–433

  7. Meisner, D., Gold, B.T., Wenisch, T. F.: PowerNap: eliminating server idle power. In: Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems. March 7–11 (2009), pp. 205–216

  8. Lang, W., Patel, J.M., Naughton, J.F.: On energy management, load balancing and replication. ACM SIGMOD Rec. 38(4), 35–42 (2009)

    Article  Google Scholar 

  9. Amur, H., Cipar, J., Gupta, V., Ganger, G.R., Kozuch, M.A., Schwan, K.: Robust and flexible power-proportional storage. In: Proceedings of the 1st ACM Symposium on Cloud Computing. June 10–11, pp. 217–228 (2010)

  10. Kim, J., Rotem, D.: Energy proportionality for disk storage using replication. In: Proceedings of the 14th International Conference on Extending Database Technology. March 21–24 (2011), pp. 81–92

  11. Chvatal, V.: A greedy heuristic for the set-covering problem. Math. Oper. Res. 4(3), 233–235 (1979)

  12. Agrawal, V., Kepler, N., Kidd, D.: Low power ARM Cortex-M0 CPU and SRAM using Deeply Depleted Channel (DDC) transistors with Vdd scaling and body bias. In: Proceedings of the IEEE Custom Integrated Circuits Conference. September 22–25 (2013), pp. 1–4

  13. Tanakamaru, S., Hung, C., Takeuchi, K.: Highly reliable and low power SSD using asymmetric coding and stripe bitline-pattern elimination programming. IEEE J. Solid-State Circuits 47(1), 85–96 (2012)

    Article  Google Scholar 

  14. Liao, X., Jin, H., Liu, H.: Towards a green cluster through dynamic remapping of virtual machines. Future Gener. Comput. Syst. 28(2), 469C477 (2012)

    Article  Google Scholar 

  15. Ding, Y., Qin, X., Liu, L., Wang, T.: Energy efficient scheduling of virtual machines in cloud with deadline constraint. Future Gener. Comput. Syst. 50(1), 62–74 (2015)

    Article  Google Scholar 

  16. Kaushik, R.T., Bhandarkar, M.A., Nahrstedt, K.: Evaluation and analysis of greenHDFS: a self-adaptive, energy-conserving variant of the hadoop distributed file system. In: Proceedings of IEEE Second International Conference on Cloud Computing Technology and Science. November 30, December 3 (2010), pp. 274–287

  17. Leverich, J., Kozyrakis, C.: On the energy (in)efficiency of hadoop clusters. SIGOPS Oper. Syst. Rev. 44(1), 61–65 (2010)

    Article  Google Scholar 

  18. Cardosa, M., Singh, A., Pucha, H., Chandra, A.: Exploiting spatio-temporal tradeoffs for energy-aware MapReduce in the cloud. IEEE Trans. Comput. 61(12), 1737–1751 (2012)

    Article  MathSciNet  Google Scholar 

  19. Hartog, J., Fadika, Z., Dede, E., Govindaraju, M.: Configuring a mapreduce framework for dynamic and efficient energy adaptation. In: Proceedings of IEEE 5th International Conference on Cloud Computing. June 24–29 (2012), pp. 914–921

  20. Maheshwari, N., Nanduri, R., Varma, V.: Dynamic energy efficient data placement and cluster reconfiguration algorithm for mapreduce framework. Future Gener. Comput. Syst. 28(1), 119–127 (2012)

    Article  Google Scholar 

  21. Mochocki, B., Hu, X., Quan, G.: Transition-overhead-aware voltage scheduling for fixed-priority real-time systems. ACM Trans. Des. Autom. Electron. Syst. 12(2), 11 (2007)

    Article  Google Scholar 

  22. Burd, T.D., Brodersen, R.W.: Design issues for dynamic voltage scaling. In: Proceedings of International Symposium on Low Power Electronics and Design, July 25–27, (2000), pp. 9–14

Download references

Acknowledgments

The research was supported by The National Natural Science Foundation of China (grant nos. 41301047, 61373015, 61300052), Research Fund for the Doctoral Program of High Education of China (grant no. 20103218110017), project funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions (grant no. PAPD), and Fundamental Research Funds for the Central Universities, NUAA (grant nos. NP2013307, NZ2013306).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, 29 Yudao Street, Nanjing, 210016, People’s Republic of China

    Youwei Ding, Xiaolin Qin, Qian Zhou, Liang Liu & Taochun Wang

Authors
  1. Youwei Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaolin Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qian Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Taochun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaolin Qin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ding, Y., Qin, X., Zhou, Q. et al. Energy-aware processing of big data in homogeneous cluster. SIViP 11, 371–379 (2017). https://doi.org/10.1007/s11760-016-0964-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-016-0964-8

Keywords

Search

Navigation