Skip to main content

Advertisement

Log in

MapReduce framework energy adaptation via temperature awareness

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

MapReduce has become a popular framework for Big Data applications. While MapReduce has received much praise for its scalability and efficiency, it has not been thoroughly evaluated for power consumption. Our goal with this paper is to explore the possibility of scheduling in a power-efficient manner without the need for expensive power monitors on every node. We begin by considering that no cluster is truly homogeneous with respect to energy consumption. From there we develop a MapReduce framework that can evaluate the current status of each node and dynamically react to estimated power usage. In so doing, we shift work toward more energy efficient nodes which are currently consuming less power. Our work shows that given an ideal framework configuration, certain nodes may consume only 62.3 % of the dynamic power they consumed when the same framework was configured as it would be in a traditional MapReduce implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)

    Article  Google Scholar 

  2. Apache hadoop. Online available: http://hadoop.apache.org

  3. Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R.H., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: OSDI, pp. 29–42 (2008)

    Google Scholar 

  4. Xie, J., Yin, S., Ruan, X., Ding, Z., Tian, Y., Majors, J., Manzanares, A., Qin, X.: Improving mapreduce performance through data placement in heterogeneous hadoop clusters. In: IPDPS Workshops, pp. 1–9 (2010)

    Google Scholar 

  5. Kaushik, R.T., Bhandarkar, M.: Greenhdfs: towards an energy-conserving, storage-efficient, hybrid hadoop compute cluster. In: Proceedings of the 2010 International Conference on Power Aware Computing and Systems, Vancouver, BC, Canada, pp. 1–9 (2010)

    Google Scholar 

  6. Leverich, J., Kozyrakis, C.: On the energy (in)efficiency of hadoop clusters. Oper. Syst. Rev. 44(1), 61–65 (2010)

    Article  Google Scholar 

  7. Lang, W., Patel, J.M.: Energy management for mapreduce clusters. Proc. VLDB Endow. 3(1), 129–139 (2010)

    Google Scholar 

  8. Fadika, Z., Dede, E., Govindaraju, M., Ramakrishnan, L.: Mariane: mapreduce implementation adapted for HPC environments. In: IEEE/ACM International Workshop on Grid Computing, vol. 12 (2011)

    Google Scholar 

  9. Chen, Y., Keys, L., Katz, R.H.: Towards energy efficient mapreduce. Electrical Engineering and Computer Science Department, University of California at Berkeley, Tech. rep. UCB/EECS-2009-109 (2009)

  10. Wirtz, T., Ge, R.: Improving mapreduce energy efficiency for computation intensive workloads. In: 2011 International Green Computing Conference and Workshops (IGCC), pp. 1–8 (2011)

    Chapter  Google Scholar 

  11. Barroso, L.A., Hölzle, U.: The case for energy-proportional computing. Computer 40(12), 33–37 (2007)

    Article  Google Scholar 

  12. Li, S., Abdelzaher, T., Yuan, M.: Tapa: temperature aware power allocation in data center with map-reduce. In: 2011 International Green Computing Conference and Workshops (IGCC), pp. 1–8. IEEE Press, New york (2011)

    Chapter  Google Scholar 

  13. PrimeNet benchmarks (GIMPS). Online available: http://www.mersenne.org/

  14. Schatz, M.C.: Cloudburst: highly sensitive read mapping with mapreduce. Bioinformatics 25(11), 1363–1369 (2009)

    Article  Google Scholar 

  15. Fadika, Z., Dede, E., Hartog, J., Govindaraju, M.: Marla: mapreduce for heterogeneous clusters. In: IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, vol. 12 (2012)

    Google Scholar 

  16. Chen, Y., Ganapathi, A.S., Fox, A., Katz, R.H., Patterson, D.A.: Statistical workloads for energy efficient mapreduce. Electrical Engineering and Computer Science Department, University of California at Berkeley, Tech. rep. UCB/EECS-2010-6 (2010)

  17. National energy research scientific computing center. http://www.nersc.gov

  18. Lm-sensors—Linux hardware monitoring. Online available: http://lm-sensors.org/

  19. Kosar, T., Livny, M.: A framework for reliable and efficient data placement in distributed computing systems. J. Parallel Distrib. Comput. 65(10), 1146–1157 (2005)

    Article  Google Scholar 

  20. Zong, Z., Briggs, M., O’Connor, N., Qin, X.: An energy-efficient framework for large-scale parallel storage systems. In: Parallel and Distributed Processing Symposium, pp. 1–7 (2007)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jessica Hartog.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hartog, J., Dede, E. & Govindaraju, M. MapReduce framework energy adaptation via temperature awareness. Cluster Comput 17, 111–127 (2014). https://doi.org/10.1007/s10586-013-0270-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-013-0270-y

Keywords

Navigation