Abstract
MapReduce has become a popular framework for Big Data applications. While MapReduce has received much praise for its scalability and efficiency, it has not been thoroughly evaluated for power consumption. Our goal with this paper is to explore the possibility of scheduling in a power-efficient manner without the need for expensive power monitors on every node. We begin by considering that no cluster is truly homogeneous with respect to energy consumption. From there we develop a MapReduce framework that can evaluate the current status of each node and dynamically react to estimated power usage. In so doing, we shift work toward more energy efficient nodes which are currently consuming less power. Our work shows that given an ideal framework configuration, certain nodes may consume only 62.3 % of the dynamic power they consumed when the same framework was configured as it would be in a traditional MapReduce implementation.
Similar content being viewed by others
References
Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Apache hadoop. Online available: http://hadoop.apache.org
Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R.H., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: OSDI, pp. 29–42 (2008)
Xie, J., Yin, S., Ruan, X., Ding, Z., Tian, Y., Majors, J., Manzanares, A., Qin, X.: Improving mapreduce performance through data placement in heterogeneous hadoop clusters. In: IPDPS Workshops, pp. 1–9 (2010)
Kaushik, R.T., Bhandarkar, M.: Greenhdfs: towards an energy-conserving, storage-efficient, hybrid hadoop compute cluster. In: Proceedings of the 2010 International Conference on Power Aware Computing and Systems, Vancouver, BC, Canada, pp. 1–9 (2010)
Leverich, J., Kozyrakis, C.: On the energy (in)efficiency of hadoop clusters. Oper. Syst. Rev. 44(1), 61–65 (2010)
Lang, W., Patel, J.M.: Energy management for mapreduce clusters. Proc. VLDB Endow. 3(1), 129–139 (2010)
Fadika, Z., Dede, E., Govindaraju, M., Ramakrishnan, L.: Mariane: mapreduce implementation adapted for HPC environments. In: IEEE/ACM International Workshop on Grid Computing, vol. 12 (2011)
Chen, Y., Keys, L., Katz, R.H.: Towards energy efficient mapreduce. Electrical Engineering and Computer Science Department, University of California at Berkeley, Tech. rep. UCB/EECS-2009-109 (2009)
Wirtz, T., Ge, R.: Improving mapreduce energy efficiency for computation intensive workloads. In: 2011 International Green Computing Conference and Workshops (IGCC), pp. 1–8 (2011)
Barroso, L.A., Hölzle, U.: The case for energy-proportional computing. Computer 40(12), 33–37 (2007)
Li, S., Abdelzaher, T., Yuan, M.: Tapa: temperature aware power allocation in data center with map-reduce. In: 2011 International Green Computing Conference and Workshops (IGCC), pp. 1–8. IEEE Press, New york (2011)
PrimeNet benchmarks (GIMPS). Online available: http://www.mersenne.org/
Schatz, M.C.: Cloudburst: highly sensitive read mapping with mapreduce. Bioinformatics 25(11), 1363–1369 (2009)
Fadika, Z., Dede, E., Hartog, J., Govindaraju, M.: Marla: mapreduce for heterogeneous clusters. In: IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, vol. 12 (2012)
Chen, Y., Ganapathi, A.S., Fox, A., Katz, R.H., Patterson, D.A.: Statistical workloads for energy efficient mapreduce. Electrical Engineering and Computer Science Department, University of California at Berkeley, Tech. rep. UCB/EECS-2010-6 (2010)
National energy research scientific computing center. http://www.nersc.gov
Lm-sensors—Linux hardware monitoring. Online available: http://lm-sensors.org/
Kosar, T., Livny, M.: A framework for reliable and efficient data placement in distributed computing systems. J. Parallel Distrib. Comput. 65(10), 1146–1157 (2005)
Zong, Z., Briggs, M., O’Connor, N., Qin, X.: An energy-efficient framework for large-scale parallel storage systems. In: Parallel and Distributed Processing Symposium, pp. 1–7 (2007)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hartog, J., Dede, E. & Govindaraju, M. MapReduce framework energy adaptation via temperature awareness. Cluster Comput 17, 111–127 (2014). https://doi.org/10.1007/s10586-013-0270-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10586-013-0270-y