Abstract
This paper presents a grid-enabled MapReduce framework called “Ussop”. Ussop provides its users with a set of C-language based MapReduce APIs and an efficient runtime system for exploiting the computing resources available on public-resource grids. Considering the volatility nature of the grid environment, Ussop introduces two novel task scheduling algorithms, namely: Variable-Sized Map Scheduling (VSMS) and Locality-Aware Reduce Scheduling (LARS). VSMS dynamically adjusts the size of the map tasks according to the computing power of grid nodes. Moreover, LARS minimizes the data transfer cost of exchanging the intermediate data over a wide-area network. The experimental results indicate that both VSMS and LARS achieved superior performance than the conventional scheduling algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Dean, J., Ghemawat, S.: MapReduce: Simplified data processing on large clusters. In: 6th USENIX Symposium on Operating Systems Design and Implementation. USENIX (2004)
Applications and organizations using Hadoop, http://wiki.apache.org/hadoop/PoweredBy
Apache Hadoop, http://hadoop.apache.org/
Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating MapReduce for Multi-core and Multiprocessor Systems. In: IEEE 13th International Symposium on High Performance Computer Architecture, pp. 13–24 (2007)
He, B., Fang, W., Luo, Q., Govindaraju, N.K., Wang, T.: Mars: a MapReduce framework on graphics processors. In: 17th International Conference on Parallel Architectures and Compilation Techniques, pp. 260–269. ACM, Toronto (2008)
Rafique, M.M., Rose, B., Butt, A.R., Nikolopoulos, D.S.: Supporting MapReduce on large-scale asymmetric multi-core clusters. ACM SIGOPS Operating Systems Review 43, 25–34 (2009)
Ibrahim, S., Jin, H., Cheng, B., Cao, H., Wu, S., Qi, L.: CLOUDLET: towards mapreduce implementation on virtual machines. In: 18th ACM International Symposium on High Performance Distributed Computing, pp. 65–66. ACM, Garching (2009)
Zaharia, M., Konwinski, A., Joseph, A., Katz, R., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: 8th USENIX Symposium on Operating Systems Design and Implementation. USENIX (2008)
Merzky, A., Stamou, K., Jha, S.: Application Level Interoperability between Clouds and Grids. In: Workshops at the Grid and Pervasive Computing Conference, pp. 143–150. IEEE Computer Society, Los Alamitos (2009)
Jin, C., Buyya, R.: MapReduce Programming Model for .NET-Based Cloud Computing. In: Sips, H., Epema, D., Lin, H.-X. (eds.) Euro-Par 2009 Parallel Processing. LNCS, vol. 5704, pp. 417–428. Springer, Heidelberg (2009)
Liang, T.Y., Wu, C.Y., Chang, J.B., Shieh, C.K.: Teamster-G: a grid-enabled software DSM system. In: IEEE International Symposium on Cluster Computing and the Grid, vol. 2 (2005)
Chen, P., Chang, J., Liang, T., Shieh, C.: A progressive multi-layer resource reconfiguration framework for time-shared grid systems. Future Generation Computer Systems 25, 662–673 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chen, PC., Su, YL., Chang, JB., Shieh, CK. (2010). Variable-Sized Map and Locality-Aware Reduce on Public-Resource Grids. In: Bellavista, P., Chang, RS., Chao, HC., Lin, SF., Sloot, P.M.A. (eds) Advances in Grid and Pervasive Computing. GPC 2010. Lecture Notes in Computer Science, vol 6104. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-13067-0_27
Download citation
DOI: https://doi.org/10.1007/978-3-642-13067-0_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-13066-3
Online ISBN: 978-3-642-13067-0
eBook Packages: Computer ScienceComputer Science (R0)