Abstract
MapReduce as a popular platform has been extensively used for solving data-intensive applications. A number of tuning parameters can be applied to improve the performance of MapReduce. Among these parameters, the number of map tasks (mappers) driven by the number of logical input splits has a dramatic effect on the performance. However, subject to one-to-one correspondence between mappers and splits, the tradeoff between mapper-level parallelism and mapper startup costs must be carefully evaluated based on the input size and the split size. Meanwhile, the manual parameter configuration is lack of flexibility to meet the performance requirements of different jobs. In this paper, an adaptive split assignment scheme is proposed to decouple the number of mappers from the number of splits. We introduce the MSMapper(Multi-Split Mapper), a modified self-tuning mapper in which multiple splits can be assigned to one mapper. And with aid of inter-MSMapper communication, we reveal the potential that map tasks can be constructed without dependence on the number of splits, while the modified MapReduce architecture can sustain fine-grained load balancing and fault tolerance, as well as coarse-grained task startup overhead. We built our prototype on top of the Hadoop MapReduce realization, and present a comprehensive evaluation that shows the benefits of the MSMapper in common scenarios where split sizing problems arise. The results show that the modified version can improve the performance by a factor of 2.5.
This work is sponsored by the National Natural Science Foundation of China (Nos. 61033007,60970070), the National High Technology Research and Development Program (863 Program) of China (No. 2012AA011004), and the National Basic Research Program (973 Program) of China (No. 2012CB316203), NWPU basic research foundation (Nos. JC20110227,JC20110225,JC201261).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Herodotou, H., Lim, H., Luo, G., Borisov, N., Dong, L., Cetin, F.B., Babu, S.: Starfish: A self-tuning system for big data analytics. In: CIDR 2011, Asilomar, CA, United states, pp. 261–272 (2011)
Wei, P., Li, Z., Sai, W., Qun, C.: Evaluating large graph processing in mapreduce based on message passing. Chinese Journal of Computers 10, 1768–1784 (2011)
Dittrich, J., Quian-Ruiz, J.A., Jindal, A., Kargin, Y., Setty, V., Schad, J.: Hadoop++: Making a yellow elephant run like a cheetah (without it even noticing). PVLDB 3(1), 518–529 (2010)
Bu, Y., Howe, B., Balazinska, M., Ernst, M.D.: Haloop: Efficient iterative data processing on large clusters. PVLDB 3(1), 285–296 (2010)
Ekanayake, J., Li, H., Zhang, B., Gunarathne, T., Bae, S.H., Qiu, J., Fox, G.: Twister: a runtime for iterative mapreduce. In: HPDC 2010, pp. 810–818. ACM (2010)
Condie, T., Conway, N., Alvaro, P., Hellerstein, J.M.: Mapreduce online. In: NSDI 2010, pp. 21–21 (2010)
Li, B., Mazur, E., Diao, Y., McGregor, A., Shenoy, P.J.: A platform for scalable one-pass analytics using mapreduce. In: SIGMOD 2011, pp. 985–996 (2011)
Babu, S.: Towards automatic optimization of mapreduce programs. In: SOCC 2010, Indianapolis, Indiana, USA, pp. 137–142 (2010)
Herodotou, H., Babu, S.: Hadoop performance models. Technical report (2010)
Herodotou, H., Babu, S.: Profiling, what-if analysis, and cost-based optimization of mapreduce programs. PVLDB 4(11), 1111–1122 (2011)
Herodotou, H., Dong, F., Babu, S.: No one (cluster) size fits all: automatic cluster sizing for data-intensive analytics. In: Proceedings of the 2nd ACM Symposium on Cloud Computing, SOCC 2011, pp. 18:1–18:14. ACM, New York (2011)
Jahani, E., Cafarella, M.J., Ré, C.: Automatic optimization for mapreduce programs. PVLDB 4(6), 385–396 (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Pan, W., Li, Z., Chen, Q., Peng, S., Suo, B., Xu, J. (2012). MSMapper: An Adaptive Split Assignment Scheme for MapReduce. In: Bao, Z., et al. Web-Age Information Management. WAIM 2012. Lecture Notes in Computer Science, vol 7419. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33050-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-642-33050-6_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33049-0
Online ISBN: 978-3-642-33050-6
eBook Packages: Computer ScienceComputer Science (R0)