Abstract
For frequent disk I/O and big data transmissions among different racks and physical nodes, the intermediate data communication has become the biggest performance bottle-neck in most running Hadoop systems. This paper proposes a reduce placement algorithm called CORP to schedule related map and reduce tasks on the near nodes or clusters or racks for the data locality. Since the number of keys cannot be counted until the input data are processed by map tasks, this paper firstly provides a sampling algorithm based on reservoir sampling to achieve the distribution of the keys in intermediate data. Through calculating the distance and cost matrices among the cross node communication, the related map and reduce tasks can be scheduled to relatively near physical nodes for data locality. Experimental results show that CORP can not only improve the balance of reduce tasks effectively, but also decrease the job execution time for the lower inner data communication.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bourguiba, M., Haddadou, K., El Korbi, I., Pujolle, G.: Improving network i/o virtualization for cloud computing. IEEE Trans. Parallel Distrib. Syst. 25(3), 673–681 (2014)
Ho, L.-Y., Wu, J.-J., Liu, P.: Optimal algorithms for cross-rackcommunication optimization in mapreduce framework. In: 2011 IEEE International Conference on Cloud Computing(CLOUD), pp. 420–427. IEEE (2011)
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: Fair scheduling for distributed computing clusters. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systemsprinciples, pp. 261–276. ACM (2009)
Ahmad, F., Lee, S., Thottethodi, M., Vijaykumar, T.: Mapreduce withcommunication overlap, technical report, Technical Report (2007)
Maheshwari, N., Nanduri, R., Varma, V.: Dynamic energy efficient data placement and cluster reconfiguration algorithm for mapreduce framework. Future Gener. Comput. Syst. 28(1), 119–127 (2012)
Sandholm, T., Lai, K.: Dynamic proportional share scheduling in hadoop. In: Frachtenberg, E., Schwiegelshohn, U. (eds.) JSSPP 2010. LNCS, vol. 6253, pp. 110–131. Springer, Heidelberg (2010)
W. [EB/OL]. http://en.wikipedia.org/wiki/greedy algorithm
Sleator, D.D., Tarjan, R.E.: Self-adjusting binary search trees. J. ACM (JACM) 32(3), 652–686 (1985)
Kwon, Y., Balazinska, M., Howe, B., Rolia, J.: Skewtune: mitigating skew inmapreduce applications. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp. 25–36. ACM (2012)
Xu, Y., Kostamaa, P.: Efficient outer join data skew handling in parallel dbms. Proc. VLDB Endowment 2(2), 1390–1396 (2009)
Palanisamy, B., Singh, A., Liu, L., Jain, B.: Purlieus: locality-awareresource allocation for mapreduce in a cloud. In: Proceedings of 2011International Conference for High Performance Computing, Networking, Storageand Analysis, p. 58. ACM (2011)
Ardizzoni, E., Bertossi, A., Pinotti, M.C., Ramaprasad, S., Rizzi, R., Shashanka, M.V., et al.: Optimal skewed data allocation on multiple channels with flat broadcast per channel. IEEE Trans. Comput. 54(5), 558–572 (2005)
Acharya, S., Gibbons, P.B., Poosala, V.: Congressional samples forapproximate answering of group-by queries. ACM SIGMOD Rec. 29(2), 487–498 (2000). ACM
Kwon, Y., Balazinska, M., Howe, B.: Skewtune: mitigating skew inmapreduce applications. In: 2012 ACM SIGMODInternational Conference on Management of Data, pp. 25–36. ACM (2012)
Chen, Q., Liu, C., Xiao, Z.: Libra: Lightweight data skew mitigation in mapreduce. IEEE Trans. Parallel Distrib. Syst. 99, 1–14 (2014)
Lin, J.: The curse of zipf and limits to parallelization: A look at thestragglers problem in mapreduce. In: 7th Workshop on Large-Scale Distributed Systems for Information Retrieval, vol. 1 (2009)
Chen, Y., Wei, H.W., Wei, M.F., Chen, Y.J.: LaSA: A locality-aware scheduling algorithm forhadoop-mapreduce resource assignment. In: 2013 International Conference on Collaboration Technologies and Systems (CTS), pp. 342–346. IEEE (2013)
Hammoud, M., Sakr, M.F.: Locality-aware reduce task scheduling formapreduce. In: 2011 IEEE Third nternational Conference on Cloud Computing Technology and Science 29 2011-December 1 2011
Grover, R., Carey, M.J.: Extending map-reduce for efficient predicate-basedsampling. In: 2012 IEEE 28th InternationalConference on Data Engineering (ICDE), pp. 486–497. IEEE (2012)
Atta, F., Viglas, S.D., Niazi, S.: Sand joina skew handling joinalgorithm for google’s mapreduce framework. In: 2011 IEEE 14th International Multitopic Conference, pp. 170–175. IEEE (2011)
Tang, Z., Jiang, L.G., Zhou, J.Q., Li, K.K., Li, K.K.: A self-adaptive scheduling algorithm for reduce start time. Future Gener. Comput. Syst. 43, 51–60 (2015)
Gufler, B., Augsten, N., Reiser, A., Kemper, A.: Load balancing in mapreducebased on scalable cardinality estimates. In: 2012 IEEE 28th International Conference on Data Engineering (ICDE), pp. 522–533. IEEE (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Tang, Z., Ma, W., Li, R., Li, K., Li, K. (2016). An Optimal Reduce Placement Algorithm for Data Skew Based on Sampling. In: Zhan, J., Han, R., Zicari, R. (eds) Big Data Benchmarks, Performance Optimization, and Emerging Hardware. BPOE 2015. Lecture Notes in Computer Science(), vol 9495. Springer, Cham. https://doi.org/10.1007/978-3-319-29006-5_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-29006-5_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-29005-8
Online ISBN: 978-3-319-29006-5
eBook Packages: Computer ScienceComputer Science (R0)