Huanshu Wang
ABSTRACT
In order to solve the problem of uneven data distribution in the shuffle stage of Spark distributed platform, this paper first analyzes the causes of data skew in Spark platform, and then establishes a skew model that can quantify the degree of data skew. Based on quantifiable skew model, a shuffle partition scheme is proposed to optimize data skew in Spark distributed platform. The working node of the partitioning scheme first samples the output data of the Map stage, summarizes the data of each working node to the master node to predict the data size of all working nodes, and then pre-partitions the intermediate data according to the load balancing partitioning algorithm and Hash partitioning algorithm to obtain a pre-partitioning table. The master node distributes the pre-partitioning table to each working node. Finally, the working node partitions all the intermediate data according to the pre-partitioning table. The experimental results under different skew conditions show that the shuffle partition scheme proposed in this paper is universal and efficient, and can effectively deal with the data skew problem of Spark distributed platform..
- Ping L. Research on University Campus Big Data Platform Based on Hadoop and Spark [J] .Software Engineering, 2018:2-34.Google Scholar
- Midoun K, Loudini M, Hidouci K W, LoEM: Improving Load Balancing for MapReduce-based Matching [J]. International Journal of Artificial Intelligence, 2019, 17 (2): 217-235.Google Scholar
- Martha V S, Zhao W, Xu X .h-MapReduce: A Framework for Workload Balancing in MapReduce [C]//IEEE Computer Society.IEEE Computer Society, 2013:637-644.Google Scholar
- Gufler B, Augsten N, Reiser A, HANDLING DATA SKEW IN MAPREDUCE [C]//CLOSER 2011 - Proceedings of the 1st International Conference on Cloud Computing and Services Science, Noordwijkerhout, Netherlands, 7-9 May, 2011:1-7.Google Scholar
- Son J, Choi H, Chung Y D .Skew-Tolerant Key Distribution for Load Balancing in MapReduce [J] .Ieice Trans.inf . & Syst, 2012, 95 (2): 677-680.Google Scholar
- Ramakrishnan S R, Swart G, Urmanov A. Balancing reducer skew in MapReduce workloads using Computsampling [C]//Third Proceedings of the Progressive ACM Symposium on Cloud Computing.ACM, 2012. 1-4.Google Scholar
- Kwon Y C, Balazinska M, Howe B, SkewTune: Mitigating Skew in MapReduce Applications [J] .Proceedings of the Vldb Endowment, 2012, 5 (12): 1934-1937.Google ScholarDigital Library
- Chen Q, Yao J, Xiao Z. LIBRA: Lightweight Data Skew Mitigation in MapReduce [J] .IEEE Transactions on Parallel & Distributed Systems, 2015, 26 (9): 2520-2533.Google ScholarCross Ref
- Yu J, Chen H, Hu F .SASM: Improving Spark Performance with Adaptive Skew Mitigation [C]//2015 IEEE International Conference on Progress in Informatics and Computing , 2015:102-107.Google Scholar
- Liu G, Zhu X, Wang J, SP-Partitioner: A novel partition method to handle intermediate data skew in spark streaming [J] .Future Generation Computer Systems, 2017, 86 (3): 1054-1063.Google Scholar
- Tang Z, Zhang X, Li K, An intermediate data placement algorithm for load balancing in the Spark environment [J] .Future Generation Computer Systems, 2016, 78 (1): 287-301.Google Scholar
- Elaheh Gavagsaz, Ali Rezaee, Hamid H. S. Javadi. Load balancing in join algorithms for skewed data in Map Reduce systems. The Journal of Supercomputing, 2019, 75(1): 220-254.Google Scholar
- Donghua Chen, Runtong Zhang. Map Reduce-based dynamic partition join with shannon entropy for data skewness. Scientific Programming, 2021, 175-209.Google Scholar
- Jianjiang Li, Yajun Liu, Jian Pan, Peng Zhang, Wei Chen, Lizhe Wang. Map-Balance-Reduce: An improved parallel programming model for load balancing of Map Reduce. Future Generation Computer Systems, 2020, 105: 973-998.Google Scholar
- Balraj Singh, Harsh K. Verma. IMSM: An interval migration based approach for skew mitigation in Map Reduce. Recent Advances in Computer Science and Communications, 2021, 14(1): 61-81.Google Scholar
- Kotoulas S, Oren E, Harmelen F V. Mind the data skew: Distributed inferencing by speeddating in elastic regions [C]/Proceedings Conference of the 19th International dating on World Wide Web, WWW 2010, Raleigh, North Carolina, USA, April 26-30, 2010.ACM, 2010. 1-33.Google Scholar
- Rana N, Deshmukh S. Shuffle Performance in Apache Spark [C]//International Journal of Engineering Research & Technology.ESRSA Publications, 2015:10-44.Google Scholar
- Aggarwal C C. On Biased Reservoir Sampling in the Presence of Stream Evolution. [C]//Very Large Data Bases Conference.2006:33-46.Google Scholar
- Adamic, Lada A.. Zipf 's law and the Internet.' glottometrics (2002): 3 (1): 143-150.Google Scholar
- Li-Jie X U .Construction and Research of Big Data Processing Platform Based on Spark [J] .Computer Knowledge and Technology, 2016:10-23.Google Scholar
- Pennebaker J W, Francis M E, Booth R J. Linguistic inquiry and word count (LIWC) [J] .Lawrence Erlbaum Associates Mahwah Nj, 2001; 24-56.Google Scholar
Index Terms
- Optimization of Spark Data Skew in Big Data Environment
Recommendations
A Spark-Based Big Data Platform for Massive Remote Sensing Data Processing
ICDS 2015: Proceedings of the Second International Conference on Data Science - Volume 9208With the fast development of remote sensing techniques, the volume of acquired data grows exponentially. This brings a big challenge to process massive remote sensing data. In the paper, an in-memory computing framework is proposed to address this ...
Intermediate Data Placement Strategy for Different Data Skew Levels Based on Random Sampling in Spark
ICBDC '19: Proceedings of the 4th International Conference on Big Data and ComputingIn recent years, the Apache Spark had been widely used in processing large-scale data. However, when the input data onto MapReduce was skewed, the default intermediate data placement algorithm of the Apache Spark could not efficiently handle the skewed ...
Geospatial Big Data Analytics Engine for Spark
BigSpatial '17: Proceedings of the 6th ACM SIGSPATIAL Workshop on Analytics for Big Geospatial DataWith the rapid development of geospatial data acquisition and processing technology, the scale of spatial data is expanding. Mass production applications put forward higher requirements for the performance of geospatial data analysis. In this study, we ...
Comments