skip to main content
10.1145/3627341.3630380acmotherconferencesArticle/Chapter ViewAbstractPublication PagesiccvitConference Proceedingsconference-collections
research-article

Optimization of Spark Data Skew in Big Data Environment

Published: 15 December 2023 Publication History

Abstract

In order to solve the problem of uneven data distribution in the shuffle stage of Spark distributed platform, this paper first analyzes the causes of data skew in Spark platform, and then establishes a skew model that can quantify the degree of data skew. Based on quantifiable skew model, a shuffle partition scheme is proposed to optimize data skew in Spark distributed platform. The working node of the partitioning scheme first samples the output data of the Map stage, summarizes the data of each working node to the master node to predict the data size of all working nodes, and then pre-partitions the intermediate data according to the load balancing partitioning algorithm and Hash partitioning algorithm to obtain a pre-partitioning table. The master node distributes the pre-partitioning table to each working node. Finally, the working node partitions all the intermediate data according to the pre-partitioning table. The experimental results under different skew conditions show that the shuffle partition scheme proposed in this paper is universal and efficient, and can effectively deal with the data skew problem of Spark distributed platform.

References

[1]
Ping L. Research on University Campus Big Data Platform Based on Hadoop and Spark [J] .Software Engineering, 2018:2-34.
[2]
Midoun K, Loudini M, Hidouci K W, LoEM: Improving Load Balancing for MapReduce-based Matching [J]. International Journal of Artificial Intelligence, 2019, 17 (2): 217-235.
[3]
Martha V S, Zhao W, Xu X .h-MapReduce: A Framework for Workload Balancing in MapReduce [C]//IEEE Computer Society.IEEE Computer Society, 2013:637-644.
[4]
Gufler B, Augsten N, Reiser A, HANDLING DATA SKEW IN MAPREDUCE [C]//CLOSER 2011 - Proceedings of the 1st International Conference on Cloud Computing and Services Science, Noordwijkerhout, Netherlands, 7-9 May, 2011:1-7.
[5]
Son J, Choi H, Chung Y D .Skew-Tolerant Key Distribution for Load Balancing in MapReduce [J] .Ieice Trans.inf . & Syst, 2012, 95 (2): 677-680.
[6]
Ramakrishnan S R, Swart G, Urmanov A. Balancing reducer skew in MapReduce workloads using Computsampling [C]//Third Proceedings of the Progressive ACM Symposium on Cloud Computing.ACM, 2012. 1-4.
[7]
Kwon Y C, Balazinska M, Howe B, SkewTune: Mitigating Skew in MapReduce Applications [J] .Proceedings of the Vldb Endowment, 2012, 5 (12): 1934-1937.
[8]
Chen Q, Yao J, Xiao Z. LIBRA: Lightweight Data Skew Mitigation in MapReduce [J] .IEEE Transactions on Parallel & Distributed Systems, 2015, 26 (9): 2520-2533.
[9]
Yu J, Chen H, Hu F .SASM: Improving Spark Performance with Adaptive Skew Mitigation [C]//2015 IEEE International Conference on Progress in Informatics and Computing, 2015:102-107.
[10]
Liu G, Zhu X, Wang J, SP-Partitioner: A novel partition method to handle intermediate data skew in spark streaming [J] .Future Generation Computer Systems, 2017, 86 (3): 1054-1063.
[11]
Tang Z, Zhang X, Li K, An intermediate data placement algorithm for load balancing in the Spark environment [J] .Future Generation Computer Systems, 2016, 78 (1): 287-301.
[12]
Elaheh Gavagsaz, Ali Rezaee, Hamid H. S. Javadi. Load balancing in join algorithms for skewed data in Map Reduce systems. The Journal of Supercomputing, 2019, 75(1): 220-254.
[13]
Donghua Chen, Runtong Zhang. Map Reduce-based dynamic partition join with shannon entropy for data skewness. Scientific Programming, 2021, 175-209.
[14]
Jianjiang Li, Yajun Liu, Jian Pan, Peng Zhang, Wei Chen, Lizhe Wang. Map-Balance-Reduce: An improved parallel programming model for load balancing of Map Reduce. Future Generation Computer Systems, 2020, 105: 973-998.
[15]
Balraj Singh, Harsh K. Verma. IMSM: An interval migration based approach for skew mitigation in Map Reduce. Recent Advances in Computer Science and Communications, 2021, 14(1): 61-81.
[16]
Kotoulas S, Oren E, Harmelen F V. Mind the data skew: Distributed inferencing by speeddating in elastic regions [C]/Proceedings Conference of the 19th International dating on World Wide Web, WWW 2010, Raleigh, North Carolina, USA, April 26-30, 2010.ACM, 2010. 1-33.
[17]
Rana N, Deshmukh S. Shuffle Performance in Apache Spark [C]//International Journal of Engineering Research & Technology.ESRSA Publications, 2015:10-44.
[18]
Aggarwal C C. On Biased Reservoir Sampling in the Presence of Stream Evolution. [C]//Very Large Data Bases Conference.2006:33-46.
[19]
Adamic, Lada A. Zipf 's law and the Internet.' glottometrics (2002): 3 (1): 143-150.
[20]
Li-Jie X U .Construction and Research of Big Data Processing Platform Based on Spark [J] .Computer Knowledge and Technology, 2016:10-23.
[21]
Pennebaker J W, Francis M E, Booth R J. Linguistic inquiry and word count (LIWC) [J] .Lawrence Erlbaum Associates Mahwah Nj, 2001; 24-56.

Index Terms

  1. Optimization of Spark Data Skew in Big Data Environment
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        ICCVIT '23: Proceedings of the 2023 International Conference on Computer, Vision and Intelligent Technology
        August 2023
        378 pages
        ISBN:9798400708701
        DOI:10.1145/3627341
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 15 December 2023

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Spark
        2. data skew
        3. partitioning algorithm
        4. shuffle

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Conference

        ICCVIT 2023

        Acceptance Rates

        ICCVIT '23 Paper Acceptance Rate 54 of 142 submissions, 38%;
        Overall Acceptance Rate 54 of 142 submissions, 38%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • 0
          Total Citations
        • 67
          Total Downloads
        • Downloads (Last 12 months)66
        • Downloads (Last 6 weeks)4
        Reflects downloads up to 20 Jan 2025

        Other Metrics

        Citations

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media