Optimization of Spark Data Skew in Big Data Environment
Abstract
References
Index Terms
- Optimization of Spark Data Skew in Big Data Environment
Recommendations
A Spark-Based Big Data Platform for Massive Remote Sensing Data Processing
ICDS 2015: Proceedings of the Second International Conference on Data Science - Volume 9208With the fast development of remote sensing techniques, the volume of acquired data grows exponentially. This brings a big challenge to process massive remote sensing data. In the paper, an in-memory computing framework is proposed to address this ...
Intermediate Data Placement Strategy for Different Data Skew Levels Based on Random Sampling in Spark
ICBDC '19: Proceedings of the 4th International Conference on Big Data and ComputingIn recent years, the Apache Spark had been widely used in processing large-scale data. However, when the input data onto MapReduce was skewed, the default intermediate data placement algorithm of the Apache Spark could not efficiently handle the skewed ...
Geospatial Big Data Analytics Engine for Spark
BigSpatial '17: Proceedings of the 6th ACM SIGSPATIAL Workshop on Analytics for Big Geospatial DataWith the rapid development of geospatial data acquisition and processing technology, the scale of spatial data is expanding. Mass production applications put forward higher requirements for the performance of geospatial data analysis. In this study, we ...
Comments
Information & Contributors
Information
Published In
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
Acceptance Rates
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 67Total Downloads
- Downloads (Last 12 months)66
- Downloads (Last 6 weeks)4
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format