Skew Aware Partitioning Techniques for Multi-way Spatial Join

Abstract

With the massive increase in the usage of location-based services, there has been a huge increase in the availability of spatial data. Extracting hidden business value inherent in the spatial data like the migration of the customer base etc. has become mandate. With the recent advancement of open-source distributed computing techniques like Hadoop the computing power is made available at ease. SpatialHadoop, Hive, Impala are the popular tools used for querying spatial data. These tools generally use indexing methods to execute queries. Extensive work on optimizing joins has been done, but as the real-world spatial datasets contain huge skew, optimizing spatial joins is still a challenging problem. We investigate the problem of skew present in the spatial datasets by providing skew aware partitioning techniques for multi-way spatial joins. We solve the problem by distributing the data symmetrically across the cluster nodes. Our algorithms implemented in Hadoop mapreduce framework offers skew aware partitioning techniques by further reducing the communication cost. We implemented a binary split partitioning approach and strip partitioned technique for multi-way spatial join and compared with the baseline approaches sequential join and controlled replicate. We observed that our approaches are in line with the existing approaches for uniformly distributed datasets, whereas for the skewed datasets, our techniques outperformed the exiting techniques. Our experiments indicate that effective partitioning strategies, distribute the data evenly across the reducers, and have a better overall turn around time compared to the baseline methods.