A Performance Comparison of Big Data Processing Platform Based on Parallel Clustering Algorithms

https://doi.org/10.1016/j.procs.2018.10.228Get rights and content
Under a Creative Commons license
open access

Abstract

The performance of three typical big data processing platform: Hadoop, Spark and DataMPI are compared based on different parallel clustering algorithms: parallel K-means, parallel fuzzy K-means and parallel Canopy. Experiments are performed on different text as well as numeric dataset and clusters of different scale. The results show that: (1) for the same data set, when the memory of each node is 4GB, DataMPI can achieve about 60% performance improvement compared with Hadoop, and can achieve about 32% performance improvement compared with Spark; (2) in order to obtain a high clustering performance, a cluster with 6 nodes and 6GB memory of each node should be selected.

Keywords

Hadoop
Spark
DataMPI
K-means
fuzzy K-means
Canopy

Cited by (0)