BC-BSP: A BSP-Based Parallel Iterative Processing System for Big Data on Cloud Architecture

Abstract

Many applications in real life can produce and collect large amount of data and many of them can be modeled by Graph. The number of vertexes of a graph could be several hundreds of millions to billions and the number of edges could be ten or more times of the number of its vertexes. A BSP-based system for large-scale data (especially graph data) parallel and iterative processing is discussed in this paper. The system has the ability to flexible configuration and the extendibility for functions and strategies (such as adjusting the parameters according to the volume of data and supporting multiple aggregation functions at the same time), to process large-scale data, to tolerate faults, to balance load, and to run clustering or classification algorithms on metric datasets. Lots of experiments are done to evaluate the extendibility of the system implemented in the paper, and the comparison between BC-BSP-based applications and MapReduce-based ones are made. The experimental results show that BSP-based applications have higher efficiency than that of MapReduce-based applications when the volume of data can be put in the memory during the course of processing; on the contrary the latter are better than the former, and the performance of BC-BSP platform outperforms Hama and Giraph.