Affiliations: [a] School of Computer Engineering and Science, Shanghai University, Shanghai, China | [b] Faculty of Information Networking and Innovation and Design, Toyo University, Tokyo, Japan | [c] College of Computer Science and Technology, Jilin University, Changchun, Jilin, China | [d] School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China
Correspondence:
[*]
Corresponding author: Yue Zhao, School of Computer Engineering and Science, Shanghai University, Shanghai, China. E-mail: [email protected].
Abstract: A number of graph-parallel computing abstractions have been proposed to address the needs of solving complex and large-scale graph computing. However, unnecessary and excessive communication and state sharing between nodes in these frameworks not only reduce the network efficiency but may also cause decrease in runtime performance. In this paper, we propose a mechanism called LightGraph, which reduces the synchronizing communication overhead for distributed graph-parallel computing abstractions. Besides identifying and eliminating the redundant synchronizing communications in existing systems, in order to minimize the required synchronizing communications LightGraph also proposes an edge direction-aware graph partitioning strategy. This new graph partitioning strategy optimally isolates the outgoing edges from the incoming edges of a vertex. We have conducted extensive experiments using real-world data, and our results verified the effectiveness of LightGraph. For example compared to PowerGraph LightGraph can not only reduce up to 31.5% synchronizing communication overhead for intra-graph synchronizations, but also cut up to 16.3% runtime for PageRank running on Livejournal dataset.
Keywords: Graph-parallel computing, big data, light communication
