TSH: Easy-to-be distributed partitioning for large-scale graphs

Highlights

• Re-formulate graph partitioning for efficient combination in Pregel-like systems.

• Discover graph locality in many real-world graphs.

• Design a distributed streaming graph partitioning algorithm based on locality.

Abstract

The big graph era is coming with strong and ever-growing demands on parallel iterative analysis. But, before that, balanced graph partitioning is a fundamental problem and is NP-complete. Till now, there have been several streaming heuristic solutions with a single full scan over the input graph. However, some of them cannot be easily parallelized to further accelerate partitioning for large-scale graphs due to complicated heuristics; while others can be run in parallel but incur expensive communication costs during iterative computation.

This paper presents Target-vertex Sensitive Hash (TSH), an easy-to-be distributed partitioning method. We first analyze the locality property naturally provided by the original input graph, which has not yet been considered by existing work. We then exploit such locality to simplify the heuristic rule. The simplified rule is implemented by a two-step framework where target vertices of edges are first logically pre-divided without accessing any graph data and then, based on the distribution of target vertices, streaming partitioning is physically performed in parallel. TSH provides the capability of quickly dividing large-scale graphs because of parallelization, as well as optimizes communication overheads due to the utilization of locality. Using a broad spectrum of real-world graphs, we conduct extensive performance studies to confirm the effectiveness of TSH over up-to-date competitors.

Introduction

Graphs have been studied in general applications in the cyber world, like intelligent transportation systems, social networks, bioinformatics, Internet, and scientific computation [1]. Many graph analysis algorithms are naturally iterative so as to refine the final solution step by step, such as PageRank, Shortest Path, and Connected Components. However, the graph data volume is increasingly growing, that poses great challenges to efficient iterative computation. For example, Google has indexed hundreds of billions of webpages.¹ Until November 2017, there are over 2.07 billion monthly active users on Facebook.² In order to handle such complex analysis over large graphs, distributed systems have been developed [2], [3], [4], [5], [6], [7], [8], [9], [10], all of which expose simple yet expressive APIs for easily programming various algorithms. Users can submit their own graph analysis jobs concurrently. Any job is typically processed by several workers in parallel which run on physical machines in a cloud computing cluster. That maximally unleashes the power of the given cluster.

Graph partitioning is a key component in all of the existing distributed systems. Distributed systems typically partition an input graph onto a group of workers as sub-graphs and then perform iterative update over them. More specifically, at each iteration, a graph algorithm exchanges intermediate results (also called messages) along edges. Then the number of cut edges across sub-graphs can reflect the network communication overhead to some extent and hence the computation efficiency. Also, the computation workload of each worker is supposed to be balanced. Otherwise, the slow, straggling worker can slow down the overall performance. Thus, as studied before, a high-quality partitioning method should decrease the number of cut edges, as well as balance the workload among subgraphs. However, besides the traditional quality factors mentioned above, now we also need to focus on the partitioning efficiency. Based on our experience, the available cloud compute resources usually dynamically change with the submission of graph jobs, and the addition and removal of physical machines. Most of systems, like Pregel [2] developed by Google, quantify available resources by worker slots. They typically provide a setWorkerNum API for users to flexibly set the number of workers when submitting their own jobs. The number of workers might not be a constant when running analysis jobs over the same graph but at different times. Therefore, graph partitioning is required by every job. That is, different jobs cannot share partitioning results to amortize partitioning costs. To summarize, balanced workload, optimized communication costs, and high partitioning efficiency, can improve the overall performance of iterative graph analysis, all of which should be considered when partitioning graphs.

However, graph partitioning is NP-complete [11], [12]. Existing heuristic solutions basically fall into two categories: offline represented by Metis [13] and streaming/online by LDG [14], Fennel [15] and FG [16]. Offline partitioning can significantly decrease the number of cut edges, and then optimize the communication overhead. It thereby attracts a lot of attention both in industry and in science. However, it requires to frequently traverse vertices, yielding expensive costs especially when processing large-scale graphs. Previous experiments have shown that the partitioning runtime may exceed the iterative computation runtime [15]. Hence, a cost-effective alternative is to accelerate the partitioning procedure by compromising the partitioning quality, i.e. streaming partitioning. It is completed by performing a single full pass over vertices and edges. This can be done along loading input graph. Hence, we also call it as online partitioning through this paper. However, streaming partitioning is usually run on a single machine so as to update heuristic information in a centralized fashion. That limits the scalability. Note that Hash as another streaming solution can be run in parallel but the quality is particularly poor in terms of communication optimization.

Hence, a naturally desirable goal for graph partitioning is to pursue a streaming mechanism that offers high partitioning quality and can be performed in an efficient distributed fashion. This paper explores a path to such a target. In particular, we re-define graph partitioning because most of Pregel-like systems have a built-in combination function to optimize communication overheads. That is, every worker combines messages sent to the same target vertex into a single one before flushing them. Communication overheads indeed are dominated by the number of messages after performing all possible combining operations. In another word, they depend on how many target vertices edges on each worker link to, instead of cut edges across workers. Existing definitions ignore the combining effect. The re-defined partitioning shifts the optimization core from cut edges to target vertices. Previous efforts have shown that the former cannot be optimized without high overheads. However, it is feasible for the latter. We discover that real-world graphs have some locality, that is, many outgoing edges from different source vertices usually link to common target vertices. By leveraging this locality, we design a Target-vertex Sensitive Hash (TSH) partitioning algorithm. Specifically, we assume that vertices as target vertices are pre-partitioned into disjoint subsets, each of which is associated with one sub-graph. Pre-partitioning is performed in a simple fashion for efficiency and the result is regarded as the heuristic rule. Accordingly, TSH can decide the placement of each vertex arriving in a streaming fashion to optimize the actual number of target vertices in all sub-graphs. But more importantly, it can be performed in parallel for efficiency since the heuristic rule has been fixed. Our experiments show that although the heuristic rule is not dynamically updated as traditional solutions, it still provides comparable communication optimization effect because of the locality property in real-world graphs.

The major contributions are summarized as below.

• We re-formulate graph partitioning for Pregel-like systems by explicitly taking their built-in combining function into account. The new definition emphasizes decreasing the number of target vertices.

• We discover the graph locality and our further investigation reveals that it exists in many real-world graphs.

• Based on the re-formulated partitioning and the locality, we design a novel streaming partitioning algorithm TSH. It can be efficiently performed in a distributed fashion, as well as optimize the communication overheads across balanced sub-graphs.

• Extensive experimental studies explore the performance features of our proposal. We demonstrate that TSH can offer comparable quality to the state-of-the-art centralized streaming partitioning solution but run faster than it.

The remainder of this paper is organized as follows. Section 2 provides necessary background regarding distributed graph processing systems and then formally gives the definition of graph partitioning. Section 3 introduces the graph locality and uses a lot of preliminary experimental results to show that it is widespread. Section 4 presents the details of our new partitioning algorithm TSH. Section 5 reports extensive evaluation results. Section 6 highlights related work. Finally, Section 7 concludes this paper.