iTurboGraph: Scaling and Automating Incremental Graph Analytics

With the rise of streaming data for dynamic graphs, large-scale graph analytics meets a new requirement of \emph{Incremental Computation} because the larger the graph, the higher the cost for updating the analytics results by re-execution. A dynamic graph consists of an initial graph $G$ and graph mutation updates $\Delta G$ of edge insertions or deletions. Given a query $Q$, its results $Q(G)$, and updates for $\Delta G$ to $G$, incremental graph analytics computes updates $\Delta Q$ such that $Q$($G \cup \Delta G)$ = $Q(G)$ $\cup$ $\Delta Q$ where $\cup$ is a union operator.In this paper, we consider the problem of large-scale incremental neighbor-centric graph analytics ({\NGA}). We solve the limitations of previous systems: lack of usability due to the difficulties in programming incremental algorithms for {\NGA} and limited scalability and efficiency due to the overheads in maintaining intermediate results for graph traversals in {\NGA}. First, we propose a domain-specific language, {\LNGA}, and develop its compiler for intuitive programming of {\NGA}, automatic query incrementalization, and query optimizations. Second, we define \emph{Graph Streaming Algebra} as a theoretical foundation for scalable processing of incremental {\NGA}. We introduce a concept of \emph{Nested Graph Windows} and model graph traversals as the generation of walk streams. Lastly, we present a system {\SystemX}, which efficiently processes incremental {\NGA} for large graphs. Comprehensive experiments show that {\SystemX} effectively avoids costly re-executions and efficiently updates the analytics results with reduced IO and computations.