skip to main content
10.1145/2907294.2907299acmconferencesArticle/Chapter ViewAbstractPublication PageshpdcConference Proceedingsconference-collections
short-paper

DD-Graph: A Highly Cost-Effective Distributed Disk-based Graph-Processing Framework

Published: 31 May 2016 Publication History

Abstract

Existing distributed graph-processing frameworks, e.g.,GPS, Pregel and Giraph, handle large-scale graphs in the memory of clusters built of commodity compute nodes for better scalability and performance. While capable of scaling out according to the size of graphs up to thousands of compute nodes, for graphs beyond a certain size, these frameworks usually require the investments of machines that are either beyond the financial capability of or unprofitable for most small and medium-sized organizations. At the other end of the spectrum of graph-processing frameworks research, the single-node disk-based graph-processing frameworks, e.g., GraphChi, handle large-scale graphs on one commodity computer, leading to high efficiency in the use of hardware but at the cost of low user performance and limited scalability. Motivated by this dichotomy, in this paper we propose a distributed disk-based graph-processing framework, called DD-Graph, that can process super-large graphs on a small cluster while achieving the high performance of existing distributed in-memory graph-processing frameworks.

References

[1]
A. Kyrola, G. E. Blelloch, and C. Guestrin. Graphchi: Large-scale graph computation on just a pc. In OSDI'12.
[2]
A. Lumsdaine, D. Gregor, B. Hendrickson, and J. Berry. Challenges in parallel graph processing. Parallel Processing Letters, 17(01):5--20, 2007.
[3]
G. Malewicz, M. H. Austern, and etc. Pregel: a system for large-scale graph processing. In Proc. ACM SIGMOD'10.
[4]
J. Malicevic, A. Roy, and W. Zwaenepoel. Scale-up graph processing in the cloud: Challenges and solutions. In Proceedings of the Fourth International Workshop on Cloud Data and Platforms. ACM, 2014.
[5]
R. Pearce, M. Gokhale, and N. M. Amato. Faster parallel traversal of scale free graphs at extreme scale with vertex delegates. In Proc. SC'14.
[6]
R. Pearce, M. Gokhale, and N. M. Amato. Multithreaded asynchronous graph traversal for in-memory and semi-external memory. In Proc. SC'10.
[7]
R. Pearce, M. Gokhale, and N. M. Amato. Scaling techniques for massive scale-free graphs in distributed (external) memory. In IPDPS'13.
[8]
A. Roy, L. Bindschaedler, J. Malicevic, and W. Zwaenepoel. Chaos: Scale-out graph processing from secondary storage. In Proc. ACM SOSP'15.
[9]
A. Roy, I. Mihailovic, and W. Zwaenepoel. X-stream: edge-centric graph processing using streaming partitions. In Proc. ACM SOSP'13.
[10]
S. Salihoglu and J. Widom. Gps: A graph processing system. In Proc. ACM SSDBM'13.

Cited By

View all
  • (2019)Using High-Bandwidth Networks Efficiently for Fast Graph ComputationIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2018.287508430:5(1170-1183)Online publication date: 1-May-2019
  • (2018)HUS-GraphProceedings of the 47th International Conference on Parallel Processing10.1145/3225058.3225108(1-10)Online publication date: 13-Aug-2018
  • (2018)A communication-reduced and computation-balanced framework for fast graph computationFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-018-6400-112:5(887-907)Online publication date: 1-Oct-2018
  • Show More Cited By

Index Terms

  1. DD-Graph: A Highly Cost-Effective Distributed Disk-based Graph-Processing Framework

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    HPDC '16: Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing
    May 2016
    302 pages
    ISBN:9781450343145
    DOI:10.1145/2907294
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 31 May 2016

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. cost-effectiveness
    2. high performance
    3. super-large graphs

    Qualifiers

    • Short-paper

    Funding Sources

    • State Key Laboratory of Computer Architecture of China
    • National High Tech- nology Research and Development Program (863 Program) of China
    • National Basic Research 973 Program of China

    Conference

    HPDC'16
    Sponsor:

    Acceptance Rates

    HPDC '16 Paper Acceptance Rate 20 of 129 submissions, 16%;
    Overall Acceptance Rate 166 of 966 submissions, 17%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)6
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2019)Using High-Bandwidth Networks Efficiently for Fast Graph ComputationIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2018.287508430:5(1170-1183)Online publication date: 1-May-2019
    • (2018)HUS-GraphProceedings of the 47th International Conference on Parallel Processing10.1145/3225058.3225108(1-10)Online publication date: 13-Aug-2018
    • (2018)A communication-reduced and computation-balanced framework for fast graph computationFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-018-6400-112:5(887-907)Online publication date: 1-Oct-2018
    • (2017)BlitzG: Exploiting high-bandwidth networks for fast graph processingIEEE INFOCOM 2017 - IEEE Conference on Computer Communications10.1109/INFOCOM.2017.8057203(1-9)Online publication date: May-2017

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media