Skip to main content
SpringerLink
Log in

DETER: Streaming Graph Partitioning via Combined Degree and Cluster Information

  • Conference paper
  • First Online:
Algorithms and Architectures for Parallel Processing (ICA3PP 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11944))

  • 1616 Accesses

Abstract

Efficient graph partitioning plays an important role in distributed graph processing systems with the rapid growth of the scale of graph data. The quality of partitioning affects the performance of systems greatly. However, most existing vertex-cut graph partitioning algorithms only focused on degree information and ignored the cluster information of a coming edge when assigning edges. It is beneficial to assign an edge to a partition with more neighbors because keeping a dense subgraph in one partition would reduce the communication cost. In this paper, we propose DETER, an efficient vertex-cut streaming graph partitioning algorithm that takes both degree and cluster information into account when assigning an edge to one partition. Our evaluations suggest that DETER algorithm owns the ability to efficiently partition large graphs and reduce communication cost significantly compared to state-of-the-art graph partitioning algorithms.

Supported in part by National Key Research and Development Program of China under Grant 2017YFB1402400, in part by graduate research and innovation foundation of Chongqing, China under Grant CYB18058, in part by the Fundamental Research Funds for the Central Universities under Grant 2018CDYJSY0055, in part by the Frontier and Application Foundation Research Program, PR China of CQ CSTC under Grant cstc2017jcyjAX0340.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Andreev, K., Racke, H.: Balanced graph partitioning. Theory Comput. Syst. 39(6), 929–939 (2006)

    Article  MathSciNet  Google Scholar 

  2. Bali, P., Kalavri, V.: Streaming graph analytics framework design (2015). http://urn.kb.se/resolve

  3. Donnelly, G.: Super-useful Facebook statistics for (75) (2018)

    Google Scholar 

  4. Fineschi, S., et al.: Metis: a novel coronagraph design for the solar orbiter mission. Proc. SPIE - Int. Soc. Opt. Eng. 8443(8), 457–469 (2012)

    Google Scholar 

  5. Gonzalez, J.E., Low, Y., Gu, H., Bickson, D., Guestrin, C.: Powergraph: distributed graph-parallel computation on natural graphs. In: USENIX Conference on Operating Systems Design & Implementation (2012)

    Google Scholar 

  6. Hu, K., Zeng, G., Jiang, H., Wang, W.: Partitioning big graph with respect to arbitrary proportions in a streaming manner. Future Gener. Comput. Syst. 80, 1–11 (2018)

    Article  Google Scholar 

  7. Jain, N., Liao, G., Willke, T.L.: Graphbuilder: scalable graph ETL framework. In: International Workshop on Graph Data Management Experiences & Systems (2013)

    Google Scholar 

  8. Kalnis, P., Awara, K., Jamjoom, H., Khayyat, Z.: Mizan: optimizing graph mining in large parallel systems. Technical report, King Abdullah University of Science and Technology (2012)

    Google Scholar 

  9. Leskovec, J., Kleinberg, J., Faloutsos, C.: Graphs over time: densification laws, shrinking diameters and possible explanations. In: Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, pp. 177–187. ACM (2005)

    Google Scholar 

  10. Low, Y., Bickson, D., Gonzalez, J., Guestrin, C., Kyrola, A., Hellerstein, J.M.: Distributed graphlab: a framework for machine learning and data mining in the cloud. Proc. VLDB Endow. 5(8), 716–727 (2012)

    Article  Google Scholar 

  11. Malewicz, G., et al.: Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, pp. 135–146. ACM (2010)

    Google Scholar 

  12. Martella, C., Logothetis, D., Loukas, A., Siganos, G.: Spinner: scalable graph partitioning in the cloud. In: IEEE International Conference on Data Engineering (2017)

    Google Scholar 

  13. Mayer, C., Tariq, M.A., Mayer, R., Rothermel, K.: Graph: traffic-aware graph processing. IEEE Trans. Parallel Distrib. Syst. 29(6), 1289–1302 (2018)

    Article  Google Scholar 

  14. Mofrad, M.H., Melhem, R., Hammoud, M.: Revolver: vertex-centric graph partitioning using reinforcement learning. In: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), pp. 818–821. IEEE (2018)

    Google Scholar 

  15. Nishimura, J., Ugander, J.: Restreaming graph partitioning: simple versatile algorithms for advanced balancing. In: ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (2013)

    Google Scholar 

  16. Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: bringing order to the web. Technical report, Stanford InfoLab (1999)

    Google Scholar 

  17. Petroni, F., Querzoni, L., Daudjee, K., Kamali, S., Iacoboni, G.: HDRF: stream-based partitioning for power-law graphs. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 243–252. ACM (2015)

    Google Scholar 

  18. Prabhakaran, V., Wu, M., Weng, X., McSherry, F., Zhou, L., Haradasan, M.: Managing large graphs on multi-cores with graph awareness. In: Presented as Part of the 2012 USENIX Annual Technical Conference (USENIX ATC 2012), pp. 41–52 (2012)

    Google Scholar 

  19. Rossi, R.A., Ahmed, N.K.: The network data repository with interactive graph analytics and visualization. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (2015). http://networkrepository.com

  20. Stanton, I., Kliot, G.: Streaming graph partitioning for large distributed graphs. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1222–1230. ACM (2012)

    Google Scholar 

  21. Thorup, M.: Undirected single-source shortest paths with positive integer weights in linear time. J. ACM (JACM) 46(3), 362–394 (1999)

    Article  MathSciNet  Google Scholar 

  22. Tsourakakis, C.: Streaming graph partitioning in the planted partition model. In: Proceedings of the 2015 ACM on Conference on Online Social Networks, pp. 27–35. ACM (2015)

    Google Scholar 

  23. Tsourakakis, C., Gkantsidis, C., Radunovic, B., Vojnovic, M.: Fennel: streaming graph partitioning for massive scale graphs. In: Proceedings of the 7th ACM International Conference on Web Search and Data Mining, pp. 333–342. ACM (2014)

    Google Scholar 

  24. Xie, C., Yan, L., Li, W.J., Zhang, Z.: Distributed power-law graph computing: theoretical and empirical analysis. In: International Conference on Neural Information Processing Systems (2014)

    Google Scholar 

  25. Xin, R.S., Gonzalez, J.E., Franklin, M.J., Stoica, I.: GraphX: a resilient distributed graph system on spark. In: First International Workshop on Graph Data Management Experiences and Systems, p. 2. ACM (2013)

    Google Scholar 

  26. Yang, J., Leskovec, J.: Defining and evaluating network communities based on ground-truth. In: IEEE International Conference on Data Mining (2012)

    Google Scholar 

  27. Yin, H., Benson, A.R., Leskovec, J., Gleich, D.F.: Local higher-order graph clustering. In: ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (2017)

    Google Scholar 

  28. Zheng, A., Labrinidis, A., Chrysanthis, P.K., Lange, J.: Argo: architecture-aware graph partitioning. In: IEEE International Conference on Big Data (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Chongqing University, Chongqing, 400044, People’s Republic of China

    Cong Hu, Jiang Zhong, Qi Li & Qing Li

  2. Key laboratory of Dependable Service Computing in Cyber Physical Society, Chongqing University, Chongqing, 400044, People’s Republic of China

    Jiang Zhong

Authors
  1. Cong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiang Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiang Zhong .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Software Engineering, Swinburne University of Technology, Hawthorn, Melbourne, VIC, Australia

    Sheng Wen

  2. School of Computer Science, The University of Sydney, Camperdown, NSW, Australia

    Albert Zomaya

  3. Department of Computer Science, St. Francis Xavier University, Antigonish, NS, Canada

    Laurence T. Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hu, C., Zhong, J., Li, Q., Li, Q. (2020). DETER: Streaming Graph Partitioning via Combined Degree and Cluster Information. In: Wen, S., Zomaya, A., Yang, L. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2019. Lecture Notes in Computer Science(), vol 11944. Springer, Cham. https://doi.org/10.1007/978-3-030-38991-8_16

Download citation

Publish with us

Policies and ethics

Search

Navigation