Skip to main content
SpringerLink
Log in

Scalable and effective peer-to-peer desktop grid system

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

We have designed a set of protocols that use peer-to-peer techniques to efficiently implement a distributed and decentralized desktop grid. Incoming jobs with different resource requirements are matched with system nodes through proximity in an N-dimensional Content-Addressable Network, where each resource type is represented as a distinct dimension. In this paper, we describe a comprehensive suite of techniques that cooperate to maximize throughput, and to ensure that load is balanced across all peers. We balance load induced by job executions through randomly generated virtual dimension values, which act to disaggregate clusters of nodes/jobs, and also by a job pushing mechanism based on an approximate global view of the system. We improve upon initial job assignments by using a job-stealing mechanism to overcome load imbalance caused by heterogeneity of nodes/jobs and stale load information. We also describe a set of optimizations that combine to reduce the system load created by the management of the underlying peer-to-peer system and the job-monitoring infrastructure. Unlike other systems, we can effectively support resource constraints of jobs during the course of load balancing since we simplify the problem of matchmaking through building a multi-dimensional resource space and mapping jobs and nodes to this space. We use extensive simulation results to show that the new techniques improve scalability, system throughput, and average response time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Anderson, D.: BOINC: a system for public-resource computing and storage. In: Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing (GRID 2004) (2004)

  2. Anderson, D.P., Christensen, C., Allen, B.: Designing a runtime system for volunteer computing. In: Proceedings of the 2006 IEEE/ACM SC06 Conference (2006)

  3. Anderson, D.P., Cobb, J., Korpela, E., Lebofsky, M., Werthimer, D.: SETI@home: an experiment in public-resource computing. Commun. ACM 45(11), 56–61 (2002)

    Article  Google Scholar 

  4. Awan, A., Ferreira, R.A., Jagannathan, S., Grama, A.: Unstructured peer-to-peer networks for sharing processor cycles. Parallel Comput. 32, 2 (2006)

    Article  MathSciNet  Google Scholar 

  5. Becchi, M., Crowley, P.: Dynamic thread assignment on heterogeneous multiprocessor architectures. In: Proceedings of the 3rd Conference on Computing Frontiers (2006)

  6. BOINC: Open-source software for volunteer computing and grid computing. http://boinc.berkeley.edu/

  7. Bridgewater, J., Boykin, P.O., Roychowdhury, V.: Balanced overlay networks (BON): an overlay technology for decentralized load balancing. IEEE Trans. Parallel Distrib. Syst. 18(7), 1122–1133 (2007)

    Article  Google Scholar 

  8. Caromel, D., di Costanzo, A., Mathieu, C.: Peer-to-peer for computational grids: mixing clusters and desktop machines. Parallel Comput. 33(4–5), 275–288 (2007)

    Article  Google Scholar 

  9. Cheema, A.S., Muhammad, M., Gupta, I.: Peer-to-peer discovery of computational resources for grid applications. In: Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing (GRID 2005) (2005)

  10. Coleman, N., Raman, R., Livny, M., Solomon, M.: Distributed policy management and comprehension with classified advertisements. Tech. Rep. UW-CS-TR-1481, University of Wisconsin - Madison Computer Sciences Department (2003)

  11. Demers, A., Greene, D., Hauser, C., Irish, W., Larson, J., Shenker, S., Sturgis, H., Swinehart, D., Terry, D.: Epidemic algorithms for replicated database maintenance. In: Proceedings of the Sixth Annual ACM Symposium on Principles of Distributed Computing (1987)

  12. Epema, D., Livny, M., van Dantzig, R., Evers, X., Pruyne, J.: A worldwide flock of condors: load sharing among workstation clusters. Future Gener. Comput. Syst. 12(1), 53–65 (1996)

    Article  Google Scholar 

  13. Folding@Home: http://folding.stanford.edu

  14. Foster, I., Iamnitchi, A.: On death, taxes, and the convergence of Peer-to-peer and grid computing. In: Proceedings of the 2nd International Workshop on Peer-to-Peer Systems (IPTPS ’03) (2003)

  15. Godfrey, B., Lakshminarayanan, K., Surana, S., Karp, R., Stoica, I.: Load balancing in dynamic structured P2P systems. In: Proceedings of the IEEE INFOCOM 2004 (2004)

  16. Gupta, R., Sekhri, V., Somani, A.K.: CompuP2P: an architecture for internet computing using peer-to-peer networks. IEEE Trans. Parallel Distrib. Syst. 17(11), 1306–1320 (2006)

    Article  Google Scholar 

  17. Kim, J.S., Keleher, P., Marsh, M., Bhattacharjee, B., Sussman, A.: Using content-addressable networks for load balancing in desktop grids. In: Proceedings of the 16th IEEE International Symposium on High Performance Distributed Computing (HPDC 2007) (2007)

  18. Kim, J.S., Nam, B., Keleher, P., Marsh, M., Bhattacharjee, B., Sussman, A.: Trade-offs in matching jobs and balancing load for distributed desktop grids. Future Gener. Comput. Syst. 24(5), 415–424 (2008)

    Article  Google Scholar 

  19. Kim, J.S., Nam, B., Marsh, M., Keleher, P., Bhattacharjee, B., Sussman, A.: Integrating categorical resource types into a P2P desktop grid system. In: Proceedings of the 9 IEEE/ACM International Conference on Grid Computing (GRID 2008) (2008)

  20. Ledlie, J., Schneidman, J., Seltzer, M., Huth, J.: Scooped, again. In: Proceedings of the 2nd International Workshop on Peer-to-Peer Systems (IPTPS ’03) (2003)

  21. Litzkow, M.J., Livny, M., Mutka, M.W.: Condor—a Hunter of idle workstations. In: Proceedings of the 8th International Conference on Distributed Computing Systems (1988)

  22. Mastroianni, C., Talia, D., Verta, O.: A super-peer model for building resource discovery services in grids: design and simulation analysis. In: Proceedings of the European Grid Conference (EGC2005) (2005)

  23. Oppenheimer, D., Albrecht, J., Patterson, D., Vahdat, A.: Design and implementation tradeoffs for wide-area resource discovery. In: Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing (HPDC-14) (2005)

  24. Raicu, I., Foster, I., Zhao, Y.: Many-task computing for grids and supercomputers. In: Proceedings of the Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS’08) (2008)

  25. Raicu, I., Zhang, Z., Wilde, M., Foster, I., Beckman, P., Iskra, K., Clifford, B.: Towards loosely-coupled programming on petascale systems. In: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (SC’08) (2008)

  26. Raicu, I., Zhao, Y., Dumitrescu, C., Foster, I., Wilde, M.: Falkon: a fast and light-weight tasK executiON framework. In: Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (SC’07) (2007)

  27. Raman, R., Livny, M., Solomon, M.: Matchmaking: distributed resource management for high throughput computing. In: Proceedings of the 7th IEEE International Symposium on High Performance Distributed Computing (HPDC-7) (1998)

  28. Ratnasamy, S., Francis, P., Handley, M., Karp, R., Shenker, S.: A scalable content addressable network. In: Proceedings of the ACM SIGCOMM (2001)

  29. Roy, A., Livny, M.: Condor and preemptive resume scheduling. Grid Resource Management: State of the Art and Future Trends, pp. 135–144. Springer, New York (2003)

    Google Scholar 

  30. Samet, H.: Foundations of Multidimensional and Metric Data Structures. Morgan-Kaufmann, Burlington (2006)

    MATH  Google Scholar 

  31. Shivaratri, N.G., Krueger, P., Singhal, M.: Load distributing for locally distributed systems. Computer 25(12), 33–44 (1992)

    Article  Google Scholar 

  32. SZTAKI Desktop grid. http://szdg.lpds.sztaki.hu/szdg/

  33. Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurr. Comput. 17(2–4), 323–356 (2005)

    Article  Google Scholar 

  34. Trunfio, P., Talia, D., Papadakis, H., Fragopoulou, P., Mordacchini, M., Pennanen, M., Popov, K., Vlassov, V., Haridi, S.: Peer-to-peer resource discovery in grids: models and systems. Future Gener. Comput. Syst. 23(7), 864–878 (2007)

    Article  Google Scholar 

  35. Walker, E., Gardner, J.P., Litvin, V., Turner, E.L.: Creating personal adaptive clusters for managing scientific jobs in a distributed computing environment. In: Proceedings of the Challenges of Large Applications in Distributed Environments (CLADE’06) (2006)

  36. Zhou, D., Lo, V.: WaveGrid: a scalable fast-turnaround heterogeneous peer-based desktop grid system. In: Proceedings of the 20th International Parallel & Distributed Processing Symposium (2006)

  37. Zhu, Y., Hu, Y.: Efficient, proximity-aware load balancing for dht-based p2p system. IEEE Trans. Parallel Distrib. Syst. 16(4), 349–361 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Institute of Supercomputing and Networking, Korea Institute of Science and Technology Information, Daejeon, South Korea

    Jik-Soo Kim

  2. School of Electrical and Computer Engineering, Ulsan National Institute of Science and Technology, Ulsan, South Korea

    Beomseok Nam

  3. UMIACS and Department of Computer Science, University of Maryland, College Park, MD, USA

    Alan Sussman

Authors
  1. Jik-Soo Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beomseok Nam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alan Sussman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jik-Soo Kim.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, JS., Nam, B. & Sussman, A. Scalable and effective peer-to-peer desktop grid system. Cluster Comput 17, 1185–1201 (2014). https://doi.org/10.1007/s10586-014-0390-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-014-0390-z

Keywords

Search

Navigation