Skip to main content
SpringerLink
Log in

Dynamic grid load sharing with adaptive dissemination protocols

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Scheduling in large scale dynamic grids comprising eclectic collections of resources is increasingly difficult. Autonomous resource neighborhoods may wish to determine the level of grid offered load that they can or will accept; different sites may wish to attract different amounts of load, to satisfy some desired property within a grid economy. This changes the traditional notion of load sharing, which generally assumes that the desired equilibrium should be an equal distribution of load across all participating machines, because they are under the jurisdiction of a single site, and therefore more likely to implement one common policy. In large-scale grids, nodes and neighborhoods should instead get a portion of the load that best matches their local policies for supporting and admitting grid jobs. This article describes information dissemination protocols that can distribute load in this way, without using load rebalancing through job migration, which is more difficult and costly in large-scale heterogeneous grids. Essentially, nodes adjust their advertising rates and aggressiveness to influence where jobs get scheduled. We report experimental results with example resource configurations in which each resource neighborhood determines its ideal grid load and disseminates accordingly. In turn, each neighborhood attracts the requisite amount of resource requests from the grid. Moreover, performance does not degrade: overall query satisfaction rates are within 9% of both adaptive dissemination protocols that use static adaptation policies, and static dissemination protocols that may be custom-tailored to specific resource and load distributions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abu-Ghazaleh N, Lewis MJ (2006) Short paper: toward self organizing grids. In: Proceedings of the 15th IEEE international symposium on high performance distributed computing (hot topics session)

    Google Scholar 

  2. Anderson DP (2004) Boinc: a system for public-resource computing and storage. In: GRID’04: proceedings of the fifth IEEE/ACM international workshop on grid computing (GRID’04), Washington, DC, USA, 2004. IEEE Computer Society, Los Alamitos, pp 4–10

    Chapter  Google Scholar 

  3. Androutsellis-Theotokis S, Spinellis D (2004) A survey of peer-to-peer content distribution technologies. ACM Comput Surv 36(4):335–371. http://portal.acm.org/citation.cfm?id=1041681

    Article  Google Scholar 

  4. Babaoglu O, Meling H, Montresor A (2002) Anthill: a framework for the development of agent-based peer-to-peer systems. In: Proceedings of the 22th international conference on distributed computing systems, Vienna, Austria, July 2002

    Google Scholar 

  5. Barak A, Laadan O (1998) The mosix multicomputer operating system for high performance cluster computing. Future Gener Comput Syst. http://www.mosix.cs.huji.ac.il

  6. Barford P, Bestavros A, Bradley A, Crovella M (1998) Changes in web client access patterns: characteristics and caching implications. Technical report

  7. Butt AR, Zhang R, Hu YC (2006) A self-organizing flock of condors. J Parallel Distrib Comput 66(1):145–161

    MATH  Google Scholar 

  8. Buyya R, Vazhkudai S (2001) Computer power market: towards a market-oriented grid. In: Proceedings of 1st IEEE international conference on cluster computing and the grid, CCGRID2001, May 2001

    Google Scholar 

  9. Buyya R, Abramson D, Giddy J (2000) Economy driven resource management architecture for computational power grids. In: Proceedings of the international conference on parallel and distributed processing techniques and applications (PDPTA2000), Las Vegas, USA

    Google Scholar 

  10. Cao J (2004) Self-organizing agents for grid load balancing. In: Proceedings of the fifth IEEE/ACM international workshop on grid computing (GRID’04), Pittsburgh, PA, November 2004

    Google Scholar 

  11. Casavant TL, Kuhl JG (1988) A taxonomy of scheduling in general-purpose distributed computing systems. IEEE Trans Softw Eng 14(2):141–154

    Article  Google Scholar 

  12. Chiu D-M, Jain R (1989) Analysis of the increase and decrease algorithms for congestion avoidance in computer networks. Comput Netw ISDN Syst 17:1–14

    Article  MATH  Google Scholar 

  13. Cowie J, Liu H, Liu J, Nicol D, Ogielski A (1999) Towards realistic million-node internet simulations. In: Proceedings of the international conference on parallel and distributed processing techniques and applications

    Google Scholar 

  14. Desai R, Tilak S, Gandhi B, Lewis MJ, Abu-Ghazaleh NB (2006) Analysis of query matching criteria and resource monitoring for grid application scheduling. In: Proceedings of CCGrid2006: IEEE international symposium on cluster computing and the grid

    Google Scholar 

  15. Eager DL, Lazowska ED, Zahorjan J (1986) A comparison of receiver-initiated and sender-initiated adaptive load sharing. Perform Eval 6(1):53–68

    Article  Google Scholar 

  16. Epema DHJ, Livny M, van Dantzig R, Evers X, Pruyne J (1995) A worldwide flock of condors: load sharing among workstation clusters. Technical Report DUT-TWI-95-130, Delft, The Netherlands

  17. Erdil DC (2007) Adaptive dissemination protocols for hybrid grid resource scheduling. PhD dissertation, dissertations & theses @ Binghamton University (SUNY), (AAT 3289113). http://www.proquest.com.proxy.binghamton.edu

  18. Erdil DC, Lewis MJ, Abu-Ghazaleh NB (2006) An adaptive algorithm for information dissemination in self-organizing grids. In: Proceedings of the 2nd IEEE international conference on e-science and Grid Computing (eScience 2006), Amsterdam, the Netherlands, 4–6 December 2006

    Google Scholar 

  19. Erdil DC, Lewis MJ, Abu-Ghazaleh NB (2006) An adaptive approach to information dissemination in self-organizing grids. In: Proceedings of the international conference on autonomic and autonomous systems (ICAS’06), Silicon Valley, CA, July 2006

    Google Scholar 

  20. Erdil DC, Lewis MJ, Abu-Ghazaleh NB (2007) Proxy-based grid information dissemination. In: Proceedings of the workshop on large-scale and volatile desktop grids (PCGrid2007) (in conjunction with IPDPS2007), Long Beach, CA, March 2007

    Google Scholar 

  21. European Union EGEE Project Group. Enabling grids for e-science. http://www.eu-egee.org

  22. Fedak G, Germain C, N’eri V, Cappello F (2001) Xtremweb: a generic global computing system. In: Proceedings of the IEEE international symposium on cluster computing and the grid (CCGRID’01), May 2001

    Google Scholar 

  23. Foster I, Kesselman C, Tuecke S (2001) The anatomy of the grid: enabling scalable virtual organizations. In: Lecture notes in computer science, vol 2150. Springer, Berlin

    Google Scholar 

  24. Iamnitchi A, Doraimani S, Garzoglio G (2006) Filecules in high-energy physics: characteristics and impact on resource management. In: Proceedings of the 15th IEEE international symposium on high performance distributed computing (HPDC-15), Paris, France, June 2006

    Google Scholar 

  25. Iyengar V, Tilak S, Abu-Ghazaleh NB, Lewis MJ (2004) Nonuniform information dissemination for dynamic grid resource discovery. In: Proceedings of IEEE NCA04: the 3rd IEEE international symposium on network computing and applications

    Google Scholar 

  26. Jung J, Krishnamurthy B, Rabinovich M (2002) Flash crowds and denial of service attacks: characterization and implications for cdns and web sites. In: Proceedings of the IEEE international world wide web conference

    Google Scholar 

  27. Kondo D, Taufer M, Brooks CL III, Casanova H, Chien AA (2004) Characterizing and evaluating desktop grids: an empirical study. In: Proceedings of the 18th international parallel and distributed processing symposium (IPDPS’04), April 2004

    Google Scholar 

  28. Kremien O, Kramer J (1992) Methodical analysis of adaptive load sharing algorithms. IEEE Trans Parallel Distrib Syst 3(6):747–760

    Article  Google Scholar 

  29. Ledlie J, Seltzer M (2005) Distributed, secure load balancing with skew, heterogeneity, and churn. In: Proceedings of the 24th annual joint conference of the IEEE computer and communications societies (INFOCOM 2005), March 2005, vol 2, pp 1419–1430

    Chapter  Google Scholar 

  30. Lenders VLC (2006) Field-based routing and its application to wireless ad hoc networks. PhD dissertation, dissertations & theses @ Swiss Federal Institute of Technology Zurich (Diss ETH No 16681). http://www.lenders.ch/publications/books/thesis.pdf

  31. Li H, Groep D, Wolters L (2004) Workload characteristics of a multi-cluster supercomputer. In: Feitelson DG, Rudolph L, Schwiegelshohn U (eds) JSSPP. LNCS, vol 3277. Springer, Berlin, pp 176–194

    Google Scholar 

  32. Li L, Halpern J, Haas Z (2002) Gossip-based ad hoc routing. In: IEEE Infocom

    Google Scholar 

  33. Livny M, Melman M (1982) Load balancing in homogeneous broadcast distributed systems. In: Proceedings of the computer network performance symposium, New York, NY, USA. ACM, New York, pp 47–55

    Chapter  Google Scholar 

  34. Lv Q, Cao P, Cohen E, Li K, Shenker S (2002) Search and replication in unstructured peer-to-peer networks. SIGMETRICS Performance Evaluation Review

  35. Molina TE, Yang G, Lin AW, Peltier ST, Ellisman MH (2005) A generalized service-oriented architecture for remote control of scientific imaging instruments. In: E-SCIENCE’05: proceedings of the first international conference on e-science and grid computing, Washington, DC, USA. IEEE Computer Society, Los Alamitos, pp 550–556

    Chapter  Google Scholar 

  36. Montresor A, Meling H, Babaoglu O (2002) Messor: load-balancing through a swarm of autonomous agents. Technical Report UBLCS-02-08, Department of Computer Science, University of Bologna

  37. Silberstein M, Geiger D, Schuster A, Livny M (2006) Scheduling mixed workloads in multi-grids: the grid execution hierarchy. In: Proceedings of the 15th IEEE symposium on high performance distributed computing (HPDC-15), Paris, France, June 2006

    Google Scholar 

  38. Tcp congestion control. http://tools.ietf.org/html/rfc2581

  39. TeraGrid Project Group. TeraGrid. http://www.teragrid.org

  40. Wang Y-T, Morris R (1985) Load sharing in distributed systems. IEEE Trans Comput C-34(3):204–217

    Article  Google Scholar 

  41. Wright D (2001) Cheap cycles from the desktop to the dedicated cluster: combining opportunistic and dedicated scheduling with Condor. In: Proceedings of the Linux clusters: the HPC revolution conference, Champaign, Urbana, IL, June 2001

    Google Scholar 

  42. Wrthimer D, Cobb J, Lebofsky M, Anderson D, Korpela E (2001) Seti@home: massively distributed computing for seti. Comput Sci Eng 3(1):78–83

    Article  Google Scholar 

  43. Zegura E, Calvert K. GT internetwork topology models (GT-ITM). http://www.cc.gatech.edu/projects/gtitm

  44. Zhang Q, Cherkasova L, Smirni E (2006) Flexsplit: a workload-aware, adaptive load balancing strategy for media clusters. In: Proceedings of SPIE, multimedia computing and networking, vol 6071

    Google Scholar 

  45. Zhou S (1988) A trace-driven simulation study of dynamic load balancing. IEEE Trans Softw Eng 14(9):1327–1341

    Article  Google Scholar 

  46. Zhou D, Lo V (2006) Wavegrid: a scalable fast-turnaround heterogeneous peer-based desktop grid system. In: Proceedings of the 20th international parallel and distributed processing symposium (IPDPS’06)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Engineering Department, İstanbul Bilgi University, 34060, Istanbul, Turkey

    D. Cenk Erdil

  2. Department of Computer Science, Binghamton University (SUNY), Binghamton, NY, 13902-6000, USA

    Michael J. Lewis

Authors
  1. D. Cenk Erdil
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael J. Lewis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to D. Cenk Erdil.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Erdil, D.C., Lewis, M.J. Dynamic grid load sharing with adaptive dissemination protocols. J Supercomput 59, 1139–1166 (2012). https://doi.org/10.1007/s11227-010-0507-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-010-0507-y

Keywords

Search

Navigation