Skip to main content
SpringerLink
Log in

Market-oriented Grids and Utility Computing: The State-of-the-art and Future Directions

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

Traditional resource management techniques (resource allocation, admission control and scheduling) have been found to be inadequate for many shared Grid and distributed systems, that consist of autonomous and dynamic distributed resources contributed by multiple organisations. They provide no incentive for users to request resources judiciously and appropriately, and do not accurately capture the true value, importance and deadline (the utility) of a user’s job. Furthermore, they provide no compensation for resource providers to contribute their computing resources to shared Grids, as traditional approaches have a user-centric focus on maximising throughput and minimising waiting time rather than maximising a providers own benefit. Consequently, researchers and practitioners have been examining the appropriateness of ‘market-inspired’ resource management techniques to address these limitations. Such techniques aim to smooth out access patterns and reduce the chance of transient overload, by providing a framework for users to be truthful about their resource requirements and job deadlines, and offering incentives for service providers to prioritise urgent, high utility jobs over low utility jobs. We examine the recent innovations in these systems (from 2000–2007), looking at the state-of-the-art in price setting and negotiation, Grid economy management and utility-driven scheduling and resource allocation, and identify the advantages and limitations of these systems. We then look to the future of these systems, examining the emerging ‘Catallaxy’ market paradigm. Finally we consider the future directions that need to be pursued to address the limitations of the current generation of market oriented Grids and Utility Computing systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abramson, D., Buyya, R., Giddy, J.: A computational economy for Grid computing and its implementation in the Nimrod-G resource broker. Future Gener. Comput. Syst. 18(8), 1061–1074 (2002)

    Article  MATH  Google Scholar 

  2. Adams, K., Agesen, O.: A comparison of software and hardware techniques for x86 virtualization. In: ASPLOS ’06: Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 2–13. ACM Press, New York, NY, USA (2006)

    Chapter  Google Scholar 

  3. Ardaiz, O., Artigas, P., Eymann, T., Freitag, F., Navarro, L., Reinicke, M.: The catallaxy approach for decentralized economic-based allocation in Grid resource and service markets. Appl. Intell. 25(2), 131–145 (2006)

    Article  Google Scholar 

  4. AuYoung, A., Chun, B., Snoeren, A., Vahdat, A.: Resource allocation in federated distributed computing infrastructures. In: OASIS ’04: Proceedings of the 1st Workshop on Operating System and Architectural Support for the Ondemand IT InfraStructure (2004) October

  5. AuYoung, A., Grit, L., Wiener, J., Wilkes, J.: Service contracts and aggregate utility functions. In: HPDC ’06: Proceedings of the 15th IEEE International Symposium on High Performance Distributed Computing, pp. 119–131 (2006) June

  6. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the art of virtualization. In: SOSP ’03: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, pp. 164–177. ACM Press, New York, NY, USA (2003)

    Chapter  Google Scholar 

  7. Barmouta, A., Buyya, R.: Gridbank: A Grid accounting services architecture (gasa) for distributed systems sharing and integration. In: ICEC ’03: Workshop on Internet Computing and E-Commerce, Proceedings of the 17th International Symposium on Parallel and Distributed Processing, pp. 245.1. IEEE Computer Society, Washington, DC, USA (2003)

    Google Scholar 

  8. Bavier, A., Bowman, M., Chun, B., Culler, D., Karlin, S., Muir, S., Peterson, L., Roscoe, T., Spalink, T., Wawrzoniak, M.: Operating system support for planetary-scale network services. In: NSDI ’04: Proceedings of the Symposium on Networked Systems Design and Implementation (2004) March

  9. Buyya, R.: Economic-based distributed resource management and scheduling for Grid computing. Ph.D. thesis, Monash University (2002) April

  10. Buyya, R., Abramson, D., Giddy, J.: Economy driven resource management architecture for computational power Grids. In: PDPTA ’00: Proceedings of the 7th International Conference on Parallel and Distributed Processing Techniques and Applications (2000) June

  11. Buyya, R., Abramson, D., Venugopal, S.: The Grid economy. Proc. IEEE 93(3), 698–714 (2005) March

    Article  Google Scholar 

  12. Chun, B., Buonadonna, P., AuYoung, A., Ng, C., Parkes, D., Shneidman, J., Snoeren, A.C., Vahdat, A.: Mirage: a microeconomic resource allocation system for sensornet testbeds. In: EMNETS ’05: Proceedings of the 2nd IEEE Workshop on Embedded Networked Sensors (2005) May

  13. Chun, B.N., Culler, D.E.: User-centric performance analysis of market-based cluster batch schedulers. In: CCGRID ’02: Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, p. 30. IEEE Computer Society, Washington, DC, USA (2002)

    Chapter  Google Scholar 

  14. Eymann, T., Reinicke, M., Streitberger, W., Rana, O., Joita, L., Neumann, D., Schnizler, B., Veit, D., Ardaiz, O., Chacin, P., Chao, I., Freitag, F., Navarro, L., Catalano, M., Gallegati, M., Giulioni, G., Schiaffino, R.C., Zini, F.: Catallaxy-based Grid markets. Multiagent Grid Systems 1(4), 297–307 (2005)

    MATH  Google Scholar 

  15. Foster, I., Kesselman, C.: The Grid: Blueprint for a New Computing Infrastructure. Morgan-Kaufmann (2001)

  16. Fu, Y., Chase, J., Chun, B., Schwab, S., Vahdat, A.: Sharp: an architecture for secure resource peering. SIGOPS Oper. Syst. Rev. 37(5), 133–148 (2003)

    Article  Google Scholar 

  17. Henderson, R.L.: Job scheduling under the portable batch system. In: IPPS ’95: Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, pp. 279–294. Springer-Verlag, London, UK (1995)

    Google Scholar 

  18. Irwin, D., Chase, J., Grit, L., Yumerefendi, A.: Self-recharging virtual currency. In: P2PECON ’05: Proceedings of the 2005 ACM SIGCOMM Workshop on Economics of Peer-to-peer systems, pp. 93–98. ACM Press, New York, NY, USA (2005)

    Chapter  Google Scholar 

  19. Irwin, D.E., Chase, J.S., Grit, L.E., Yumerefendi, A.R., Becker, D., Yocum, K.: Sharing networked resources with brokered leases. In: USENIX ’06: Proceedings of the USENIX Annual Technical Conference, General Track, pp. 199–212 (2006)

  20. Irwin, D.E., Grit, L.E., Chase, J.S.: Balancing risk and reward in a market-based task service. In: HPDC ’04: Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing, pp. 160–169. IEEE Computer Society, Washington, DC, USA (2004)

    Google Scholar 

  21. Huberman, B.A., Lai, K., Fine, L.: Tycoon: A Distributed Market-based Resource Allocation System. Technical Report arXiv:cs.DC/0404013, HP Labs, Palo Alto, CA, USA (2004) April

  22. Lai, K.: Markets are dead, long live markets. SIGecom Exchanges 5(4), 1–10 (2005)

    Article  Google Scholar 

  23. Lai, K., Rasmusson, L., Adar, E., Zhang, L., Huberman, B.A.: Tycoon: an implementation of a distributed, market-based resource allocation system. Multiagent Grid Systems 1(3), 169–182 (2005)

    MATH  Google Scholar 

  24. Li, C., Li, L., Lu, Z.: Utility driven dynamic resource allocation using competitive markets in computational Grid. Adv. Eng. Softw. 36(6), 425–434 (2005)

    Article  Google Scholar 

  25. Mu’alem, A.W., Feitelson, D.G.: Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling. IEEE Trans. Parallel Distrib. Syst. 12(6), 529–543 (2001)

    Article  Google Scholar 

  26. Nelson, M., Lim, B.-H., Hutchins, G.: Fast transparent migration for virtual machines. In: USENIX ’05: Proceedings of the 2005 USENIX Annual Technical Conference, pp. 391–394 (2005)

  27. Ng, C., Buonadonna, P., Chun, B.N., Snoeren, A.C., Vahdat, A.: Addressing strategic behavior in a deployed microeconomic resource allocator. In: P2PECON ’05: Proceedings of the 2005 ACM SIGCOMM Workshop on Economics of Peer-to-peer Systems, pp. 99–104. ACM Press, New York, NY, USA (2005)

    Chapter  Google Scholar 

  28. Nisan, N.: Bidding and allocation in combinatorial auctions. In: EC ’00: Proceedings of the 2nd ACM Conference on Electronic Commerce, pp. 1–12. ACM Press, New York, NY, USA (2000)

    Chapter  Google Scholar 

  29. Oppenheimer, D., Albrecht, J., Patterson, D., Vahdat, A.: Design and implementation tradeoffs for wide-area resource discovery. In: HPDC ’05: Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing. IEEE Computer Society, Washington, DC, USA (2005)

    Google Scholar 

  30. Peterson, L., Anderson, T., Culler, D., Roscoe, T.: A blueprint for introducing disruptive technology into the internet. Comput. Commun. Rev. 33(1), 59–64 (2003)

    Article  Google Scholar 

  31. Placek, M., Buyya, R.: Storage exchange: a global trading platform for storage services. In: EUROPAR ’06: Proceedings of the 12th International European Parallel Computing Conference. Springer-Verlag (2006) August

  32. Popovici, F.I., Wilkes, J.: Profitable services in an uncertain world. In: SC ’05: Proceedings of the 2005 ACM/IEEE conference on Supercomputing, pp. 36. IEEE Computer Society, Washington, DC, USA (2005)

    Google Scholar 

  33. Sarmiento, E.: Securing FreeBSD using jail. Syst. Admin. 10(5), 31–37 (2001)

    Google Scholar 

  34. Schnizler, B., Neumann, D., Veit, D., Weinhardt, C.: A multiattribute combinatorial exchange for trading Grid resources. In: RSEEM ’05: Proceedings of the 12th Research Symposium on Emerging Electronic Markets (2005)

  35. Sherwani, J., Ali, N., Lotia, N., Hayat, Z., Buyya, R.: Libra: a computational economy-based job scheduling system for clusters. Software Practice and Experience 34(6), 573–590 (2004)

    Article  Google Scholar 

  36. Shneidman, J., Ng, C., Parkes, D.C., AuYoung, A., Snoeren, A.C., Vahdat, A., Chun, B.N.: Why markets could (but don’t currently) solve resource allocation problems in systems. In: USENIX ’05: Proceedings of the 10th USENIX Workshop on Hot Topics in Operating Systems (2005) June

  37. Tsafrir, D., Etsion, Y., Feitelson, D.G.: Modeling user runtime estimates. In: JSSPP ’05: Proceedings of the 11th International Workshop on Job Scheduling Strategies for Parallel Processing, pp. 1–35. Cambridge, MA, USA (2005)

  38. Waldspurger, C.A.: Memory resource management in vmware esx server. In: OSDI ’02: Proceedings of the 5th Symposium on Operating Systems Design and Implementation, pp. 181–194. ACM Press, New York, NY, USA (2002)

    Google Scholar 

  39. Wolski, R., Plank, J.S., Brevik, J., Bryan, T.: Analyzing market-based resource allocation strategies for the computational Grid. Int. J. High Perform. Comput. Appl. 15(3), 258–281 (2001) Fall

    Article  Google Scholar 

  40. Wolski, R., Plank, J.S., Brevik, J., Bryan, T.: G-commerce: Market formulations controlling resource allocation on the computational Grid. In: IPDPS ’01: Proceedings of the 15th International Parallel and Distributed Processing Symposium. IEEE, San Francisco (2001) April

    Google Scholar 

  41. Yeo, C.S., Buyya, R.: Pricing for utility-driven resource management and allocation in clusters. In: ADCOM ’04: Proceedings of the 12th International Conference on Advanced Computing and Communications, pp. 32–41. Allied Publishers: New Delhi, India (2004) December

    Google Scholar 

  42. Yeo, C.S., Buyya, R.: Service level agreement based allocation of cluster resources: Handling penalty to enhance utility. In: CLUSTER ’05: Proceedings of the 7th IEEE International Conference on Cluster Computing. IEEE Computer Society: Los Alamitos, CA (2005) September

    Google Scholar 

  43. Yeo, C.S., Buyya, R.: Managing risk of inaccurate runtime estimates for deadline constrained job admission control in clusters. In: ICPP ’06: Proceedings of the 2006 International Conference on Parallel Processing, pp. 451–458. IEEE Computer Society, Washington, DC, USA (2006)

    Google Scholar 

  44. Yeo, C.S., Buyya, R.; A taxonomy of market-based resource management systems for utility-driven cluster computing. Software Practice and Experience 36(13), 1381–1419 (2006)

    Article  Google Scholar 

  45. Yeo, C.S., Buyya, R., de Assuncao, M.D., Yu, J., Sulistio, A., Venugopal, S., Placek, M.: Utility computing on global Grids. In: The Handbook of Computer Networks. John Wiley & Sons, New York, USA (2007)

    Google Scholar 

  46. Yu, J., Buyya, R.: Scheduling scientific workflow applications with deadline and budget constraints using genetic algorithms. Sci. Program. 14(3-4), 217–230 (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Grid Computing and Distributed Systems (GRIDS) Laboratory, Department of Computer Science and Software Engineering, The University of Melbourne, Melbourne, Australia

    James Broberg, Srikumar Venugopal & Rajkumar Buyya

Authors
  1. James Broberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Srikumar Venugopal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rajkumar Buyya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James Broberg.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Broberg, J., Venugopal, S. & Buyya, R. Market-oriented Grids and Utility Computing: The State-of-the-art and Future Directions. J Grid Computing 6, 255–276 (2008). https://doi.org/10.1007/s10723-007-9095-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-007-9095-3

Keywords

Search

Navigation