Skip to main content
Springer Nature Link
Log in

Self-Adaptive Resource Management for Large-Scale Shared Clusters

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

In a shared cluster, each application runs on a subset of nodes and these subsets can overlap with one another. Resource management in such a cluster should adaptively change the application placement and workload assignment to satisfy the dynamic applications workloads and optimize the resource usage. This becomes a challenging problem with the cluster scale and application amount growing large. This paper proposes a novel self-adaptive resource management approach which is inspired from human market: the nodes trade their shares of applications' requests with others via auction and bidding to decide its own resource allocation and a global high-quality resource allocation is achieved as an emergent collective behavior of the market. Experimental results show that the proposed approach can ensure quick responsiveness, high scalability, and application prioritization in addition to managing the resources effectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Tsai W T. Service-oriented system engineering: A new paradigm. In Proc. IEEE International Workshop on Service-Oriented System Engineering (SOSE), Beijing, China, Oct. 20-21, 2005, pp.3–6.

  2. Foster I, Kesselman C (eds.). The Grid: Blueprint for a New Computing Infrastructure. San Francisco: Morgan Kaufmann Publishers Inc., 1999, p.677.

  3. Armbrust M, Fox A, Griffith R, Joseph A, Katz R, Konwinski A, Lee G. Patterson D, Rabkin A, Stoica I, Zaharia M. Above the clouds: A Berkeley view of cloud computing. University of California, Berkeley, Tech. Rep., 2009, http://d1smfj0g31qzek.cloudfront.net/abovetheclouds.pdf.

  4. Urgaonkar B, Shenoy P, Roscoe T. Resource overbooking and application profiling in shared hosting platforms. In Proc. Fifth Symposium on Operating Systems Design and Implementation, Boston, USA, Dec. 9-11, 2002, pp.239–254.

  5. Crovella M E, Bestavros A. Self-similarity inWorld WideWeb traffic: Evidence and possible causes. IEEE/ACM Trans. Networking, 1997, 5(6): 835–846.

    Article  Google Scholar 

  6. Karve A, Kimbrel T, Pacifici G, Spreitzer M, Steinder M, Sviridenko M, Tantawi A. Dynamic application placement for clustered Web applications. In Proc. the International World Wide Web Conference, Edinburgh, UK, May 23-26, 2006, pp.595–604.

  7. Tang C, Steinder M, Spreitzer M, Pacifici G. A scalable application placement controller for enterprise data centers. In Proc. the International World Wide Web Conference, Banff, Canada, May 8-12, 2007, pp.331–340.

  8. Rolia J, Anderzejak A, Arlitt M. Automating enterprise application placement in resource utilities. In Proc. the 14th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management (DSOM), 2003, Heidelberg, Germany, Oct. 20-22, pp.118–129.

  9. Tan W, Fan Y. Decentralized workflow execution for virtual enterprises in grid environment. In Proc. the 5th International Conference on Grid and Cooperative Computing Workshops (GCCW2006), Changsha, China, 2006, pp.308–314.

  10. Andrzejak A, Graupner S, Kotov V, Trinks H. Algorithms for self-organization and adaptive service placement in dynamic distributed systems. Technical Report HPL 2002-259, Hewlett Packard Labs Palo Alto, 2002.

  11. Clearwater S H (ed.). Market-Based Control: A Paradigm for Distributed Resource Allocation. Singapore: World Scientific, 1996.

  12. De Wolf T, Holvoet T. Design patterns for decentralised coordination in self-organising emergent systems. In Proc. the 4th International Workshop on Engineering Self-Organing Systems (ESOA), Hakodate, Japan, May 9, 2006, pp.28–49.

  13. Appleby K, Fakhouri S, Fong L, Goldszmidt G, Kalantar M, Krishnakumar S, Pazel D, Pershing J, Rochwerger B. Oceano: SLA based management of a computing utility. In Proc. International Symposium on Integrated Network Management, Seattle, USA, May 24-25, 2001, pp.14–18.

  14. Chandra A, Goyal P, Shenoy P. Quantifying the benefits of resource multiplexing in on-demand data centers. In the 1st ACM Workshop on Algorithms and Architectures for Self-Managing Systems, San Diego, USA, Jun. 11, 2003.

  15. Adam C, Stadler R. Service middleware for self-managing large-scale systems. IEEE Trans. Network And Service Management, 2007, 4(3): 50–64.

    Article  Google Scholar 

  16. Broberg J, Venugopal S, Buyya R. Market-oriented grids and utility computing: The state-of-the-art and future directions. 2007, http://gridbus.org/reports/MarketGridUtilitySurvey2007.pdf.

  17. Teo Y, Ayani R. Comparison of load balancing strategies on cluster-based web servers. The Journal of the Society for Modeling and Simulation International, 2001, 77(5/6): 185–195.

    Google Scholar 

  18. Ungureanu V, Melamed B, Katehakis M. Effective load balancing for cluster-based servers employing job preemption. Performance Evaluation, 2008, 65(8): 606–622.

    Article  Google Scholar 

  19. Cormen T H, Leiserson C E, Rivest R L, Stein C. Introduction to Algorithms. Second Edition, MIT Press, 2001.

  20. PeKing University Application Server. http://forge.objectweb.org/projects/pkuas.

  21. Pacifici G, Segmuller W, Spreitzer M, Tantawi A. Dynamic estimation of CPU demand of Web traffic. In Proc. the 1st International Conference on Performance Evaluation Methodologies and Tools (VALUETOOLS), Pisa, Italy, Oct. 11-13, 2006, pp.26.

  22. Clark C, Fraser K, Hand S, Hansen J G, Jul E, Limpach C, Pratt I, Warfield A. Live migration of virtual machines. In Proc. the 2nd ACM/USENIX Symp. Networked Systems Design and Implementation (NSDI), Boston, USA, May 2-4, 2005, pp.273–286.

Download references

Author information

Authors and Affiliations

  1. Key Laboratory of High Confidence Software Technologies, Ministry of Education, Institute of Software School of Electronics Engineering and Computer Science, Peking University, Beijing, 100871, China

    Yan Li, Feng-Hong Chen, Xi Sun, Ming-Hui Zhou (Member, CCF), Wen-Pin Jiao (Senior Member, CCF), Dong-Gang Cao & Hong Mei (Senior Member, CCF)

Authors
  1. Yan Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Feng-Hong Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Xi Sun
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Ming-Hui Zhou
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Wen-Pin Jiao
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Dong-Gang Cao
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Hong Mei
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Ming-Hui Zhou.

Additional information

Supported by the National Basic Research 973 Program of China under Grant No. 2009CB320700, the National High Technology Research and Development 863 Program of China under Grant Nos. 2007AA010301, 2008AA01Z139, 2009AA01Z139-1, and the National Natural Science Foundation of China under Grant Nos. 60603038, 60773151.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Li, Y., Chen, FH., Sun, X. et al. Self-Adaptive Resource Management for Large-Scale Shared Clusters. J. Comput. Sci. Technol. 25, 945–957 (2010). https://doi.org/10.1007/s11390-010-9379-0

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-010-9379-0

Keywords