Skip to main content
SpringerLink
Log in

Adaptive resource provisioning method using application-aware machine learning based on job history in heterogeneous infrastructures

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

With the remarkable growth in cloud computing, computing resources can be manipulated on-demand in most scientific fields. This enables scientists to strategically select their experimental environment. Since it is hard to offer cloud resources in accordance with application characteristics, efficient resource provisioning methods are needed. This paper proposes an adaptive resource provisioning method using an application-aware machine learning technique that is based on the job history in heterogeneous infrastructures. The proposed resource provisioning method is built on two main concepts. First, it provides application-aware resource provisioning through the profiling of scientific application in a heterogeneous computing infrastructure. A resource provisioning model uses the resource usage profiles of scientific applications and job history data in heterogeneous computing infrastructures. In addition to the multilayer perceptron machine learning method, an error back-propagation approach is applied to analyze job history to re-learn the error of the output value. Second, it offers an adaptive resource scaling that is invoked by the availability of resource changes. An adaptive resource management method results in an enhancement of the overall balance between the performance and utilization of a system. For the experiments with the two CPU-intensive applications according to the method, a heterogeneous infrastructure comprising clusters and cloud environments is used. Experimental results indicate that the use of the proposed method can gratify user requests (cost and execution time) regarding its application and enhance resource usage effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Lee, K., et al.: Standardization trends and strategies on cloud computing. Commun. Korean Inst. Inf. Sci. Eng. 28(12), 27–33 (2010)

    Google Scholar 

  2. Lim, C.-S.: Cloud computing security technology. Rev. KIISC 19(3), 14–17 (2009)

    Google Scholar 

  3. Mateescu, G., et al.: Hybrid computing where HPC meets grid and cloud computing. Future Gener. Comput. Syst. 27(5), 440–453 (2011)

    Article  Google Scholar 

  4. Moca, M., et al.: Multi-criteria and satisfaction oriented scheduling for hybrid distributed computing infrastructures. Future Gener. Comput. Syst. 55, 428–443 (2016)

    Article  Google Scholar 

  5. Delamare, S., et al.: SpeQuloS: a QoS service for BoT applications using best effort distributed computing infrastructures. In: Proceedings of the 21st International Symposium on HPDC, pp. 173–186 (2012)

  6. Choi, J., et al.: A job dispatch optimization method on cluster and cloud for large-scale high-throughput computing service. In: IEEE International Conference on Cloud and Autonomic Computing, Cambridge, MA, USA, 21–24 September (2015)

  7. SGE. http://www.oracle.com/technetwork/oem/grid-engine-166852.html

  8. OpenStack. https://www.openstack.org/

  9. DIRAC. http://diracgrid.org/files/docs/Overview/index.html?highlight=pilot

  10. Condor, G.: A computation management agent for multi-institutional grids. Clust. Comput. 5(3), 237–246 (2002)

    Article  Google Scholar 

  11. gUSE/WS-PGRADE. http://guse.hu/about/home

  12. Kim, S.K., et al.: HTCaaS (high throughput computing as a service) in supercomputing environment. J. Korean Contents Assoc. 14(5), 817 (2014)

    Google Scholar 

  13. Hwang, S., et al.: HTCaaS: Efficient and simplified large-scale scientific computing over supercomputers, grids and cloud. The ACM Cloud and Autonomic Computing Conference Miami, Florida, USA, 59 August (2013)

  14. Kim, J., et al.: Fuzzy logic-driven virtual machine resource evaluation method for cloud provisioning service. J. Korea Soc. Simul. 22(1), 77–86 (2013)

    Article  Google Scholar 

  15. Rao, J., et al.: A distributed self-learning approach for elastic provisioning of virtualized cloud resources. In: IEEE 19th Annual International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, pp. 45–54 (2011)

  16. Tesauro, G., et al.: A hybrid reinforcement learning approach to autonomic resource allocation: 2006 IEEE International Conference on Autonomic Computing, pp. 65–73 (2006)

  17. Sukhija, N., et al.: A learning-based selection for portfolio scheduling of scientific applications on heterogeneous computing systems. J. Parallel Cloud Comput. 3(4), 66–81 (2014)

    Article  Google Scholar 

  18. Amazon CloudWatch. http://aws.amazon.com/cloudwatch/

  19. Samuel, A.A., Akindele, B.A.: Proactive prediction models for web application resource provisioning in the cloud. In: International Conference on Transition from Observation to Knowledge to Intelligence, pp. 17–35 (2014)

  20. Bashar, A.: Autonomic scaling of cloud computing resources using BN-based prediction models. In: 2013 IEEE 2nd International Conference on Cloud Networking, pp. 200–204 (2013)

  21. Nikravesh, A.Y., et al.: Cloud resource auto-scaling system based on hidden markov model (HMM). In: 2014 IEEE International Conference on Semantic Computing, pp. 124–127, (2014)

  22. Lorido-Botran, T., et al.: A review of auto-scaling techniques for elastic applications in cloud environments. J. Grid Comput. 12(4), 559–592 (2014)

    Article  Google Scholar 

  23. Choi, J., et al.: VVM auto-scaling methods for high throughput computing on hybrid infrastructure. Clust. Comput. 18(3), 1063–1073 (2015)

    Article  Google Scholar 

  24. Simon, P.: Too Big to Ignore: The Business Case for Big Data. Wiley, Hoboken (2013)

    Google Scholar 

  25. Zhang, B.T.: Next-generation machine learning technologies. Commun. Korean Inst. Inf. Sci. Eng. 25(3), 96–107 (2007)

    Google Scholar 

  26. Choi, J.: An adaptive resource provisioning method based on application-aware job history learning technique, Masters Thesis, Sookmyung Womens University (2016)

  27. Weka. http://www.cs.waikato.ac.nz/ml/weka/

  28. Landis, J.R., et al.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)

    Article  MATH  Google Scholar 

  29. Amazon EC2. http://aws.amazon.com/

  30. Scientific computing. https://en.wikipedia.org/wiki/Computational_science

  31. Autodock. http://autodock.scripps.edu/

  32. Sjstrand, T., Mrenna, S., Skands, P.Z.: PYTHIA 6.4 Physics and Ma. J. High Energy Phys. 2006, 026 (2006)

    Article  MATH  Google Scholar 

  33. Lee, K.W.: Practical introduction to computational biology. Korean. Biophys. Soc. Newsl. 10(1), 10–22 (2004)

    Google Scholar 

  34. Calheiros, R.N., et al.: CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms, software: practice and experience, vol. 41, No. 1, pp. 23–50 (2011)

Download references

Acknowledgements

This research was supported by the Next-Generation Information Computing Development Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Science, ICT & Future Planning (2015M 3C 4A7065646).

Author information

Authors and Affiliations

  1. National Institute of Supercomputing and Networking, KISTI, Daejeon, 34141, Korea

    Jieun Choi

  2. Department of Computer Science, Sookmyung Women’s University, Seoul, 140-742, Korea

    Yoonhee Kim

Authors
  1. Jieun Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yoonhee Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yoonhee Kim.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Choi, J., Kim, Y. Adaptive resource provisioning method using application-aware machine learning based on job history in heterogeneous infrastructures. Cluster Comput 20, 3537–3549 (2017). https://doi.org/10.1007/s10586-017-1148-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-017-1148-1

Keywords

Search

Navigation