Skip to main content
SpringerLink
Log in

VM auto-scaling methods for high throughput computing on hybrid infrastructure

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Cloud computing provides on-demand resource provisioning and scalable resources dynamically for the efficient use of computing resources. Scientific applications recently need a very large number of loosely coupled tasks to be handled efficiently. In response, current computing environments often consist of heterogeneous resources such as cloud computing. To effectively use cloud resources, auto-scaling methods that consider diverse metrics such as CPU utilization and costs of resource usage have been studied widely. However it still remains a challenge to automatically and timely allocate resources such that deadline violation and application types are considered. In this paper, we propose auto-scaling methods that consider specific conditions such as application types, task dependency, user-defined deadlines and data transfer times within a hybrid computing infrastructure. Our hybrid computing infrastructure consists of local cluster and cloud resources using HTCaaS. We observe noticeable improvements in performance when our auto-scaling methods for bag-of-tasks and workflow applications is applied.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Cirne, W., Brasileiro, F., Sauve, J., Andrade, N., Paranhos, D., Santos-Neto, E., Medeiros, R.: Grid computing for bag of jobs applications. In: Proceedings of the 3rd IFIP Conference on E-Commerce, E-Business and E-Government, 21–23 Sept 2003

  2. O’Brien, A., Newhouse, S., Darlington, J.: Mapping of Scientific Workflow Within the E-Protein project to Distributed Resources. In: UK E-Sceince All Hands Meeting, Nottingham (2004)

  3. Kang, H., Koh, J., Kim, Y.: “A SLA driven VM auto-scaling method in hybrid cloud environment. In: Network Operations and Management Symposium (APNOMS), 2013 15th Asia-Pacific, Hiroshima, Japan, 25–28 Sept 2013

  4. High-Throughput Computing as a Service(HTCaaS), http://htcaas.kisti.re.kr/

  5. Liu, C.-Y., Shie, M.-R., Lee, Y.-F., Lin, Y.-C., Lai, K.-C.: Vertical/horizontal resource scaling mechanism for federated clouds (2014)

  6. Lorido-Bortran, T., Miguel-Alonso, J., Lozano, J.A.: A review of auto-scaling techniques for elastic applications in cloud environments. J. Grid Comput. 12(4), 559–592 (2014)

    Article  Google Scholar 

  7. Bao, J., Lu, Z., Wu, J., Zhang, S., Zhong, Y.: Implementing a novel load-aware auto scale scheme for private cloud resource management platform. In: Network Operations and Management Symposium (NOMS) (2014)

  8. Yang, J., Liu, C., Shang, Y., Cheng, B., Mao Z., et al.: A cost-aware auto-scaling approach using the workload prediction in service clouds. In: 6th IEEE International Conference on Cloud Computing (CLOUD), pp. 810–815 (2013)

  9. Saleh, O., GropengieBer, F., Betz, H., Mandarawi W., Sattler, K.: Monitoring and auto-scaling iaas clouds: a case for complex event processing on data streams. In: 6th IEEE/ACM International Conference on Utility and Cloud Computing, pp. 387–392 (2013)

  10. Dutta, S., Gera, S., Vermam A., Viswanathan, B.: Smartscale: automatic application scaling in enterprise cloud. In: 5th IEEE International Conference on Cloud Comuting (CLOUD), pp. 221–228 (2012)

  11. Mao, M., Humphrey, M.: Scaling and Scheduling to Maximize Application Performance within Budget Constraints in Cloud Workflows. In: IEEE 27th International Symposium on Parallel and Distributed Processing (2013)

  12. Yu, J., Buyya R., Tham, C.K.: Cost-based scheduling of scientific workflow applications on utility grids. In: 1st IEEE International Conference on E-Science and Grid Computing, Melbourne, 5–8 Dec 2005

  13. Abrishami, S., Naghibzadeh, M., Epema, D.H.: Deadline-donstrained workflow scheduling algorithms for infrastructure as a service clouds. J. Future Gener. Comput. Syst. 29(1), 158–169 (2013)

    Article  Google Scholar 

  14. Bittencourt, L.F., Madeira, E.R.: A performance oriented adaptive scheduler for dependent tasks on grids. Concurr. Comput. 20(9), 1029–1049 (2008)

    Article  Google Scholar 

  15. Rizos, S., et al.: Integrated Research in GRID Computing. Scheduling workflows with budget constraints. Springer, Berlin (2007)

    Google Scholar 

  16. Niu, S., et al.: Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters. In: Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, p. 56 (2013)

  17. OpenStack, https://www.openstack.org/

  18. HMMER, http://hmmer.janelia.org/

  19. IMPALA, http://www.cloudera.com/content/cloudera/en/products-and-services/cdh/impala.html

  20. Johnson, M., Zaretskaya, I., Raytselis, Y., Merezhuk, Y., McGinnis, S., Maddent, T.L.: NCBI BLAST: a better web interface. Nucl. Acids Res. 36, W5–W9 (2008)

    Article  Google Scholar 

  21. Bergman, N.H., Bhagwat, M., Aravind, L.: PSI-BLAST Tutorial (2007)

  22. PSI-PRED, http://bioinf.cs.ucl.ac.uk/index.php?id=779

  23. Calheiros, R.N., Ranjan, R., Beloglazov, A., De Rose, C.A.F., Buyya, R.: Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw. Pract. Exp. 41(1), 23–50 (2011)

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (NRF-2013R1A1A3007866).

Author information

Authors and Affiliations

  1. Department of Computer Science, Sookmyung Women’s University, Seoul, 140-742, Korea

    Jieun Choi, Younsun Ahn & Yoonhee Kim

  2. National Institute of Supercomputing and Networking, KISTI, Daejeon, 305-806, Korea

    Seoyoung Kim

  3. School of Computer Science & Engineering, Soongsil University, Seoul, 156-743, Korea

    Jaeyoung Choi

Authors
  1. Jieun Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Younsun Ahn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seoyoung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yoonhee Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jaeyoung Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yoonhee Kim.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Choi, J., Ahn, Y., Kim, S. et al. VM auto-scaling methods for high throughput computing on hybrid infrastructure. Cluster Comput 18, 1063–1073 (2015). https://doi.org/10.1007/s10586-015-0462-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-015-0462-8

Keywords

Search

Navigation