Skip to main content
SpringerLink
Log in

Towards optimal resource provisioning for Hadoop-MapReduce jobs using scale-out strategy and its performance analysis in private cloud environment

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Cloud computing always provides IT resources on demand basis, without additional waiting time. Therefore, data analytics is one of the most significant areas that can be benefited from Cloud Computing. MapReduce programs in the cloud computing to optimize the resource provisioning and finish the MapReduce jobs with quantified time. The efficacy as well as the accuracy of performance of the performance model based on regression used for predicting the MapReduce job completion time has been suggested in our OpenStack private cloud Hadoop cluster using linear regression method. In order to satisfy the user jobs with deadline requirements, Cloud service providers do not have a resource provisioning technique or polices. The contemporary system requires a cloud user to estimate the needed quantity of resources for running jobs in the cloud. Our proposed scalability strategy of Scale-Out methods used to obtain the accurate prediction of job completion times through our experimental results shows the performance level of MapReduce benchmark in the open stack private cloud. The regression based performance model predicting and evaluating the execution time of 5 popular MapReduce benchmark applications over our private cloud environment with better resource utilization which depicts 99% of accuracy results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Khan, M., Jin, Y., Li, M., Xiang, Y., Jiang, C.: Hadoop performance modeling for job estimation and resource provisioning. IEEE Trans. Parallel Distrib. Syst. 27(2), 441–454 (2016)

    Article  Google Scholar 

  2. Lin, X., Meng, Z., Xu, C., Wang, M.: A practical performance model for Hadoop MapReduce. In: Proceedings of IEEE International Conference on Cluster Computing. Workshops, pp. 231–239 (2012)

  3. Cui, X., Lin, X., Hu, C., Zhang, R., Wang, C.: Modeling the performance of MapReduce under resource contentions and task failures. In: Proceedings of IEEE 5th International Conference on Cloud Computing Technology and Science, vol. 1, pp. 158–163 (2013)

  4. Liu, J., Zhang, Y., Zhou, Y., Zhang, D., Liu, H.: Aggressive resource provisioning for ensuring QoS in virtualized environments. IEEE Trans. Cloud Comput. 3(2), 119–131 (2015)

    Article  Google Scholar 

  5. Mao, M., Humphrey, M.: Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In: IEEE Xplore SC ‘11, Proceedings of 2011 International Conference for High Performance Computing, Networking Storage and Analysis, p. 49 (2011)

  6. Zhang, Q., Cherkasova, L., Smimi, E.: Regression-based analytic model for dynamic resource provisioning of multi-tier applications. In: Proceedings of the Fourth International conference on Autonomic Computing, Jacksonville, Florida, USA (2007)

  7. Davis, I.J., Hemmati, H., Holt, R.C., Godfrey, M.W., Neuse D.M., Mankovskii, S.: Regression-based utilization prediction algorithms: an empirical investigation. CASCON’13 Proceedings of the 2013, ACM, (2013)

  8. Marshall, P., Keahey, K., Freeman,T.: Elastic site using clouds to elastically extend site resources. InCluster, Cloud and Grid Computing (CCGrid), 2010 10th IEEE/ACM International Conference on IEEE, pp. 43–52 (2010)

  9. Hwang, K., Bai, X., Shi, Y., Li, M., Chen, W.G., Wu, Y.: Cloud performances modeling with benchmark evaluation of elastic scaling strategies. IEEE Trans. Parallel Distrib. Syst. 27(1), 130–143 (2016)

    Article  Google Scholar 

  10. Ostermann, S., Iosup, A., Yigitbasi, N., Prodan, R., Fahringer, T., Epema, D.: A performance analysis of EC2 cloud computing services for scientific computing, International Conference on Cloud Computing, vol. 34, pp. 115–131. Springer, New York (2009)

    Chapter  Google Scholar 

  11. Chen, K., Powers, J., Guo, S., Tian, F.: CRESP: towards optimal resource provisioning for MapReduce computing in public clouds. IEEE Trans. Parallel Distrib. Syst. 25(6), 1403–1412 (2014)

    Article  Google Scholar 

  12. Li, D., Chen, C., Guan, J., Zhang, Y., Zhu, J., Yu, R.: DCloud: deadline-aware resource allocation for cloud computing jobs. IEEE Trans. Parallel Distrib. Syst. 27(8), 2248–2260 (2016)

    Article  Google Scholar 

  13. da Rosa Right, R., Rodrigues, V.F., Da Costa, C.A., Galante, G., de Bona, L.C.E., Ferreto, T.: AutoElastic:automatic resource elasticity for high performance applications in the cloud. IEEE Trans. Cloud Comput. 4(1), 16–19 (2016)

    Google Scholar 

  14. Rodriguez, M.A., Buyya, R.: Deadline based resource provisioning and scheduling algorithm for scientific workflows on cloud. IEEE Trans. Cloud Comput. 2(2), 222–235 (2014)

    Article  Google Scholar 

  15. Mashayekhy, L., Nejad, M.M., Grosu, D.: A PTAS mechanism for provisioning and allocation of heterogeneous cloud resources. IEEE Trans. Parallel Distrib. Syst. 26(9), 2386–2399 (2015)

    Article  Google Scholar 

  16. Dai, W., Bassiouni, M.: An improved task assignment scheme for Hadoop running in the clouds. J. Cloud Comput. 2(1), 23 (2013)

    Article  Google Scholar 

  17. Pastorelli, M., Carra, D., Dell Amico, M., Michiardi, P.: HFSP: bringing size-based scheduling to hadoop. IEEE Trans. Cloud Comput. 5(1), 43–56 (2013)

    Article  Google Scholar 

  18. Calheiros, R.N., Masoumi, E., Ranjan, R., Buyya, R.: Workload prediction using ARIMA model and its impact on cloud applications’ QoS. IEEE Trans. Cloud Comput. 3(4), 449–458 (2015)

    Article  Google Scholar 

  19. Ji, C., Li, Y., Qiu, W., Awada, U., Li, K: Big data processing in cloud computing environments. In: International Symposium Pervasive Systems, Algorithms and Networks, pp. 17–23 (2012)

  20. Zhang, Z., Cherkasova, L., Loo, B.T.: Performance modeling of MapReduce jobs in heterogeneous cloud environments. In: IEEE Sixth international Conference on Cloud Computing (2013)

  21. Assuncao, M.D., Calheiros, R.N., Bianchi, S., Netto, M.A., Buyya, R.: Big data computing and clouds: trends and future directions. J. Parallel Distrib. Comput. 79, 3–15 (2015)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology, Dhanalakshmi College of Engineering, Chennai, India

    Ramakrishnan Ramanathan

  2. Department of Computer Science and Engineering, Sri Sairam Engineering College, Chennai, India

    B. Latha

Authors
  1. Ramakrishnan Ramanathan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. B. Latha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ramakrishnan Ramanathan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ramanathan, R., Latha, B. Towards optimal resource provisioning for Hadoop-MapReduce jobs using scale-out strategy and its performance analysis in private cloud environment. Cluster Comput 22 (Suppl 6), 14061–14071 (2019). https://doi.org/10.1007/s10586-018-2234-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-018-2234-8

Keywords

Search

Navigation