Abstract
Cloud services are emerging today as an innovative IT provisioning model, offering benefits over the traditional approach of provisioning infrastructure. However, the occurrence of multi-tenancy, virtualization and resource sharing issues raise certain difficulties in providing performance estimation during application design or deployment time. In order to assess the performance of cloud services and compare cloud offerings, cloud benchmarks are required. The aim of this paper is to present a mechanism and a benchmarking process for measuring the performance of various cloud service delivery models, while describing this information in a machine understandable format. The suggested framework is responsible for organizing the execution and may support multiple cloud providers. In our work context, benchmarking measurement results are demonstrated from three large commercial cloud providers, Amazon EC2, Microsoft Azure and Flexiant in order to assist with provisioning decisions for cloud users. Furthermore, we present approaches for measuring service performance with the usage of specialized metrics for ranking the services according to a weighted combination of cost, performance and workload.
This is a preview of subscription content, log in via an institution to check access.
References
ARTIST Consortium (2013), Deliverable D7.2 v1.0- PaaS/IaaS Metamodelling Requirements and SOTA. http://www.artist-project.eu/sites/default/files/D7.2%20PaaS%20IaaS%20metamodeling%20requirements%20and%20SOTA_M4_31012013.pdf
REMICS Consortium (2012), Deliverable D4.1 v2.0 - PIM4Cloud. http://www.remics.eu/system/files/REMICS_D4.1_V2.0_LowResolution.pdf
Kousiouris, G., Kyriazis, D., Menychtas, A., Varvarigou, T.: Legacy applications on the cloud: challenges and enablers focusing on application performance analysis and providers characteristics. In: Proceedings of the 2012 2nd IEEE International Conference on Cloud Computing and Intelligence Systems (IEEE CCIS 2012), Hangzhou, China, 30 October–1 November 2012
Hauck, M., Huber, M., Klems, M., Kounev, S., Muller-Quade, J., Pretschner, A., Reussner, R., Tai, S.: Challenges and opportunities of Cloud computing. Karlsruhe Reports in Informatics 19, Karlsruhe Institute of Technology - Faculty of Informatics (2010)
Ou, Z., Zhuang, H., Nurminen, J.K., Ylä-Jääski, A., Hui, P.: Exploiting hardware heterogeneity within the same instance type of Amazon EC2. In: Proceedings of the 4th USENIX Conference on Hot Topics in Cloud Computing (HotCloud 2012), p. 4. USENIX Association, Berkeley (2012)
Kousiouris, G., Cucinotta, T., Varvarigou, T.: The effects of scheduling, workload type and consolidation scenarios on virtual machine performance and their prediction through optimized artificial neural networks. J. Syst. Softw. 84(8), 1270–1291 (2011). doi:10.1016/j.jss.2011.04.013. Elsevier
Koh, Y., Knauerhase, R., Brett, P., Bowman, M., Wen, Z., Pu, C.: An analysis of performance interference effects in virtual environments. In: IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 200–209, April 2007
Schaffrath, G., Schmid, S., Vaishnavi, I., Khan, A., Feldmann, A.: A resource description language with vagueness support for multi-provider cloud networks. In: International Conference on Computer Communication Networks (ICCCN 2012), Munich, Germany (2012)
Charlton, S.: Model driven design and operations for the Cloud. In: OOPSLA 2009, 14th Conference Companion on Object Oriented Programming Systems Languages and Applications, pp. 17–26 (2009)
Charão, A.S., Primet, P.V.-B., Koslovski, G.P.: VXDL: virtual resources and interconnection networks description language. In: Kudoh, T., Mambretti, J., Vicat-Blanc Primet, P. (eds.) GridNets 2008. LNICST, vol. 2, pp. 138–154. Springer, Heidelberg (2009)
Mirkovic, J., Faber, T., Hsieh, P., Malayandisamu, G., Malavia, R.: DADL: Distributed Application Description Language. USC/ISI Technical report ISI-TR-664 (2010)
Garg, S.K., Versteeg, S., Buyya, R.: A framework for ranking of Cloud computing services. Future Gener. Comput. Syst. 29(4), 1012–1023 (2013). ISSN 0167-739X http://dx.doi.org/10.1016/j.future.2012.06.006
Iosup, A., Prodan, R., Epema, D.: Iaas cloud benchmarking: approaches, challenges, and experience. In: Proceedings of the International Conference on High Performance Networking and Computing (SC), MTAGS 2012, pp. 1–8. IEEE/ACM (2012)
Li, A., Yang, X., Kandula, S., Zhang, M.: CloudCmp: comparing public Cloud providers. In: Proceedings of the 10th ACM SIGCOMM Conference on Internet measurement (IMC 2010), pp. 1–14. ACM, New York (2010). doi:10.1145/1879141.1879143, http://doi.acm.org/10.1145/1879141.1879143
ARTIST Consortium, Deliverable D7.2.1 v1.0- Cloud services modelling and performance analysis framework (2013). http://www.artist-project.eu/sites/default/files/D7.2.1%20Cloud%20services%20modeling%20and%20performance%20analysis%20framework_M12_30092013.pdf
Folkerts, E., Alexandrov, A., Sachs, K., Iosup, A., Markl, V., Tosun, C.: Benchmarking in the Cloud: what it should, can, and cannot be. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 173–188. Springer, Heidelberg (2013)
Milenkoski, A., Iosup, A., Kounev, S., Sachs, K., Rygielski, P., Ding, J., Cirne, W., Rosenberg, F.: Cloud Usage Patterns: A Formalism for Description of Cloud Usage Scenarios. Technical report SPEC-RG-2013-001 v.1.0.1, SPEC Research Group - Cloud Working Group, Standard Performance Evaluation Corporation (SPEC), April 2013
Ardagna, D., Di Nitto, E., Casale, G., Petcu, D., Mohagheghi, P., Mosser, S., Matthews, P., Gericke, A., Balligny, C., D’Andria, F., Nechifor, C.-S., Sheridan, C.: MODACLOUDS: a model-driven approach for the design and execution of applications on multiple clouds. In: ICSE MiSE: International Workshop on Modelling in Software Engineering, pp. 50–56. IEEE/ACM (2012)
Acknowledgements
The research leading to these results is partially supported by the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 317859, in the context of the ARTIST Project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Evangelinou, A. et al. (2015). Experimenting with Application-Based Benchmarks on Different Cloud Providers via a Multi-cloud Execution and Modeling Framework. In: Helfert, M., Desprez, F., Ferguson, D., Leymann, F., Méndez Munoz, V. (eds) Cloud Computing and Services Sciences. CLOSER 2014. Communications in Computer and Information Science, vol 512. Springer, Cham. https://doi.org/10.1007/978-3-319-25414-2_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-25414-2_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-25413-5
Online ISBN: 978-3-319-25414-2
eBook Packages: Computer ScienceComputer Science (R0)