Abstract
The analysis and optimization of public clouds gains momentum as an important research topic, due to their widespread exploitation by individual users, researchers and companies for their daily tasks. We identify primitive algorithmic operations that should be part of a cloud analysis and optimization tool, such as resource profiling, performance spike detection and prediction, resource resizing, and others, and we investigate ways the collected monitoring information can be processed towards these purposes. The analyzed information is valuable in driving important virtual resource management decisions. We also present an open-source tool we developed, called SuMo,which contains the necessary functionalities for collecting monitoring data from Amazon Web Services (AWS), analyzing them and providing resource optimization suggestions. SuMo makes easy for anyone to analyze AWS instances behavior, incorporating a set of basic modules that provide profiling and spikef detection functionality. It can also be used as a basis for the development of new such analytic procedures for AWS. SuMo contains a Cost and Utilization Optimization (CUO) mechanism, formulated as an Integer Linear Programming (ILP) problem, for optimizing the cost and the utilization of a set of running Amazon EC2 instances. This CUO mechanism receives information on the currently used set of instances (their number, type, utilization) and proposes a new set of instances for serving the same load that minimizes cost and maximizes utilization and performance efficiency.
Similar content being viewed by others
References
Buyya, R., et al.: Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Futur. Gener. Comput. Syst. 25, 599–616 (2009)
Rimal, B.P., Jukan, A., Katsaros, D., Goeleven, I.: Architecture requirementes for cloud computing systems: An enterprise cloud approach. J. Grid Comput. 9(1), 3–26 (2011)
Amazon Web Services – AWS: aws.amazon.com, last seen January 2014
RackSpace: www.rackspace.com, last seen January 2014
Openstack: www.openstack.org, last seen January 2014
OpenNebula: opennebula.org, last seen January 2014
Eucaluptus: www.eucalyptus.com, last seen January 2014
Amazon data center size: huanliu.wordpress.com/2012/03/13/amazon-data-center-size, last seen January 2014
Designs, Lessons and Advice from Building Large Distributed Systems www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf
Amazon Case Studies: aws.amazon.com/solutions/case-studies/
Wang, H., et al.: Distributed systems meet economics: pricing in the cloud USENIX Hot Topics in Cloud Computing (HotCloud) (2010)
Chen, J., et al.: Tradeoffs Between Profit and Customer Satisfaction for Service Provisioning in the Cloud International Symposium on High performance Distributed Computing (HPDC) (2011)
Zanikolas, S., Sakellariou, R.: A Taxonomy of grid monitoring systems. FGCS 21(1), 163–188 (2005)
Kung, H.T., Lin, C.-K., Vlah, D.: CloudSense: Continuous fine-grain cloud monitoring with compressive sensing. USENIX HotCloud (2011)
Petcu, D., et al.: Experiences in building a mOSAIC of clouds. J. Cloud Comput. 2(1) (2013)
Ferrer, A.J., et al.: OPTIMIS: A holistic approach to cloud service provisioning. Futur. Gener. Comput. Syst. 28(1), 66–77 (2012)
Mallick, S.: Virtualization based cloud capacity prediction. HPCS, 849–852 (2011)
De Chaves, S., et al.: Toward an architecture for monitoring private clouds. IEEE Comm. Mag. 49(12), 130–137 (2011)
Ward, J. S., Barker, A.: Semantic Based Data Collection for Large Scale Cloud Systems. DIDC, pp. 13–22 (2012)
Shao, J., Wei, H., Wang, Q., Mei, H.: A Runtime Model Based Monitoring Approach for Cloud. IEEE CLOUD, pp. 313–320 (2010)
Meng, S., et al.: Reliable State Monitoring in Cloud Datacenters. IEEE CLOUD, pp. 951–958 (2012)
Weng, J., et al.: Event Detection in Twitter. HP Laboratories (2011)
Kokkinos, P., Kretsis, A., Varvarigou, T., Varvarigos, E.: Social-like Analysis on Virtual Machine Communication Traces. IEEE Cloudnet (2012)
Malkowski, S., Hedwig, M., Jayasinghe, D., Pu, C., Neumann, D.: CloudXplor: A tool for configuration planning in clouds based on empirical data. ACM Symposium on Applied Computing (SAC) (2010)
M. Frîncu: Scheduling highly available applications on cloud environments. Futur. Gener. Comput. Syst. 32, 138–153 (2014)
Kokkinos, P., et al.: Cost and Utilization Optimization of Amazon EC2 instances. IEEE Sixth International Conference on Cloud Computing, pp. 518–525 (2013)
Amazon Elastic Compute Cloud: aws.amazon.com/ec2, last seen January 2014
Amazon CloudWatch: aws.amazon.com/cloudwatch, last seen January 2014
Nagios: www.nagios.org, last seen January 2014
Newvem: www.newvem.com, last seen January 2014
Cloudability: cloudability.com, last seen January 2014
Cloudvertical. www.cloudvertical.com, last seen January 2014
boto - A Python interface to Amazon Web Services: docs.pythonboto.org, last seen January 2014
SciPy: www.scipy.org, last seen January 2014
NumPy: numpy.scipy.org, last seen January 2014
IBM ILOG CPLEX Optimizer: www-01.ibm.com/software/integration/optimization/cplex-optimizer, last seen January 2014
On demand instances pricing: aws.amazon.com/ec2/pricing/pricing-on-demand-instances.json, last seen January 2014
Pinterest use case: www.theregister.co.uk/2012/04/30/inside_pinterest_virtual_data_center, last seen January 2014
mOSAIC project: www.mosaic-cloud.eu, last seen January 2014
Optimis project: www.optimis-project.eu, last seen January 2014
Aeolus project: www.aeolusproject.org, last seen January 2014
Data Set for IMC 2010 Data Center Measurement: pages.cs.wisc.edu/%7Etbenson/IMC10_Data.html, last seen January 2014
Google. Google Cluster Data V1. Available: http://code.google.com/p/googleclusterdata/wiki/TraceVersion1, last seen January 2014
Kavulya, S., et al.: An analysis of traces from a production mapreduce cluster. Cluster, Cloud and Grid Comput. (CCGrid), pp. 94–103 (2010)
Kertesz, A., Kecskemeti, G., Oriol, M., Kotcauer, P., Acs, S., Rodrıguez, M., Merce, O., Marosi, A.C., Marco, J., Franch, X.: Enhancing Federated Cloud Management with an Integrated Service Monitoring Approach. J. Grid Comput. 11(4), 699–720 (2013)
Mendez, V., Casajus, A., Fernandez, V., Graciani, R., Merino, G.: Rafhyc: An Architecture for Constructing Resilient Services on Federated Hybrid Clouds. J. Grid Comput. 11(4), 753–770 (2013)
SuMo-tool: https://github.com/SuMo-tool, last seen January 2014
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kokkinos, P., Varvarigou, T.A., Kretsis, A. et al. SuMo: Analysis and Optimization of Amazon EC2 Instances. J Grid Computing 13, 255–274 (2015). https://doi.org/10.1007/s10723-014-9311-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10723-014-9311-x