Abstract
Resource demands are a core aspect of performance models. They describe how an operation utilizes a resource and therefore influence the systems performance metrics: response time, resource utilization and throughput. Such demands can be determined by two extraction classes: direct measurement or demand estimation. Selecting the best suited technique depends on available tools, acceptable measurement overhead and the level of granularity necessary for the performance model. This work compares two direct measurement techniques and an adaptive estimation technique based on multiple statistical approaches to evaluate strengths and weaknesses of each technique. We conduct a series of experiments using the SPECjEnterprise2010 industry benchmark and an automatic performance model generator for architecture-level performance models based on the Palladio Component Model. To compare the techniques we conduct two experiments with different levels of granularity on a standalone system, followed by one experiment using a distributed SPECjEnterprise2010 deployment combining both extraction classes for generating a full-stack performance model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Becker, S., Koziolek, H., Reussner, R.: The Palladio Component Model for Model-Driven Performance Prediction. Journal of Systems and Software 82(1), 3–22 (2009). special Issue: Software Performance - Modeling and Analysis
Brosig, F., Gorsler, F., Huber, N., Kounev, S.: Evaluating approaches for performance prediction in virtualized environments. In: 2013 IEEE 21st International Symposium on Modeling, Analysis Simulation of Computer and Telecommunication Systems (MASCOTS), pp. 404–408, August 2013
Brosig, F., Kounev, S., Krogmann, K.: Automated extraction of palladio component models from running enterprise java applications. In: Proceedings of the 1st International Workshop on Run-time Models for Self-managing Systems and Applications (ROSSA 2009). ACM, New York (2009)
Brunnert, A., Krcmar, H.: Continuous Performance Evaluation and Capacity Planning Using Resource Profiles (under review). Journal of Systems and Software (2015)
Brunnert, A., Neubig, S., Krcmar, H.: Evaluating the Prediction Accuracy of Generated Performance Models in Up- and Downscaling Scenarios, pp. 113–130 (2014)
Brunnert, A., Vögele, C., Krcmar, H.: Automatic performance model generation for java enterprise edition (EE) applications. In: Balsamo, M.S., Knottenbelt, W.J., Marin, A. (eds.) EPEW 2013. LNCS, vol. 8168, pp. 74–88. Springer, Heidelberg (2013)
Dynatrace: Dynatrace Agent Timers. https://community.compuwareapm.com/community/display/DOCDT60/Agent+Timers (accessed: May 14, 2015)
Harchol-Balter, M.: Performance Modeling and Design of Computer Systems. Cambridge University Press, New York (2013)
Hofer, P., Hörschläger, F., Mössenböck, H.: Sampling-based steal time accounting under hardware virtualization. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015), pp. 87–90. ACM, New York (2015)
Huber, N., von Quast, M., Hauck, M., Kounev, S.: Evaluating and modeling virtualization performance overhead for cloud environments. In: Proceedings of the 1st International Conference on Cloud Computing and Services Science (CLOSER 2011), pp. 563–573. SciTePress, May 2011
Kuperberg, M.: Quantifying and Predicting the Influence of Execution Platform on Software Component Performance, vol. 5. KIT Scientific Publishing (2010)
Menascé, D.A.: Computing missing service demand parameters for performance models. In: Proceedings of the 2008 Computer Measurement Group Conference (CMG 2008), Las Vegas, NV, USA, pp. 241–248 (2008)
Rolia, J., Vetland, V.: Parameter estimation for performance models of distributed application systems. In: Proceedings of the 1995 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON 1995), p. 54. IBM (1995)
Spinner, S.: Evaluating Approaches to Resource Demand estimation. Master’s thesis, Karlsruhe Institute of Technology (KIT) (2011)
Spinner, S., Casale, G., Zhu, X., Kounev, S.: LibReDE: A library for resource demand estimation. In: Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014), pp. 227–228. ACM, New York (2014)
Wang, W., Huang, X., Qin, X., Zhang, W., Wei, J., Zhong, H.: Application-level CPU consumption estimation: towards performance isolation of multi-tenancy web applications. In: Proceedings of the 5th International Conference on Cloud Computing (CLOUD), pp. 439–446. IEEE (2012)
Willnecker, F., Brunnert, A., Gottesheim, W., Krcmar, H.: Using dynatrace monitoring data for generating performance models of java EE applications. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015), pp. 103–104. ACM, New York (2015)
Zheng, T., Woodside, M., Litoiu, M.: Performance Model Estimation and Tracking Using Optimal Filters. IEEE Transactions on Software Engineering 34(3), 391–406 (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Willnecker, F. et al. (2015). Comparing the Accuracy of Resource Demand Measurement and Estimation Techniques. In: Beltrán, M., Knottenbelt, W., Bradley, J. (eds) Computer Performance Engineering. EPEW 2015. Lecture Notes in Computer Science(), vol 9272. Springer, Cham. https://doi.org/10.1007/978-3-319-23267-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-23267-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23266-9
Online ISBN: 978-3-319-23267-6
eBook Packages: Computer ScienceComputer Science (R0)