research-article

Deriving Parameters for Open and Closed QN Models of Operational Systems Through Black Box Optimization

Authors:

Daniel A. MenasceAuthors Info & Claims

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Pages 127 - 138

https://doi.org/10.1145/3030207.3030208

Published: 17 April 2017 Publication History

Abstract

Black-box modeling techniques are used when modeling computer systems with unknown internal structure or behavior and/or when it is not feasible or too time consuming to monitor a running computer system. The main challenge in these situations lies in estimating values for the parameters of these models, especially the values of service demands at the various devices for each transaction class. These estimates have to be compliant with the input-output relationships observed through measurements. This means that solving a model of the system with the estimated parameters should yield the same outputs (e.g., response times) for the same inputs (e.g., arrival rates or concurrency level). This paper presents a method for automatically estimating service demands for open, closed, single and multiclass queuing networks (QN). The method is based on casting the estimation problem as a non-linear optimization problem. However, because the solution of closed QNs does not have a closed form, we need to resort to black-box optimization techniques. The parameter estimation method presented here is part of iModel, a framework for automatically deriving performance models of systems whose detailed characteristics (structure and behavior) are unknown. Other portions of the framework were discussed in detail in previous publications by the authors. This paper illustrates the ideas through several numerical examples and then applies them to a multi-tiered operational system running OFBiz. The estimated service demands closely satisfy the input-output relationships at various workload intensity levels and can be used for prediction purposes.

References

[1]

Awad, M. and Menascé, D.A., "Performance Model Derivation of Operational Systems Through Log Analysis,"IEEE Intl. Symp. Modeling, Analysis and Simulation of Computer Systems and Telecommunication Systems (MASCOTS 2016), Imperial College, London, UK, Sept. 19--21, 2016.

[2]

Awad, M. and Menascé, D.A., "On the Predictive Properties of Performance Models Derived Through Input-Output Relationships", Proc. European Performance Engineering Workshop (EPEW 2014), Venice, Italy, 2014.

[3]

Awad, M. and Menascé, D.A., "Dynamic Derivation of Analytical Performance Models in Autonomic Computing Environments", Proc. Computer Measurement Group Conf. Performance and Capacity, Atlanta, GA, 2014.

[4]

Barna, C., Litoiu, M., Ghanbari, H., "Autonomic load-testing framework," Proc. 8th ACM Intl. Conf. Autonomic Computing,Karlsruhe, Germany, 2011, pp. 91--100.

Digital Library

[5]

Begin, T., Brandwajn, A., Baynat, B., Wolfinger, B.E., Fdida, S., "Towards an automatic modeling tool for observed system behavior," Formal Methods and Stochastic Models for Performance Evaluation, pp. 200--212. Springer Berlin Heidelberg (2007)\balance

[6]

Begin, T., Brandwajn, A., Baynat, B., Wolfinger, B. E., Fdida, S., "High-level approach to modeling of observed system behavior," ACM SIGMETRICS Performance Evaluation Review, 35(3), 34--36 (2007)

Digital Library

[7]

Begin, T., Baynat, B., Sourd, F., Brandwajn, A., "A DFO technique to calibrate queuing models," Computers & Operations Research, 37, no. 2, 2010, pp. 273--281.

Digital Library

[8]

Durillo,J.J. and Nebro, A.J., "jMetal: a Java Framework for Multi-Objective Optimization,"Advances in Engineering Software, 42(2011), pp. 760--771.

Digital Library

[9]

Ghanbari, H., Barna, C., Litoiu, M., Woodside, M., Zheng, T., Wong, J., Iszlai, G., "Tracking Adaptive Performance Models Using Dynamic Clustering of User Classes," Proc. 2nd ACM/SPEC Intl. Conf. Performance Engineering, Karlsruhe, Germany, March 14--16, 2011.

Digital Library

[10]

Gmach, D., J. Rolia, L. Cherkasova, and A. Kemper. "Workload Analysis and Demand Prediction of Enterprise Data Center Applications," Proc. 2007 IEEE 10th Intl. Symp. Workload Characterization (IISWC '07), Boston, MA, Sept. 27--29, 2007.

Digital Library

[11]

Herbst, N.R., N. Huber, S. Kounev, and E. Amrehn. "Self-adaptive workload classification and forecasting for proactive resource provisioning." Proc. 4th ACM/SPEC Intl. Conf. Performance Engineering (ICPE '13),Prague, Czech Republic, April 21--24, 2013.

Digital Library

[12]

Huber, N., F. Brosig, S. Spinner, S. Kounev, and M. Bahr, "Model-Based Self-Aware Performance and Resource Management Using the Descartes Modeling Language,"to appear in the IEEE Tr. Software Engineering, DOI 10.1109/TSE.2016.2613863

Digital Library

[13]

Incerto, E., M. Tribastone, and C. Trubiani, "Symbolic Performance Adaptations"Proc. 11th Intl. Symp. Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2016), May 16--17, 2016, Austin, TX.

[14]

Khan A., X. Yan, S. Tao and N. Anerousis, "Workload characterization and prediction in the cloud: A multiple time series approach," 2012 IEEE Network Operations and Management Symposium, Maui, HI, 2012

[15]

itoiu, M., Woodside, M., Zheng, T., "Hierarchical model-based autonomic control of software systems," ACM SIGSOFT Software Engineering Notes, Vol. 30, No. 4, pp. 1--7 (2005)\balance

[16]

Little, J.D.C., "A Proof for the Queuing Formula: L = ł W," Operations Research. 9 (3), 1961, pp 383--387.

Digital Library

[17]

Menascé, D., "Computing Missing Service Demand Parameters for Performance Models," Proc. Thirty-fourth Intl. Computer Measurement Group Conf., Las Vegas, NV, Dec. 7--12, 2008, pp. 7--12.

[18]

Menascé, D.A., Almeida, V.A.F., and Dowdy, L., "Performance by Design: Computer Capacity Planning By Example." Prentice Hall, 2004.

[19]

Menascé, D.A. and Almeida, V.A.F., "Scaling for E-Business: technologies, models, performance, and capacity planning," Prentice Hall, 2000.

[20]

Reiser, M. and Lavenberg, S. S. "Mean-Value Analysis of Closed Multichain Queuing Networks," J. ACM., 27(2), April 1980, pp. 313--322.

Digital Library

[21]

Pérez, J.F., Pacheco-Sanchez, S., Casale, G., "An Offline Demand Estimation Method for Multi-Threaded Applications," Proc. 2013 IEEE 21st Intl. Symp. Modeling, Analysis and Simulation of Computer and Telecommunication Systems (pp. 21--30), IEEE Computer Society, 2013.

Digital Library

[22]

Schweitzer, P. "Approximate analysis of multiclass closed networks of queues." Proc. Intl. Conf. Stochastic Control and Optimization, 1979.

[23]

Seidmann, A., P. Schweitzer, and S. Shalev-Oren, "Computerized Closed Queueing Network Models of Flexible Manufacturing," Large Scale System J., North Holland, Vol. 12, 1987, pp. 91--107.

[24]

Spinner S., G. Casale, F. Brosig, and S. Kounev., "Evaluating Approaches to Resource Demand Estimation," Performance Evaluation, 92:51 - 71, October 2015, Elsevier B.V.

Digital Library

[25]

Woodside, M., Zheng, T., Litoiu, M., "The use of optimal filters to track parameters of performance models," Second Intl. Conf. Quantitative Evaluation of Systems, Torino, Italy, Sept. 19--22, 2005, pp. 74--83.

Digital Library

[26]

Zheng, T., Yang, J., Woodside, M., Litoiu, M., Iszlai, G. "Tracking time-varying parameters in software systems with extended Kalman filters," Proc. 2005 Conf. of the Centre for Advanced Studies on Collaborative Research, 2005, pp. 334--345.

Digital Library

Cited By

Incerto EPizziol RTribastone M(2023)μOpt: An Efficient Optimal Autoscaler for Microservice Applications2023 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS)10.1109/ACSOS58161.2023.00024(67-76)Online publication date: 25-Sep-2023
https://doi.org/10.1109/ACSOS58161.2023.00024
Incerto ETribastone MApte VDi Marco ALitoiu MMerseguer J(2019)Model-based Performance Self-adaptationCompanion of the 2019 ACM/SPEC International Conference on Performance Engineering10.1145/3302541.3310293(49-52)Online publication date: 27-Mar-2019
https://dl.acm.org/doi/10.1145/3302541.3310293
Chatley RField TWei D(2019)Continuous Performance Testing in Virtual Time2019 IEEE International Conference on Software Architecture Companion (ICSA-C)10.1109/ICSA-C.2019.00027(109-115)Online publication date: Mar-2019
https://doi.org/10.1109/ICSA-C.2019.00027
Show More Cited By

Index Terms

Deriving Parameters for Open and Closed QN Models of Operational Systems Through Black Box Optimization
1. Applied computing
  1. Operations research
2. Mathematics of computing
  1. Mathematical analysis
    1. Nonlinear equations
  2. Probability and statistics
    1. Statistical paradigms
      1. Cluster analysis
      2. Queueing theory

Recommendations

iModel: Automatic Derivation of Analytic Performance Models

Deriving analytic performance models requires detailed knowledge of the architecture and behavior of the computer system being modeled as well as modeling skills. This detailed knowledge may not be readily available (or it may be impractical to gather) ...
A Unifying Framework for the Approximate Solution of Closed Multiclass Queuing Networks

Queuing network models of modern computing systems must consider a large number of components (e.g., Web servers, DB servers, application servers, firewall, routers, networks) and hundreds of customers with very different resource requirements. The ...
Estimation of parameters for trend-renewal processes

Methods of estimating unknown parameters of a trend function for trend-renewal processes are investigated in the case when the renewal distribution function is unknown. If the renewal distribution is unknown, then the likelihood function of the trend-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

April 2017

450 pages

ISBN:9781450344043

DOI:10.1145/3030207

General Chairs:
Walter Binder
Università della Svizzera Italiana, Switzerland
,
Vittorio Cortellessa
Università dell'Aquila, Italy
,
Program Chairs:
Anne Koziolek
Karlsruhe Institute of Technology, Germany
,
Evgenia Smirni
College of William & Mary, USA
,
Meikel Poess
Oracle Corporation, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation
SIGSOFT: ACM Special Interest Group on Software Engineering
SPEC: SPEC Research Group

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

United States Air Force Office of Scientific Research

Conference

ICPE '17

Sponsor:

ICPE '17: ACM/SPEC International Conference on Performance Engineering

April 22 - 26, 2017

L'Aquila, Italy

Acceptance Rates

ICPE '17 Paper Acceptance Rate 27 of 83 submissions, 33%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
126
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Incerto EPizziol RTribastone M(2023)μOpt: An Efficient Optimal Autoscaler for Microservice Applications2023 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS)10.1109/ACSOS58161.2023.00024(67-76)Online publication date: 25-Sep-2023
https://doi.org/10.1109/ACSOS58161.2023.00024
Incerto ETribastone MApte VDi Marco ALitoiu MMerseguer J(2019)Model-based Performance Self-adaptationCompanion of the 2019 ACM/SPEC International Conference on Performance Engineering10.1145/3302541.3310293(49-52)Online publication date: 27-Mar-2019
https://dl.acm.org/doi/10.1145/3302541.3310293
Chatley RField TWei D(2019)Continuous Performance Testing in Virtual Time2019 IEEE International Conference on Software Architecture Companion (ICSA-C)10.1109/ICSA-C.2019.00027(109-115)Online publication date: Mar-2019
https://doi.org/10.1109/ICSA-C.2019.00027
Wang WCasale GKattepur ANambiar M(2018)QMLEACM Transactions on Modeling and Performance Evaluation of Computing Systems10.1145/32331803:4(1-28)Online publication date: 22-Aug-2018
https://dl.acm.org/doi/10.1145/3233180
Incerto ENapolitano ATribastone M(2018)Moving Horizon Estimation of Service Demands in Queuing Networks2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)10.1109/MASCOTS.2018.00040(348-354)Online publication date: Sep-2018
https://doi.org/10.1109/MASCOTS.2018.00040
Tribastone M(2018)Towards Software Performance by ConstructionLeveraging Applications of Formal Methods, Verification and Validation. Modeling10.1007/978-3-030-03418-4_27(466-470)Online publication date: 29-Oct-2018
https://doi.org/10.1007/978-3-030-03418-4_27

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten