Abstract
Model-based performance prediction methods aim at evaluating the expected response time, throughput, and resource utilisation of a software system at design time, before implementation. Existing performance prediction methods use monolithic, throw-away prediction models or component-based, reusable prediction models. While it is intuitively clear that the development of reusable models requires more effort, the actual higher amount of effort has not been quantified or analysed systematically yet. To study the effort, we conducted a controlled experiment with 19 computer science students who predicted the performance of two example systems applying an established, monolithic method (Software Performance Engineering) as well as our own component-based method (Palladio). The results show that the effort of model creation with Palladio is approximately 1.25 times higher than with SPE in our experimental setting, with the resulting models having comparable prediction accuracy. Therefore, in some cases, the creation of reusable prediction models can already be justified, if they are reused at least once.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Balsamo, S., Di Marco, A., Inverardi, P., Simeoni, M.: Model-Based Performance Prediction in Software Development: A Survey. IEEE TSE 30(5), 295–310 (2004)
Balsamo, S., Marzolla, M.: A Simulation-Based Approach to Software Performance Modeling. In: Proc. of ESEC/FSE, pp. 363–366. ACM Press, New York (2003)
Balsamo, S., Marzolla, M., Di Marco, A., Inverardi, P.: Experimenting different software architectures performance techniques. In: Proc. of WOSP, pp. 115–119. ACM Press, New York (2004)
Basili, V.R., Caldiera, G., Rombach, H.D.: The Goal Question Metric Approach. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering - 2 Volume Set, pp. 528–532. John Wiley & Sons, Chichester (1994)
Becker, S., Grunske, L., Mirandola, R., Overhage, S.: Performance Prediction of Component-Based Systems: A Survey from an Engineering Perspective. In: Reussner, R., Stafford, J.A., Szyperski, C.A. (eds.) Architecting Systems with Trustworthy Components. LNCS, vol. 3938, pp. 169–192. Springer, Heidelberg (2006)
Becker, S., Koziolek, H., Reussner, R.: Model-based Performance Prediction with the Palladio Component Model. In: Proc. of WOSP, February 5–8, 2007, pp. 54–65. ACM Sigsoft (2007)
Bertolino, A., Mirandola, R.: CB-SPE Tool: Putting Component-Based Performance Engineering into Practice. In: Crnković, I., Stafford, J.A., Schmidt, H.W., Wallnau, K. (eds.) CBSE 2004. LNCS, vol. 3054, pp. 233–248. Springer, Heidelberg (2004)
Bondarev, E., de With, P.H.N., Chaudron, M.: Predicting Real-Time Properties of Component-Based Applications. In: Proc. of RTCSA (2004)
Gorton, I., Liu, A.: Performance Evaluation of Alternative Component Architectures for Enterprise JavaBean Applications. IEEE Internet Computing 7(3), 18–23 (2003)
Höst, M., Regnell, B., Wohlin, C.: Using students as subjects - A comparative study of students and professionals in lead-time impact assessment. Empirical Software Engineering 5(3), 201–214 (2000)
Koziolek, H., Firus, V.: Empirical Evaluation of Model-based Performance Predictions Methods in Software Development. In: Reussner, R., Mayer, J., Stafford, J.A., Overhage, S., Becker, S., Schroeder, P.J. (eds.) QoSA 2005 and SOQUA 2005. LNCS, vol. 3712, pp. 188–202. Springer, Heidelberg (2005)
Martens, A.: Empirical Validation of the Model-driven Performance Prediction Approach Palladio. Master’s thesis, Carl-von-Ossietzky Universität Oldenburg (November 2007)
Martens, A., Becker, S., Koziolek, H., Reussner, R.: An empirical investigation of the applicability of a component-based performance prediction method. In: EPEW 2008, Palma de Mallorca, Spain (accepted, 2008)
Menascé, D.A., Almeida, V.A.F., Dowdy, L.W.: Performance by Design. Prentice-Hall, Englewood Cliffs (2004)
Petriu, D.C., Wang, X.: From UML description of high-level software architecture to LQN performance models. In: Nagl, M., Schürr, A., Münch, M. (eds.) AGTIVE 1999. LNCS, vol. 1779, Springer. Heidelberg (2000)
Poulin, J.S.: Measuring software reuse: principles, practices, and economic models. Addison-Wesley Longman Publishing Co., Inc., Boston (1996)
Rausch, A., Reussner, R., Mirandola, R., Plasil, F. (eds.): The Common Component Modeling Example: Comparing Software Component Models. LNCS. Springer, Heidelberg (to appear, 2008)
Sachs, L.: Applied Statistics: A Handbook of Techniques. Springer, New York (1982)
Smith, C.U.: Performance Engineering of Software Systems. Addison-Wesley, Reading (1990)
Smith, C.U., Williams, L.G.: Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software. Addison-Wesley, Reading (2002)
Szyperski, C.: Component Software: Beyond Object-Oriented Programming. ACM Press, Addison-Wesley, Reading (1998)
Welch, B.L.: The generalization of student’s problem when several different population variances are involved. Biometrika 34, 28–35 (1947)
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering: an Introduction. Kluwer Academic Publishers, Dordrecht (2000)
Wu, X., Woodside, M.: Performance Modeling from Software Components. SIGSOFT SE Notes 29(1), 290–301 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Martens, A., Becker, S., Koziolek, H., Reussner, R. (2008). An Empirical Investigation of the Effort of Creating Reusable, Component-Based Models for Performance Prediction. In: Chaudron, M.R.V., Szyperski, C., Reussner, R. (eds) Component-Based Software Engineering. CBSE 2008. Lecture Notes in Computer Science, vol 5282. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87891-9_2
Download citation
DOI: https://doi.org/10.1007/978-3-540-87891-9_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-87890-2
Online ISBN: 978-3-540-87891-9
eBook Packages: Computer ScienceComputer Science (R0)