Skip to main content
Springer Nature Link
Log in

Performance and reliability prediction for evolving service-oriented software systems

Industrial experience report

  • Experience Report
  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

During software system evolution, software architects intuitively trade off the different architecture alternatives for their extra-functional properties, such as performance, maintainability, reliability, security, and usability. Researchers have proposed numerous model-driven prediction methods based on queuing networks or Petri nets, which claim to be more cost-effective and less error-prone than current practice. Practitioners are reluctant to apply these methods because of the unknown prediction accuracy and work effort. We have applied a novel model-driven prediction method called Q-ImPrESS on a large-scale process control system from ABB consisting of several million lines of code. This paper reports on the achieved performance prediction accuracy and reliability prediction sensitivity analyses as well as the effort in person hours for achieving these results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Notes

  1. http://www.q-impress.eu/wordpress/demonstrators/abb-demonstrator/

References

  • Balsamo S, Di Marco A, Inverardi P, Simeoni M (2004) Model-based performance prediction in software development: a survey. IEEE Trans Softw Eng 30(5):295–310

    Article  Google Scholar 

  • Basili VR, Caldiera G, Rombach HD (2002) The goal question metric approach. In: Marciniak JJ (ed) Encyclopedia of software engineering, 2nd edn. Wiley, New York, pp 578–583

    Google Scholar 

  • Bause F (1993) Queueing Petri nets—a formalism for the combined qualitative and quantitative analysis of systems. In: Proceedings of the 5th international workshop on Petri nets and performance models, pp 14–23

  • Becker S (2008) Coupled model transformations for QoS enabled component-based software design. PhD thesis, University of Oldenburg, Germany

  • Becker S, Bulej L, Bures T, Hnetynka P, Kapova L, Kofron J, Koziolek H, Kraft J, Mirandola R, Stammel J, Tamburelli G, Trifu M (2008) Q-ImPrESS project deliverable D2.1: service architecture meta model (SAMM). Tech. Rep. 1.0, Q-ImPrESS Consortium. http://bit.ly/bfSGrh

  • Becker S, Koziolek H, Reussner R (2009) The Palladio component model for model-driven performance prediction. J Syst Softw 82(1):3–22

    Article  Google Scholar 

  • Brosch F, Koziolek H, Buhnova B, Reussner R (2010) Parameterized reliability prediction for component-based software architectures. In: Proc. 6th int. conf. on the quality of software architectures (QoSA’10). LNCS, vol 6093. Springer, New York, pp 36–51

    Google Scholar 

  • Cheung L, Roshandel R, Medvidovic N, Golubchik L (2008) Early prediction of software component reliability. In: ICSE ’08: proceedings of the 30th international conference on software engineering. ACM, pp 111–120. doi:10.1145/1368088.1368104

  • Cheung RC (1980) A user-oriented software reliability model. IEEE Trans Softw Eng SE-6(2):118–125

    Article  MATH  Google Scholar 

  • Cortellessa V, Marco AD, Inverardi P (2011) Model-based software performance analysis. Springer, New York

    Book  Google Scholar 

  • Fenton NE, Neil M (1999) A critique of software defect prediction models. IEEE Trans Softw Eng 25(5):675–689

    Article  Google Scholar 

  • Franks G, Al-Omari T, Woodside M, Das O, Derisavi S (2009) Enhanced modeling and solution of layered queueing networks. IEEE Trans Softw Eng 35(2):148–161. doi:10.1109/TSE.2008.74

    Article  Google Scholar 

  • Gokhale SS (2007) Architecture-based software reliability analysis: overview and limitations. IEEE Trans Dependable Secure Comput 4(1):32–40. doi:10.1109/TDSC.2007.4

    Article  Google Scholar 

  • Gokhale SS, Wong WE, Horgan JR, Trivedi KS (2004) An analytical approach to architecture-based software performance and reliability prediction. Perform Eval 58(4):391–412

    Article  Google Scholar 

  • Goseva-Popstojanova K, Hamill M, Perugupalli R (2005) Large empirical case study of architecture-based software reliability. In: Proc. 16th IEEE int. symp. on software reliability engineering (ISSRE’05). IEEE Computer Society, pp 43–52

  • Goseva-Popstojanova K, Hamill M, Wang X (2006) Adequacy, accuracy, scalability, and uncertainty of architecture-based software reliability: lessons learned from large empirical case studies. In: Proc. 17th int. symp. on software reliability engineering (ISSRE’06). IEEE, pp 197–203

  • Goseva-Popstojanova K, Mathur K, Trivedi K (2001) Comparison of architecture-based software reliability models. In: Proceedings. 12th international symposium on software reliability engineering, 2001. ISSRE 2001, pp 22–31

  • Goseva-Popstojanova K, Trivedi KS (2001) Architecture-based approach to reliability assessment of software systems. Perform Eval 45(2–3):179–204

    Article  MATH  Google Scholar 

  • Grassi V, Mirandola R, Sabetta A (2007) Filling the gap between design and performance/reliability models of component-based systems: a model-driven approach. J Syst Softw 80(4):528–558. doi:10.1016/j.jss.2006.07.023

    Article  Google Scholar 

  • Happe J, Becker S, Rathfelder C, Friedrich H, Reussner RH (2010) Parametric performance completions for model-driven performance prediction. Perform Eval 67(8):694–716

    Article  Google Scholar 

  • Hatvani L, Jansen A, Seceleanu C, Pettersson P (2010) An integrated tool for trade-off analysis of quality-of-service attributes. In: Proc. 2nd int. workshop on the quality of service-oriented software systems (QUASOSS’10). ACM Digital Library

  • Huber N, Becker S, Rathfelder C, Schweflinghaus J, Reussner RH (2010) Performance modeling in industry: a case study on storage virtualization. In: Proc. 32nd int. conf. on software engineering (ICSE’10). ACM, pp 1–10

  • IEEE (2008) Recommended practice on software reliability, IEEE std 1633-2008. Tech. rep., The Institute of Electrical and Electronics Engineers, Inc.

  • Immonen A, Niemelä E (2008) Survey of reliability and availability prediction methods from the viewpoint of software architecture. Journal on Softw. and Syst. Modeling 7(1):49–65

    Article  Google Scholar 

  • Jain R (1991) The art of computer systems performance analysis: techniques for experimental design, measurement, simulation, and modeling. Wiley, New York

    MATH  Google Scholar 

  • Jin Y, Tang A, Han J, Liu Y (2007) Performance evaluation and prediction for legacy information systems. In: Proceedings of the 29th international conference on software engineering, ICSE ’07. IEEE Computer Society, Washington, DC, USA, pp 540–549. doi:10.1109/ICSE.2007.64

  • Kounev S (2006) Performance modeling and evaluation of distributed component-based systems using queueing Petri nets. IEEE Trans Softw Eng 32(7):486–502

    Article  Google Scholar 

  • Koziolek H (2010) Performance evaluation of component-based software systems: a survey. Perform Eval 67(8):634–658

    Article  Google Scholar 

  • Koziolek H, Doppelhamer J, Weiss R, Bilich C, Schlich B, Skuliber I, Zemljic M, Desic S, Huljenic D (2010) Q-ImPrESS project deliverable D7.3: demonstrator results documentation. Tech. rep., Q-ImPrESS Consortium

  • Koziolek H, Reussner R (2008) A model-transformation from the palladio component model to layered queueing networks. In: Proc. of the SPEC international workshop on performance engineering (SIPEW’08). LNCS, vol 5119. Springer, pp 58–78

  • Koziolek H, Schlich B, Bilich C (2010) A large-scale industrial case study on architecture-based software reliability analysis. In: Proc. 21st IEEE international symposium on software reliability engineering (ISSRE’10). IEEE Computer Society, pp 279–288. doi:10.1109/ISSRE.2010.15

  • Koziolek H, Schlich B, Bilich C, Weiss R, Becker S, Krogmann K, Trifu M, Mirandola R, Martens A (2011) An industrial case study on quality impact prediction for evolving service-oriented software. In: Proc. 33rd ACM/IEEE Int. Conf. on Software Engineering (ICSE’11) software engineering in practice track. ACM

  • Littlewood B, Verrall JL (1973) A Bayesian reliability growth model for computer software. Appl Stat 22(3):332. doi:10.2307/2346781

    Article  MathSciNet  Google Scholar 

  • Liu Y, Fekete A, Gorton I (2005) Design-level performance prediction of component-based applications. IEEE Trans Softw Eng 31(11):928–941. doi:10.1109/TSE.2005.127

    Article  Google Scholar 

  • Martens A, Koziolek H, Becker S, Reussner RH (2010) Automatically improve software models for performance, reliability and cost using genetic algorithms. In: Proc. 1st int. conf. on performance engineering (ICPE’10). ACM, pp 105–116. doi:10.1145/1712605.1712624

  • Martens A, Koziolek H, Prechelt L, Reussner R (2010) From monolithic to component-based performance evaluation of software architectures: a series of experiments analysing accuracy and effort. Empir Softw Eng 1–36. doi:10.1007/s10664-010-9142-8

  • Masetti M, Becker S, Skuliber I, Hauck M, Kofron J, Krogmann K, Stammel J, Seceleanu C, Tysiak J, Mirandola R, Ardagna D (2009) Q-ImPrESS project deliverable D6.1: method and abstract workflow. Tech. rep., Q-ImPrESS Consortium

  • McConnell S (1993) Code Complete: a practical handbook of software construction. Microsoft Press, Buffalo, NY

  • Miller KW, Morell LJ, Noonan RE, Park SK, Nicol DM, Murrill BW, Voas JM (1992) Estimating the probability of failure when testing reveals no failures. IEEE Trans Softw Eng 18(1):33–43. doi:10.1109/32.120314

    Article  Google Scholar 

  • Mizan A, Franks G (2011) An automatic trace based performance evaluation model building for parallel distributed systems. In: Proc. 2nd Int. Conference on Performance Engineering (ICPE’11), pp 61–72

  • Musa JD (2004) Software reliability engineering: more reliable software faster and cheaper, 2nd edn. AuthorHouse

  • Musa JD, Iannino A, Okumoto K (1987) Software reliability: measurement, prediction, application. McGraw-Hill, New York

    Google Scholar 

  • Nagappan N, Ball T, Zeller A (2006) Mining metrics to predict component failures. In: Proc. 28th Int. Conf. on Softw. Eng. (ICSE’06). ACM, pp 452–461. doi:10.1145/1134285.1134349

  • Object Management Group (OMG) (2007) UML Profile for MARTE, Beta 1. http://www.omg.org/cgi-bin/doc?ptc/2007-08-04

  • Pietrantuono R, Russo S, Trivedi KS (2010) Software reliability and testing time allocation: an architecture-based approach. IEEE Trans Softw Eng 36(3):323–337

    Article  Google Scholar 

  • Q-ImPrESS Consortium: Project website. http://www.q-impress.eu. Accessed 06 June 2012

  • Rolia JA, Sevcik KC (1995) The method of layers. IEEE Trans Softw Eng 21(8):689–700. doi:10.1109/32.403785

    Article  Google Scholar 

  • Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131–164

    Article  Google Scholar 

  • Sharma VS, Trivedi KS (2007) Quantifying software performance, reliability and security: an architecture-based approach. J Syst Softw 80(4):493–509

    Article  Google Scholar 

  • Singh H, Cortellessa V, Cukic B, Gunel E, Bharadwaj V (2001) A bayesian approach to reliability prediction and assessment of component based systems. In: Proc. 12th International Symposium on Software Reliability Engineering (ISSRE’01), pp 12–21

  • Skuliber I, Huljenic D, Desic S (2009) Black-box and gray-box components as elements for performance prediction in telecommunications system. In: 10th international conference on telecommunications, 2009. ConTEL 2009, pp 131–134

  • Smith C (1990) Performance engineering of software systems. Addision-Wesley

  • Smith CU (2002) Performance solutions: a practical guide to creating responsive, scalable software. Addison-Wesley

  • Snipes W, Robinson B, Brooks P (2009) Approximating deployment metrics to predict field defects and plan corrective maintenance activities. In: 20th Int. Symp. on Software Reliability Engineering (ISSRE’09). IEEE, pp 90–98

  • Stammel J, Reussner R (2009) KAMP: Karlsruhe architectural maintainability prediction. In: Proc. 1st workshop GI-Arbeitskreis Langlebige Softwaresysteme (L2S2’09), pp 87–98. http://ftp.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-537/

  • Trivedi K (2001) Probability and statistics with reliability, queuing, and computer science applications, 2nd edn. Wiley, New York

    Google Scholar 

  • Wang WL, Pan D, Chen MH (2006) Architecture-based software reliability modeling. J Syst Softw 79(1):132–146

    Article  Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell, MA, USA

    Book  MATH  Google Scholar 

  • Woodside M, Franks G, Petriu DC (2007) The future of software performance engineering. In: Future of software engineering (FOSE ’07). IEEE Computer Society, pp 171–187. doi:10.1109/FOSE.2007.32

  • Xu J, Oufimtsev A, Woodside M, Murphy L (2006) Performance modeling and prediction of enterprise javabeans with layered queuing network templates. SIGSOFT Softw Eng Notes 31(2):5

    Article  Google Scholar 

  • Zhu L, Aurum A, Gorton I, Jeffery R (2005) Tradeoff and sensitivity analysis in software architecture evaluation using analytic hierarchy process. Softw Qual J 13(4):357–375

    Article  Google Scholar 

Download references

Acknowledgements

This work was funded within the Q-ImPrESS research project (FP7-215013) by the European Union under the Information and Communication Technologies priority of FP7.

Author information

Authors and Affiliations

  1. Industrial Software Systems, ABB Corporate Research, Wallstadter Str. 59, 68526, Ladenburg, Germany

    Heiko Koziolek & Bastian Schlich

  2. University of Paderborn, Paderborn, Germany

    Steffen Becker

  3. Forschungszentrum Informatik (FZI), Karlsruhe, Germany

    Michael Hauck

Authors
  1. Heiko Koziolek
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Bastian Schlich
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Steffen Becker
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Michael Hauck
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Heiko Koziolek.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Koziolek, H., Schlich, B., Becker, S. et al. Performance and reliability prediction for evolving service-oriented software systems. Empir Software Eng 18, 746–790 (2013). https://doi.org/10.1007/s10664-012-9213-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-012-9213-0

Keywords