Skip to main content
SpringerLink
Log in

Evaluating probabilistic models with uncertain model parameters

  • Theme Section Paper
  • Published:
Software & Systems Modeling Aims and scope Submit manuscript

Abstract

Probabilistic models are commonly used to evaluate quality attributes, such as reliability, availability, safety and performance of software-intensive systems. The accuracy of the evaluation results depends on a number of system properties which have to be estimated, such as environmental factors or system usage. Researchers have tackled this problem by including uncertainties in the probabilistic models and solving them analytically or with simulations. The input parameters are commonly assumed to be normally distributed. Accordingly, reporting the mean and variances of the resulting attributes is usually considered sufficient. However, many of the uncertain factors do not follow normal distributions, and analytical methods to derive objective uncertainties become impractical with increasing complexity of the probabilistic models. In this work, we introduce a simulation-based approach which uses Discrete Time Markov Chains and probabilistic model checking to accommodate a diverse set of parameter range distributions. The number of simulation runs automatically regulates to the desired significance level and reports the desired percentiles of the values which ultimately characterises a specific quality attribute of the system. We include a case study which illustrates the flexibility of this approach using the evaluation of several probabilistic properties.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Alur, R., Courcoubetis, C., Dill, D.: Model-checking in dense real-time. Inf. Comput. 104(1), 2–34 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  2. Ardagna, D., Ghezzi, C., Mirandola, R.: Rethinking the use of models in software architecture. In: Quality of Software Architectures. Models and Architectures, pp. 1–27. Springer, Berlin (2008)

  3. Assayad, I., Girault, A., Kalla, H.: A bi-criteria scheduling heuristic for distributed embedded systems under reliability and real-time constraints. In: Dependable Systems and Networks, pp. 347–356. IEEE (2004)

  4. Axelsson, J.: Cost models with explicit uncertainties for electronic architecture trade-off and risk analysis. In: Current Practice (2006)

  5. Aziz, A., Singhal, V., Balarin, F.: It usually works: The temporal logic of stochastic systems. In: Wolper, P. (ed.) Proceedings of 7th International Conference on Computer Aided Verification, CAV 95. LNCS, vol 939, pp. 155–165. Springer, Berlin (1995)

  6. Baier, C., Katoen, J.-P., Hermanns, H.: Approximate symbolic model checking of continuous-time markov chains. In: Baeten, J.C.M., Mauw, S. (eds.) Proceedings of 10th International Conference on Concurrency Theory, CONCUR 99. LNCS, vol. 1664, pp. 146–161. Springer, Berlin (1999)

  7. Balsamo, S., Di Marco, A., Inverardi, P., Simeoni, M.: Model-based performance prediction in software development: a survey. IEEE Trans. Softw. Eng. 30(5), 295–310 (2004)

    Article  Google Scholar 

  8. Basseur, M., Zitzler, E.: A preliminary study on handling uncertainty in indicator-based multiobjective optimization. In: Appl. of Evol. Computing, pp. 727–739. Springer, Berlin (2006)

  9. Becker, S., Grunske, L., Mirandola, R., Overhage, S.: Performance prediction of component-based systems—a survey from an engineering perspective. In: Architecting Systems with Trustworthy Components. LNCS, vol. 3938, pp. 169–192. Springer, Berlin (2006)

  10. Beyer, H., Sendhoff, B.: Robust optimization: a comprehensive survey. Comput. Methods Appl. Mech. Eng. 196(33–34), 3190–3218 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  11. Bhunia, A., Sahoo, L., Roy, D.: Reliability stochastic optimization for a series system with interval component reliability via genetic algorithm. Appl. Math. Comput. 216(3), 929–939 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  12. Birolini, A.: Reliability Engineering: Theory and Practice. Springer, Berlin (2010)

    Book  Google Scholar 

  13. Bolch, G., Greiner, S., de Meer, H., Trivedi, K.S.: Queuing Network and Markov Chains. Wiley, New York (1998)

  14. Cheung, R.: A user-oriented software reliability model. IEEE Trans. Softw. Eng. 6(2), 118–125 (1980)

    Article  MATH  Google Scholar 

  15. Ciesinski, F., Größer, M.: On probabilistic computation tree logic. In: Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.-P., Siegle, M. (eds.) Validation of Stochastic Systems: A Guide to Current Research. LNCS, vol. 2925, pp. 147–188. Springer, Berlin (2004)

    Chapter  Google Scholar 

  16. Coit, D., Jin, T., Wattanapongsakorn, N.: System optimization with component reliability estimation uncertainty: a multi-criteria approach. IEEE Trans. Reliab. 53(3), 369–380 (2004)

    Article  Google Scholar 

  17. Coit, D.W., Smith, A.E.: Genetic algorithm to maximize a lower-bound for system time-to-failure with uncertain component Weibull parameters. Comput. Ind. Eng. 41 (2002)

  18. Cortellessa, V., Grassi, V.: A modeling approach to analyze the impact of error propagation on reliability of component-based systems. In: Component-Based Software Engineering, pp. 140–156. Springer, Berlin (2007)

  19. Filieri, A., Ghezzi, C., Tamburrelli, G.: Run-time efficient probabilistic model checking. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu, HI, USA, May 21–28, 2011, pp. 341–350. ACM, New York (2011)

  20. Fiondella, L., Gokhale, S.S.: Software reliability with architectural uncertainties. In: Parallel and Distributed Processing, pp. 1–5. IEEE (2008)

  21. Förster, M., Trapp, M.: Fault tree analysis of software-controlled component systems based on second-order probabilities. In: International Symposium on Software Reliability Engineering, pp. 146–154. IEEE, Nov 2009

  22. Frolund, S., Koistinen, J.: Quality-of-service specification in distributed object systems. Distrib. Syst. Eng. J. 5(4), 179–202 (1998)

    Article  Google Scholar 

  23. Goeva-Popstojanova, K., Trivedi, K.: Architecture-based approach to reliability assessment of software systems. Perform. Eval. 45(2–3), 179–204 (2001)

    Google Scholar 

  24. Goseva-Popstojanova, K., Hamill, M.: Architecture-based software reliability: why only a few parameters matter?. In: Computer Software and Applications Conference, 2007, vol. 1, pp. 423–430. IEEE (2007)

  25. Goseva-Popstojanova, K., Hamill, M., Perugupalli, R.: Large empirical case study of architecture-based software reliability. In: International Symposium on Software Reliability Engineering, vol. 54, pp. 10–52. IEEE (2005)

  26. Goseva-Popstojanova, K., Hamill, M., Wang, X.: Adequacy, accuracy, scalability, and uncertainty of architecture-based software reliability: lessons learned from large empirical case studies. In: International Symposium on Software Reliability Engineering, pp. 197–203. IEEE (2006)

  27. Goseva-Popstojanova, K., Kamavaram, S.: Assessing uncertainty in reliability of component-based software systems. In: ISSRE 2003, pp. 307–320. IEEE (2003)

  28. Goseva-Popstojanova, K., Kamavaram, S.: Software reliability estimation under uncertainty:generalization of the method of moments. High Assur. Syst. Eng. 2004, 209–218 (2004)

    Google Scholar 

  29. Grunske, L.: Specification patterns for probabilistic quality properties. In: Proceedings of the 13th International Conference on Software Engineering, ICSE ’08 61(0), 31 (2008)

  30. Grunske, L., Han, J.: A comparative study into architecture-based safety evaluation methodologies using AADL’s error annex and failure propagation models. High Assurance Systems Engineering, Symposium, pp. 283–292 (2008)

  31. Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects Comput. 6(5), 512–535 (1994)

    Article  MATH  Google Scholar 

  32. Jhumka, A., Hiller, M., Suri, N.: Component-based synthesis of dependable embedded software. Lect. Notes Comput. Sci. 2469, 111–128 (2002)

    Article  Google Scholar 

  33. Koymans, R.: Specifying real-time properties with metric temporal logic. Real-Time Syst. 2(4), 255–299 (1990)

    Article  Google Scholar 

  34. Kwiatkowska, M., Norman, G., Parker, D.: Probabilistic symbolic model checking with PRISM: a hybrid approach. Int. J. Softw. Tools Technol. Transfer 6(2), 128–142 (2004)

    Article  Google Scholar 

  35. Kwiatkowska, M., Norman, G., Parker, D.: Probabilistic model checking in practice: case studies with PRISM. ACM SIGMETRICS Perform. Eval. Rev. 32(4), 16–21 (2005)

    Article  Google Scholar 

  36. Kwiatkowska, M., Norman, G., Parker, D., Sproston, J.: Performance analysis of probabilistic timed automata using digital clocks. Formal Methods Syst. Des. 29(1), 33–78 (2006)

    Article  MATH  Google Scholar 

  37. Limbourg, P.: Multi-objective optimization of generalized reliability design problems using feature models: a concept for early design stages. Reliab. Eng. Syst. Saf. 93(6), 815–828 (2008)

    Article  Google Scholar 

  38. Marseguerra, M., Zio, E., Podofillini, L., Coit, D.: Optimal design of reliable network systems in presence of uncertainty. IEEE Trans. Reliab. 54(2), 243–253 (2005)

    Article  Google Scholar 

  39. Meedeniya, I., Buhnova, B., Aleti, A., Grunske, L.: Architecture-driven reliability and energy optimization for complex embedded systems. In: Quality of Software Architectures, QoSA 2010. LNCS, vol. 6093, pp. 52–67. Springer, Berlin (2010)

  40. Meedeniya, I., Bühnova, B., Aleti, A., Grunske, L.: Reliability-driven deployment optimization for embedded systems. J. Syst. Softw. 84(5), 835–846 (2011)

    Article  Google Scholar 

  41. Meedeniya, I., Moser, I., Aleti, A., Grunske, L.: Architecture-based reliability evaluation under uncertainty. In: 7th International Conference on the Quality of Software Architectures, QoSA 2011 and 2nd International Symposium on Architecting Critical Systems, ISARCS 2011, pp. 85–94 (2011)

  42. Montgomery, D., Runger, G.: Applied Statistics and Probability for Engineers. Wiley, India (2007)

    MATH  Google Scholar 

  43. Oberkampf, W., Helton, J., Joslyn, C., Wojtkiewicz, S., Ferson, S.: Challenge problems: uncertainty in system response given uncertain parameters. Reliab. Eng. Syst. Saf. 85(1–3), 11–19 (2004)

    Article  Google Scholar 

  44. Puterman, M.L.: Markov Decision Processes. Wiley, New York (1994)

    Book  MATH  Google Scholar 

  45. Roshandel, R., Banerjee, S., Cheung, L., Medvidovic, N., Golubchik, L.: Estimating software component reliability by leveraging architectural models. In: International Conference on Software Engineering, p. 853. ACM, New York (2006)

  46. Roshandel, R., Medvidovic, N., Golubchik, L.: A Bayesian model for predicting reliability of software systems at the architectural level. LNCS 4880, 108–126 (2007)

    Google Scholar 

  47. Rubinstein, R., Kroese, D.: Simulation and the Monte Carlo Method. Wiley-Interscience, New York (2008)

    MATH  Google Scholar 

  48. Sanchez, A., Carlos, S., Martorell, S., Villanueva, J.: Addressing imperfect maintenance modelling uncertainty in unavailability and cost based optimization. Reliab. Eng. Syst. Saf. 94(1), 22–32 (2009)

    Article  Google Scholar 

  49. Trivedi, K.: Probability& Statistics with Reliability. Queuing and Computer Science Applications. Wiley, India (2009)

    Google Scholar 

  50. Wang, W., Wu, Y., Chen, M.: An architecture-based software reliability model. In: Proceedings of 1999 Pacific Rim International Symposium on Dependable Computing, pp. 143–150. IEEE (2002)

  51. Wattanapongskorn, N., Coit, D.W.: Fault-tolerant embedded system design and optimization considering reliability estimation uncertainty. Rel. Eng. 92, 395–407 (2007)

    Google Scholar 

  52. Yin, L., Smith, M., Trivedi, K.: Uncertainty analysis in reliability modeling. In: Symposium on Reliability and Maintainability, pp. 229–234 (2001)

Download references

Acknowledgments

This original research was proudly supported by the Commonwealth of Australia, through the Cooperative Research Center for Advanced Automotive Technology (projects C4-501: Safe and Reliable Integration and Deployment Architectures for Automotive Software Systems). Furthermore, the research was supported by the Center for Mathematical and Computational Modelling \(\text{(CM)}^2\) at the University of Kaiserslautern.

Author information

Authors and Affiliations

  1. Faculty of Information and Communication Technologies, Swinburne University of Technology, Hawthorn, VIC, 3122, Australia

    Indika Meedeniya & Irene Moser

  2. Faculty of Information Technology, Monash University, Clayton, VIC, 3800, Australia

    Aldeida Aleti

  3. Faculty of Computer Science and Center for Mathematical and Computational Modelling (CM), University of Kaiserslautern, 67653 , Kaiserslautern, Germany

    Lars Grunske

Authors
  1. Indika Meedeniya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Irene Moser
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aldeida Aleti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lars Grunske
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Indika Meedeniya.

Additional information

Communicated by Prof. Dr. Dorina Petriu and Dr. Jens Happe.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Meedeniya, I., Moser, I., Aleti, A. et al. Evaluating probabilistic models with uncertain model parameters. Softw Syst Model 13, 1395–1415 (2014). https://doi.org/10.1007/s10270-012-0277-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10270-012-0277-5

Keywords

Search

Navigation