Skip to main content
SpringerLink
Log in

Integrating risk-based testing in industrial test processes

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Risk-based testing has a high potential to improve the software development and test process as it helps to optimize the allocation of resources and provides decision support for the management. But for many organizations, its integration into an existing test process is a challenging task. In this article, we provide a comprehensive overview of existing work and present a generic testing methodology enhancing an established test process to address risks. On this basis, we develop a procedure on how risk-based testing can be introduced in a test process and derive a stage model for its integration. We then evaluate our approach for introducing risk-based testing by means of an industrial study and discuss benefits, prerequisites and challenges to introduce it. Potential benefits of risk-based testing identified in the studied project are faster detection of defects resulting in an earlier release, a more reliable release quality statement as well as the involved test-process optimization. As necessary prerequisites for risk-based testing, we identified an inhomogeneous distribution of risks associated with the various parts of the tested software system as well as consolidated technical and business views on it. Finally, the identified challenges of introducing risk-based testing are reliable risk assessment in the context of complex systems, the availability of experts for risk assessment as well as established tool supports for test management.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Ahern, D., Clouse, A., & Turner, R. (2008). CMMI distilled: A practical introduction to integrated process improvement. Boston, MA: Addison-Wesley Professional.

    Google Scholar 

  • Amland, S. (2000). Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. Journal of Systems and Software, 53(3), 287–295.

    Article  Google Scholar 

  • Bach, J. (1999). Heuristic risk-based testing. Software Testing and Quality Engineering Magazine, 11, 99.

    Google Scholar 

  • Black, R. (2009). Advanced software testing vol. 2: Guide to the ISTQB advanced certification as an advanced test manager. Santa Barbara, CA: Rocky Nook.

    Google Scholar 

  • Boehm, B. W. (1988). A spiral model of software development and enhancement. Computer, 21(5), 61–72.

    Article  Google Scholar 

  • Cangussu, J. W., Karcich, R. M., Mathur, A. P., & DeCarlo, R. A. (2004). Software release control using defect based quality estimation. In 15th international symposium on software reliability engineering.

  • Cavano, J. P., & McCall, J. A. (1978). A framework for the measurement of software quality. ACM SIGMETRICS Performance Evaluation Review, 7(3–4), 133–139.

    Article  Google Scholar 

  • Chen, Y., Probert, R. L., & Sims, D. P. (2002). Specification-based regression test selection with risk analysis. In Conference of the Centre for Advanced Studies on collaborative research 2002.

  • Dorling, A. (1993). SPICE: Software process improvement and capability determination. Software Quality Journal, 2(4), 209–224.

    Article  Google Scholar 

  • Felderer, M., Haisjackl, C., Breu, R., & Motz, J. (2012). Integrating manual and automatic risk assessment for risk-based testing. Software quality. Process automation in software development. In 4th international conference SWQD 2012 (pp. 159–180).

  • Felderer, M., & Ramler, R. (2013). Experiences and challenges of introducing risk-based testing in an industrial project. Software quality. Increasing value in software and systems development. In 5th international conference SWQD 2013 (pp. 10–29).

  • Fenton, N. E., & Ohlsson, N. (2000). Quantitative analysis of faults and failures in a complex software system. IEEE Transactions on Software Engineering, 26(8), 797–814.

    Article  Google Scholar 

  • Gerrard, P., & Thompson, N. (2002). Risk based e-business testing. Norwood: Artech House. Inc.

    Google Scholar 

  • IEEE. (2008). IEEE standard 829-2008: IEEE standard for software and system test documentation. New York, NY: The Institute of Electrical and Electronics Engineers.

  • ISO. (2005). ISO/IEC 25000 software and system engineering–software product quality requirements and evaluation (SQuaRE)-guide to SQuaRE. Geneva: International Organization for Standardization.

    Google Scholar 

  • ISO. (2010). ISO/IEC/IEEE 24765:2010 system and software engineering—Vocabulary. Geneva: International Organization for Standardization.

  • ISO/IEC. (2013). ISO/IEC 29119 software testing. Draft available online at http://www.softwaretestingstandard.org/. Accessed on September 4, 2013.

  • ISTQB. (2012). Standard glossary of terms used in software testing. Version 2.2. Brussels: International Software Testing Qualifications Board.

  • Kan, S. H. (2002). Metrics and models in software quality engineering (2nd ed.). Boston, MA: Addison-Wesley Longman.

    Google Scholar 

  • Karolak, D. W. (1995). Software engineering risk management. New York: Wiley–IEEE Computer Society Press.

    Google Scholar 

  • Kontio, J. (1999). Risk management in software development: A technology overview and the Riskit method. In 21st international conference on Software engineering (ICSE 1999).

  • Koomen, T., & Pol, M. (1999). Test process improvement: A practical step-by-step guide to structured testing. Boston, MA: Addison-Wesley Professional.

    MATH  Google Scholar 

  • Koomen, T., van der Aalst, L., Broekman, B., & Vroon, M. (2006). TMap next, for result-driven testing. Singapore: UTN Publishers.

    Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.

    Article  MATH  MathSciNet  Google Scholar 

  • Letouzey, J. L., & Coq, T. (2010). The scale analysis model: An analysis model compliant with the representation condition for assessing the quality of software source code. In Second international conference on advances in system testing and validation lifecycle (VALID 2010).

  • Li, Q., Li, M., Yang, Y., Wang, Q., Tan, T., Boehm, B., et al. (2009). Bridge the gap between software test process and business value: A case study. In International conference on software process: Trustworthy software development processes (ICSP ‘09).

  • Li, Q., Yang, Y., Li, M., Wang, Q., Boehm, B., & Hu, C. (2010). Improving software testing process: Feature prioritization to make winners of success-critical stakeholders. Journal of Software: Evolution and Process, 24(7), 783–801.

    Google Scholar 

  • McCabe, T. J. (1976). A complexity measure. IEEE Transactions on Software Engineering, 2(4), 308–320.

    Article  MATH  MathSciNet  Google Scholar 

  • OMG. (2012). CISQ specifications for automated quality characteristic measures. CISQ-TR-2012-01.

  • Pfleeger, S. L. (2000). Risky business: What we have yet to learn about risk management. Journal of Systems and Software, 53(3), 265–273.

    Article  Google Scholar 

  • Ramler, R., Biffl, S., & Grünbacher, P. (2006). Value-based management of software testing. In S. Biffl, A. Aurum, B. Boehm, H. Erdogmus, & P. Grünbacher (Eds.), Value-based software engineering (pp. 225–244). Berlin: Springer.

    Chapter  Google Scholar 

  • Ramler, R., & Felderer, M. (2013). Experiences from an initial study on risk probability estimation based on expert opinion. In Joint conference of the 23rd international workshop on software measurement (IWSM) and the eighth international conference on software process and product measurement (Mensura) (pp. 93–97).

  • Ramler, R., Kopetzky, T., & Platz, W. (2012). Value-based coverage measurement in requirements-based testing: Lessons learned from an approach implemented in the TOSCA test suite. In 38th Euromicro conference on software engineering and advanced applications (SEAA 2012).

  • Ramler, R., Larndorfer, S., Natschläger, T. (2009). What software repositories should be mined for defect predictors? In 35th Euromicro conference on software engineering and advanced applications (SEAA 2009).

  • Redmill, F. (2004). Exploring risk-based testing and its implications. Software Testing, Verification and Reliability, 14(1), 3–15.

    Article  Google Scholar 

  • Redmill, F. (2005). Theory and practice of risk-based testing: Research articles. Software Testing, Verification and Reliability, 15(1), 3–20.

    Article  Google Scholar 

  • Saaty, T. L. (1980). The analytic hierarchy process. New York City: McGraw-Hill.

    MATH  Google Scholar 

  • Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York City: McGraw-Hill.

    MATH  Google Scholar 

  • Souza, E., Gusmão, C., & Venâncio, J. (2010). Risk-based testing: A case study. In Seventh international conference on information technology: New generations (ITNG) (pp. 1032–1037).

  • Souza, E., Gusmão, C., Venâncio, J., & Melo, R. (2009). Measurement and control for risk-based test cases and activities. In 10th Latin American test workshop (LATW’09) (pp. 1–6).

  • Spillner, A., Rossner, T., Winter, M., & Linz, T. (2007). Software testing practice: Test management: A study guide for the certified tester exam ISTQB advanced level. Santa Barbara, CA: Rocky Nook.

    Google Scholar 

  • Stallbaum, H., & Metzger, A. (2007). Employing requirements metrics for automating early risk assessment. In Workshop on measuring requirements for project and product success (pp. 1–12).

  • Stallbaum, H., Metzger, A., & Pohl, K. (2008). An automated technique for risk-based test case generation and prioritization. In 3rd international workshop on automation of software test (pp. 67–70).

  • Steiner, M., Blaschke, M., Philipp, M., & Schweigert, T. (2012). Make test process assessment similar to software process assessment—The test SPICE approach. Journal of Software: Evolution and Process, 24(5), 471–480.

    Google Scholar 

  • van Veenendaal, E. (2012). The PRISMA approach. Uitgeverij Tutein Nolthenius.

  • van Veenendaal, E., Goslin, A., Olsen, K., O’Hara, F., Miller, M., Thompson, G., et al. (2008). Test Maturity Model integration (TMMi) version 1.0. Princeton, IN: TMMi Foundation.

    Google Scholar 

  • Wagner, S., Lochmann, K., Heinemann, L., Kläs, M., Trendowicz, A., Plösch, R., et al. (2012). The Quamoco product quality modelling and assessment approach. In 34th international conference on software engineering (ICSE 2012) (pp. 1133–1142).

  • Wendland, M.-F., Kranz, M., & Schieferdecker, I. (2012). A systematic approach to risk-based testing using risk-annotated requirements models. In The seventh international conference on software engineering advances (ICSEA 2012) (pp. 636–642).

  • Wiegers, K. E. (1999). First things first: Prioritizing requirements. Software Development, 7(10), 24–30.

    Google Scholar 

  • Yoon, H., & Choi, B. (2011). A test case prioritization based on degree of risk exposure and its empirical evaluation. International Journal of Software Engineering and Knowledge Engineering, 21(02), 191–209.

    Article  Google Scholar 

Download references

Acknowledgments

This work has been supported by the COMET Competence Center program of the Austrian Research Promotion Agency (FFG), the project QE LaB—Living Models for Open Systems (www.qe-lab.at) funded by the Austrian Federal Ministry of Economics (Bundesministerium für Wirtschaft und Arbeit), the project MOBSTECO funded by the Austrian Science Fund (FWF) as well as the competence network Softnet Austria (www.soft-net.at) funded by the Austrian Federal Ministry of Economics (Bundesministerium für Wirtschaft und Arbeit), the province of Styria, the Steirische Wirtschaftsförderungsgesellschaft mbH (SFG), and the city of Vienna’s Center for Innovation and Technology (ZIT).

Author information

Authors and Affiliations

  1. Institute of Computer Science, University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  2. Software Competence Center Hagenberg, Hagenberg, Austria

    Rudolf Ramler

Authors
  1. Michael Felderer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rudolf Ramler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Felderer.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Felderer, M., Ramler, R. Integrating risk-based testing in industrial test processes. Software Qual J 22, 543–575 (2014). https://doi.org/10.1007/s11219-013-9226-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-013-9226-y

Keywords

Search

Navigation