Skip to main content
SpringerLink
Log in

Investigating the effect of variations in the test development process: a case from a safety-critical system

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Variation is inherent to a process, and process management demands understanding the nature of variation in quantitative terms, for evaluation and prediction purposes. This understanding requires the identification of process indicators that build the system of variation. To utilize quantitative techniques to understand and improve a software process, more indicators are needed than in a manufacturing process. The need to identify the indicators of a software process and the lack of a generic approach to assess the ability of a software process for quantitative management encouraged us to carry out a sequence of studies that resulted in the development of an Assessment Approach for Quantitative Process Management (A2QPM). This paper explains an application of the A2QPM to the test development process of an avionics software project and presents the results. The study aimed at understanding the effect of the test design stage and the effect of internal reviews as verification activities in test development, with respect to process productivity and product quality measures. The measurement data collected during the execution of the processes were analyzed by control charts to observe the evidence of process stability. The mean values of measurement data were utilized to make performance comparisons between the various executions of the test development process. The results showed that process productivity was unaffected, but the test procedure quality was positively influenced by the application of test design and internal reviews. The utilization of the A2QPM as a guide for the quantitative implementation enabled the systematic evaluation of the test development process and measures prior to analysis. This resulted in the identification of process clusters having stable variation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Baldassarre, M. T., Boffoli, N., Caivano, D., & Visaggio, G. (2004). Managing software process improvement (SPI) through statistical process control (SPC). Lecture Notes in Computer Science, Book Chapter, 3009, 30–46.

    Article  Google Scholar 

  • Baldassarre, M. T., Caivano, D., Kitchenham, B., & Visaggio, G. (2007). Systematic review of statistical process control: An experience report. In Proceedings of the 11th international conference on evaluation and assessment in software engineering (EASE’07). British Computer Society, United Kingdom.

  • Barnard, J., & Carleton, A. (1999). Analyzing a mature software inspection process using statistical process control. In Proceedings of the national SEPG conference, Pittsburgh.

  • Burgess, S. M., & Drabick R. D. (1996). The I.T.B.G. testing capability maturity model.

  • Burnstein, I., Suwanassart, T., & Carlson C. R. (1996). Developing a testing maturity model. Crosstalk, Part I: August, 21–24; Part II: September, 19–26.

  • Card, D. (1994). Statistical Process Control for Software? IEEE Software, 11(3), 95–97.

    Article  Google Scholar 

  • Card, D. N., & Berg, R. A. (1989). An industrial engineering approach to software development. Journal of Systems and Software, 10(3), 159–167.

    Article  Google Scholar 

  • Card, D. N., Domzalski, K., & Davies, G. (2008). Making statistics part of decision making in an engineering organization. IEEE Software, 25(3), 37–47.

    Article  Google Scholar 

  • Demirors, O., & Sargut, K. U. (2006). Utilization of statistical process control (SPC) in emergent software organizations: pitfalls and suggestions. Software Quality Journal, 14(2), 135–157.

    Article  Google Scholar 

  • Demmy, W. S., & Petrini, A. B. (1989). Statistical process control in software quality assurance. Proceedings of the Aerospace and Electronics Conference, 4, 1585–1590.

    Article  Google Scholar 

  • Do, H., & Rothermel, G. (2006). An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models. In Proceedings of the 14th ACM SIGSOFT international symposium on foundations of software engineering (SIGSOFT ‘06/FSE-14). ACM, New York, pp. 141–151.

  • Do, H., & Rothermel, G. (2008). Using sensitivity analysis to create simplified economic models for regression testing. In Proceedings of the 2008 international symposium on software testing and analysis (ISSTA ‘08). ACM, New York, pp. 51–62.

  • Ebenau, R. G. (1994). Predictive quality control with software inspections. Crosstalk, 7(6).

  • Ebert, C., & Dumke, R. (2007). Software measurement. Berlin: Springer.

    MATH  Google Scholar 

  • Ericson, T., Subotic, A., & Ursing, S. (1997). TIM: A test improvement model. Software Testing, Verification and Reliability, 7(4), 229–246.

    Article  Google Scholar 

  • Farooq, A., Georgieva, K., & Dumke, R. (2008). A meta-measurement approach for software test processes. In Proceedings of multitopic conference (INMIC ‘2008), IEEE international, pp. 333–338.

  • Florac, A. W., & Carleton A. D. (1999). Measuring the software process: Statistical process control for software process improvement. Pearson Education, New Jersey.

  • Florac, A. W., & Carleton, A. D. (2000). Statistical process control: Analyzing a space shuttle onboard software process. IEEE Software, 17(4), 97–106.

    Article  Google Scholar 

  • Gelperin, D. (1996). Software testability support model. http://twin-spin.cs.umn.edu/previous_presentations.php?id=89.

  • Holzmann, G. J. (2001). Economics of software verification. In Proceedings of the 2001 ACM SIGPLAN-SIGSOFT Workshop on Program Analysis For Software Tools and Engineering (PASTE ‘01). ACM, New York, pp. 80–89.

  • IEEE Computer Society (2004). Software engineering body of knowledge.

  • IEEE Software (2000). Process diversity. 17(4).

  • IEEE Std 610.12 (1990). IEEE standard glossary of software engineering terminology.

  • IEEE Std 1012 (2004). IEEE standard for software verification and validation.

  • Itkonen, J., Rautiainen, K., & Lassenius, C. (2005). Towards understanding quality assurance in agile software development. In Proceedings of the international conference on agility, pp. 201–207.

  • Jacob, L., & Pillai, S. K. (2003). Statistical process control to improve coding and code review. IEEE Software, 20(3), 50–55.

    Article  Google Scholar 

  • Jalote, P., Dinesh, K., Raghavan, S., Bhashyam, M.R., & Ramakrishnan, M. (2000). Quantitative quality management through defect prediction and statistical process control. In Proceedings of the 2nd world quality congress for software, Yokohama, Japan.

  • Jalote, P., & Saxena, A. (2002). Optimum control limits for employing statistical process control in software process. IEEE Transactions on Software Engineering, 28(12), 1126–1134.

    Article  Google Scholar 

  • Karlstrom, D., Runeson, P., & Norden, S. (2005). A minimal test practice framework for emerging software organizations. Software Testing, Verification and Reliability, 15, 145–166.

    Article  Google Scholar 

  • Kasurinen, J., Taipale, O., & Smolander, K. (2009). Analysis of problems in testing practices. In Proceedings of Software Engineering Conference (APSEC ‘09), Asia-Pacific, 1–3 Dec., pp. 309–315.

  • Kettunen, V., Kasurinen, J., Taipale, O., & Smolander, K. (2010). A study on agility and testing processes in software organizations. In Proceedings of the 19th international symposium on software testing and analysis (ISSTA ‘10). ACM, New York, pp. 231–240.

  • Kirbas, S., Tarhan, A., & Demirors, O. (2007). An assessment and analysis tool for statistical process control of software processes. In Proceedings of SPICE conference, Seoul, Korea.

  • Koomen, T., & Pol, M. (1999). Test process improvement: A practical step-by-step guide to structured testing. Boston, MA: Addison-Wesley.

    MATH  Google Scholar 

  • Krause, M. E. (1994). A maturity model for automated software testing. Medical Device & Diagnostic Industry Magazine, http://www.mddionline.com/print/2452.

  • Lantzy, M. A. (1992). Application of statistical process control to software processes. In Proceedings of the 9th Washington Ada symposium on empowering software users and developers, pp. 113–123.

  • Lee, C. (2009). Adapting and adjusting test process reflecting characteristics of embedded software and industrial properties based on referential models. In Proceedings of the 2nd international conference on interaction sciences: Information technology, culture and human (ICIS ‘09), vol. 403. ACM, New York, NY, pp. 1372–1377.

  • Monteiro, L. F. S., & de Oliveira, K. M. (2010). Defining a catalog of indicators to support process performance analysis. Journal of Software Maintenance and Evolution: Research and Practice, DOI: 10.1002/spip.435.

  • Paulk, M. C., Weber, C.V., Curtis, B., & Chrissis, M.B. (1993). Capability maturity model for software, Version 1.1. Technical Report, CMU/SEI-93-TR-024 ESC-TR-93-177.

  • RTCA/DO-178B (1992). Software Considerations in Airborne Systems and Equipment Certification.

  • Shewhart, W. A. (1986). Statistical Method from the Viewpoint of Quality Control. Mineola, NY: Dover Publications.

    Google Scholar 

  • Swinkels, R. (2000). A comparison of TMM and other test process improvement models. http://wwwbruegge.informatik.tu-muenchen.de/static/contribute/Lehrstuhl/documents/12-4-1-FPdef.pdf.

  • Talby, D., Hazzan, O., Dubinsky, Y., & Keren, A. (2006). Agile software testing in a large-scale project. IEEE Software, 23(4), 30–37.

    Google Scholar 

  • Tarhan, A., & Demirors, O. (2006). Investigating suitability of software process and metrics for statistical process control. In Proceedings of EuroSPI conference, LNCS. Springer, Berlin, vol. 4257, pp. 87–98.

  • Tarhan, A., & Demirors, O. (2008). Assessment of software process and metrics for quantitative understanding. In Proceedings of IWSM-Mensura Conference, LNCS. Springer, Berlin, vol. 4895, pp. 102–113.

  • Weller E. (2000a). Applying quantitative methods to software maintenance. ASQ Software Quality Professional, 3(1).

  • Weller, E. (2000b). Practical applications of statistical process control. IEEE Software, 17(3), 48–55.

    Article  Google Scholar 

  • Wheeler, D.J. (1995). Advanced topics in statistical process control. SPC Press.

Download references

Acknowledgments

The authors would like to thank Erhan Yuceer for his valuable contribution to the implementation of this study during the phases of data collection and data analysis interpretation.

Author information

Authors and Affiliations

  1. Computer Engineering Department, Hacettepe University, Beytepe Kampusu, 06532, Ankara, Turkey

    Ayca Tarhan

  2. Informatics Institute, Middle East Technical University, Inonu Bulvari, 06531, Ankara, Turkey

    Onur Demirors

Authors
  1. Ayca Tarhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Onur Demirors
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayca Tarhan.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tarhan, A., Demirors, O. Investigating the effect of variations in the test development process: a case from a safety-critical system. Software Qual J 19, 615–642 (2011). https://doi.org/10.1007/s11219-011-9129-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-011-9129-8

Keywords

Search

Navigation