Skip to main content
SpringerLink
Log in

Challenges of software process and product quality improvement: catalyzing defect root-cause investigation by process enactment data analysis

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

It is claimed by software quality management that the quality of a software product is highly influenced by the quality of the software process followed to develop it. Since measurement of the software process is a challenging task, it is frequently the defects in the software product that are used to measure development quality. By extracting semantic information from defect records, practitioners can investigate and address root causes of software defects to improve development process and product quality. Investigating root causes requires effort for a detailed analysis into the components of the development process that originated the software defects, and is therefore encouraged only at higher maturity levels by most known process improvement models such as Capability Maturity Model Integration (CMMI). This practice, however, postpones the benefits that root-cause analysis would bring in gaining process awareness to improve the software development process and product quality in emergent organizations or organizations residing at lower maturity levels (MLs). In this article, we present a method for and results from applying root-cause analysis for software defects recorded in a software-intensive project of a CMMI ML3 certified institute. The suggested method combines process enactment data collection and analysis with Orthogonal Defect Classification which is a known technique in defect root-cause analysis. Prior to and after implementing the method in the study, defect attributes were analyzed and compared in order to understand any improvements in development performance and product quality. The results of the comparison indicate that the suggested method was efficient in the effort it required and effective in improving development performance and product quality. Defect triggers have become more active in identifying software defects in the earlier phases of software development, and the cost of quality due to software defects has decreased in consequence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Andersen, B., & Fagerhaug, T. (2006). Root cause analysis: simplified tools and techniques. Milwaukee: ASQ Quality Press.

    Google Scholar 

  • Aslan, D., Tarhan, A., & Demirörs, O. (2014). How process enactment data affects product defectiveness prediction—a case study. In R. Lee (Ed.), Software engineering research, management and applications (pp. 151–166). Heidelberg: Springer International Publishing. doi:10.1007/978-3-319-00948-3_10.

    Chapter  Google Scholar 

  • Bassin KA, Santhanam P (1997) Use of software triggers to evaluate software process effectiveness and capture customer usage profiles. Proceedings The Eighth International Symposium on Software Reliability Engineering—Case Studies 103–114. doi:10.1109/CSSRE.1997.637852

  • Bassin, K. A., Kratschmer, T., & Santhanam, P. (1998). Evaluating software development objectively. IEEE Software, 15(6), 66–74. doi:10.1109/52.730846.

    Article  Google Scholar 

  • Boehm, W. B. (1984). Software engineering economics. IEEE Transactions on Software Engineering, SE-10(1), 4–21. doi:10.1109/TSE.1984.5010193.

    Article  Google Scholar 

  • Bridge, N., & Miller, C. (1998). Orthogonal defect classification using defect data to improve software development. Software Quality, 3, 1997–1998.

    Google Scholar 

  • Buglione, L., & Abran, A. (2006). Introducing root-cause analysis and orthogonal defect classification at lower CMMI maturity levels. In Proceedings of International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement (IWSM-MENSURA 2006) (p. 29). Cadiz, Spain. Retrieved from http://www.gelog.etsmtl.ca/publications/pdf/1037.pdf

  • Butcher, M., Munro, H., & Kratschmer, T. (2002). Improving software testing via ODC: Three case studies. IBM Systems Journal, 41(1), 31–44. doi:10.1147/sj.411.0031.

    Article  Google Scholar 

  • Carleton, A., & Florac, W. (1999). Measuring the software process: Statistical process control for software process improvement. Boston: Addison-Wesley Professional.

    Google Scholar 

  • Chillarege, R. (1996). Handbook of software reliability engineering. In M. R. Lyu (Ed.), (pp. 359–400). Hightstown, NJ, USA: McGraw-Hill, Inc. Retrieved from http://dl.acm.org/citation.cfm?id=239425.239453

  • Chillarege, R. (2013). Using ODC to diagnose an Agile enterprise application development project. IEEE International Symposium on Software Reliability Engineering Workshops, (ISSREW 2013), p. 45. doi:10.1109/ISSREW.2013.6688862

  • Chillarege, R., Bhandari, I. S., Chaar, J. K., Halliday, M. J., Moebus, D. S., Ray, B. K., et al. (1992). Orthogonal defect classification—A concept for in-process measurements. IEEE Transactions on Software Engineering, 18(11), 943–956. doi:10.1109/32.177364.

    Article  Google Scholar 

  • Chillarege, R., & Prasad, K. R. (2002). s_Test and development process retrospective—a case study using ODC triggers. Proceedings of International Conference on Dependable Systems and Networks,. doi:10.1109/DSN.2002.1029012.

    Google Scholar 

  • Dubey, A. (2012). Towards adopting ODC in automation application development projects. Proceedings of the 5th India Software Engineering Conference (ISEC’12),. doi:10.1145/2134254.2134282.

    Google Scholar 

  • Feiler, P. H., & Humphrey, W. S. (1993). Software process development and enactment: Concepts and definitions. Software Process 1993 Continuous Software Process Improvement Second International Conference on the, Berlin, Germany, pp. 28–40. doi:10.1109/SPCON.1993.236824

  • Freimut, B. (2001). Developing and using defect classification schemes. Fraunhofer IESE. Retrieved from http://en.scientificcommons.org/20203575

  • Gómez, O., Oktaba, H., Piattini, M., & García, F. (2006). A systematic review measurement in software engineering: State-of-the-art in measures. In International Conference on Software and Data Technologies (ICSOFT 2006) (pp. 165–176). doi:10.1007/978-3-540-70621-2

  • Grundy, J. C., Mugridge, W. B., & Hosking, J. G. (1997). Utilising past event histories in a process-centred software engineering environment. Proceedings of the Australian Software Engineering Conference,. doi:10.1109/ASWEC.1997.623764.

    Google Scholar 

  • Gürgen, T., Tarhan, A., & Karagöz, N. A. (2014). An integrated infrastructure using process mining techniques for software process verification. In R. Perez-Castillo & M. Piattini (Eds.), Uncovering essential software artifacts through business process archaeology (pp. 364–382). Hershey: IGI Global. doi:10.4018/978-1-4666-4667-4.ch014.

    Chapter  Google Scholar 

  • Hommes, B. J. (2004). The evaluation of business process modeling techniques. Ph.D. thesis, Delft University of Technology, Delft.

  • Huang, L., Ng, V., Persing, I., Geng, R., Bai, X., & Tian, J. (2015). ac_AutoODC: Automated generation of orthogonal defect classifications. Automated Software Engineering, 22, 3–46. doi:10.1109/ASE.2011.6100086.

    Article  Google Scholar 

  • Humphrey, W. S. (1989). Managing the software process. Boston: Addison-Wesley Professional.

    Google Scholar 

  • Huo, M., He, Z., & Jeffery, R. (2006). A systematic approach to process enactment analysis as input to software process improvement or tailoring. Proceedings—Asia-Pacific Software Engineering Conference, APSEC, pp. 401–408. doi:10.1109/APSEC.2006.14

  • IEEE. (1990). IEEE standard glossary of software engineering terminology (IEEE Std 610.12-1990). Los Alamitos. CA: IEEE Computer Society (Vol. 121990). Retrieved from http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:IEEE+Standard+Glossary+of+Software+Engineering+Terminology+(IEEE+Std+610.12-1990)#0

  • Ishikawa, K. (1986). Guide to quality control (2nd ed.). Clearwater: Quality Resources.

    Google Scholar 

  • ISO. (2015). ISO 9001: Quality management system-requirements. Geneva: International Organization for Standardization.

    Google Scholar 

  • Jalote, P., & Agrawal, N. (2005). Using defect analysis feedback for improving quality and productivity in iterative software development. In Proceedings of 3rd International Conference on Information and Communications Technology (ITI 2005)—Enabling technologies for the new knowledge society. (pp. 703–713). doi:10.1109/ITICT.2005.1609661

  • Jones, C. (2008). Applied software measurement: Global analysis of productivity and quality (3rd ed.). McGraw-Hill Osborne Media. Retrieved from http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/0071502440

  • Kabbaj, M., Lbath, R., & Coulette, B. (2008). A deviation management system for handling software process enactment evolution. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5007 LNCS, pp. 186–197. doi:10.1007/978-3-540-79588-9_17

  • Kalinowski, M., Card, D. N., & Travassos, G. H. (2012). Evidence-based guidelines to defect causal analysis. IEEE Software, 29(4), 16–18. doi:10.1109/MS.2012.72.

    Article  Google Scholar 

  • Kan, S. H. (2002). Metrics and models in software quality engineering (2nd ed.). Boston, MA: Addison-Wesley Longman Publishing Co., Inc.

    MATH  Google Scholar 

  • Kanniappan, H. S., & Mishra, U. (2011). Orthogonal defects classification of observed defects in C-DAC Noida projects. In Proceedings of Annual Seminar of C-DAC Noida Technologies (ASCNT-CDAC 2011). Noida, India.

  • Kuhrmann, M., Kalus, G., & Then, M. (2014). The process enactment tool framework—Transformation of software process models to prepare enactment. Science of Computer Programming, 79, 172–188. doi:10.1016/j.scico.2012.03.007.

    Article  Google Scholar 

  • Kumaresh, S., & Baskaran, R. (2010). Defect analysis and prevention for software process quality improvement. International Journal of Computer Applications, 8(7), 42–47. doi:10.5120/1218-1759.

    Article  Google Scholar 

  • Li, Z. B., Hou, X. M., Yu, L., Du, Z. P., & Xu, B. (2011). Analysis of software process effectiveness based on orthogonal defect classification. Procedia Environmental Sciences (PART A), 10, 765–770. doi:10.1016/j.proenv.2011.09.124.

    Article  Google Scholar 

  • Poncin, W., Serebrenik, A., & Brand, M. Van Den. (2011). Process mining software repositories. 2011 15th European Conference on Software Maintenance and Reengineering, pp. 5–14. doi:10.1109/CSMR.2011.5

  • Rubin, V. A., Günther, C. W., Van Der Aalst, W. M. P., Kindler, E., Van Dongen, B. F., & Schäfer, W. (2007). Process mining framework for software processes. In Software process dynamics and agility (Vol. 4470, pp. 169–181). doi:10.1007/978-3-540-72426-1_15

  • Rubin, V. A., Mitsyuk, A. A., Lomazova, I. A., & van der Aalst, W. M. P. (2014). Process mining can be applied to software too! Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement—ESEM’14, pp. 1–8. doi:10.1145/2652524.2652583

  • SEI. (2010). CMMI for development, Version 1.3

  • Shenvi, A. A. (2009). Defect prevention with orthogonal defect classification. In Proceedings of the 2nd India Software Engineering Conference (pp. 83–88). New York, NY, USA: ACM. doi:10.1145/1506216.1506232

  • Si, Q., & Yan, G. (2013). Research on quality assurance method based on software defect analysis. In Proceedings of the 26th Conference of Spacecraft TT&C Technology in China (Vol. 187, pp. 371–378). doi:10.1007/978-3-642-33663-8

  • Söylemez, M., & Tarhan, A. (2013). Using process enactment data analysis to support orthogonal defect classification for software process improvement. In Proceedings of International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement (IWSM-MENSURA 2013) (pp. 120–125). doi:10.1109/IWSM-Mensura.2013.27

  • Tarhan, A., & Demirors, O. (2011). Investigating the effect of variations in the test development process: A case from a safety-critical system. Software Quality Journal, 19(4), 615–642.

    Article  Google Scholar 

  • Tarhan, A., & Demirors, O. (2012). Apply quantitative management now. IEEE Software, 29(3), 77–85. doi:10.1109/MS.2011.91.

    Article  Google Scholar 

  • Van der Aalst, W. M. P., & Weijters, A. J. M. M. (2004). Process mining: A research agenda. Computers in Industry, 53(3), 231–244. doi:10.1016/j.compind.2003.10.001.

    Article  Google Scholar 

  • Wagner, S. (2008). Defect classification and defect types revisited. In Proceedings of the 2008 Workshop on Defects in Large Software Systems (pp. 39–40). New York, NY, USA: ACM. doi:10.1145/1390817.1390829

  • Wolf, A. L., & Rosenblum, D. S. (1993). A study in software process data capture and analysis. In Proceedings of the Second International Conference on the Software Process—Continuous Software Process Improvement. doi:10.1109/SPCON.1993.236817

  • Yin, R. K. (2013). Case study research: Design and methods (5th ed.). Thousand Oaks: Sage Publications.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. TÜBİTAK - BİLGEM/Software Technologies Research Institute, 06100, Ankara, Turkey

    Mehmet Söylemez

  2. Computer Engineering Department, Hacettepe University, Beytepe Kampusu, 06800, Ankara, Turkey

    Ayca Tarhan

Authors
  1. Mehmet Söylemez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayca Tarhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayca Tarhan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Söylemez, M., Tarhan, A. Challenges of software process and product quality improvement: catalyzing defect root-cause investigation by process enactment data analysis. Software Qual J 26, 779–807 (2018). https://doi.org/10.1007/s11219-016-9334-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9334-6

Keywords

Search

Navigation