A systematic mapping study on the combination of static and dynamic quality assurance techniques

https://doi.org/10.1016/j.infsof.2011.06.003Get rights and content

Abstract

Context

A lot of different quality assurance techniques exist to ensure high quality products. However, most often they are applied in isolation. A systematic combination of different static and dynamic quality assurance techniques promises to exploit synergy effects, such as higher defect detection rates or reduced quality assurance costs. However, a systematic overview of such combinations and reported evidence about achieving synergy effects with such kinds of combinations is missing.

Objective

The main goal of this article is the classification and thematic analysis of existing approaches that combine different static and dynamic quality assurance technique, including reported effects, characteristics, and constraints. The result is an overview of existing approaches and a suitable basis for identifying future research directions.

Method

A systematic mapping study was performed by two researchers, focusing on four databases with an initial result set of 2498 articles, covering articles published between 1985 and 2010.

Results

In total, 51 articles were selected and classified according to multiple criteria. The two main dimensions of a combination are integration (i.e., the output of one quality assurance technique is used for the second one) and compilation (i.e., different quality assurance techniques are applied to ensure a common goal, but in isolation). The combination of static and dynamic analyses is one of the most common approaches and usually conducted in an integrated manner. With respect to the combination of inspection and testing techniques, this is done more often in a compiled way than in an integrated way.

Conclusion

The results show an increased interest in this topic in recent years, especially with respect to the integration of static and dynamic analyses. Inspection and testing techniques are currently mostly performed in an isolated manner. The integration of inspection and testing techniques is a promising research direction for the exploitation of additional synergy effects.

Highlights

Systematic mapping study on the combination of static and dynamic quality assurance. ► 51 articles were found and grouped into compilation and integration approaches. ► Combined approaches mainly aim at improving effectiveness, efficiency, and coverage. ► Twice as many articles were published between 2005 and 2009 than in the 20 years before. ► About 50% of the approaches were evaluated.

Introduction

Nowadays, software and software-intensive systems can be found all around us. Since they are growing both in size and complexity, developing high-quality software is becoming more challenging and expensive. In order to achieve the desired time, cost, and quality goals, the development approach, including the quality assurance (QA) activities, has to be optimized. According to Burnstein [49], in this article, QA activities are understood as all kinds of analytical activities conducted during software development with the intention of finding and removing defects. One important strategy in this direction is a systematic combination of existing QA techniques in order to obtain certain synergy effects, such as reduced costs or higher effectiveness.

Various kinds of QA techniques and methods to ensure software quality exist that involve preventing defects or detecting existing defects. Those QA activities that focus on detecting existing defects are also called verification and validation activities, which is defined by the IEEE Standard Glossary of Software Engineering Terminology as “the process of determining whether the requirements for a system or component are complete and correct, the products of each development phase fulfill the requirements or conditions imposed by the previous phase, and the final system or component complies with specified requirements” [37].

With respect to software verification and validation activities, static and dynamic QA techniques can be distinguished. Static QA techniques (e.g., inspections, reviews, walkthroughs, or static analyses such as program slicing) do not need models or code to be executed, but rather examine artifacts such as requirements documents, design models, or code without running them. In contrast, dynamic QA techniques (e.g., equivalence partitioning, boundary value analysis, control-flow based testing techniques, or dynamic analyses such as program profiling) need to execute programs or program parts.

The use of both static and dynamic QA activities is well known to software engineering practitioners. However, usually there is no systematic combination of such activities to exploit further benefits.

Today, a large number of well-established static and dynamic QA techniques exist, such as various inspection and testing techniques [48], [49], [56]. However, the effort for applying these techniques sometimes consumes more than 50% of the overall development effort, especially for conducting testing activities [34], [55], [64]. Thus, one objective is often to improve the efficiency of testing and to reduce the overall QA effort. Besides this, further desirable goals include improving overall effectiveness (i.e., finding as many defects as possible – especially critical ones – before distributing software), planning and controlling QA activities, and improving the overall quality.

In the past, a lot of research has been performed to develop and improve a variety of static and dynamic QA techniques. Juristo et al. [52] examined 25 years of empirical studies with respect to a large number of different testing techniques, classified them, and summarized the main findings. They conclude that the current testing knowledge is very limited. With respect to software inspections, Aurum et al. [58] examined software inspection processes published during the 25 years since inspection as a QA technique was first published by Fagan in 1976 [24]. They identified different inspection processes and support for the inspection, such as reading techniques, tools, and support for deciding to perform a re-inspection. In conclusion, Aurum et al. [58] stated that the identified studies contribute to the evolution of software inspections, but many research questions remain open. Another examination of software inspection research, covering the period between 1991 and 2005, was performed by Kollanus and Koskinen [60]. They classified the identified articles into a technical view (e.g., reading techniques, effectiveness factors), a management view (e.g., inspection impact on development process), and other topics (e.g., defect estimation, inspection tools). The two authors concluded that much research has been performed with respect to software inspections, but that empirical knowledge remained low.

One fundamental observation with respect to research on inspection and testing techniques, which are two of the best-established static, respectively dynamic, QA techniques, is that most often, this research is done to improve inspections or testing themselves. In contrast, some studies compare different inspection and testing techniques [23], [73], which often resulted in the conclusion to apply them in combination [2], [59]. Other studies calculated effectiveness values when applying them in combination to demonstrate the benefit of a joint application [6], [21]. However, except for suggestions to apply both, no concrete process or additional advices is usually provided.

Conventional software engineering, particularly standard lifecycles (such as the waterfall and “V” models), have emphasized static methods during early (pre-code) development phases and dynamic development during later (post-code) development phases. However, from our point of view, combining different static and dynamic QA techniques, such as inspections and testing, is a promising way to improve QA and to cope with problems such as high QA costs. The connection between static and dynamic QA seems intuitively clear and obvious, but in practice it is often lost or obscured. The result is poorly prioritized and often redundant QA effort. It is perfectly possible that combining static and dynamic quality assurance techniques may have been used in practice already, because the underlying reasoning is grounded on well-known software engineering practices. However, even in this case, it is questionable whether existing approaches actually rely on explicit, well-grounded and evaluated approaches instead of merely being based on common sense and unsystematic procedures.

The main objective of this systematic mapping study is to obtain a profound overview of existing approaches that combine static and dynamic QA techniques, and of their goals (e.g., reduction of effort, improvement of defect detection). To the best of our knowledge, no systematic mapping study of the combination of static and dynamic QA techniques exists. One survey [73] summarizes some defect detection studies and compares inspection and testing techniques. However, this study has a narrower scope, as it only considers inspection and testing (and not static and dynamic quality assurance activities in general), and the survey did not explicitly follow procedures for a systematic mapping study, which raises questions regarding its completeness. Aurum et al. [58], for example, mentioned ten open research questions in their conclusion. One of these is about the relationship between inspections and software testing and the best way these techniques might complement each other. Thus, the results of this mapping study may be a substantial starting point for comprehensive systematic literature reviews and future research in the area of combining static and dynamic QA techniques.

The article is structured as follows. Section 2 presents the research methodology pursued to perform the systematic mapping study. The results are shown and described in Section 3. Section 4 comprises a discussion of the results and their implications. Section 5 concludes the article and presents directions for future work.

Section snippets

Research methodology

According to Petersen et al. [61], a systematic mapping study is considered as a kind of secondary study that reviews articles related to a specific research topic, and which aims at providing a classification, conducting a thematic analysis, or presenting publication channels. A number of research questions must be defined in order to obtain these objectives in a systematic manner. Consequently, presenting an overview of a certain research area and identifying research gaps are the main goals

Results

The results of the systematic mapping study are presented next, ordered by the six research questions.

General findings

Based on the results of the systematic mapping study, one main observation with respect to the combination of static and dynamic QA techniques is that this research area has gained increased attention since 2005. Some articles published before suggested combining, for instance, inspection and testing techniques in order to improve overall quality assurance. Nevertheless, concrete combination approaches were mainly published during the last few years.

The combination of static and dynamic

Conclusion

In this article, the results of a systematic mapping study regarding the combination of static and dynamic quality assurance techniques were presented. Six detailed research questions were answered. The main results are as follows:

  • RQ1. 51 identified articles were mainly classified into two groups of combination approaches, namely compilation and integration approaches. Both groups contain the same number of articles.

  • RQ2. The topic has gained increasing interest since 2005, with some articles

Acknowledgments

Parts of this work have been funded by the Stiftung Rheinland-Pfalz für Innovation project “Qualitäts-KIT” (Grant: 925). We would like to thank Sonnhild Namingha for proofreading.

References (74)

  • M. Roper et al.

    An empirical evaluation of defect detection techniques

    Information and Software Technology

    (1997)
  • C. Artho et al.

    Combined static and dynamic analysis

    Electronic Notes in Theoretical Computer Science

    (2005)
  • T.F. Chang, A. Danylyzsn, S. Norimatsu, J. Rivera, D. Shepard, A. Lattanze, J. Tomayko, “Continuous verification” in...
  • A. Endres et al.

    A Handbook of Software and Systems Engineering

    (2003)
  • Y. Chen, S. Liu, W.E Wong, A method combining review and testing for verifying software systems, in: Proceedings of the...
  • A. Hanna, H.Z. Ling, X. Yang, M. Debbabi, A synergy between static and dynamic analysis for the detection of software...
  • D.M. Zimmerman, J.R. Kiniry, A verification-centric software development process for Java, in: Proceedings of the 9th...
  • S.S. So et al.

    An empirical evaluation of six methods to detect faults in software

    Software Testing, Verification and Reliability

    (2002)
  • E. Kamsties, C.M. Lott, An empirical evaluation of three defect-detection techniques, in: 5th European Software...
  • R. Conradi, A.S. Marjara, B. Skatevik, An empirical study of inspection and testing data at Ericsson, Norway, in: 24th...
  • C. Andersson, T. Thelin, P. Runeson, N. Dzamashvili, An experimental evaluation of inspection and testing for detection...
  • T. Berling, T. Thelin, An industrial case study of the verification and validation activities, in: Proceedings of the...
  • T.Y. Chen et al.

    Applying testing to requirements inspection for software quality assurance

    Information Systems Control Journal

    (2006)
  • P. Massicotte et al.

    Aspects-classes integration testing strategy: an incremental approach

    Rapid Integration of Software Engineering Techniques

    (2006)
  • P. Godefroid et al.

    Automating software testing using program analysis

    IEEE Software

    (2008)
  • C. Csallner, Y. Smaragdakis, Check ‘n’ crash: combining static checking and testing, in: Proceedings of the 27th...
  • J. Chen, H. Zhou, S.D. Bruda, Combining model checking and testing for software analysis, in: Proceedings of the 2008...
  • O. Chebaro et al.

    Combining static analysis and test generation for C program debugging

    Tests and Proofs

    (2010)
  • P. Centonze, R. Flynn, M. Pistoia, Combining static and dynamic analysis for automatic identification of precise...
  • F.D. Anger, R.V. Rodriguez, M. Young, Combining static and dynamic analysis of concurrent programs, in: Proceedings of...
  • M. Wood, M. Roper, A. Brooks, J. Miller, Comparing and combining software defect detection techniques – a replicated...
  • S. Wagner, J. Jürjens, C. Koller, P. Trischberger, Comparing bug finding tools with reviews and test, in: Proceedings...
  • V.R. Basili et al.

    Comparing the effectiveness of software testing strategies

    IEEE Transactions on Software Engineering

    (1987)
  • M.E. Fagan

    Design and code inspections to reduce errors in program development

    IBM Systems Journal

    (1976)
  • P. Runeson, A. Andrews, Detection or isolation of defects? An experimental comparison of unit testing and code...
  • C. Csallner et al.

    DSD-Crasher: a hybrid analysis tool for bug finding

    ACM Transactions on Software Engineering and Methodology

    (2008)
  • S. Zhang, Y. Lin, Z. Gu, J. Zhao, Effective identification of failure-inducing changes: a hybrid approach, in:...
  • M. Gopinathan, S.K. Rajamani, Enforcing object protocols by combining static and runtime analysis, in: Proceedings of...
  • Engineering Village....
  • L. Franz et al.

    Estimating the value of inspections and early testing for software projects

    Hewlett-Packard Journal

    (1994)
  • N. Juristo et al.

    Functional testing, structural testing, and code reading: what fault type do they each detect?

    Empirical Methods and Studies in Software Engineering

    (2003)
  • B.A. Kitchenham, S. Charters, Guidelines for Performing Systematic Literature Reviews in Software Engineering,...
  • Q. Chen, L. Wang, Z. Yang, S.D. Stoller, HAVE: detecting atomicity violations via integrated dynamic and static...
  • Health, Social, and Economic Research, The Economic Impacts of Inadequate Infrastructure for Software Testing, National...
  • Q. Chen, L. Wang, Z. Yang, HEAT: an integrated static and dynamic approach for thread escape analysis, in: Proceedings...
  • P.D. Kumar, A. Nema, R. Kumar, Hybrid analysis of executables to detect security vulnerabilities: security...
  • IEEE Standard 610.12-1990. IEEE Standard Glossary of Software Engineering Terminology,...
  • Cited by (70)

    • Personal health data: A systematic mapping study

      2018, International Journal of Medical Informatics
      Citation Excerpt :

      In mapping studies this thread concerns inaccurate data extraction or exclusion of relevant studies during the selection phase [161]. The control over conclusion validity has also implications to reliability of the study, so that if a systematic mapping study is performed by other researchers, they will arrive at the same conclusions [158,159]. To handle this thread, the selection and extraction process of the data performed according to the established scheme as described in Section 3.

    • Spreadsheet quality assurance: a literature review

      2024, Frontiers of Computer Science
    View all citing articles on Scopus
    View full text