A systematic mapping study on the combination of static and dynamic quality assurance techniques
Highlights
► Systematic mapping study on the combination of static and dynamic quality assurance. ► 51 articles were found and grouped into compilation and integration approaches. ► Combined approaches mainly aim at improving effectiveness, efficiency, and coverage. ► Twice as many articles were published between 2005 and 2009 than in the 20 years before. ► About 50% of the approaches were evaluated.
Introduction
Nowadays, software and software-intensive systems can be found all around us. Since they are growing both in size and complexity, developing high-quality software is becoming more challenging and expensive. In order to achieve the desired time, cost, and quality goals, the development approach, including the quality assurance (QA) activities, has to be optimized. According to Burnstein [49], in this article, QA activities are understood as all kinds of analytical activities conducted during software development with the intention of finding and removing defects. One important strategy in this direction is a systematic combination of existing QA techniques in order to obtain certain synergy effects, such as reduced costs or higher effectiveness.
Various kinds of QA techniques and methods to ensure software quality exist that involve preventing defects or detecting existing defects. Those QA activities that focus on detecting existing defects are also called verification and validation activities, which is defined by the IEEE Standard Glossary of Software Engineering Terminology as “the process of determining whether the requirements for a system or component are complete and correct, the products of each development phase fulfill the requirements or conditions imposed by the previous phase, and the final system or component complies with specified requirements” [37].
With respect to software verification and validation activities, static and dynamic QA techniques can be distinguished. Static QA techniques (e.g., inspections, reviews, walkthroughs, or static analyses such as program slicing) do not need models or code to be executed, but rather examine artifacts such as requirements documents, design models, or code without running them. In contrast, dynamic QA techniques (e.g., equivalence partitioning, boundary value analysis, control-flow based testing techniques, or dynamic analyses such as program profiling) need to execute programs or program parts.
The use of both static and dynamic QA activities is well known to software engineering practitioners. However, usually there is no systematic combination of such activities to exploit further benefits.
Today, a large number of well-established static and dynamic QA techniques exist, such as various inspection and testing techniques [48], [49], [56]. However, the effort for applying these techniques sometimes consumes more than 50% of the overall development effort, especially for conducting testing activities [34], [55], [64]. Thus, one objective is often to improve the efficiency of testing and to reduce the overall QA effort. Besides this, further desirable goals include improving overall effectiveness (i.e., finding as many defects as possible – especially critical ones – before distributing software), planning and controlling QA activities, and improving the overall quality.
In the past, a lot of research has been performed to develop and improve a variety of static and dynamic QA techniques. Juristo et al. [52] examined 25 years of empirical studies with respect to a large number of different testing techniques, classified them, and summarized the main findings. They conclude that the current testing knowledge is very limited. With respect to software inspections, Aurum et al. [58] examined software inspection processes published during the 25 years since inspection as a QA technique was first published by Fagan in 1976 [24]. They identified different inspection processes and support for the inspection, such as reading techniques, tools, and support for deciding to perform a re-inspection. In conclusion, Aurum et al. [58] stated that the identified studies contribute to the evolution of software inspections, but many research questions remain open. Another examination of software inspection research, covering the period between 1991 and 2005, was performed by Kollanus and Koskinen [60]. They classified the identified articles into a technical view (e.g., reading techniques, effectiveness factors), a management view (e.g., inspection impact on development process), and other topics (e.g., defect estimation, inspection tools). The two authors concluded that much research has been performed with respect to software inspections, but that empirical knowledge remained low.
One fundamental observation with respect to research on inspection and testing techniques, which are two of the best-established static, respectively dynamic, QA techniques, is that most often, this research is done to improve inspections or testing themselves. In contrast, some studies compare different inspection and testing techniques [23], [73], which often resulted in the conclusion to apply them in combination [2], [59]. Other studies calculated effectiveness values when applying them in combination to demonstrate the benefit of a joint application [6], [21]. However, except for suggestions to apply both, no concrete process or additional advices is usually provided.
Conventional software engineering, particularly standard lifecycles (such as the waterfall and “V” models), have emphasized static methods during early (pre-code) development phases and dynamic development during later (post-code) development phases. However, from our point of view, combining different static and dynamic QA techniques, such as inspections and testing, is a promising way to improve QA and to cope with problems such as high QA costs. The connection between static and dynamic QA seems intuitively clear and obvious, but in practice it is often lost or obscured. The result is poorly prioritized and often redundant QA effort. It is perfectly possible that combining static and dynamic quality assurance techniques may have been used in practice already, because the underlying reasoning is grounded on well-known software engineering practices. However, even in this case, it is questionable whether existing approaches actually rely on explicit, well-grounded and evaluated approaches instead of merely being based on common sense and unsystematic procedures.
The main objective of this systematic mapping study is to obtain a profound overview of existing approaches that combine static and dynamic QA techniques, and of their goals (e.g., reduction of effort, improvement of defect detection). To the best of our knowledge, no systematic mapping study of the combination of static and dynamic QA techniques exists. One survey [73] summarizes some defect detection studies and compares inspection and testing techniques. However, this study has a narrower scope, as it only considers inspection and testing (and not static and dynamic quality assurance activities in general), and the survey did not explicitly follow procedures for a systematic mapping study, which raises questions regarding its completeness. Aurum et al. [58], for example, mentioned ten open research questions in their conclusion. One of these is about the relationship between inspections and software testing and the best way these techniques might complement each other. Thus, the results of this mapping study may be a substantial starting point for comprehensive systematic literature reviews and future research in the area of combining static and dynamic QA techniques.
The article is structured as follows. Section 2 presents the research methodology pursued to perform the systematic mapping study. The results are shown and described in Section 3. Section 4 comprises a discussion of the results and their implications. Section 5 concludes the article and presents directions for future work.
Section snippets
Research methodology
According to Petersen et al. [61], a systematic mapping study is considered as a kind of secondary study that reviews articles related to a specific research topic, and which aims at providing a classification, conducting a thematic analysis, or presenting publication channels. A number of research questions must be defined in order to obtain these objectives in a systematic manner. Consequently, presenting an overview of a certain research area and identifying research gaps are the main goals
Results
The results of the systematic mapping study are presented next, ordered by the six research questions.
General findings
Based on the results of the systematic mapping study, one main observation with respect to the combination of static and dynamic QA techniques is that this research area has gained increased attention since 2005. Some articles published before suggested combining, for instance, inspection and testing techniques in order to improve overall quality assurance. Nevertheless, concrete combination approaches were mainly published during the last few years.
The combination of static and dynamic
Conclusion
In this article, the results of a systematic mapping study regarding the combination of static and dynamic quality assurance techniques were presented. Six detailed research questions were answered. The main results are as follows:
RQ1. 51 identified articles were mainly classified into two groups of combination approaches, namely compilation and integration approaches. Both groups contain the same number of articles.
RQ2. The topic has gained increasing interest since 2005, with some articles
Acknowledgments
Parts of this work have been funded by the Stiftung Rheinland-Pfalz für Innovation project “Qualitäts-KIT” (Grant: 925). We would like to thank Sonnhild Namingha for proofreading.
References (74)
- et al.
An empirical evaluation of defect detection techniques
Information and Software Technology
(1997) - et al.
Combined static and dynamic analysis
Electronic Notes in Theoretical Computer Science
(2005) - T.F. Chang, A. Danylyzsn, S. Norimatsu, J. Rivera, D. Shepard, A. Lattanze, J. Tomayko, “Continuous verification” in...
- et al.
A Handbook of Software and Systems Engineering
(2003) - Y. Chen, S. Liu, W.E Wong, A method combining review and testing for verifying software systems, in: Proceedings of the...
- A. Hanna, H.Z. Ling, X. Yang, M. Debbabi, A synergy between static and dynamic analysis for the detection of software...
- D.M. Zimmerman, J.R. Kiniry, A verification-centric software development process for Java, in: Proceedings of the 9th...
- et al.
An empirical evaluation of six methods to detect faults in software
Software Testing, Verification and Reliability
(2002) - E. Kamsties, C.M. Lott, An empirical evaluation of three defect-detection techniques, in: 5th European Software...
- R. Conradi, A.S. Marjara, B. Skatevik, An empirical study of inspection and testing data at Ericsson, Norway, in: 24th...
Applying testing to requirements inspection for software quality assurance
Information Systems Control Journal
Aspects-classes integration testing strategy: an incremental approach
Rapid Integration of Software Engineering Techniques
Automating software testing using program analysis
IEEE Software
Combining static analysis and test generation for C program debugging
Tests and Proofs
Comparing the effectiveness of software testing strategies
IEEE Transactions on Software Engineering
Design and code inspections to reduce errors in program development
IBM Systems Journal
DSD-Crasher: a hybrid analysis tool for bug finding
ACM Transactions on Software Engineering and Methodology
Estimating the value of inspections and early testing for software projects
Hewlett-Packard Journal
Functional testing, structural testing, and code reading: what fault type do they each detect?
Empirical Methods and Studies in Software Engineering
Cited by (70)
Unfolding the blockchain era: Timeline, evolution, types and real-world applications
2022, Journal of Network and Computer ApplicationsExploring the intersection between software industry and Software Engineering education - A systematic mapping of Software Engineering Trends
2021, Journal of Systems and SoftwarePersonal health data: A systematic mapping study
2018, International Journal of Medical InformaticsCitation Excerpt :In mapping studies this thread concerns inaccurate data extraction or exclusion of relevant studies during the selection phase [161]. The control over conclusion validity has also implications to reliability of the study, so that if a systematic mapping study is performed by other researchers, they will arrive at the same conclusions [158,159]. To handle this thread, the selection and extraction process of the data performed according to the established scheme as described in Section 3.
Spreadsheet quality assurance: a literature review
2024, Frontiers of Computer Science