Mutation Analysis and Constraint-Based Criteria: Results from an Empirical Evaluation in the Context of Software Testing

Abstract

Several software testing criteria have been proposed during last years with the goal of aiming the test set generation and revealing many faults as possible. They are considered complementary because can reveal different kind of faults and are based on different principles. For example, structural criteria use the internal structure of the program for deriving test cases; Mutation Analysis is a fault-based criterion; and Constraint Based Criteria use constraints to be satisfied during the program execution. Because of this, some questions can be posed, such as: “What criterion should be used or be first applied?”. Many research works compare criteria with the goal of answering these questions. However, some criteria as Mutation Analysis and Constraint Based Criteria are theoretically incomparable and only empirical studies can point out the relation between them. This work presents results from an empirical evaluation of Mutation Analysis and All-Constrained-Potential-Uses criterion considering the factors: cost (number of test cases), efficacy (number of revealed faults) and strength (difficulty of satisfying a criterion, given that another one has been satisfied). The obtained results show an empirical relation, which is used to propose a strategy for application of different testing criteria.