An externally replicated experiment to evaluate software testing methods

Context: Many empirical studies have been carried out to evaluate software testing methods in the past decades. However, we are still not able to generalize the results, as most studies are not complete and differ significantly from one another. Objective: To contribute to the existing knowledge base of testing techniques by evaluating three software testing methods a) code reading by stepwise abstraction, b) functional testing using equivalence partitioning and boundary value analysis, and c) structural testing using 100% branch, multiple-condition, loop, and relational-operator coverage using a well-defined and standard schema. Method: A controlled experiment was carried out with eighteen subjects who applied three techniques to three C programs in a fractional factorial experimental design. Results: There is no difference in techniques in terms of failure observation and fault finding effectiveness and the effectiveness depends on the program rather than technique. In case of failure observation efficiency, all the techniques performed equally; however, in terms of fault isolation efficiency, code reading performed better than structural testing which in turn out performed functional testing. With respect to the fault types, all techniques performed equally except in case of cosmetic faults where functional testing performed better than other two testing methods. Conclusion: We conclude that all techniques are all equivalent in terms of effectiveness. However, the techniques differ partially in terms of efficiency. The effect of the program was significant in almost all cases. We need to build some standardized and better laboratory packages which should represent actual software engineering practices. Carry out experiments on such packages will help in deriving realistic results.