Cost Effectiveness of Coverage-Guided Test-Suite Reduction for Safety-Relevant Systems

Abstract

Test-suite reduction targets to find a minimal subset of the test suite that is sufficient regarding specific test requirements. In general, test-suite reduction causes some kind of loss in the fault-detection ability of the reduced test suite. Cost-Effectiveness for test-suite reduction is only given if the trade-off between the savings in the testing effort and the resulting degradation in the fault-detection ability is reasonable. In this work we define a cost function, summarizing the beneficial output of a test suite depending on the test effort and the fault-detection ability. We apply this cost function to test suites that are defined by different commonly used coverage criteria, like MCC (multiple condition coverage), MC/DC (modified condition/decision coverage), and DC (decision coverage) and evaluate the cost function for a safety-relevant system from automotive. The results mainly demonstrate two fundamental aspects of test-suite reduction: First, the cost effectiveness is very sensitive to the assumed costs for the test-case execution and the penalty for undetected faults. Secondly, for non-safety-relevant systems test-suite reduction for DC is still cost effective, but this does not hold for safety-relevant systems. The cost effectiveness of test-suite reduction for safety-relevant systems for MCC and MC/DC is almost the same.

This work has been partially funded by the ARTEMIS Joint Undertaking and the National Funding Agency of Austria for the project VeTeSS under the funding ID ARTEMIS-2011-1-295311.