Quantitative evaluation of safety critical software testability based on fault tree analysis and entropy

Abstract

One of the definitions for testability is the probability whether tests will detect a fault, given that a fault in the program exists. The testability can be estimated from the probability of each statement fault leading to output failure. The probability of the test detecting a fault depends on the probability of individual statement faults appearing as an output failure when a fault exists at a statement. The testability measure of the software has been introduced based on output failure probability and the entropy of the importance of basic statements to the output failure from the software fault tree analysis. The output failure probability and the importance of statements are calculated from software fault tree analysis. The suggested testability measure has been applied to the two modules of the safety system in a nuclear power plant. The proposed testability measure can be used for the selection of output variables or to determine the modules that are more vulnerable to undetected faults.

Introduction

The software life cycle is defined from the functional requirements phase to the operation and maintenance phases, where testing is one of the major processes. The testing is performed in a unit entity that can be a module in an assembled entity or can be a program with combined modules for a specific function. Depending on the importance of the software, the unit testing can be merged into program-level testing. There have been efforts to model the software failure to derive test cases that can detect any software faults included in the program. Richardson et al. (1993) proposed the RELAY model of faults and failures based on the incorrect intermediate state from a faulty statement to output. The model defines necessary and sufficient conditions for detecting certain classes of faults. In the RELAY model, a fault needs three distinct steps to appear as a failure: introduction of an origination of a fault, computational transfer of a fault, and propagation of an error to the output. The introduction of a fault and its computational transfer occurs when the statement containing the fault evaluates incorrectly. Even if the error state is introduced, certain transfer conditions to output should be met for an error state to result in failure. By analyzing these conditions, the test case can be selected to guarantee fault detection for chosen fault classes. With the safety critical software, the number of faults detected has been very few. If there has been detection of an error, the testing should be repeated to make sure that there have been no faults introduced with modifications. Littlewood and Wright (1997) derives the number of test cases that should be repeated after a fault is detected in safety critical software. Even though no fault has been detected, it might be necessary to verify the effectiveness of the testing. The effectiveness of the testing might be checked with fault injection testing or justified with a testability measure showing that the module or software has high testability. The testability to justify the testing should demonstrate that the testing is effective in detecting a fault. But in the IEEE glossary of terminology, the software testability is defined as the degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met. In this definition, the testability is focused on the ease of generating test sets and satisfying the criteria. There have been efforts to measure testability in this respect, trying to measure the testability early in the design stage based on the specification. Voas and Miller (1993) estimated testability of the software from the software input domain and software output range. If the ratio of the domain and range is big, then the testability is low. If the ratio is small, the testability is high. The testability measure based on the domain-range ratio can be calculated from the program specification using a simple relationship, but it does not consider the detail specification of the program. Freedman (1991) introduced the testability by measuring the observability and controllability of the software. The observability of the program is defined as the ease of determining if specified inputs affect the outputs, and the controllability is defined as the ease of producing a specified output from a specified input. Programs can be made more observable by introducing more input based on implicit states, and programs can be made more controllable by restricting the range of outputs. The testability proposed is based on observability and controllability of the program specification. It represents testing efforts required to modify a component to a domain testable form which is more testable from the aspects of the observability and controllability. It is argued that it takes less time to build and test a program if it is more observable and controllable. Voas and Miller (1995) defines software testability as the probability that a piece of software will fail on its next execution during testing if the software includes a fault. He proposes software testability predicted by sensitivity analysis that seeds the fault and observes the propagation into failures. The testability estimation is based on the empirical sensitivity analysis which consists of three empirical analyses: program execution analysis, infection analysis, and propagation analysis. The mutation testing is performed for infection analysis and random testing is performed to monitor the propagation analysis. As design heuristics for testability, he suggests three methods: specification decomposition to reduce error cancellation, minimizing variable reuse, and increasing out-parameters. Out-parameters mean specifying special output variables that are specified and implemented specifically and exclusively for testing. For the suggestion of making software easy to reveal faults, Bertolino and Strigini (1996) argues to make programs more robust, increase coverage by improving oracle or observability of the internal state, and choose test input distribution to increase error infection probability. He argues that increasing testability by raising the failure probability is dangerous and should not increase failure probability while increasing probability of detecting faults. He recommends more refined testing tools and to improve the internal error detection capability. The testability measures proposed so far have been based on the program specification of input/output variables (Freedman, 1991; Voas and Miller, 1993), or they required sensitivity analysis with program execution (Voas and Miller, 1995). The testability based on program specification can be obtained simply from specification on inputs and outputs, but it does not consider the internal complexity of the specification. The testability from sensitivity analysis is based on the program and requires mutation analysis and random testing in addition. In this paper, a measure for testability based on the program is proposed, which measures the ease of detecting a fault when the program includes a fault. The measure can be obtained from the fault tree analysis of the program. It is based on the output failure probability and entropy of the importance of statements leading to output failure in software fault tree analysis. It considers the internal complexity of the program but does not require the execution of the program. In the case where software safety analysis is required using the fault tree analysis method, the testability can be easily estimated from the fault tree analysis.

In the following section, the software testability measure based on the fault tree analysis is introduced. Section 3 shows the application of testability for simple programs with AND, OR logic. Section 4 shows the application of the testability measure to the safety software modules and its testing. Section 5 concludes the paper.