Quantitative risk-based requirements reasoning

Abstract

At NASA we have been developing and applying a risk management framework, "Defect Detection and Prevention" (DDP). It is based on a simple quantitative model of risk and is supported by custom software. We have used it to aid in study and planning for systems that employ advanced technologies. The framework has proven successful at identifying problematic requirements (those which will be the most difficult to attain), at optimizing the allocation of resources so as to maximize requirements attainment, at identifying areas where research investments should be made, and at supporting tradeoff analyses among major alternatives. We describe the DDP model, the information that populates a model, how DDP is used, and its tool support. DDP has been designed to aid decision making early in development. Detailed information is lacking at this early stage. Accordingly, DDP exhibits a number of strategic compromises between fidelity and tractability. The net result is an approach that appears both feasible and useful during early requirements decision making.

Appendix: DDP concepts and example

Requirements are whatever the system under scrutiny is to achieve, and operational constraints on the system's construction and operation. Each requirement has a title, a position in the Requirements tree, a weight (representing its relative importance), on/off status, and optional further information such as description, notes, and reference. Only those Requirements whose status is on are taken into account in the qualitative calculations.

Failure Modes are the things that, should they occur, will lead to lack of attainment of Requirements. Each Failure Mode has a title, a position in the Failure Modes tree, an a priori likelihood (the chance of the Failure Mode occurring, if nothing is done to inhibit it), a repair cost per phase (what it would cost to remove an instance of that Failure Mode at that phase in the project), on/off status, and optional further information such as description, notes, and reference. Only Failure Modes whose status is on are taken into account in the qualitative calculations.

PACTs are the activities that could be done to reduce the likelihood of Failure Modes and/or reduce their impact on Requirements. Each PACT has a title, a position in the PACTs tree, a cost, the phase in which it applies, on/off status, and optional further information such as description, notes, and reference. PACTs are classified into preventions (which reduce the likelihood of Failure Modes), detections (which discover instances of failure modes so that those detected failure modes can be corrected prior to release/use), and alleviations (which reduce the severity of Failure Modes). Only those PACTs whose status is on are taken into account in the qualitative calculations, with the exception of calculations specifically to reveal what would be the net effect (in terms of risk reduction) were an "off" PACT to be turned on.

Impacts are the qualitative relationships between Requirements and Failure Modes. Each impact has the Requirement and Failure Mode it links, a value representing the proportion of loss of attainment of the Requirement should the Failure Mode occur, and optional further information such as description, notes, and reference. The value may be non-numeric, in which case although it shows up on displays it is ignored in the quantitative calculations; the usual use for this is as a placeholder and reminder for further scrutiny (e.g., a value "to be determined").

Effects are the qualitative relationships between PACTs and Failure Modes. Each impact has the PACT and Failure Mode it links, a value representing the proportion of reduction of the Failure Mode should that PACT be applied, and optional further information such as description, notes, and reference. If the value is negative, it denotes an effect of increasing, rather than decreasing, a Failure Mode's likelihood. As for impacts, the value may be non-numeric, and if so is ignored in quantitative calculations.

We use a hypothetical example for illustration. This avoids any proprietary issues that would arise from reporting one of the actual DDP applications, and permits the use of somewhat smaller amounts of data than would arise in practice. Nevertheless, this example will serve to illustrate the elements of DDP referenced throughout the paper. The figures are annotated fragments of screenshots taken from the DDP tool running on this example.

Figure 11 shows the Requirements tree, the Failure Modes tree, and the matrix of Impact values between these trees' elements. The blue coloring (reproduced here as the darker gray border of the highlighted row and background of its header cells) highlights one of the Requirements, "Get to the target", and the red coloring (reproduced here as the darker gray border of the highlighted column and background of its header cells) highlights one of the Failure Modes, "Tolerance Issues".

Fig. 11.

Requirements, failure modes and impact matrix, between requirements (rows) and failure modes (cols)

The matrix header rows and columns show the titles of the items, and some totals computed by DDP.

The third row down contains values (39.4, 39.4, 15.9, etc.) that are the computed totals of loss of Requirements attainment that each Failure Mode causes. For a given Failure Mode F, this value is computed as:

F. APrioriLikelihood*Σ(R∈AllRequirements):R.Weight*Impact(F, R)where Impact(F, R) is the impact value of Failure Mode F on Requirement R (zero if there is no numerical impact asserted between them). For the highlighted Failure Mode, the calculation is:

1*((10*0.3)+(10*0.1)+(10*0.1)+(10*0.3)+(8*0.7)+(2*1.0)+(3*0.1))=3+1+1+3+5.6+2+0.3=15.9

The third column across contains values (71, 73, 78, etc.) that are the computed totals of loss of each Requirements Attainment caused by Failure Modes. For a given Requirement R, this value is computed as:

R.Weight*Σ(F∈AllFailureModes):Impact(F, R)*F.APrioriLikelihood

In the example data, all FM's a priori likelihoods are 1, so for the highlighted Requirement, this calculation is:

10*((0.3*1)+(0.3*1)+(0.3*1)+(0.3*1)+(1.0*1)+…)=

10*(0.3+0.3+0.3+0.3+1.0+0.3+1.0+1.0+0+0+0.1+0.7+0.1+0.7+1.0)=10*7.1=71

These are the totals for risk in the extreme case that nothing is done to prevent the Failure Modes from occurring. The first requirement has an assigned weight of 10, while its loss of attainment is calculated as 71, indicating that it is more than totally at risk.

This example's tree of PACTs is shown in Fig. 12 (top). PACTs' costs are listed in the column to the left of the tree. PACTs' effects on reducing Failure Modes are shown in the matrix in Fig. 12 (bottom). The green coloring (reproduced here as the darker gray border of the highlighted row and background of its header cells) highlights one of the PACTs "Environmental Tests", and the red coloring (reproduced here as the darker gray border of the highlighted column and background of its header cells) highlights one of the Failure Modes, "Tolerance Issues".

Fig. 12.

PACTs and effect matrix, between PACTs (rows) and failure modes (cols)

For the sake of illustration, we have checked those and only those PACTs in the "Tests" folder. DDP calculates their combined cost (176) and combined effect on reducing the likelihood of the Failure Modes. For example, the highlighted Failure Mode is effected by three of those PACTs: 0.7 by Environmental Tests, 0.7 by Functional Tests, and 0.9 by Component Test/Characterize. These are all detection-style PACTs, meaning that their effect is to reduce Failure Modes' likelihoods.

For a Failure Mode F, its "PACTed" likelihood, i.e., taking into account effects of PACTs, is computed as:

F.AprioriLikelihood*(Π(P∈PACTs when P.Status=On): (1−Effect(P, F) )

(If there are PACTs that induce Failure Modes, then a slightly more complicated formula must be used.)

For the highlighted Failure Mode, and the five checked PACTs in the Tests folder, the calculation is:

1*((1–0.7)*(1–0.7)*(1–0.9)*(1–0)*(1–0))=1*(0.3*0.3*0.1*1*1)=1*0.009=0.009

This was the Failure Mode whose (unreduced) total contribution to loss of Requirements attainment we calculated earlier to be 15.9. The corresponding "PACTed" calculation that takes into account the beneficial effects of the selected PACTs, substituting the Failure Mode's "PACTed" likelihood for its a priori likelihood, thus:

F. PACTedLikelihood*Σ (R∈AllRequirements):R.Weight*Impact(F, R)

For the highlighted Failure Mode this is 0.009*15.9=0.1431, i.e., considerably reduced.

Because these PACTs were all detections, the reduction in likelihood of this Failure Mode is accomplished by repairing (prior to launch of course!) the problems those PACTs detect. DDP takes this into account in computing the sum total costs. Repair costs for a Failure Mode F detected and repaired in phase PH are calculated as: (F.PACTedLikelihood prior to PH−F.PACTedLikelihood after PH)*F.RepairCost (PH).

In the example data, Tolerance Issues' RepairCost in the test phase is 250, so DDP computes its cost of repair due to these PACTs as: (1−0.009)*250=247.75. PACTs from the Preventative Measures folder are prevention PACTs, so would avoid incurring repair costs such as these, and leave far fewer such problems for detection and repair.

These fragmentary examples illustrate DDP's core calculations. Their results are displayed to users via bar charts (e.g., Fig. 9) and the risk region plot (Fig. 10) shown earlier, the overall aim being to aid users in their decision making.