On the relationship between two control-flow coverage criteria: all JJ-paths and MCDC

Abstract

Coverage criteria may be used to assess the adequacy of software test data. Improved test data, that takes account of any inadequacies identified by lack of coverage, may then be developed. It is natural to seek ways of comparing different criteria and the ‘subsumes’ relationship is one such way: one criterion subsumes another, if satisfying the first always implies satisfaction of the second. This paper considers two criteria: ‘all jump-to-jump paths’ (all JJ-paths) and ‘modified condition/decision coverage’ (MCDC). It might be anticipated that there would be a relationship between these criteria since both are based on advanced control-flow concepts. MCDC has particular importance since it is involved in the DO-178B standard for avionics software. However, it is shown that ‘all JJ-paths’ and MCDC are, in general, incomparable, but for programs written under certain specific constraints ‘all JJ-paths’ subsumes MCDC.

Introduction

Measurement of coverage in software testing refers to the practice of determining the extent to which all occurrences of some feature of interest have been exercised by a given test data set. Different features lead to different coverage criteria. Henceforth in this document, coverage criteria will be regarded in the traditional sense of being program-based [8], although coverage notions can be applied to other software artefacts, such as state-based specifications [4], [9]. Despite the fact that complete coverage provides no guarantees about correctness, test experts continue to recommend coverage measurement in order: (1) to identify test data sets that are inadequate, possibly as a result of testers having ‘blind spots’, and (2) to discover ‘surprises’, i.e. aspects of the implementation that do not form part of the requirements [1]. Thus, ascertaining coverage can be seen as a way of ensuring basic adequacy of test data by indicating deficiencies in testing where further investigation is needed.

Given the existence of a variety of coverage criteria, the ‘subsumes’ relationship provides one mechanism for comparing them. A criterion C1 is said to subsume criterion C2 if, whenever C1 is satisfied, C2 is also satisfied. Such subsumption is often denoted using the logical implication symbol, i.e. C1⇒C2. If neither C1 nor C2 subsumes the other, they are said to be incomparable.

The focus in this short note is on two particular program coverage criteria whose relationship was hitherto unknown, but which might have been suspected of being related since they are both defined in terms of control flow. One is the ‘all jump-to-jump paths’ (all JJ-paths) criterion, originally introduced as coverage of ‘linear code sequence and jumps’ (LCSAJs) [15]; the other is ‘modified condition/decision coverage’ (MCDC) [2], which forms part of the DO-178B standard for avionics software [12]. It is shown that, in general, ‘all JJ-paths’ and MCDC are incomparable but that, for programs written under certain specific constraints, ‘all JJ-paths’ subsumes MCDC. The next section provides the necessary background by describing the terminology, the two criteria, and related work. Section 3 provides the main results concerning the comparison of the two criteria and the final section provides a brief discussion and some conclusions.