Examining random and designed tests to detect code mistakes in scientific software

Abstract

Successfully testing computational software to detect code mistakes is impacted by multiple factors. One factor is the tolerance accepted in test output. Other factors are the nature of the code mistake, the characteristics of the code structure, and the choice of test input. We have found that randomly generated test input is a viable approach to testing for code mistakes and that simple structural metrics have little predictive power in the type of testing required. We provide further evidence that reduction of tolerance in expected test output has a much larger impact than running many more tests to discover code mistakes.

Introduction

In scientific software, any number of sources [10] can contribute to error in output. Typical contributors are round-off error from finite representations in the hardware, truncation error from computational techniques, measurement error and approximations in empirically based models and input data, and simplifications to realise the computational solution. All these contributors can readily mask an unintended source of error: a mistake in the source code. We define a mistake in the source code (code mistake) as any divergence from what was intended.

The most common software assessment activity is testing. Scientists routinely test their software to validate their scientific models (e.g. [3], [23]). The goal is to ensure the science is correct. As well, there is considerable work on code verification, where the goal is to ensure the solution technique is correct (e.g. [15], [22]). However, as far as we are aware, there is no work on identifying techniques for testing scientific software specifically for code mistakes. This is despite such examples as [8], where nine different companies wrote software based on the same algorithm and produced nine significantly different answers. Most of the discrepancies came from code mistakes, particularly what are called “one-off” mistakes. In another example, the discovery of a code mistake in analysis software for molecular structures caused the retraction of five journal papers [18]. In our own research into industrial software, we have come across code mistakes that have reduced the ability of analysis software to reliably identify artifacts in images [12]. In other examples, researchers incorrectly altered their scientific models to compensate for an undetected code mistake [5], [13].

In the work described in this paper, we confine ourselves to code mistakes and finding them by testing.

We suggest that searching for code mistakes should precede both validation testing and verification testing as commonly carried out by scientists. Preferably, the search should be carried out at the module level, where the length of executions and the extent of code being exercised are shorter. This also removes some of the masking of error from other sources other than the code mistake. However, even at this smaller scope of testing, decisions about what is exactly the “right” answer may be difficult to make, given the nature of most examples of computational software.

Successfully testing computational application software is confounded by what we call the “oracle assumption”. Weyuker [24] describes an oracle as the mechanism used to determine the correctness of a program for given test data. Weyuker further explains that the “.. belief that the tester is routinely able to determine whether or not the test output is correct is the oracle assumption.” Determining whether or not the results of a test are correct for computational software is hampered by unexpected interactions between underlying hardware, software code, input data, scientific model, and solution technique [3]. Weyuker [24] called such software “untestable” and suggested special considerations were needed.

Many testing techniques described in software engineering literature assume accurate oracles for tests are always available, and available for possibly copious numbers of tests.

Related to the oracle assumption, Hook [9] describes the tolerance problem. This deals with the fact that scientists often have an incomplete oracle (i.e., the mechanism to check test results only exists for a limited number of possible cases) as well as an approximate oracle. An approximate oracle is a mechanism that provides only approximate checks for tests, or in other words, checks with large tolerances. Work by Hook and Kelly [11] showed that reducing the tolerances in oracles could significantly improve the ability of the scientist to notice increased and unexpected error in test output values.

The work described in this paper extends work started by Hook (e.g. [9]). Our ultimate goal is to provide scientists with advice on testing for code mistakes in computational software. This would help fill the gap in testing literature where there is currently no research in testing computational codes for code mistakes. Our research towards this goal includes developing a research technique to evaluate different testing techniques. This paper largely describes this research technique and its application to a series of small MATLAB functions in order to evaluate the effectiveness of randomly generated tests versus designed tests. We use Hook's tool, MATMute [16] to generate a test-bed of code mistakes as part of our research technique.

Our works confirms Hook's results on the impact of tolerance on testing, finds no useful correlation between our testing experience and typical code structure metrics, and provides evidence that random testing is a viable approach for finding code mistakes in our generated test-bed.

In the next section we discuss the tools we used for our research. Section 3 contains our observations from using these tools. Section 4 discusses the results and concludes with a recommended approach to testing that could be the basis for further research.