Partition testing with usage models

https://doi.org/10.1016/S0950-5849(00)00100-2Get rights and content

Abstract

The fundamental statistical strategy of improving sampling efficiency through partitioning the population is applied to software testing. Usage models make it possible to apply this strategy to improve the efficiency of testing. The testing budget is allocated to the blocks of the partition, and the software is executed on the sample of uses drawn from each block or sub-population of potential uses. Usage models support many strategies for automated partitioning and generating test cases from the partitioned population. Two strategies are shown here with the efficiency gains demonstrated.

Section snippets

Statistical testing

Statistical testing is the comprehensive application of statistical science to the solution of software testing problems. The population of interest is all possible uses of the software under test, and this population must be represented in a form that facilitates posing and solving problems about the software and the testing process. The representation we use is the Markov chain usage model, which is described below. Further information about Markov chain usage models is presented in Ref. [1].

Partition testing

Sampling theory is a mature area of statistics. To take advantage of stratified sampling methods it is necessary to define the sample in terms of a mathematical partition of the population. Sample allocations are then calculated for the blocks of the partition.

Testing software by using random samples of test cases drawn from blocks of a partition defined on the population of all test cases is called “partition testing.” In this paper testing the use of a random sample drawn from the

Model-based partitioning strategies

A good partitioning strategy is one that results in blocks with different proportions of failures. The greatest gain in precision is realized when all blocks of the partition are homogeneous, i.e. all tests within a block either fail or all of the tests succeed [2], [3], [4], [5].

Given usage model M, let W be a matrix where Wi,j is a pre-test estimate of the probability of encountering a failure while executing the code to achieve transition from state-i to state-j, that is, while traversing

Conclusions

If tests are allocated to partition blocks in the optimal manner, partition random testing will always perform at least as well as simple random testing. Gains are realized from partition testing when the partitioning strategy creates near-homogeneous blocks, i.e. blocks in which all tests either fail or succeed. Therefore the choice of a partitioning scheme is very important.

The partitioning schemes presented in this paper use pre-test estimates of the failure rate of the software to create

References (9)

  • J.H Poore et al.

    Application of Statistical Science to Testing and Evaluating Software Intensive Systems

    (1998)
  • D Hamlet et al.

    Partition testing does not inspire confidence

    IEEE Trans. Software Engng

    (1990)
  • E.J Weyuker et al.

    Analyzing partition testing strategies

    IEEE Trans. Software Engng

    (1991)
  • W.G Cochran

    Sampling Techniques

    (1977)
There are more references available in the full text version of this article.

Cited by (7)

  • Impartial evaluation in software reliability practice

    2005, Journal of Systems and Software
  • Stopping criteria for statistical testing

    2000, Information and Software Technology
  • A mathematical modeling framework for software reliability testing

    2007, International Journal of General Systems
  • Software partition testing based on execution profiles filtering

    2006, Hunan Daxue Xuebao/Journal of Hunan University Natural Sciences
  • Using information about functions in selecting test cases

    2005, Proceedings of the 1st International Workshop on Advances in Model-Based Testing, A-MOST '05
View all citing articles on Scopus
View full text