Efficient Uniform Sampling of Traces in Presence of Infeasibilities

Abstract

Coverage biased random testing of software consists of drawing uniformly at random a large sample of paths from the control graph of a program, or from traces of its specification or model. In order to obtain input values that exercise them, the conditions for following each path/trace are incrementally collected into a formula, which characterises the values that lead to traverse it at run time. A solver is then used for obtaining a test set that ensures the coverage of the sample.

A well-known problem is that not all paths/traces correspond to feasible runs. Such infeasible paths/traces must be rejected and others must be drawn to ensure a sufficiently large number of test inputs. This severely limits the interest of the method when there is a high ratio of infeasible paths or traces, as it is often the case.

The new technique we propose uses knowledge about the shortest infeasible prefixes of infeasible paths that have been drawn to prevent future drawing of extensions of such prefixes. Starting with uniform drawing from all the paths/traces, the new algorithm incrementally shrinks the drawing domain and remains uniform among the paths/traces that do not have a known infeasible prefix. As the number of infeasible paths/traces is often large, their elimination from the subsequent drawings is a drastic improvement with respect to the simple path rejection method above.

Appendix: Transposition of the method to Boltzmann samplers.

The method can be used to enrich other drawing methods than the classical recursive one considered in this paper, for instance Boltzmann samplers. This would make possible, instead of drawing among bounded paths, to have some probability distributions on the length of the paths. Such samplers exist in two varieties, the ordinary version and the exponential version. In this appendix we explain how our method is applicable to ordinary Boltzmann sampling [16]. The same approach is can be done for exponential Boltzmann sampling.

1.1 Ordinary Boltzmann Samplers in a Nutshell

Instead of using Eq. (1), which is based on the number of paths of a given length from a given vertex, Boltzmann samplers use recursive equations on generating functions of the sets of all paths from a given vertex. In contrast with the recursive method, which allows to uniformly draw objects of a fixed size (or in our context paths of bounded lengths), such samplers return objects of random sizes and ensure uniformity for the objects of the same size. It is possible to tune these samplers to favour objects of a size in the vicinity of a given value. When drawing paths, using such samplers would avoid introducing a bound on their lengths.

Consider the set of paths starting from a given vertex s as a language noted $\mathcal {L}_s$. The ordinary generating function of parameter z of this language is:

$$L(z)_s = \varSigma _{m \in \mathbb {N}}l_m z^m $$

where z is a complex variable and $l_m$ the number of words of length m. Such a function is defined for values $0 < z < \rho _L$ where $\rho _L$ is called the convergence radius of L.

Given a value of z such that $0 < z < \rho _L$, an ordinary Boltzmann sampler draws a path p with probability $z^{|p|}/L(z)$. The choice of the value of z determines the distribution on the lengths of the paths. For more details and complexity results, see [16] and [6].

1.2 Taking into Account Infeasibilities in a Control Flow Graph

The generating functions for the set of vertices of $\mathcal {G}$ satisfy the following equation:

$$\begin{aligned} L_s(z) = \big ( z \sum _{s \rightarrow t \in \mathcal {G} } L_t(z) \big ) + \textrm{1}_{s = s_f} \end{aligned}$$

(2)

Similarly to what was done on the base of Eq. (1) in Sect. 2.3, Eq. (2) can be used to recursively define a drawing algorithm: if $s = s_f$, since $s_f$ has no successors, the random generation stops, otherwise a successor t of s is drawn with probability $z L_t(z)/L_s(z)$, then the drawing algorithm is called recursively with t.

We observe that the enrichment of $\mathcal {G}$ with an edge from $s_f$ to itself, introduced in Subsect. 2.3 for drawing paths of lengths less or equal to bound n, is no longer necessary: as said above, Boltzmann samplers return paths of different lengths whose distribution, i.e. average value and variance, is adjustable.

One can note that when the aim is to draw only paths of an exact length, the sampler can be tuned via an adequate choice of z to a distribution of the lengths with small variance, and combined with a rejection method to filter unwanted paths, at the cost of a slightly increased complexity.

As noted in [16], a Boltzmann sampler requires as input the value of the parameter z, and the finite collection of the values at z of the generating functions used in the specification. These values need only to be computed once. In our case, it is the vector $\mathbb {L}(z) = (L_s(z))_{s\in \mathcal {G}}$. It means that the memory requirement is significantly lower than for the recursive method O(q) instead of $O(n \times q)$.

After detecting the infeasibility of some prefix $p.s.s'$, the probability of drawing $s'$ after p.s must become 0, and the probabilities of the vertices of p.s must be decreased, taking into account the size of $\mathcal {L}_{s'}$, i.e. $\mathbb {L}(z)_{s'} = L_{s'}(z)$. This can be achieved by complementing vector $\mathbb {L}(z)$ by the same trie data structure as in Sect. 3. The contents of the vector and of the trie are different since the lengths of the paths, and therefore of the suffixes of the infeasible prefixes, are no longer taken into account. But the principles of algorithms for drawing with the trie, updating it, and pruning it remain unchanged.

We cannot report experiments with such samplers due to the lack of easily available and well documented implementations for the choice of z and the computation of $\mathbb {L}(z)$.