Tests and Refutation

Abstract

The purpose of testing a system with respect to a requirement is to refute the hypothesis that the system satisfies the requirement. We build a theory of tests and refutation based on the elementary notions of satisfaction and refinement. We use this theory to characterize the requirements that can be refuted through black-box testing and, dually, verified through such tests. We consider refutation in finite time and obtain the well-known finite falsifiability of hyper-safety temporal requirements as a special case. We extend our theory with computational constraints and separate refutation from enforcement in the context of temporal hyper-properties. Overall, our theory provides a basis to analyze the scope and reach of black-box tests and to bridge results from areas including testing, verification, and enforcement.

A Proofs

We first present the proofs of the lemmas and theorems that are given in the paper. Afterwards, we formally state and prove the claim that a requirement is semi-monotone iff it is the intersection of its upper set and its lower set, which is mentioned in Sect. 5.

Proof

(Lemma 3’s Proof). If no system satisfies R, then R is trivial. Suppose that some system S satisfies R. Then, every system in $\lfloor {\lceil {S}\rceil }\rfloor$ satisfies R, because R is an obligation and a prohibition. As $\lfloor {\lceil {S}\rceil }\rfloor =\mathcal {S}$, for any $S\in \mathcal {S}$, we conclude that every system satisfies R. That is, R is trivial.    $\square$

Proof

(Theorem  7’s Proof). Suppose R is $\mathbf {T}$-refutable, with $\mathbf {T}=(T,\alpha )$. We prove that R is a prohibition. If R is empty, then R is a trivial prohibition. Assume that R is nonempty and let $S\in R$. Now, suppose $S'\preceq S$. All we need to prove is that $S'\in R$. We present a proof by contradiction.

Assume that $S'\not \in R$. Then $\exists t\in T.\ \hat{\alpha }(t)\cap R=\emptyset$ simply because R is $\mathbf {T}$-refutable. Since $\alpha$ is order-preserving and $S'\preceq S$, we have $t\in \alpha (S)$. Therefore, $S\in \hat{\alpha }(t)$. This entails $S\not \in R$, which contradicts the assumption $S\in R$. We conclude that $S'\in R$. Therefore, R is a prohibition.

Proof

(Lemma  9’s Proof). Fix a system model $\mathsf {M}=(\mathcal {S},\preceq ,\bot ,\top )$, and assume that R is prohibition. We show that R is $\mathbf {T}_r^\mathsf {M}$-refutable, where $\mathbf {T}_r^\mathsf {M}=(\mathcal {S},\lfloor {\cdot }\rfloor )$. Assume that some system S violates R. Since R is a prohibition, any system that abstracts S violates R. Moreover, $S\in \lfloor {S}\rfloor$. We conclude that $\exists S_w\in \lfloor {S}\rfloor .\ \lceil {S_w}\rceil \cap R=\emptyset$, namely $S_w=S$. Therefore, R is $\mathbf {T}_r^\mathsf {M}$-refutable.

Proof

(Lemma  10’s Proof). Suppose R is a nontrivial obligation. We prove by contradiction that R is not refutable in any test setup.

Assume that R is $\mathbf {T}$-refutable in some test setup $\mathbf {T}$. By Theorem 7, R is a prohibition. Then, R must be trivial by Lemma 3, because R is both a prohibition and an obligation. That R is a trivial contradicts the assumption that R is a nontrivial obligation. We conclude that R is not refutable in any test setup.

Proof

(Theorem  14’s Proof). Suppose R is $\mathbf {T}$-verifiable, with $\mathbf {T}=(T,\alpha )$. We prove that R is an obligation. If R is empty, then R is a trivial obligation. Assume that R is nonempty and let $S\in R$. Now, suppose $S\preceq S'$. All we need to prove is that $S'\in R$. Since R is $\mathbf {T}$-verifiable, from $S\in R$ we conclude $\exists t\in \alpha (S).\ \hat{\alpha }(S)\subseteq R$. As $\alpha$ is order-preserving and $S\preceq S'$, we have $t\in \alpha (S')$. That is, $S'\in \hat{\alpha }(S)$. We conclude that $S'\in R$. Therefore, R is an obligation.

Proof

(Lemma  15’s Proof). Suppose R is a nontrivial prohibition. We prove by contradiction that R is not verifiable in any test setup.

Assume that R is $\mathbf {T}$-verifiable in some test setup $\mathbf {T}$. By Theorem 14, R is an obligation. Then, R must be trivial by Lemma 3, because R is both a prohibition and an obligation. That R is a trivial contradicts the assumption that R is a nontrivial prohibition. We conclude that R is not verifiable in any test setup.

Proof

(Theorem  20’s Proof). We split the proof into several parts.

• (1) Suppose that a property $\phi$ is $\mathbf {T}_*$-refutable. We show that $\phi$ is safety. Assume that $\pi \not \in \phi$, for some $\pi \in \varSigma ^\omega$. Then, the system $S_\pi =\{\pi \}$ violates $\phi$. Now, by $\phi$’s $\mathbf {T}_*$-refutability, there exists a finite set t of $\phi$’s finite prefixes that demonstrates $S_\pi \not \in R_\phi$, where $R_\phi =\lfloor {\phi }\rfloor$. Let $\sigma$ be the longest element in t; note that since $\{\pi \}$ is a singleton, there always exists a single longest element in t. Then, for any $\pi '\in \varSigma ^\omega$, the system $S_{\pi '}=\{\sigma \pi '\}$ violates $\phi$, simply because t belongs to $\alpha (S_{\pi '})$. We conclude that $\sigma \pi '\not \in \phi$, for all $\pi '\in \varSigma ^\omega$. That is, $\phi$ is a safety temporal property.

• (2) Suppose that $\phi$ is safety. We show that $\phi$ is $\mathbf {T}_*$-refutable. Assume that a system S violates $\phi$. That is, $\exists \pi \in S.\ \pi \not \in \phi$. Since $\phi$ is safety, a finite prefix of $\pi$, say $\sigma$, satisfies the following condition: $\forall \pi '\in \varSigma ^\omega .\ \sigma \pi '\not \in \phi$. Now, define the observation $t\in T_*$ as $\{\sigma \}$. Note that $t\in \alpha (S)$, and moreover $\hat{\alpha }(t)\cap R_\phi =\emptyset$ due to the above condition. This shows that $\phi$ is $\mathbf {T}_*$-refutable.

• (3) Any temporal property $\phi$ is $\mathbf {T}_r$-refutable because $R_\phi$’s satisfaction is abstraction-closed, for any $\phi$. Then, by Lemmas 3 and 15, any $\mathbf {T}_r$-verifiable or $\mathbf {T}_*$-verifiable property must be trivial.

Theorem 26

A requirement R is semi-monotone iff $R=\lfloor {R}\rfloor \wedge \lceil {R}\rceil$.

Proof

We split the proof into two parts, reflecting the theorem’s two statements.

• (1) Assume that R is semi-monotone. We show that $R=\lfloor {R}\rfloor \wedge \lceil {R}\rceil$. Clearly $R\subseteq \lceil {R}\rceil \wedge \lfloor {R}\rfloor$, for any requirement R. All we need to prove is that $\lceil {R}\rceil \wedge \lfloor {R}\rfloor \subseteq R$. If $\lceil {R}\rceil \wedge \lfloor {R}\rfloor =\emptyset$, then the claim trivially holds. Suppose $S\in \lceil {R}\rceil \wedge \lfloor {R}\rfloor$ for some system S. From $S\in \lceil {R}\rceil$, we conclude $\exists S_-\in R.\ S_-\preceq S$. Similarly, from $S\in \lfloor {R}\rfloor$, we conclude $\exists S_+\in R.\ S\preceq S_+$. In short, we have $S_-\preceq S\preceq S_+$, $S_-\in R$, and $S_+\in R$. Then, Lemma 27 below implies that $S\in R$, simply because R is semi-monotone. Therefore, if R is semi-monotone, then $R=\lceil {R}\rceil \wedge \lfloor {R}\rfloor$.

• (2) Assume that $R=\lfloor {R}\rfloor \wedge \lceil {R}\rceil$. We prove that R is semi-monotone. Note that for any requirement Q, the requirement $\lfloor {Q}\rfloor$ is a prohibition, hence monotone. Moreover, $\lceil {Q}\rceil$ is an obligation, hence monotone. Therefore, $\lfloor {Q}\rfloor \wedge \lceil {Q}\rceil$ is semi-monotone, that is the intersection of two monotone requirements, for any requirement Q. In particular, R is semi-monotone because $R=\lfloor {R}\rfloor \wedge \lceil {R}\rceil$.

Lemma 27

If R is a semi-monotone requirement, then for any three systems $S_-$, S, and $S_+$ the following condition holds.

$$\begin{aligned} S_-\preceq S\preceq S_+\ \wedge \ S_-\in R\ \wedge \ S_+\in R\ \rightarrow \ S\in R \end{aligned}$$

Proof

Either (1) R is monotone, that is R is the conjunction of two prohibitions or the conjunction of two obligations, or (2) R is the conjunction of a prohibition P and an obligation O. The lemma’s claim is immediate for case (1). Let us consider case (2). Suppose $S_-\preceq S\preceq S_+\ \wedge \ S_-\in R\ \wedge \ S_+\in R$. Note that $S_-\in R$ implies that $S_-\in O$. Then, $S_-\preceq S$ implies that $S\in O$. Similarly, $S_+\in R$ implies that $S_+\in P$. Then, $S\preceq S_+$ implies that $S\in P$. These two statements show that $S\in P\wedge O$. That is, $S\in R$.