Safer Than Perception: Assuring Confidence in Safety-Critical Decisions of Automated Vehicles

Abstract

We address one of the key challenges in assuring safety of autonomous cyber-physical systems that rely on learning-enabled classification within their environmental perception: How can we achieve confidence in the perception chain, especially when dealing with percepts safe-guarding critical manoeuvres? We present a methodology which allows to mathematically prove that the risk of misevaluating a safety-critical guard conditions referring to environmental artefacts can be bounded to a considerably lower frequency than the risk of individual misclassifications, and can thereby be adjusted to a value less than a given level of societally accepted risk.

M. Fränzle—Supported by the State of Lower Saxony within the Zukunftslabor Mobilität as well as by Deutsche Forschungsgemeinschaft under grant no. DFG FR 2715/5-1.

M. Swaminathan—Contribution while employed at the University of Oldenburg.

A Proofs

Proof of

Lemma 1. We use induction over n to show that the inequality

$$\begin{aligned}\begin{gathered} \min _{i=1,\dots ,n}\left\{ \frac{a_i}{b_i}\right\} \le \frac{\sum _{i=1}^n a_i}{\sum _{i=1}^n b_i} \le \max _{i=1,\dots ,n}\left\{ \frac{a_i}{b_i}\right\} \end{gathered}\end{aligned}$$

holds for any positive integer n and fractions $\frac{a_1}{b_1}, \dots , \frac{a_n}{b_n}$ with real nominators $a_1,\dots ,a_n$ and positive real denominators $b_1,\dots ,b_n$. The base case $n=1$ is trivial and the case $n=2$ follows immediately from the mediant inequality $\frac{a}{b} \le \frac{c}{d} \implies \frac{a}{b} \le \frac{a + c}{b + d} \le \frac{c}{d}$ for real numbers a, b and positive real numbers b, d. Assume that the induction hypothesis holds for $n-1$. For any fraction $\frac{a_n}{b_n}$ with real $a_n,b_n>0$ at least one of the inequalities (i) $\frac{\sum _{i=1}^{n-1} a_i}{\sum _{i=1}^{n-1} b_i} \le \frac{a_n}{b_n}$ or (ii) $\frac{a_n}{b_n} \le \frac{\sum _{i=1}^{n-1} a_i}{\sum _{i=1}^{n-1} b_i}$ holds. From case (i) it follows

$$\begin{aligned} \min _{i=1,\dots ,n}\left\{ \frac{a_i}{b_i}\right\} = \min _{i=1,\dots ,n-1}\left\{ \frac{a_i}{b_i}\right\} \le \frac{\sum _{i=1}^{n-1}a_i}{\sum _{i=1}^{n-1}b_i} \overset{(*)}{\le }\frac{\sum _{i=1}^{n}a_i}{\sum _{i=1}^{n}b_i} \overset{(*)}{\le }\frac{a_n}{b_n} = \max _{i=1,\dots ,n}\left\{ \frac{a_i}{b_i}\right\} \end{aligned}$$

and from case (ii) it follows

$$\begin{aligned} \min _{i=1,\dots ,n}\left\{ \frac{a_i}{b_i}\right\} = \frac{a_n}{b_n} \overset{(*)}{\le }\frac{\sum _{i=1}^{n}a_i}{\sum _{i=1}^{n-1}b_i} \overset{(*)}{\le }\frac{\sum _{i=1}^{n-1}a_i}{\sum _{i=1}^{n-1}b_i} \le \max _{i=1,\dots ,n-1}\left\{ \frac{a_i}{b_i}\right\} = \max _{i=1,\dots ,n}\left\{ \frac{a_i}{b_i}\right\} \end{aligned}$$

where $(*)$ denotes the application of the mediant inequality.    $\square$

Proof of

Lemma 2. We have to show that the inequalities

hold for all disjoint events $B_1,\dots ,B_n$. Using the identity , an application of Lemma 1 yields

$$\begin{aligned}\begin{gathered} \min _{i=1,\dots ,n}\left\{ \frac{P(A,B_i)}{P(B_i)} \right\} \le \frac{ \sum _{i=1}^n P(A,B_i)}{ \sum _{i=1}^n P(B_i)} \le \max _{i=1,\dots ,n}\left\{ \frac{P(A,B_i)}{P(B_i)} \right\} , \end{gathered}\end{aligned}$$

which finally rewrites to the asserted inequalities.    $\square$

Proof of

Lemma 3. We have to show that the identity

$$\begin{aligned} P( A_i^{\lambda _i} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n}, A_j^{\lambda _j}) = P( A_i^{\lambda _i} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n}) \end{aligned}$$

(3)

is equivalent to the identity

$$\begin{aligned} P( A_i^{\lambda _i}, A_j^{\lambda _i} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n}) = P( A_i^{\lambda _i} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n}) P( A_j^{\lambda _j} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n}) \end{aligned}$$

(4)

for all positive integers n, atoms $A_1,\dots A_n$, and $i\ne j$, $i\le n$, $j\le n$. Note that we implicitly assume the well-definedness of Eq. (3) and Eq. (4). I.e., both identities stipulate $P(A_1^{\pi _1},\dots ,A_n^{\pi _n})>0$ and Eq. (3) additionally stipulates $P(A_j^{\lambda _j} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n}) > 0$. The equivalent transformation from Eq. (3) to Eq. (4) is obtained by multiplying both sides of Eq. (3) with $P(A_j^{\lambda _j} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n})$. The equivalent transformation from Eq. (4) to Eq. (3) by division is valid as long as the stronger stipulation $P(A_j^{\lambda _j} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n}) > 0$ imposed by Eq. (3) holds. Finally, note that for $P(A_j^{\lambda _j} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n}) = 0$ the identity Eq. (4) does not contain any deeper findings, as it degenerates to the trivial identity $0=0$ in this case.    $\square$

Proof of

Lemma 4. We have to show that the identities

$$\begin{aligned}\begin{gathered} \underline{P}( A_i^{\lambda _i}, A_j^{\lambda _j} \mid A_i^{\pi _i}, A_j^{\pi _j} )= \underline{P}( A_i^{\lambda _i}\mid A_i^{\pi _i} )\underline{P}( A_j^{\lambda _j} \mid A_j^{\pi _j} ),\\ \overline{P}( A_i^{\lambda _i}, A_j^{\lambda _j} \mid A_i^{\pi _i}, A_j^{\pi _j} )= \overline{P}( A_i^{\lambda _i}\mid A_i^{\pi _i} )\overline{P}( A_j^{\lambda _j} \mid A_j^{\pi _j} ) \end{gathered}\end{aligned}$$

hold for the limit probabilities $\underline{P}$ and $\overline{P}$ for all $i\ne j$. To see this consider the following chain of rewritings.

$$\begin{aligned} \underline{P}( A_i^{\lambda _i} \mid A_i^{\pi _i})\underline{P}(A_j^{\lambda _j} \mid A_j^{\pi _j})&= \underline{P}( A_i^{\lambda _i} \mid A_i^{\pi _i}, A_j^{\pi _j} ) \underline{P}( A_j^{\lambda _j} \mid A_i^{\pi _i}, A_j^{\pi _j} )\\&= \underline{P}( A_i^{\lambda _i},A_j^{\lambda _j} \mid A_i^{\pi _i}, A_j^{\pi _j} ). \end{aligned}$$

The corresponding identity for $\overline{P}$ follows analogously.    $\square$

Proof of

Thm. 1. We have to show that

$$\begin{aligned}\begin{gathered} \underline{P}( \phi ^{\lambda _\phi } \mid \phi ^{\pi _\phi } ) \le P( \phi ^{\lambda _\phi } \mid \phi ^{\pi _\phi } ) \le \overline{P}( \phi ^{\lambda _\phi } \mid \phi ^{\pi _\phi } ) \end{gathered}\end{aligned}$$

holds for any formula $\phi$ with label $\phi ^{\lambda _\phi }$ and truth value $\phi ^{\pi _\phi }$. In Eq. (1) we already showed

$$\begin{aligned}\begin{gathered} \min _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}P(l \mid m) \le P(\phi ^{\lambda _\phi } \mid \phi ^{\pi _\phi }) \le \max _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}P(l \mid m). \end{gathered}\end{aligned}$$

As all involved probabilities are nonnegative, monotonicity of $\le$ yields

$$\begin{aligned} \min _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}\underline{P}(l \mid m) \le \min _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}P(l \mid m)\\ \max _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}P(l \mid m) \le \max _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}\overline{P}(l \mid m) \end{aligned}$$

Combining the inequalities yields the asserted estimation.    $\square$

Proof of

Thm. 2. Let $\phi$ be an arbitrary formula with given label $\phi ^\lambda _\phi$ and truth value $\phi ^{\pi _\phi }$. Further let $m=(A_1^{\pi _1},\dots ,A_n^{\pi _n})$ be a truth assignment and $l=(A_1^{\lambda _1},\dots ,A_n^{\lambda _n})$ a label assignment for all atoms $A_i$. In order to establish the theorem we show

$$\begin{aligned} \min _{m \models \phi ^{\pi _\phi }}\!\sum _{l \models \phi ^{\lambda _\phi }}\prod _{i=1}^n \underline{P}({A_i}^{\lambda _i} \mid {A_i}^{\pi _i})&\le P({\phi }^{\lambda _\phi } \mid {\phi }^{\pi _\phi }) \le \max _{m \models \phi ^{\pi _\phi }}\!\sum _{l \models \phi ^{\lambda _\phi }}\prod _{i=1}^n \overline{P}({A_i}^{\lambda _i} \mid {A_i}^{\pi _i}). \end{aligned}$$

The bounds

$$\begin{aligned}\begin{gathered} \min _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}P(l \mid m) \le P({\phi }^{\lambda _\phi } \mid {\phi }^{\pi _\phi }) \le \max _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}P(l \mid m) \end{gathered}\end{aligned}$$

can be obtained without any further assumption and have already been established in Eq. (1). In Eq. (2) we already argued that the bounds

$$\begin{aligned}\begin{gathered} \min _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}\prod _{i=1}^n P(A_i^{\lambda _i} \mid m) \le P({\phi }^{\lambda _\phi } \mid {\phi }^{\pi _\phi }) \le \max _{m \models \phi ^{\pi _\phi }}\sum _{l \models \phi ^{\lambda _\phi }}\prod _{i=1}^n P(A_i^{\lambda _i} \mid m) \end{gathered}\end{aligned}$$

can be derived under the Independent Labelling Assumption 1. Finally, Assumption 2 allows us to bound each term of the form $P(A_i^{\lambda _i} \mid m) = P(A_i^{\lambda _i} \mid A_1^{\pi _1},\dots ,A_n^{\pi _n})$ by its respective lower and upper bound $\underline{P}({A_i}^{\lambda _i} \mid {A_i}^{\pi _i})$ and $\overline{P}({A_i}^{\lambda _i} \mid {A_i}^{\pi _i})$. As all involved terms are nonnegative, we obtain

$$\begin{aligned} \min _{m \models \phi ^{\pi _\phi }}\!\sum _{l \models \phi ^{\lambda _\phi }}\prod _{i=1}^n \underline{P}({A_i}^{\lambda _i} \mid {A_i}^{\pi _i})&\le P({\phi }^{\lambda _\phi } \mid {\phi }^{\pi _\phi }) \le \max _{m \models \phi ^{\pi _\phi }}\!\sum _{l \models \phi ^{\lambda _\phi }}\prod _{i=1}^n \overline{P}({A_i}^{\lambda _i} \mid {A_i}^{\pi _i}). \end{aligned}$$

   $\square$

Proof of

Thm. 4. We show the identities for the lower limiting rates $\underline{\textrm{TPR}}_{\phi \wedge \psi }=\underline{\textrm{TPR}}_\phi \underline{\textrm{TPR}}_\psi$ and $\underline{\textrm{FNR}}_{\phi \wedge \psi }=\underline{\textrm{FNR}}_\phi + \underline{\textrm{FNR}}_\psi - \underline{\textrm{FNR}}_\phi \underline{\textrm{FNR}}_\psi$. The identities for the upper limiting rates follow analogously. The remaining estimations for $\underline{\textrm{TNR}}_{\phi \vee \psi }$, $\underline{\textrm{FPR}}_{\phi \vee \psi }$, $\overline{\textrm{TNR}}_{\phi \vee \psi }$, and $\overline{\textrm{FPR}}_{\phi \vee \psi }$ can be obtained from the identity $\phi \vee \psi \equiv \lnot (\lnot \phi \wedge \lnot \psi )$ and Thm. 3.

We decompose the conditional probabilities into proper assignments using Lemma 3 and Thm. 1.

$$\begin{aligned} \underline{\textrm{TPR}}_{\phi \wedge \psi }&= \underline{P}((\phi \wedge \psi )^{+} \mid (\phi \wedge \psi )^\top ) = \underline{P}(\phi ^+, \psi ^+\mid \phi ^\top , \psi ^\top )\\&= \underline{P}(\phi ^+\mid \phi ^\top )\underline{P}(\psi ^+\mid \psi ^\top ) = \underline{\textrm{TPR}}_{\phi }\underline{\textrm{TPR}}_{\psi },\\ \underline{\textrm{FNR}}_{\phi \wedge \psi }&= \underline{P}((\phi \wedge \psi )^-\mid (\phi \wedge \psi )^\top )\\&= \underline{P}(\phi ^-\mid \phi ^\top , \psi ^\top ) + \underline{P}(\psi ^-\mid \phi ^\top , \psi ^\top ) - \underline{P}(\phi ^-\mid \phi ^\top , \psi ^\top ) \underline{P}(\psi ^-\mid \phi ^\top , \psi ^\top )\\&= \underline{P}(\phi ^-\mid \phi ^\top ) + \underline{P}(\psi ^-\mid \psi ^\top ) - \underline{P}(\phi ^-\mid \phi ^\top )\underline{P}(\psi ^-\mid \psi ^\top )\\&= \underline{\textrm{FNR}}_\phi + \underline{\textrm{FNR}}_\psi - \underline{\textrm{FNR}}_\phi \underline{\textrm{FNR}}_\psi . \end{aligned}$$

   $\square$

Proof of

Theorem 5. We show the inequalities for the lower limiting rates only. The corresponding inequalities for the upper limiting rates follow analogously. Note that for the conditional probabilities $\underline{\textrm{TPR}}_{\phi \vee \psi }=\underline{P}((\phi \vee \psi )^+\mid (\phi \vee \psi )^\top )$ and $\underline{\textrm{FNR}}_{\phi \vee \psi }=\underline{P}((\phi \vee \psi )^-\mid (\phi \vee \psi )^\top ))$ the conditioning event $(\phi \vee \psi )^\top$ can be decomposed into disjoint models yielding

Lemma 2 allows us to infer a lower estimate of the rates:

$$\begin{aligned} \min \left\{ \begin{array}{l} \underline{P}((\phi \vee \psi )^+\mid \phi ^\top ,\psi ^\top ),\\ \underline{P}((\phi \vee \psi )^+\mid \phi ^\top ,\psi ^\bot ),\\ \underline{P}((\phi \vee \psi )^+\mid \phi ^\bot ,\psi ^\top ) \end{array} \right\}&\le \underline{\textrm{TPR}}_{\phi \vee \psi }, \\ \min \left\{ \begin{array}{l} \underline{P}((\phi \vee \psi )^-\mid \phi ^\top ,\psi ^\top ),\\ \underline{P}((\phi \vee \psi )^-\mid \phi ^\top ,\psi ^\bot ),\\ \underline{P}((\phi \vee \psi )^-\mid \phi ^\bot ,\psi ^\top ) \end{array} \right\}&\le \underline{\textrm{FNR}}_{\phi \vee \psi }. \end{aligned}$$

We decompose the labelling of $(\phi \vee \psi )^+$ and $(\phi \vee \psi )^-$ of each term in the minimum and maximum expression individually, where conjunctive events can further be decomposed into products using Lemma 3 E.g., the first term of the estimate for $\underline{\textrm{TPR}}_{\phi \vee \psi }$ is rewritten as follows:

$$\begin{aligned}&\ \ \ \underline{P}((\phi \vee \psi )^+\mid \phi ^\top ,\psi ^\top )\\&= \underline{P}(\phi ^+\mid \phi ^\top ,\psi ^\top ) + \underline{P}(\psi ^+\mid \phi ^\top ,\psi ^\top ) - \underline{P}(\phi ^+, \psi ^+\mid \phi ^\top ,\psi ^\top )\\&= \underline{P}(\phi ^+\mid \phi ^\top ) + \underline{P}(\psi ^+\mid \psi ^\top ) - \underline{P}(\phi ^+\mid \phi ^\top )P(\psi ^+\mid \psi ^\top )\\&= \underline{\textrm{TPR}}_\phi + \underline{\textrm{TPR}}_\psi - \underline{\textrm{TPR}}_\phi \underline{\textrm{TPR}}_\psi , \end{aligned}$$

and the second term as follows:

$$\begin{aligned}&\ \ \ \underline{P}((\phi \vee \psi )^+\mid \phi ^\top ,\psi ^\bot )\\&= \underline{P}(\phi ^+\mid \phi ^\top ,\psi ^\bot ) + \underline{P}(\psi ^+\mid \phi ^\top ,\psi ^\bot ) - \underline{P}(\phi ^+, \psi ^+\mid \phi ^\top ,\psi ^\bot )\\&= \underline{P}(\phi ^+\mid \phi ^\top ) + \underline{P}(\psi ^+\mid \psi ^\bot ) - \underline{P}(\phi ^+\mid \phi ^\top )P(\psi ^+\mid \psi ^\bot )\\&= \underline{\textrm{TPR}}_\phi + \underline{\textrm{FPR}}_\psi - \underline{\textrm{TPR}}_\phi \underline{\textrm{FPR}}_\psi . \end{aligned}$$

After rewriting all terms accordingly, the lower bounds for $\underline{\textrm{TPR}}_{\phi \vee \psi }$ and $\underline{\textrm{FNR}}_{\phi \vee \psi }$ are established. The upper bounds for $\overline{\textrm{TPR}}_{\phi \vee \psi }$ and $\overline{\textrm{FNR}}_{\phi \vee \psi }$ follow analogously, and the remaining bounds for for $\underline{\textrm{TNR}}_{\phi \wedge \psi }$, $\overline{\textrm{TNR}}_{\phi \wedge \psi }$, $\underline{\textrm{FPR}}_{\phi \wedge \psi }$, and $\underline{\textrm{FPR}}_{\phi \wedge \psi }$ are obtained using De Morgan’s law.    $\square$