Scalable offline monitoring of temporal specifications

Abstract

We propose an approach to monitoring IT systems offline where system actions are logged in a distributed file system and subsequently checked for compliance against policies formulated in an expressive temporal logic. The novelty of our approach is that monitoring is parallelized so that it scales to large logs. Our technical contributions comprise a formal framework for slicing logs, an algorithmic realization based on MapReduce, and a high-performance implementation. We evaluate our approach analytically and experimentally, proving the soundness and completeness of our slicing techniques and demonstrating its practical feasibility and efficiency on real-world logs with 400 GB of relevant data.

Appendices

Appendix 1: Additional details: slicing time

In the following, we define a slicer that splits a log in its temporal dimension. We also provide soundness and completeness guarantees for it.

To define the slicer, we first determine a time range for a given formula that suffices to evaluate the formula on a single time point of a temporal structure. The time range depends on the formula’s temporal operators and their intervals. To define this time range, we extend our notation for intervals over $\mathbb {N}$ to intervals over $\mathbb {Z}$. For example, for $b,b'\in \mathbb {Z}$, $[b,b']$ denotes the set $\{a\in \mathbb {Z}\mathbin {|}b\le a\le b'\}$. Moreover, for intervals I and J over $\mathbb {Z}$, let $I \oplus J := \{i+j\mathbin {|}i \in I \text { and } j \in J\}$, and let $I \Cup J$ be the smallest interval containing I and J.

Definition A1.1

The relative interval of the formula $\varphi$, ${\text {RI}}(\varphi )\subseteq \mathbb {Z}$, is defined recursively over the formula structure:

We give intuition for Definition A1.1. The relative interval of $\varphi$ specifies a time range, which contains relative timestamps. These relative timestamps describe time points that are sufficient to evaluate $\varphi$ on the current time point. Relative timestamps that refer to the future are positive integers and relative timestamps that refer to the past are negative integers. In the following, we give some intuition about the different cases of ${\text {RI}}$’s definition.

The evaluation of an atomic formula $\varphi$ only depends on the current time point. Therefore, it suffices to consider time points with equal timestamps, and hence ${\text {RI}}(\varphi )=\{0\}$. To evaluate a formula of the form $\lnot \psi$, $\exists x.\,\psi$, or $\psi \vee \,\chi$, it suffices to consider the time points needed to evaluate its subformulas. Hence, we choose the smallest interval subsuming the relative intervals of the subformulas.

The evaluation of depends only on the time points whose timestamps fall in the interval needed for $\psi$’s evaluation, shifted by the interval I. Moreover, the timestamp of the next time point must be the same in the time slice as in the original log. This is ensured by considering the interval from 0 to the furthest value from 0 in I. Considering only an interval I with $0 \not \in I$ would allow for additional time points to be inserted in the time slice between the current time point and the original next time point. The evaluation of $\psi \mathbin {\mathsf {U}}_{I} \chi$, with $I=[a,b)$, depends on having the same timestamps for the time points in the time slice as in the original log between the current time point and the one furthest away, but with its timestamp still falling within I. This is ensured by [0, b). The subformula $\psi$ is evaluated on time points between the current time point and the furthest time point with a timestamp that falls into I, so we must consider the relative interval of this subformula shifted by [0, b). The subformula $\chi$ is evaluated only on time points whose relative timestamps fall within I, so we must consider the relative interval of this subformula shifted by [a, b). Formulas of the form and $\psi \mathbin {\mathsf {S}}_{I} \chi$ are treated similarly to formulas with the corresponding future operators. However, their relative intervals are mirrored over 0, since these temporal operators refer to the past.

The next lemma establishes that 0 is included in the relative interval of every formula. Its proof, which we omit, is a straightforward induction over the formula structure.

Lemma A1.1

For every formula $\varphi$, $0 \in {\text {RI}}(\varphi )$.

We have now the definitions at hand to formalize slicing a log by time (Definition A1.2) and the time slicer (Definition A1.3).

Definition A1.2

Let $T\subseteq \mathbb {Z}$ be an interval and $(\bar{\mathscr {D}},\bar{\tau })\in \mathbf {T}$. The T -slice of $(\bar{\mathscr {D}},\bar{\tau })$ is the time slice $(\bar{\mathscr {D}}',\bar{\tau }')$ of $(\bar{\mathscr {D}},\bar{\tau })$, where $s:[0,\ell )\rightarrow \mathbb {N}$ is the function $s(i') = i'+ c$, $\ell = |\{i \in \mathbb {N}\mathbin {|}\tau _i \in T\}|$, and $c = \min \{i \in \mathbb {N}\mathbin {|}\tau _i \in T\}$. We also require that $\tau '_{\ell } \not \in T$ and $\mathscr {D}'_{i'} = \mathscr {D}_{s(i')}$, for all $i' \in [0,\ell )$.

Fig. 3

Illustration of a T-slice

Figure 3 illustrates Definition A1.2, where the original log refers to the temporal structure $(\bar{\mathscr {D}},\bar{\tau })$ and a T-slice of the original log to $(\bar{\mathscr {D}}',\bar{\tau }')$. Intuitively, the first time point in a T-slice is the first time point in $(\bar{\mathscr {D}}, \bar{\tau })$ with the timestamp in T. There are $\ell$ time points in $(\bar{\mathscr {D}}, \bar{\tau })$ whose timestamps fall into T. Those time points are identical in the T-slice. To ensure the soundness and completeness of time slices, the $\ell$th time point in the T-slice must have a timestamp that lies outside of T, just like the corresponding time point in $(\bar{\mathscr {D}},\bar{\tau })$.

Definition A1.3

The time slicer $\mathfrak {t}_{\varphi ,(I^k)_{k \in K}}$ for the formula $\varphi$ and the family of intervals $(I^k)_{k \in K}$ is the function mapping $(\bar{\mathscr {D}},\bar{\tau })\in \mathbf {T}$ and $\mathcal {R}\in \mathbf {R}$ to the family of temporal structures $(\bar{\mathscr {D}}^k,\bar{\tau }^k)_{k \in K}$ and the family of restrictions $(\mathcal {R}^k)_{k \in K}$, where $(\bar{\mathscr {D}}^k,\bar{\tau }^k)$ is the $T^k$-slice of $(\bar{\mathscr {D}},\bar{\tau })$, with $T^k$ the smallest interval containing $\big (I^k \cap \{t \in \mathbb {N}\mathbin {|}(v,t) \in \mathcal {R}, {\text { for some valuation }}$v$\}\big )\oplus {\text {RI}}(\varphi )$, and $\mathcal {R}^k= \{(v,t)\mathbin {|}(v,t)\in \mathcal {R}{\text { with }}t \in I^k\}$, for each $k \in K$.

The following theorem establishes that a time slicer is a slicer.

Theorem A1.1

The time slicer $\mathfrak {t}_{\varphi ,(I^k)_{k \in K}}$ is a slicer for the formula $\varphi$, if $\bigcup _{k \in K} I^k = \mathbb {N}$.

To prove Theorem A1.1, we first introduce additional machinery.

Definition A1.4

Let $I \subseteq \mathbb {Z}$ be an interval and $c, i \in \mathbb {N}$. The temporal structures $(\bar{\mathscr {D}},\bar{\tau })\in \mathbf {T}$ and $(\bar{\mathscr {D}}',\bar{\tau }')\in \mathbf {T}$ are (I, c, i)-overlapping if the following conditions hold.

• 1. $j \ge c$, $\mathscr {D}_j = \mathscr {D}'_{j-c}$, and $\tau _j = \tau '_{j-c}$, for all $j \in \mathbb {N}$ with $\tau _j - \tau _i \in I$.

• 2. $\mathscr {D}_{j'+c} = \mathscr {D}'_{j'}$ and $\tau _{j'+c} = \tau '_{j'}$, for all $j' \in \mathbb {N}$ with $\tau '_{j'} - \tau _i \in I$.

Intuitively, two temporal structures are (I, c, i)-overlapping if their time points (timestamps and structures) are “the same” on an interval of timestamps. This is the case for time slices. The value c here corresponds to the c in Definition A1.2. It specifies how many time points the two temporal structures are “shifted” relative to each other. The interval I specifies the timestamps for which time points must be “the same”, i.e. those timestamps whose difference to the timestamp $\tau _i$ are within I.

The next three lemmas establish that (1) time slices overlap, (2) if temporal structures overlap for an interval I, then they also overlap for other time points in I and for subintervals of I, and (3) a formula’s truth value match at the overlapping time points i and $i-c$.

Lemma A1.2

Let $T \subseteq \mathbb {N}$ and $I \subseteq \mathbb {Z}$ be intervals, $(\bar{\mathscr {D}},\bar{\tau })\in \mathbf {T}$, and $(\bar{\mathscr {D}}',\bar{\tau }')\in \mathbf {T}$ be a $(T \oplus I)$-slice of $(\bar{\mathscr {D}},\bar{\tau })$. The temporal structures $(\bar{\mathscr {D}}',\bar{\tau }')$ and $(\bar{\mathscr {D}},\bar{\tau })$ are (I, c, i)-overlapping, for all $i \in \mathbb {N}$ with $\tau _i \in T$, where $c \in \mathbb {N}$ is the value in Definition A1.2 used by the function s with respect to $(\bar{\mathscr {D}},\bar{\tau })$ and its time slice $(\bar{\mathscr {D}}',\bar{\tau }')$.

Proof

We first show that Condition 1 in Definition A1.4 is satisfied. For all $i \in \mathbb {N}$ with $\tau _i \in T$ and all $j \in \mathbb {N}$ with $\tau _j - \tau _i \in I$, it holds that $\tau _j \in T \oplus I$. From $c = \min \{k \in \mathbb {N}\mathbin {|}\tau _k \in T \oplus I\}$ in Definition A1.2, it follows that $j \ge c$. Let $j' := j - c$. It also follows from $\tau _j \in T \oplus I$ that $j' \in [0,\ell )$. Therefore, $\mathscr {D}_j = \mathscr {D}_{s(j')} = \mathscr {D}'_{j'} = \mathscr {D}'_{j-c}$ and $\tau _j = \tau _{s(j')} = \tau '_{j'} = \tau '_{j-c}$.

Next, we show that Condition 2 is satisfied. For all $i \in \mathbb {N}$ with $\tau _i \in T$ and all $j' \in \mathbb {N}$ with $\tau '_{j'} - \tau _i \in I$, it holds that $\tau '_{j'} \in T \oplus I$. Since $\tau '_\ell \not \in T \oplus I$, it follows that $j' \in [0,\ell )$. Therefore, $\mathscr {D}_{j'+c} = \mathscr {D}_{s(j')} = \mathscr {D}'_{j'}$ and $\tau _{j'+c} = \tau _{s(j')} = \tau '_{j'}$. $\square$

Lemma A1.3

Let $(\bar{\mathscr {D}},\bar{\tau })\in \mathbf {T}$ and $(\bar{\mathscr {D}}',\bar{\tau }')\in \mathbf {T}$ be temporal structures that are (I, c, i)-overlapping, for some $I \subseteq \mathbb {Z}$, $c \in \mathbb {N}$, and $i \in \mathbb {Z}$. Then $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are (K, c, k)-overlapping, for each $k \in \mathbb {N}$ with $\tau _k - \tau _i \in I$ and $K \subseteq \{\tau _i - \tau _k\} \oplus I$.

Proof

For all $j \in \mathbb {N}$ with $\tau _j - \tau _k \in K$, it follows from $\tau _j - \tau _k \in K$ that $\tau _j - \tau _k + \tau _k - \tau _i \in \{\tau _k - \tau _i\} \oplus K$ and hence $\tau _j - \tau _i \in \{\tau _k - \tau _i\} \oplus K$. From the assumption $K \subseteq \{\tau _i - \tau _k\} \oplus I$, it follows that $\{\tau _k - \tau _i\} \oplus K \subseteq \{\tau _k - \tau _i\} \oplus \{\tau _i - \tau _k\} \oplus I = I$ and hence $\tau _j - \tau _i \in I$. Since $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are (I, c, i)-overlapping, Condition 1 in Definition A1.4 holds for them to be (K, c, k)-overlapping. Similarly, for all $j' \in \mathbb {N}$ with $\tau '_{j'} - \tau _k \in K$, it follows that $\tau '_{j'} - \tau _i \in I$ and hence Condition 2 in Definition A1.4 holds. $\square$

Lemma A1.4

Let $\varphi$ be a formula and $(\bar{\mathscr {D}},\bar{\tau }), (\bar{\mathscr {D}}',\bar{\tau }')\in \mathbf {T}$. If $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(\varphi ),c,i)$-overlapping, for some c and i, then for all valuations v, it holds that $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \varphi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \varphi$.

Proof

Note that the lemma’s statement is well-defined. Namely, $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \varphi$ is defined. It follows from Lemma A1.1 that $0 \in {\text {RI}}(\varphi )$ and from Condition 1 in Definition A1.4 that $i \ge c$ and hence $i-c \in \mathbb {N}$.

We prove the lemma by structural induction on the formula $\varphi$. We have the following cases.

• $t \approx t'$, where $t,t'\in V\cup C$. Since the satisfaction of the formula $t \approx t'$ depends only on the valuation v, it follows that $(\bar{\mathscr {D}},\bar{\tau },v,i) \models t \approx t'$ iff $v(t) = v(t')$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models t \approx t'$, for all valuations v.

• $t \prec t'$, where $t,t'\in V\cup C$. This case is similar to the previous one.

• $r(\bar{t})$, where $t_1,\ldots ,t_{\iota (r)}\in V\cup C$. Since $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(r(\bar{t})),c,i)$-overlapping and $0 \in {\text {RI}}(r(\bar{t}))$, it also follows from Condition 1 in Definition A1.4 that $\mathscr {D}_{i} = \mathscr {D}'_{i-c}$ and hence $(\bar{\mathscr {D}},\bar{\tau },v,i) \models r(\bar{t})$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models r(\bar{t})$, for all valuations v.

• $\lnot \psi$. $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(\lnot \psi ),c,i)$-overlapping and ${\text {RI}}(\lnot \psi ) = {\text {RI}}(\psi )$. By the inductive hypothesis, $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \psi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \psi$, for all valuations v. Therefore $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \lnot \psi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \lnot \psi$.

• $\psi \vee \,\chi$. $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(\psi ) \Cup {\text {RI}}(\chi ),c,i)$-overlapping. From ${\text {RI}}(\psi ) \subseteq {\text {RI}}(\psi ) \Cup {\text {RI}}(\chi )$, ${\text {RI}}(\chi ) \subseteq {\text {RI}}(\psi ) \Cup {\text {RI}}(\chi )$, and Lemma A1.3 it follows that $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(\psi ),c,i)$-overlapping and $({\text {RI}}(\psi ),c,i)$-overlapping. By the inductive hypothesis, $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \psi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \psi$ and $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \chi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \chi$, for all valuations v. Hence $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \psi \vee \,\chi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \psi \vee \,\chi$.

• $\exists x.\,\psi$. From ${\text {RI}}(\exists x.\,\psi ) = {\text {RI}}(\psi )$ it follows that $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(\psi ),c,i)$-overlapping. By the inductive hypothesis, $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \psi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \psi$, for all valuations v. Hence, for all $d \in \mathbb {D}$ we have that $(\bar{\mathscr {D}},\bar{\tau },v[x \mapsto d],i) \models \psi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v[x \mapsto d],i-c) \models \psi$. It follows that $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \exists x.\,\psi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \exists x.\,\psi$, for all valuations v.

• . $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are -overlapping, where . From and Condition 1 in Definition A1.4, it follows that $\tau _{i} = \tau '_{i-c}$.

  We make a case split on the value of i. If $i = 0$, then , for all valuations v. From $c \in \mathbb {N}$, $i-c \in \mathbb {N}$, and $i=0$, it follows that $i-c = 0$. Trivially, , for all valuations v. Next, we consider the case that $i > 0$ and make a case split on whether $\tau _i - \tau _{i-1}$ is included in the interval [a, b).

  • If $\tau _i - \tau _{i-1} \in [a,b)$, then and from Condition 1 in Definition A1.4 it follows that $i-1 \ge c$, $\tau _{i-1} = \tau '_{i-c-1}$, and hence $\tau '_{i-c} - \tau '_{i-c-1} \in [a,b)$. From $\tau _i - \tau _{i-1} \in [a,b)$ it also follows that and hence by Lemma A1.3 $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(\psi ),c,i-1)$-overlapping. By the inductive hypothesis, $(\bar{\mathscr {D}},\bar{\tau },v,i-1) \models \psi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c-1) \models \psi$, for all valuations v. Because $\tau _{i} = \tau '_{i-c}$ and $\tau _{i-1} = \tau '_{i-c-1}$, it follows that iff , for all valuations v.

  • If $\tau _i - \tau _{i-1} \not \in [a,b)$ then , for all valuations v. Recall that, from Definition A1.4, $i \ge c$. We make a case split on whether $i = c$ or $i > c$. If $i = c$ then $i-c = 0$ and hence , for all valuations v. Consider the case $i > c$. To achieve a contradiction, suppose that $\tau '_{i-c} - \tau '_{i-c-1} \in [a,b)$. From Condition 2 in Definition A1.4 it follows that $\tau _{i-1} = \tau '_{i-c-1}$ and hence $\tau _{i} - \tau _{i-1} = \tau '_{i-c} - \tau '_{i-c-1} \in [a,b)$.

    This contradicts $\tau _{i} - \tau _{i-1} \not \in [a,b)$, so it must be the case that $\tau '_{i-c} - \tau '_{i-c-1} \not \in [a,b)$. It follows that , for all valuations v.

• . This case is similar to the previous one, but it is simpler because we need not consider $i=0$ and $i-c=0$ as a special case. We omit its details.

• $\psi \mathbin {\mathsf {S}}_{[a,b)} \chi$. $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(\psi \mathbin {\mathsf {S}}_{[a,b)} \chi ),c,i)$-overlapping, where ${\text {RI}}(\psi \mathbin {\mathsf {S}}_{[a,b)} \chi ) = (-b,0] \Cup \big ( (-b,0] \oplus {\text {RI}}(\psi ) \big ) \Cup \big ( (-b,-a] \oplus {\text {RI}}(\chi ) \big )$. From $0 \in {\text {RI}}(\psi \mathbin {\mathsf {S}}_{[a,b)} \chi )$ and with Condition 1 in Definition A1.4 it follows that $\tau _{i} = \tau '_{i-c}$.

  We show the following two claims, which we use later in the proof.

  • I. For all $j \in \mathbb {N}$ with $j \le i$ and $\tau _i - \tau _j \in [a,b)$, it holds that ${\text {RI}}(\chi ) \subseteq \{\tau _i - \tau _j\} \oplus \{\tau _j - \tau _i\} \oplus {\text {RI}}(\chi ) \subseteq \{\tau _i - \tau _j\} \oplus (-b,-a] \oplus {\text {RI}}(\chi ) \subseteq \{\tau _i - \tau _j\} \oplus {\text {RI}}(\psi \mathbin {\mathsf {S}}_{[a,b)} \chi )$ and $j \ge c$. By Lemma A1.3, $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(\chi ),c,j)$-overlapping. It follows from the inductive hypothesis that $(\bar{\mathscr {D}},\bar{\tau },v,j) \models \chi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,j-c) \models \chi$, for all valuations v.

  • II. For all $k \in \mathbb {N}$ with $k \le i$ and $\tau _i - \tau _k \in [0,b)$, it holds that ${\text {RI}}(\psi ) \subseteq \{\tau _i - \tau _k\} \oplus \{\tau _k - \tau _i\} \oplus {\text {RI}}(\psi ) \subseteq \{\tau _i - \tau _k\} \oplus (-b,0] \oplus {\text {RI}}(\psi ) \subseteq \{\tau _i - \tau _k\} \oplus {\text {RI}}(\psi \mathbin {\mathsf {S}}_{[a,b)} \chi )$ and $k \ge c$.

    By Lemma A1.3 $(\bar{\mathscr {D}},\bar{\tau })$ and $(\bar{\mathscr {D}}',\bar{\tau }')$ are $({\text {RI}}(\psi ),c,k)$-overlapping. It follows from the inductive hypothesis that $(\bar{\mathscr {D}},\bar{\tau },v,k) \models \psi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,k-c) \models \psi$, for all valuations v.

  We first show the direction from left to right of the claimed equivalence $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \psi \mathbin {\mathsf {S}}_{[a,b)} \chi$ iff $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \psi \mathbin {\mathsf {S}}_{[a,b)} \chi$, for all valuations v. If $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \psi \mathbin {\mathsf {S}}_{[a,b)} \chi$ then there is some $j \le i$ with $\tau _i - \tau _j \in [a,b)$ such that $(\bar{\mathscr {D}},\bar{\tau },v,j) \models \chi$ and $(\bar{\mathscr {D}},\bar{\tau },v,k) \models \psi$, for all $k \in [j+1,i+1)$.

  From $\tau _i - \tau _j \in [a,b)$ it follows that $\tau _j - \tau _i \in {\text {RI}}(\psi \mathbin {\mathsf {S}}_{[a,b)} \chi )$ and from Condition 1 in Definition A1.4 we see that $j \ge c$ and $\tau _j = \tau '_{j-c}$. From Claim I above and from $(\bar{\mathscr {D}},\bar{\tau },v,j) \models \chi$ it follows that $(\bar{\mathscr {D}}',\bar{\tau }',v,j-c) \models \chi$.

  For all $k' \in [j+1-c,i+1-c)$, it holds that $\tau '_{k'} - \tau '_{i-c} = \tau '_{k'} - \tau _{i} \in (-b, 0]$ and hence $\tau '_{k'} - \tau _i \in {\text {RI}}(\psi \mathbin {\mathsf {S}}_{[a,b)} \chi )$. From Condition 2 in Definition A1.4 we see that $\tau _{k'+c} = \tau '_{k'}$. From Claim II above and from $(\bar{\mathscr {D}},\bar{\tau },v,k'+c) \models \psi$ it follows that $(\bar{\mathscr {D}}',\bar{\tau }',v,k') \models \psi$. Therefore, $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \models \psi \mathbin {\mathsf {S}}_{[a,b)} \chi$.

  It remains to show the right-to-left direction of the claimed equivalence. We do this by contraposition. If $(\bar{\mathscr {D}},\bar{\tau },v,i) \not \models \psi \mathbin {\mathsf {S}}_{[a,b)} \chi$ then there are two possibilities:

  • 1. For all $j \le i$ with $\tau _i - \tau _j \in [a,b)$ it holds that $(\bar{\mathscr {D}},\bar{\tau },v,j) \not \models \chi$. Then for all $j' \le i-c$ with $\tau '_{i-c} - \tau '_{j'} = \tau _{i} - \tau '_{j'} \in [a,b)$, it holds that $\tau '_{j'} - \tau _i \in {\text {RI}}(\psi \mathbin {\mathsf {S}}_{[a,b)} \chi )$. From Condition 2 in Definition A1.4, it follows that $\tau '_{j'} = \tau _{j'+c}$. That is, there are no additional time points with a timestamp within the interval [a, b) in $(\bar{\mathscr {D}}',\bar{\tau }')$ that would not be present in $(\bar{\mathscr {D}},\bar{\tau })$.

    Since $\tau _{i} - \tau _{j'+c} \in [a,b)$, it follows from Claim I above and from $(\bar{\mathscr {D}},\bar{\tau },v,j'+c) \not \models \chi$ that $(\bar{\mathscr {D}}',\bar{\tau }',v,j') \not \models \chi$. Therefore, $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \not \models \psi \mathbin {\mathsf {S}}_{[a,b)} \chi$.

  • 2. For all $j \le i$ with $\tau _i - \tau _j \in [a,b)$ and $(\bar{\mathscr {D}},\bar{\tau },v,j) \models \chi$, there is some $k \in \mathbb {N}$ with $k \in [j+1,i+1)$ and $(\bar{\mathscr {D}},\bar{\tau },v,k) \not \models \psi$. Then for every $j' \in \mathbb {N}$ with $j' \le i-c$, $\tau '_{i-c} - \tau '_{j'} \in [a,b)$, and $(\bar{\mathscr {D}}',\bar{\tau }',v,j') \models \chi$, there is a $j\in \mathbb {N}$ with $j = j' + c$. We show that $\tau '_{j'} = \tau _j$ and $j \le i$.

    From $\tau '_{i-c} - \tau '_{j'} \in [a,b)$ and $\tau '_{i-c} = \tau _i$, it follows that $\tau '_{j'} - \tau _i \in (-b,-a]$ and hence $\tau '_{j'} - \tau _i \in {\text {RI}}(\psi \mathbin {\mathsf {S}}_{[a,b)} \chi )$. From Condition 2 in Definition A1.4, $\tau '_{j'} = \tau _{j'+c} = \tau _j$. From $j = j'+c$ and $j' \le i-c$, it follows that $j \le i$.

    Since $\tau '_{j'} = \tau _j$ and $j \le i$, we can use Claim I above for j. From Claim I and $(\bar{\mathscr {D}}',\bar{\tau }',v,j-c) \models \chi$ it follows that $(\bar{\mathscr {D}},\bar{\tau },v,j) \models \chi$. As a consequence, there is a $k \in \mathbb {N}$ with $k \in [j+1,i+1)$ and $(\bar{\mathscr {D}},\bar{\tau },v,k) \not \models \psi$. If follows from $k \in [j+1,i+1)$ that $k \le i$. Furthermore, from $\tau '_{i-c} - \tau '_{j'} \in [a,b)$ it follows that $\tau _i - \tau _j \in [a,b)$ and hence $\tau _i - \tau _k \in [0,b)$. Therefore, we can use Claim II above for k. From Claim II and $(\bar{\mathscr {D}},\bar{\tau },v,k) \not \models \psi$ it follows that $(\bar{\mathscr {D}}',\bar{\tau }',v,k-c) \not \models \psi$. From $k \in [j+1,i+1)$ it follows that $k-c \in [j'+1,i-c+1)$ and hence $(\bar{\mathscr {D}}',\bar{\tau }',v,i-c) \not \models \psi \mathbin {\mathsf {S}}_{[a,b)} \chi$.

• $\psi \mathbin {\mathsf {U}}_{[a,b)} \chi$. This case is analogous to the previous one. $\square$

We prove Theorem A1.1 by showing that $\mathfrak {t}_{\varphi ,(I^k)_{k \in K}}$ satisfies the conditions (S1) to (S3) from Definition 3.3 if $\bigcup _{k \in K} I^k = \mathbb {N}$. (S1), i.e. $\mathcal {R}=\bigcup _{k\in K}\mathcal {R}^k$, follows from the definition of $\mathcal {R}^k$ and the assumption that $\bigcup _{k\in K}I^k=\mathbb {N}$. (S2) and (S3) follow from the Lemmas A1.2 and A1.4.

Appendix 2: Additional details: filtering empty time points

We first introduce a filter that removes empty time points.

Definition A2.5

The function $\mathfrak {f}_{\varphi }$ for the formula $\varphi$ maps $(\bar{\mathscr {D}},\bar{\tau })\in \mathbf {T}$ and $\mathcal {R}\in \mathbf {R}$ to a family that contains only the temporal structure $(\bar{\mathscr {D}}',\bar{\tau }')$ and a family that contains only the restriction $\mathcal {R}$, where $(\bar{\mathscr {D}}',\bar{\tau }')$ is the empty-time-point-filtered slice of $(\bar{\mathscr {D}},\bar{\tau })$.

Next, we present a fragment of formulas for which the empty-time-point-filtered slice is sound and complete with respect to the original temporal structure. To define the fragment, we use the sets $\text {FT}$, $\text {FF}$, and $\text {FE}$, defined in Definition A2.6. Membership of a formula in these sets reflects whether the formula is satisfied at an empty time point. In a nutshell, at an empty time point, a formula in the set $\text {FF}$ is not satisfied, a formula in the set $\text {FT}$ is satisfied, and the satisfaction of a formula in the set $\text {FE}$ is not affected by adding or removing empty time points in the temporal structure.

Definition A2.6

The sets $\text {FT}$, $\text {FF}$, and $\text {FE}$ of formulas are defined as follows.

• $\varphi \in \text {FT}$ iff $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \varphi$, for all $(\bar{\mathscr {D}},\bar{\tau })\in \mathbf {T}$, all valuations v, and all empty time points i of $(\bar{\mathscr {D}},\bar{\tau })$.

• $\varphi \in \text {FF}$ iff $(\bar{\mathscr {D}},\bar{\tau },v,i) \not \models \varphi$, for all $(\bar{\mathscr {D}},\bar{\tau })\in \mathbf {T}$, all valuations v, and all empty time points i of $(\bar{\mathscr {D}},\bar{\tau })$.

• $\varphi \in \text {FE}$ iff the equivalence

  $$\begin{aligned} (\bar{\mathscr {D}}',\bar{\tau }',v,i') \models \varphi \quad {\text {iff}}\quad (\bar{\mathscr {D}},\bar{\tau },v,s(i')) \models \varphi \end{aligned}$$

  holds, for all $(\bar{\mathscr {D}},\bar{\tau }), (\bar{\mathscr {D}}',\bar{\tau }')\in \mathbf {T}$, all valuations v, and all nonempty time points $i'$ of $(\bar{\mathscr {D}}',\bar{\tau }')$, where $(\bar{\mathscr {D}}',\bar{\tau }')$ is the empty-time-point-filtered slice of $(\bar{\mathscr {D}},\bar{\tau })$ and s is the function used in the filtering of $(\bar{\mathscr {D}},\bar{\tau })$.

We approximate membership in the sets $\text {FT}$, $\text {FF}$, and $\text {FE}$ with syntactic fragments. Such an approximation is necessary since these sets are undecidable, which follows from the undecidability of the satisfiability problem of MFOTL. The fragments are defined in terms of a labeling algorithm that assigns the labels $\mathsf {FT}$, $\mathsf {FF}$, and $\mathsf {FE}$ to formulas. The fragments are sound in the sense that if a formula is assigned to a label ($\mathsf {FT}$, $\mathsf {FF}$, $\mathsf {FE}$) then the formula is in the corresponding set ($\text {FT}$, $\text {FF}$, $\text {FE}$, respectively). However, the fragments are incomplete: not every formula in one of the sets is assigned by the algorithm to the corresponding label. The algorithm labels the atomic subformulas of a formula and propagates the labels bottom-up to the formula’s root. The labeling rules are shown in Fig. 4, where the expression $\varphi :\ell$ denotes that the formula $\varphi$ is labeled with $\ell$. Note that a formula can have multiple labels. We prove next the soundness of the labeling rules.

Fig. 4

Labeling rules (empty-time-point filter)

Theorem A2.2

For all formulas $\varphi$, if the derivation rules shown in Fig. 4 assign the label $\mathsf {FT}$, $\mathsf {FF}$, or $\mathsf {FE}$ to $\varphi$ then $\varphi$ is in the set $\text {FT}$, $\text {FF}$, or $\text {FE}$, respectively.

Proof

We begin with the labels $\mathsf {FT}$ and $\mathsf {FF}$. We proceed by induction on the size of the derivation tree assigning label $\ell$ to the formula $\varphi$. We make a case distinction based on the rules applied to label the formula, that is, the rule at the tree’s root. However, for clarity, we group cases by the formula’s form. For readability, and without loss of generality, we fix the temporal structure $(\bar{\mathscr {D}},\bar{\tau })$, a time point $i \in \mathbb {N}$, and a valuation v.

A formula $r(\bar{t})$ is labeled $\mathsf {FF}$. If i is an empty time point in $(\bar{\mathscr {D}},\bar{\tau })$ then clearly $(\bar{\mathscr {D}},\bar{\tau },v,i) \not \models r(\bar{t})$, for any predicate symbol $r\in R$ and any terms $\bar{t}$. The formula true is labeled $\mathsf {FT}$. Trivially, $(\bar{\mathscr {D}},\bar{\tau },v,i) \models {{true}}$. The other rules propagate the assigned label of the subformulas through the non-temporal connectives according to their semantics. The rules’ correctness is straightforward.

We consider next the label $\mathsf {FE}$. Again, we proceed by induction on the size of the derivation tree assigning label $\mathsf {FE}$ to formula $\varphi$. We make a case distinction based on the rules applied to label the formula, that is, the rule at the tree’s root. However, for clarity, we again group cases by the formula’s form.

For every valuation v and $i' \in \mathbb {N}$, the evaluation of the formulas $r(\bar{t})$, $t \approx t'$, and $t \prec t'$ only depends on the current time point and hence they are in $\text {FE}$. The other rules not involving temporal operators depend only on the value of their subformulas at the current time point. If the subformulas are labeled with $\mathsf {FE}$, then by the inductive hypothesis the subformulas are in $\text {FE}$, so the formula is also in $\text {FE}$.

For readability, and without loss of generality, we already fix the temporal structure $(\bar{\mathscr {D}},\bar{\tau })$ and its empty-time-point-filtered slice $(\bar{\mathscr {D}}',\bar{\tau }')$.

The proof is trivial for the case where s is the identity function. In the rest of the proof, we assume that $(\bar{\mathscr {D}},\bar{\tau })$ has infinitely many nonempty time points and hence s is not the identity function.

• $\varphi \mathbin {\mathsf {S}}_I \psi$: We show separately that, for every valuation v and $i' \in \mathbb {N}$, (1) $(\bar{\mathscr {D}}',\bar{\tau }',v,i') \models \varphi$ implies $(\bar{\mathscr {D}},\bar{\tau },v,s(i')) \models \varphi$, and (2) $(\bar{\mathscr {D}},\bar{\tau },v,s(i')) \models \varphi$ implies $(\bar{\mathscr {D}}',\bar{\tau }',v,i') \models \varphi$.

  • (1) From $(\bar{\mathscr {D}}',\bar{\tau }',v,i') \models \varphi \mathbin {\mathsf {S}}_I \psi$ we know that there is a $j' \le i'$ such that $\tau '_{i'} - \tau '_{j'} \in I$ and $(\bar{\mathscr {D}}',\bar{\tau }',v,j') \models \psi$ and, for every $k'$ with $j' < k' \le i'$, we have that $(\bar{\mathscr {D}}',\bar{\tau }',v,k') \models \varphi$.

    Since $\psi$ is labeled $\mathsf {FE}$, it follows from the inductive hypothesis that $\psi$ is in $\text {FE}$ and hence $(\bar{\mathscr {D}},\bar{\tau },v,s(j')) \models \psi$. For each k with $s(j') < k \le s(i')$ either k is an empty or a nonempty time point in $(\bar{\mathscr {D}}, \bar{\tau })$. If it is an empty time point then from $\varphi$ being labeled $\mathsf {FT}$ and hence in $\text {FT}$ we know that $(\bar{\mathscr {D}}, \bar{\tau }, v, k) \models \varphi$. If it is a nonempty time point then we know that there is a time point $k'$ in $(\bar{\mathscr {D}}', \bar{\tau }')$ with $j' < k' \le i'$ and $k = s(k')$.

    From $\varphi$ being labeled $\mathsf {FE}$ and hence in $\text {FE}$ we know that $(\bar{\mathscr {D}}, \bar{\tau }, v, k) \models \varphi$. In both cases $(\bar{\mathscr {D}}, \bar{\tau }, v, k) \models \varphi$ and therefore $(\bar{\mathscr {D}},\bar{\tau },v,s(i')) \models \varphi \mathbin {\mathsf {S}}_I \psi$.

  • (2) From $(\bar{\mathscr {D}},\bar{\tau },v,s(i')) \models \varphi \mathbin {\mathsf {S}}_I \psi$ it follows that there is a $j \le s(i')$ with $\tau _{s(i')} - \tau _j \in I$ and $(\bar{\mathscr {D}},\bar{\tau },v,j) \models \psi$, and that, for every k with $j < k \le s(i')$, we have that $(\bar{\mathscr {D}},\bar{\tau },v,k) \models \varphi$.

    Since $(\bar{\mathscr {D}},\bar{\tau },v,j) \models \psi$ and $\psi$ is labeled $\mathsf {FF}$, so that $\psi$ is in $\text {FF}$, we know that j cannot be an empty time point in $(\bar{\mathscr {D}},\bar{\tau })$. Therefore, there is a $j'$ such that $j = s(j')$. We have that $j' \le i'$ because s is monotonically increasing. From $\psi$ being labeled $\mathsf {FE}$ it follows that $\psi$ is in $\text {FE}$ and hence $(\bar{\mathscr {D}},\bar{\tau }, v, j) \models \psi$ implies $(\bar{\mathscr {D}}',\bar{\tau }', v, j') \models \psi$.

    Furthermore, for every $k'$ with $j' < k' \le i'$ there is a corresponding time point k in $(\bar{\mathscr {D}}, \bar{\tau })$ such that $k = s(k')$. As s is a monotonously increasing function we have that $s(j') < k \le s(i')$. From $(\bar{\mathscr {D}},\bar{\tau }, v, s(i')) \models \varphi \mathbin {\mathsf {S}}_I \psi$ it follows that $(\bar{\mathscr {D}},\bar{\tau }, v, k) \models \varphi$. From $\varphi$ being labeled $\mathsf {FE}$ it follows that $\varphi$ is in $\text {FE}$ and hence $(\bar{\mathscr {D}}',\bar{\tau }', v, k') \models \varphi$. Therefore, $(\bar{\mathscr {D}},\bar{\tau },v,s(i')) \models \varphi \mathbin {\mathsf {S}}_I \psi$.

• $\varphi \mathbin {\mathsf {U}}_I \psi$: This case is similar to $\varphi \mathbin {\mathsf {S}}_I \psi$.

• and with $0 \in I \cap J$: These formulas can both be rewritten to ), which can be labeled with the rules proven above.

• and with $0 \in I \cap J$: These formulas can both be rewritten to ), which can be labeled with the rules proven above. $\square$

From Theorem A2.2 and the following Theorem A2.3, it follows that the empty-time-point filter is a slicer for all formulas that can be labeled with $\mathsf {FE}$ and $\mathsf {FT}$.

Theorem A2.3

The empty-time-point filter $\mathfrak {f}_{\varphi }$ is a slicer for the formula $\varphi$, if the formula $\varphi$ is in both $\text {FE}$ and $\text {FT}$.

Proof

We show that $\mathfrak {f}_{\varphi }$ satisfies the conditions (S1) to (S3) from Definition 3.3. (S1) follows trivially because $\mathfrak {f}_{\varphi }$ does not modify the given restriction. For showing (S2) and (S3), let $(\bar{\mathscr {D}}',\bar{\tau }')$ be the empty-time-point filtered slice of $(\bar{\mathscr {D}},\bar{\tau })$.

For (S2), we show that for all valuations v and timestamps $t \in \mathbb {N}$, it holds that $(\bar{\mathscr {D}},\bar{\tau },v,i)\models \varphi$, for all $i\in \mathbb {N}$ with $\tau _i = t$, implies $(\bar{\mathscr {D}}',\bar{\tau }',v,i')\models \varphi$, for all $i'\in \mathbb {N}$ with $\tau '_{i'} = t$ using contraposition. Assume that $(\bar{\mathscr {D}}',\bar{\tau }',v,i') \not \models \varphi$, for some $i' \in \mathbb {N}$ with $\tau '_{i'} = t$. As $(\bar{\mathscr {D}}',\bar{\tau }')$ is the empty-time-point filtered slice of $(\bar{\mathscr {D}},\bar{\tau })$, there is some $i \in \mathbb {N}$ such that $i = s(i')$ and $\tau _i = \tau '_{i'} = t$. From $\varphi \in \text {FE}$ it follows that $(\bar{\mathscr {D}},\bar{\tau },v,i) \not \models \varphi$.

For (S3), we show that for all valuations v and timestamps $t \in \mathbb {N}$, it holds that $(\bar{\mathscr {D}},\bar{\tau },v,i)\not \models \varphi$, for some $i\in \mathbb {N}$ with $\tau _i = t$, implies $(\bar{\mathscr {D}}',\bar{\tau }',v,i')\not \models \varphi$, for some $i'\in \mathbb {N}$ with $\tau '_{i'} = t$. Let $i\in \mathbb {N}$. There is nothing to prove if i is empty in $(\bar{\mathscr {D}},\bar{\tau })$, since $\varphi$ is in $\text {FT}$ and hence $(\bar{\mathscr {D}},\bar{\tau },v,i) \models \varphi$. If i is nonempty in $(\bar{\mathscr {D}},\bar{\tau })$ then there exists a time point $i'$ in $(\bar{\mathscr {D}}',\bar{\tau }')$ such that $i = s(i')$ and $\tau _i = \tau '_{i'}$. Since $\varphi$ is in $\text {FE}$ and if $(\bar{\mathscr {D}},\bar{\tau },v,i) \not \models \varphi$ then it follows that $(\bar{\mathscr {D}}',\bar{\tau }',v,i') \not \models \varphi$. $\square$

Appendix 3: Additional details: algorithmic realization

We present slicing functions and restriction modifiers for time slicing (Sect. 3.2.2 and Appendix 1) and filtering empty time points (Sect. 3.2.3 and Appendix 2).

Algorithm A3.1 describes the pointwise slicing function $\textit{fs}^{{{time}}}_{\varphi ,I}$ and the restriction modifier $\textit{fr}^{{{time}}}_{\varphi ,I}$ for time slicing. The body of $\textit{fs}^{{{time}}}_{\varphi ,I}$ first determines whether the timestamp $\tau$ is within the time interval $I\oplus {\text {RI}}(\varphi )=\{i+j\mathbin {|}i \in I {\text { and }} j \in {\text {RI}}(\varphi )\}$, where ${\text {RI}}(\varphi )$ is the relative interval of $\varphi$ (see Definition A1.1). Note that this check corresponds to the condition on timestamps in our definition of a T-slice, with $T = I \oplus {\text {RI}}(\varphi )$ (see Definition A1.2). If $\tau$ is within the computed interval, then $\textit{fs}^{{{time}}}_{\varphi ,I}$ returns ${\mathscr {D}}$ unmodified and, otherwise, $\bot$ to indicate that the log entry shall be deleted. The restriction modifier $\textit{fr}^{{{time}}}_{\varphi ,I}$ removes all violations with timestamps outside I from a given restriction $\mathcal {R}$.

Algorithm A3.2 describes the pointwise slicing function $\textit{fs}^{{{empty}}}$ and the restriction modifier $\textit{fr}^{{{empty}}}$ for filtering empty time points. The function $\textit{fs}^{{{empty}}}$ returns $\mathscr {D}$ if there is at least one $\textit{r}\in \textit{R}$ for which $r^{\mathscr {D}}$ is nonempty, and otherwise $\bot$ to indicate that the time point shall be deleted. The check $\{\textit{r}\in \textit{R} \mid r^{\mathscr {D}}\ne \emptyset \} \ne \emptyset$ in its body corresponds to the condition for nonempty time points. For efficiency, one should not explicitly construct the set $\{\textit{r}\in \textit{R} \mid r^{\mathscr {D}}\ne \emptyset \}$ when implementing it. The restriction modifier $\textit{fr}^{{{empty}}}$ returns $\mathcal {R}$ without modification.