A Thread-Safe Term Library

Abstract

Terms are one of the fundamental mathematical concepts in computing. E.g. every expression characterisable by a context free grammar is a term. We developed a thread-safe Term Library. The biggest challenge is to implement hyper-efficient multi-reader/single-writer mutual exclusion for which we designed the new busy-forbidden protocol. Model checking is used to show both the correctness of the protocol and the Term Library. Benchmarks show this Term Library has little overhead compared to sequential versions and outperforms them already on two processors. Using the new library in an existing state space generation tool, very substantial speed ups can be obtained.

Supported by projects 612.001.751 (NWO, AVVA) and 00795160 (TTW, MASCOT).

Appendices

AThe mCRL2 Language and Modal Formulas

The models are written in mCRL2. This is a modelling language based on CCS (Calculus of Communicating Processes) [24] and ACP (Algebra of Communicating Processes) [1]. It is based on atomic actions. Every occurrence of an atomic action causes a state change. Typically, calling a function, or returning from a function, setting or reading a global variable are modelled by atomic actions. Each action consists of a label and a possible set of data parameters, e.g., the $\textit{lock}(p)$ action has p as a data parameter. The tau or hidden action $\tau$ has a special status, as it is an action of which the occurrence cannot be observed directly.

Actions can be sequentially composed using the dot (‘$\cdot$’) operator. Alternative composition, where nondeterministically one of the options can be chosen, is denoted using a plus (‘$+$’). The $\sum _{e{:}S}$ operator denotes the application of the (‘$+$’) operator over all elements of some set S. The if-then-else is written as $c{\rightarrow }p{\diamond }q$ where p is executed if c is true, otherwise the process q takes place.

Parallel composition is denoted by $\parallel$ and allows the actions of two processes to both interleave and occur simultaneously. Using the comm and allow operators, we can enforce that only specific actions can and must occur simultaneously. For example, ${\textbf {comm}}(\{a,b\}{\rightarrow }c\}, {\textbf {allow}}(\{c,d,e\}, (d\cdot a) || (b\cdot e)))$ enforces that the actions d and e can not occur simultaneously, while a and b must occur simultaneously with any other action, signified with c. As a result, $d\cdot c \cdot e$ is the only possible sequence of actions that can occur.

Recursive behaviour is denoted using equations, typically of the form $X=p$, e.g., $X=a{\cdot }X$ is the process that can perform an infinite number of a’s. Similar to actions, the process variables X can contain data parameters. A counter can thus be described as $C(n{:}\mathbb {N})=\textit{up}{\cdot }X(n+1)$. An important type of data parameter that we use is a function. For example, the process variable $Y(m : \mathbb {N}\rightarrow \mathbb {B})$ uses a mapping m from natural numbers to booleans. The function update $m[ n \mapsto b]$ specifies that $m[ n \mapsto b](k)$ equals b if $k= n$, and otherwise, it equals m(k).

The safety and liveness properties that we verify, are written in the modal mu-calculus. These consist of conjunctions ($\wedge$), disjunctions ($\vee$), implications ($\rightarrow$), negations ($\lnot$), quantification ($\exists ,\forall$) and $\textit{true}$ and $\textit{false}$, each with their usual meaning. Besides this there is a modality $\langle a \rangle \phi$ that is valid if we can take an action a after which $\phi$ holds. Similarly, the modality $[a]\phi$ holds iff after every possible a action $\phi$ holds. The action a inside these modalities can also consist of possibly multiple actions. This can be done through sequential composition $(\cdot )$, choice $(\cup )$, intersection $(\cap )$ and complement $(\overline{a})$. For example, the formula $\langle \overline{a} \cup \overline{b} \rangle \textit{true}$ only holds if we can do some action that is neither a or b. The expression $\textit{true}$ in a modality represents the set of all actions. Using Kleene’s star on a set of actions, all sequences over the action in this set are expressed. An often occurring pattern is $[\textit{true}^*]\phi$ expressing that $\phi$ must hold in all states reachable via a sequence of actions.

We can also write recursive formulas using the minimal fixed point operator $\mu X.\phi$ and the maximal fixed point operator $\nu X. \phi$. For example, the formula $\nu X.\langle a \rangle X$ expresses that we must be able to perform action a after which the same formula still holds. Thus this formula only holds if we can perform an infinite amount of a actions. The difference between a minimal and a maximal fixed point is that iteration through the fixed point variable must be bounded in a minimal fixed point.

A fixed point construction used in several properties is

$$ \nu X.\mu Y.([\overline{\textit{succes} \cup \textit{interrupt}}]Y \wedge [\textit{interrupt}]X \wedge \langle \textit{true}^* .\textit{succes} \rangle \textit{true}) $$

This says that an action $\textit{succes}$ will always occur within a finite amount of steps, unless an action $\textit{interrupt}$ continuously occurs. But even in that case $\textit{succes}$ must remain possible. This construction is useful for properties in which we state that something must eventually happen given fair scheduling.

The fixed point operators also allow us to pass on parameters in the same way we can do for process variables. This allows us, for example, to keep track of the number of times that a given action has occurred. We discuss one such fixed point operator in Appendix B.

Table 3. Specification of the busy-forbidden protocol corresponding to Fig. 2.

BmCRL2 Specifications for the Busy-Forbidden Protocol

In this section we give the formal mCRL2 specifications of the implementation and the external behaviour, i.e., the specification, of the busy-forbidden protocol that are used to perform the model and equivalence checking. The process specification given in Table  3 exactly matches the external behaviour shown in Fig. 2. We define P to be the (finite) set of threads and we define $\textit{S}$ to be a data set representing the set of states:

The mapping s maps each thread to their current state. Initially, $s(p) = \textit{Free}$ for all $p \in P$. The conditions for performing transitions are the same as the conditions in the diagram of the external behaviour.

Observe that we use a typewriter font (for example enter_shared_call) to indicate visible actions and an italics font (for example $\textit{store}_p$) to indicate internal actions that will be hidden for divergence-preserving branching bisimulation reductions.

The mCRL2 specification of the implementation is separated per function. Entering the shared section is specified in Table  4 and leaving it in Table  5. Entering the exclusive section is specified in Table  6 and leaving it in Table  7.

Note that we use actions to model the assignments to variables. For example $\textit{store}_p(\textit{Busy}(p),\textit{true},p)$ corresponds to the assignment of $\textit{true}$ to $\textit{p.busy}$ in the implementation pseudocode. The process algebra has no global variables and we use an additional process and actions to read from and write to these variables. For the atomic flags we introduce a struct F that is defined below to declare a busy and a forbidden flag per thread.

$$\begin{aligned} \texttt {sort}~F = \texttt {struct}~\textit{Busy}(P)~|~\textit{Forbidden}(P) \end{aligned}$$

Table  8 shows the behaviour of the Busy and Forbidden flags for every thread and the mutex variable. We model the ‘while’ construction in the pseudocode by recursion and have added the improbable action to ensure equivalence modulo divergence-preserving branching bisimulation. When entering the exclusive section, we use a set forbidden (and for leaving allowed) to keep track of the threads whose forbidden flag have already been set to $\textit{true}$ ($\textit{false}$ when leaving).

Table  9 shows the specification for the behaviour of a thread. Each thread repeatedly chooses (non-deterministically) to enter and leave either the shared or exclusive section. Finally, Table  10 contains the complete mCRL2 specification of the various processes in a parallel composition and the necessary communication to deal with the atomic flags and mutex.

As the next step, we transformed the six requirements discussed in Sect. 3.3 into modal logic formulas, and verified them on the specification. Note that these properties are preserved by divergence-preserving branching bisimulation, so verifying the implementation is not necessary. We discuss property 2 in detail as an illustration of what such formulas look like. The informal description of the property reads:

• 2. There should never be a thread present in the exclusive section while one or more threads are present in the shared section.

The corresponding modal formula is shown in Table  12. We use a maximal fixpoint with two data parameters, namely $n_{shared}$ and $n_{exclusive}$, both initially 0. The argument $n_{shared}$ indicates the number of threads present in the shared section, and $n_{exclusive}$ the number in the exclusive section. At lines 2 through 5, we keep track of the amount of threads present in each section, updating the variables after each respective action. At lines 6 through 11, we state that our variables stay the same, after any action that is not one of the four aforementioned actions. Finally, at line 12, we state that threads are only allowed to be either present in exclusive or are present in shared.

The formula for property 1 is shown in Table  11 and states that when a thread enters the exclusive section, no other thread may enter that section till it leaves the section. The formulas for properties 3 and 4 are presented in Tables  13 and 15 and use data parameters to count the number of threads in the exclusive section or in any section respectively. Note that these are two subformulas with identical structure for shared and exclusive sections respectively. Finally, properties 4 and 6 presented in Tables  14 and 16 use boolean parameters to keep track of whether any thread is in the shared or exclusive sections, respectively. This is more efficient than keeping track of the exact amount of threads.

The properties were verified for up to 7 threads. The specification model has about three million states and the implementation model about 11 billion states.

Table 4. mCRL2 specification for the implementation of enter_shared.

Table 5. mCRL2 specification for the implementation of leave_shared.

Table 6. mCRL2 specification for the enter_exclusive function in Table  2.

Table 7. mCRL2 specification for the leave_exclusive function in Table  2.

Table 8. mCRL2 specifications for the atomic flags and the mutex.

Table 9. mCRl2 specification for a thread p interacting with the protocol.

Table 10. mCRL2 specification for the busy-forbidden protocol.

Table 11. Modal formula for property 1: “There should never be more than one thread present in the exclusive section”.

Table 12. The modal formula for property 2: “There should never be a thread present in the exclusive section while one or more threads are present in the shared section”.

Table 13. Modal formula for property 3: “When a thread requests to enter the shared section, it will be granted access within a bounded number of steps, unless there is another thread in the exclusive section”.

Table 14. Modal formula for property 5: “When a thread requests to leave the exclusive/shared section, it will leave it within a bounded number of steps”.

Table 15. Modal formula for property 4: “When a thread requests to enter the exclusive section, it will be granted access within a bounded number of steps, unless there is another thread in the shared or in the exclusive section”.

Table 16. Modal formula for property 6: “A thread not in the exclusive or shared section can instantly start to enter the exclusive or shared section”.

CmCRL2 Specifications for the Term Library

In this section we give the formal mCRL2 specifications of the implementation of the term library that is used to perform model checking. Creating a term is specified in Table  17 and destroying a term in Table  18. In this model, the set P corresponds to the set containing all threads, T to the set containing all terms and A to the set containing all memory addresses. The set $A_\bot = A \cup \{\bot \}$ with $\bot \not \in A$ contains the extra element $\bot$ meaning no address or a NULL pointer. To ensure finiteness and reduce the complexity of the model, the set T only contains a finite amount of constants, i.e., terms of arity zero.

Table 17. mCRL2 specification for the $\textsf {create}$ function shown in Table  1.

First of all, we introduce processes $\textit{EnterShared}$, $\textit{LeaveShared}$, $\textit{EnterExclusive}$ and $\textit{LeaveExclusive}$ to interact with the busy-forbidden specification $\textit{BF}$ specified in Table  3. To distinguish between the term library and the protocol all actions such as $\texttt {enter\_shared\_call}$ are split into action $\texttt {enter\_shared\_call}_\textit{bf}$ for the protocol and $\texttt {enter\_shared\_call}_\textit{p}$ for the term library. Finally, we have the process $\textit{MainMemory}$ to model the main memory by keeping track of used memory addresses, the process $\textit{HashTable}$ to model a hash table as an associative array, and process $\textit{ReferenceCounter}$ to track a reference counter for every address (or term). Destroying a term is specified in Table  18, which uses the same other processes as the creation function. Again, there are two separate processes to model the behaviour of the while loop. In Table  19 the behaviour of the main memory, the hash table and the reference counter are specified.

The specification in Table  20 models the behaviour of each thread. Each thread repeatedly tries to either creates a term it does not yet know, or it destroys a known term. Finally, Table  21 shows the complete specification including the communication between various processes used to model the thread-safe term library.

Table 18. mCRL2 specification for the $\textsf {destroy}$ function shown in Table  1.

To verify the model of the thread-safe term library we again specify a number of modal formulas for the properties described in Sect. 2.2. The modal formula for property 1 specified in Table  22 uses a mapping a from addresses to terms and the finite set $\textit{owners}$ containing all threads that own/protect term t as data parameters. If at any point in time a $\texttt {create(t)}$ returns a different address than the current address, then the term must not be owned by any thread. The modal formula in Table  23 for property 2 uses the same constructs to check whether terms on the same address are also equivalent.

The formula for property 3 shown in Table  24 uses a boolean parameter $\textit{busy}$ to keep track of whether the thread p is creating (or destroying) a term. Furthermore, the parameter $\textit{known}$ is a finite set containing all terms that thread p knows. If at any point in time $\textit{busy}$ is false, then the process must be able to start destroying any term in $\textit{known}$ and start creating any term not currently in $\textit{known}$. Finally, for property 4 the formula shown in Table  25 uses again the construction which (under fairness) indicates that term creation and destruction will finish within a finite number of steps. Note that the subformulas for creation and destruction have an identical structure.

Table 19. mCRL2 specifications of the main memory, hash table and reference counters used in the term library specification.

Table 20. mCRL2 specification of a thread p interacting with the term library.

Table 21. mCRL2 specification for the thread-safe term library.

Table 22. Formulation of property 1: “A term and all its subterms remain in existence at exactly the same address, with unchanged function symbol and arguments, as long as it is not destroyed”.

Table 23. Modal formula for property 2: “Two stored terms $t_1$ and $t_2$ always have the same non-null address iff they are equal”.

Table 24. Modal formula for property 3: “Any thread that is not busy creating or destroying a term, can always initiate the construction of a new term or the destruction of an owned term, i.e., a term that this thread has exclusive access to”.

Table 25. Modal formula(s) for property 4: “Any thread that started creating a term or destroying a term, will eventually successfully finish this task provided there is enough memory to store one more term than those that are in use. But it is required that other threads behave fairly, in the sense that they will not continually create and destroy terms or stall other threads by busy waiting”.

DBenchmark Data

The benchmark tests and information shown in Figs. 4 and 5 are hard to read exactly. Therefore, we repeat the corresponding precise benchmark numbers in Table  28 up to and including Table  26. Each wall-clock time is measured in seconds.

The measurements in Table  27 came from benchmarking performed on an Intel i7-7700HQ processor. All other measurements were obtained through benchmarking on an AMD EPYC 7452 32-Core processor.

The benchmark results in Table  28 were obtained by having each thread create a term $t_{400~000}$, with $t_0$ being a constant, and $t_{i+1}$ equal to $f(t_i,t_i)$. No garbage collection was performed during the benchmark. Note that only one copy of the term is actually stored in memory. So, most threads wanting to construct some term $f(t_i,t_i)$ detect that the term already exists, and only need to return its address, without actually creating it.

The benchmark results in Table  29 were obtained by having each thread create its own copy of the term $t_{400~000 / \#\textit{threads}}$, and measuring the wall-clock time. Note that although each thread creates its own term, all terms are stored in the data structures in an intermixed way. Note that as there is no sharing here, each thread stores a full copy of the term in memory.

The benchmark results in Table  30 and 31 were obtained by measuring the wall-clock time of creating $1000/\#\textit{threads}$ instances of the terms used in Table  28 and 29. Before we start measuring the wall-clock times, the terms and subterms have already been inserted into the hash table, thus we are only measuring the cost of performing repeated lookups in our hash table. The experiment reported in Table  30 is the same as the one in Table  27, but the former is run on an AMD EPYC 7452 processor whereas the latter uses an Intel i7-7700HQ processor.

The benchmark results in Table  32 were obtained by having each thread perform $1000/\#\textit{threads}$ breadth-first traversals of the term $t_{20}$ and measuring the wall-clock time. The traversal does not make use of the shared structure of terms, meaning that approximately $10^9$ term nodes are visited. Similarly, the benchmark results in Table  33 were obtained by having each thread perform $1000/\#\textit{threads}$ breadth-first traversals of a term $t_{20}$ that is unique for each thread.

We also measured the wall-clock time of the state space generation of the 1394 firewire protocol using a parallel prototype of the mCRL2 toolset. The results are listed in Table  26.

Table 26. Wall-clock time for state space exploration.

Table 27. Wall-clock time for creating existing terms (shared, Intel).

Table 28. Wall-clock time for creating new terms (shared).

Table 29. Wall-clock time for creating new terms (distinct).

Table 30. Wall-clock time for creating existing terms (shared).

Table 31. Wall-clock time for creating existing terms (distinct).

Table 32. Wall-clock time for traversing terms (shared).

Table 33. Wall-clock time for traversing terms (distinct).