Extending the power of datalog recursion

Abstract

Supporting aggregates in recursive logic rules represents a very important problem for Datalog. To solve this problem, we propose a simple extension, called Datalog$^{FS}\,$(Datalog extended with frequency support goals), that supports queries and reasoning about the number of distinct variable assignments satisfying given goals, or conjunctions of goals, in rules. This monotonic extension greatly enhances the power of Datalog, while preserving (i) its declarative semantics and  (ii) its amenability to efficient implementation via differential fixpoint and other optimization techniques presented in the paper. Thus, Datalog$^{FS}\,$enables the efficient formulation of queries that could not be expressed efficiently or could not be expressed at all in Datalog with stratified negation and aggregates. In fact, using a generalized notion of multiplicity called frequency, we show that diffusion models and page rank computations can be easily expressed and efficiently implemented using Datalog$^{FS}\,$.

Appendices

Appendix A: Abstract semantics by example

The abstract semantics of a Datalog$^{FS}\,$program P is defined by its translation into a set of Horn clauses called the expansion of P. For example, let us consider the query in Example 11 that counts how many copies of each basic part are contained in assemblies:

$$\begin{aligned} \begin{array}{ll} \mathtt{cassb(Part, Sub)\!:\!\,\, Qty \, } \leftarrow&\mathtt{ assbl(Part, Sub, Qty).~~~~~~~~ } \\ \mathtt{cbasic(Pno)\!:\!\,\, 1 } \leftarrow&\mathtt{\quad basic(Pno,\_). } \\ \mathtt{cbasic(Part)\!:\!\,\, K } \leftarrow&\mathtt{\quad K\!:\![cassb(Part, Sub),cbasic(Sub)]. } \\ \end{array} \end{aligned}$$

We rewrite all multi-occuring predicates using lessthan, obtaining

$$\begin{aligned}\begin{array}{ll} \mathtt{\overline{cassb}(Part, Sub,J) } \leftarrow&\mathtt{assbl(Part, Sub, Qty), } \\ \mathtt{ }&\mathtt{lessthan(J,Qty). } \\ \mathtt{\overline{cbasic}(Pno,J) } \leftarrow&\mathtt{lessthan(J,1), basic(Pno, \_). } \\ \mathtt{\overline{cbasic}(Part,J) } \leftarrow&\mathtt{lessthan(J,K), } \\ \mathtt{ }&\mathtt{\quad K\!:\![\overline{cassb}(Part, Sub, J1), \overline{cbasic}(Sub, J2)]. } \\ \end{array} \end{aligned}$$

The third rule evaluates the join of $\mathtt{cassb(Part, Sub, \_) }$ and $\mathtt{cbasic(Sub, \_) }$ on $\mathtt{Sub }$, and from their multiplicity of the two operands derives the counts associated with each Part. But in terms of formal abstract semantics no count aggregate is used: we instead use the $\mathtt{conj }$ predicate specific to each b-expression. For the case at hand we obtain (using lists to assemble the values of the three local variables into an object):

$$\begin{aligned} \begin{array}{ll} \mathtt{conj(1, Part, [[Sub, J1, J2]]) } \leftarrow&\mathtt{\overline{cassb}(Part, Sub, J1), } \\ \mathtt{ }&\mathtt{ \overline{cbasic}(Sub, J2). } \\ \mathtt{conj(N1, Part, [[Sub, J1, J2] | T]) } \leftarrow&\mathtt{\overline{cassb}(Part, Sub, J1), } \\ \mathtt{ }&\mathtt{ \overline{cbasic}(Sub, J2), } \\ \mathtt{ }&\mathtt{conj(N, Part, T), } \\ \mathtt{ }&\mathtt{ notin( [Sub, J1, J2], T), } \\ \mathtt{ }&\mathtt{ N1=N+1. } \\ \mathtt{notin(Z, [~]). }&\mathtt{ } \\ \mathtt{notin(Z, [V|T]) } \leftarrow&\mathtt{ \quad Z \ne V , notin(Z, T). } \\ \end{array} \end{aligned}$$

Therefore, the semantic-defining rewriting of our FS-literals expand our Datalog$^{FS}\,$program into standard Horn Clauses, for which model-theoretic, proof-theoretic and fixpoint semantics exist and are equivalent. Thus the FS terms can be used freely in recursive rules. However this rewriting does not provide a good starting point for implementation since it is extremely inefficient. Thus, in the example above, we have triples [Sub, J1, J2], and we build all lists of such triplets without repetitions; as we do that, we also count the N elements in each of the N! lists.¹⁰

Efficient implementations that are fully consistent with the formal semantics, are however possible starting directly from the FS constructs in the rules and we suggest that this is the approach that should be taken in actual implementations of Datalog$^{FS}\,$. In fact, working directly with the FS constructs in the rules is much preferable in terms of performance and it also more attractive in terms of usability. Indeed, concepts such as multiplicity are simple and enough that users would rather program with running-FS and FS-assert terms, rather than their abstract definitions based on lists and recursive predicates. Although Prolog-like implementation approaches are also possible, in this paper we focused on Datalog implementation technology.

Appendix B: The expressive power of running-FS

In Sect. 5, we claimed that the increased expressive power of stratified Datalog$^{FS}\,$is due only to the running-FS construct. In this section we prove such assertion by showing that final-FS goals and FS-assert constructs can be rewritten using only running-FS goals and stratified negation.

In Sects. 3.3 and 3.4 we have shown the rewriting in logic programming without lists of final-FS goals and FS-assert, respectively. However, the rewritings that we have shown use integer comparisons (operator $>$ for final-FS goals) and the sum operator (for FS-assert constructs). Now we will show how it is possible to rewrite final-FS goals and FS-assert constructs using only running-FS goals without integer comparison and built-in integer operation as the sum.

As shown in Sect. 3.4, a final-FS goal $\mathtt{Kj=![exprj(Xj, Yj)] }$ is rewritten as the conjunction:

$$\begin{aligned} \mathtt{Kj:[exprj(Xj, Yj)] },\lnot \mathtt{morethan(Y_j,K) } \end{aligned}$$

where the predicate morethan is defined as follows:

$$\begin{aligned} \begin{array}{ll} \mathtt{morethan(Y_j,K) } \leftarrow&\mathtt{K1:[exprj(\_,Y_j)],K1>K. } \\ \end{array} \end{aligned}$$

this rule uses the integer comparison $>$ and this could lead to think that its use adds expressive power to our language. However, this is not the case and in the following we provide a further rewriting that only uses the inequality operator $\ne$:

$$\begin{aligned} \begin{array}{ll} \mathtt{morethan(Y_j,K) } \leftarrow&\mathtt{exprj(Z_j,Y_j), } \\ \mathtt{ }&\mathtt{K:[exprj(X_j,Y_j),X_j\ne Z_j]. } \\ \end{array} \end{aligned}$$

Similarly, in the case of FS-assert constructs we have shown, in Sect. 3.3, their rewriting which uses integer comparison and the sum operator as follows:

$$\begin{aligned} \begin{array}{ll} \mathtt{ lessthan(1, K) } \leftarrow&\mathtt{\! K \ge 1. } \\ \mathtt{ lessthan(J1, K) } \leftarrow&\mathtt{\! lessthan(J, K), K\! > \! J, J1=J+1. } \\ \end{array} \end{aligned}$$

these rules can be rewritten as follows:

$$\begin{aligned} \begin{array}{ll} \mathtt{ a(1,0). }&\mathtt{ } \\ \mathtt{ a(1,1). }&\mathtt{ } \\ \mathtt{ a(K,1) } \leftarrow&\mathtt{\!K:[a(K1,X)] . } \\ \mathtt{ b(1,K,0) } \leftarrow&\mathtt{a(K,1). } \\ \mathtt{ b(1,K,1) } \leftarrow&\mathtt{a(K,1). } \\ \mathtt{ b(K,K1,1) } \leftarrow&\mathtt{a(K1,\_),\!K:[b(K2,K1,X),K1\ne K2]. } \\ \mathtt{ lessthan(K,K1) } \leftarrow&\mathtt{b(K,K1,1). } \\ \end{array} \end{aligned}$$

where predicate a generates all integer numbers from 1 until the maximum precision (as the first rule of the lessthan predicate in the previous two rules). Then, the 4th and 5th rule, generate, for each integer, two b predicates and the 6th rule, given a number K1, generates as many occurrences of predicate b as K1. Finally, the last rule will generate as many predicates lessthan as the required K.

Appendix C: Stratified aggregates

The expressive power of $D^a$ on unordered domains (genericity), is also of significant theoretical interest. This issue has been studied by Mumick and Shmueli in [22], where the basic arithmetic functions $(+,-,*, \ldots )$ were also allowed as built-in. Even with these extensions, $D^a$ cannot express the generalized part explosion query of Example 15. In the rest of this section, we will express Datalog aggregates in Datalog$^{FS}\,$. Then, from Example 15 we can conclude that, for unordered domains:

Theorem 7

$D^a \subsetneqq$ Datalog$^{FS}\,$

Let us now discuss how to express $D^a$ in Datalog$^{FS}\,$. Mumick and Shmueli describe Datalog with stratified aggregates as an extension of Datalog that permits a “Groupby” predicates of the form

$$\begin{aligned} \mathtt{GROUPBY }(\mathtt{r }(\overline{\mathtt{t }}),[\mathtt{Y }_\mathtt{1 }, \mathtt{Y }_\mathtt{2,\ldots , }\mathtt{Y }_\mathtt{m }],\mathtt{Z }_\mathtt{1 }= \mathtt{A }_\mathtt{1 }(\mathtt{E }_\mathtt{1 }(\overline{\mathtt{t }})), \mathtt{\ldots ,Z }_\mathtt{n }=\mathtt{A }_\mathtt{n }(\mathtt{E }_\mathtt{n } (\overline{\mathtt{t }}))) \end{aligned}$$

to appear as subgoals in Datalog body rules. The “Groupby” predicate takes as arguments: a predicate r with its attribute list $\overline{\mathtt{t }}$, a grouping list $[\mathtt{Y }_\mathtt{1 },\mathtt{Y }_\mathtt{2,\ldots },\mathtt{Y }_\mathtt{m }]$ contained in $\overline{\mathtt{t }}$ and a set of aggregation terms $\mathtt{Z }_\mathtt{1 }=\mathtt{A }_\mathtt{1 }(\mathtt{E }_\mathtt{1 }),\mathtt{\ldots ,Z } _\mathtt{n }=\mathtt{A }_\mathtt{n }(\mathtt{E }_\mathtt{n })$. For each aggregation term $\mathtt{Z }_\mathtt{i }=\mathtt{A }_\mathtt{i } (\mathtt{E }_\mathtt{i }(\overline{\mathtt{t }})), \mathtt{Z }_\mathtt{i }$ is a new variable, $\mathtt{E }_\mathtt{i }(\overline{\mathtt{t }})$ is an arithmetic expression that uses the variables $\overline{\mathtt{t }}$ and $\mathtt{A }_\mathtt{i }$ is an aggregate operator, for example sum, count, max, min and avg. Stratification means that if a derived relation $r1$ is defined by applying aggregation on a derived relation $r2$, then $r2$’s definition does not depend, syntactically, on relation $r1$.

Example 23

Stratified aggregate.

$$\begin{aligned} \begin{array}{ll} \mathtt{p(Y,Z_1,Z_2) } \leftarrow&\mathtt{GROUPBY(r(\_,Y,F),[Y],Z_1=avg(F),Z_2=sum(F)). } \\ \end{array} \end{aligned}$$

This program returns for each Y value in r the average and sum of value assumed by F in r. $\square$

We can express the predicate “Groupby” by using Datalog$^{FS}\,$. Without loss of generality we can consider only one aggregation term. In fact for each rule of the form

$$\begin{aligned} \begin{array}{ll} \mathtt{p(Y_1,\ldots ,Y_m,Z_1,\ldots ,Z_n) } \leftarrow&\mathtt{GROUPBY(r(\overline{t}), [Y_1,\ldots ,Y_m], } \\ \mathtt{ }&\mathtt{Z_1=A_1(E_1(\overline{t})), \ldots ,Z_n=A_n(E_n(\overline{t}))). } \\ \end{array} \end{aligned}$$

we can rewrite the program in the following way:

$$\begin{aligned} \begin{array}{ll} \mathtt{r_0:p(Y_1,\ldots ,Y_m,Z_1,\ldots ,Z_n) } \leftarrow&\mathtt{p_1(Y_1,\ldots ,Y_m,Z_1),\ldots } \\ \mathtt{ }&\mathtt{\ldots ,p_n(Y_1,\ldots ,Y_m,Z_n). } \\ \mathtt{r_1:p_1(Y_1,\ldots ,Y_m,Z_1) } \leftarrow&\mathtt{\!\!\! GROUPBY(r(\overline{t}),[Y_1,\ldots ,Y_m],Z_1=A_1(E_1(\overline{t}))). } \\ \mathtt{\vdots }&\mathtt{ } \\ \mathtt{r_n:p_n(Y_1,\ldots ,Y_m,Z_n) } \leftarrow&\mathtt{\!\!\! GROUPBY(r(\overline{t}), [Y_1,\ldots ,Y_m],Z_n=A_n(E_n(\overline{t}))). } \\ \end{array} \end{aligned}$$

where $r_1,\ldots ,r_n$ are rules with “GROUPBY” predicate with only one aggregation term.

Count aggregate We start by showing how the “GROUPBY” predicate with count aggregate can be express in Datalog$^{FS}\,$. Consider the following rule

$$\begin{aligned} \begin{array}{ll} \mathtt{p(\overline{y},Z) } \leftarrow&\mathtt{GROUPBY(r(\_,\overline{y}, \overline{x}),[\overline{y}],Z=count(\overline{x})). } \\ \end{array} \end{aligned}$$

where $\overline{\mathtt{y }}$ is the list of grouping variables, $\overline{\mathtt{x }}$ is the list of count variables. The following program can be rewrite in this way:

$$\begin{aligned} \begin{array}{ll} \mathtt{p( \overline{y},Z) } \leftarrow&\mathtt{r(\_, \overline{y},\_ ), Z=![r^{\prime }( \overline{y},\overline{x})]. } \\ \mathtt{r^{\prime }( \overline{y},\overline{x}) } \leftarrow&\mathtt{r(\_ , \overline{y},\overline{x} ). } \\ \end{array} \end{aligned}$$

where the rule with predicate $r^{\prime }$ represents a projection over relation r.

Sum aggregate The next step consists to show how to use the aggregate sum on decimal number. Suppose that we have the scale-up factor $\mathtt{d=10 }^\mathtt{n }$ that represent the decimal precision number (see Sect. 7) where n is large enough to represent the decimal numbers used. The rule with sum aggregate has the following form:

$$\begin{aligned} \begin{array}{ll} \mathtt{p(\overline{y},Z) } \leftarrow&\mathtt{GROUPBY(r(\_,\overline{y}, \overline{x}),[\overline{y}],Z=sum(E(\overline{y},\overline{x})). } \\ \end{array} \end{aligned}$$

where E is a expression composed by built-in functions $(+,-,*, \ldots )$ that use some variables in $\overline{\mathtt{y }}$ and $\overline{\mathtt{x }}$. Initially, we compute for each fact of relation r the expression $\mathtt{E }(\overline{\mathtt{y }},\overline{\mathtt{x }})$ and we convert such value in a multiplicity by distinguishing its sign.

$$\begin{aligned} \begin{array}{ll} \mathtt{r^{\prime \prime }(\overline{y},\overline{x},+):K } \leftarrow&\mathtt{r(\_,\overline{y},\overline{x}),Z=E(\overline{y},\overline{x}), Z>0,K=Z*d. } \\ \mathtt{r^{\prime \prime }(\overline{y},\overline{x},-):K } \leftarrow&\mathtt{r(\_,\overline{y},\overline{x}),Z=E(\overline{y}, \overline{x}),Z<0,K=-Z*d. } \\ \end{array} \end{aligned}$$

As last we obtain the sum aggregate value by subtract the sum $\mathtt{Z }^\mathtt{- }$ of negative numbers from the sum of positive numbers $\mathtt{Z }^\mathtt{+ }$ in the following way:

$$\begin{aligned} \begin{array}{ll} \mathtt{p(\overline{y},Z) } \leftarrow&\mathtt{r(\_, \overline{y},\_ ),Z^+=![r^{\prime \prime }(\overline{y},\overline{x},+)], } \\ \mathtt{ }&\mathtt{Z^-=![r^{\prime \prime }(\overline{y},\overline{x},-)],Z=(Z^+-Z^-)/D. } \\ \end{array} \end{aligned}$$

Note that the result value is scaled up by using a factor D.

Average aggregate The average aggregate can be obtained by combining the sum and count aggregate. Thus given the average aggregate rule:

$$\begin{aligned} \begin{array}{ll} \mathtt{p(\overline{y},Z) } \leftarrow&\mathtt{GROUPBY(r(\_,\overline{y}, \overline{x}),[\overline{y}],Z=avg(E(\overline{y},\overline{x})). } \\ \end{array} \end{aligned}$$

it is possible to rewrite it in Datalog$^{FS}\,$by using the following rule:

$$\begin{aligned} \begin{array}{ll} \mathtt{p(\overline{y},Z) } \leftarrow&\mathtt{r(\_, \overline{y},\_ ),Z^+=![r^{\prime \prime }(\overline{y},\overline{x},+)],Z^-=![r^{\prime \prime }(\overline{y},\overline{x},-)], } \\ \mathtt{ }&\mathtt{K=![r(\_, \overline{y},\overline{x})],Z=(Z^+-Z^-)/(K*D). } \\ \end{array} \end{aligned}$$

where the rule that defines the predicates $r^{\prime \prime }$ is the rule used in sum aggregate.

Min and max aggregates For min and max aggregate it is sufficient to use the Negation-stratified Datalog. It follows that for the min aggregate rule:

$$\begin{aligned} \begin{array}{ll} \mathtt{p(\overline{y},Z) } \leftarrow&\mathtt{GROUPBY(r(\_,\overline{y}, \overline{x}),[\overline{y}],Z=min(E(\overline{y},\overline{x})). } \\ \end{array} \end{aligned}$$

we have the following Negation-stratified Datalog:

$$\begin{aligned} \begin{array}{ll} \mathtt{p(\overline{y},Z) } \leftarrow&\mathtt{r(\_, \overline{y}, \overline{x}), Z=E(\overline{y},\overline{x}), \lnot p^{\prime }(\overline{y},\overline{x}). } \\ \mathtt{p^{\prime }(\overline{y},\overline{x}) } \leftarrow&\mathtt{r(\_, \overline{y}, \overline{x}),r(\_, \overline{y}, \overline{x}^{\prime }),E(\overline{y},\overline{x}^{\prime }) <E(\overline{y},\overline{x}). } \\ \end{array} \end{aligned}$$

We can obtain the max aggregate by change the $\mathtt{E }(\overline{\mathtt{y }},\overline{\mathtt{x }}^{\prime } )<\mathtt{E }(\overline{\mathtt{y,x }})$ atom in the second rule with $\mathtt{E }(\overline{\mathtt{y,x }}^{\prime })>E(\overline{y},\overline{x})$.