HIFUN - a high level functional query language for big data analytics

Abstract

We present a high level query language, called HIFUN, for defining analytic queries over big datasets, independently of how these queries are evaluated. An analytic query in HIFUN is defined to be a well-formed expression of a functional algebra that we define in the paper. The operations of this algebra combine functions to create HIFUN queries in much the same way as the operations of the relational algebra combine relations to create algebraic queries. The contributions of this paper are: (a) the definition of a formal framework in which to study analytic queries in the abstract; (b) the encoding of a HIFUN query either as a MapReduce job or as an SQL group-by query; and (c) the definition of a formal method for rewriting HIFUN queries and, as a case study, its application to the rewriting of MapReduce jobs and of SQL group-by queries. We emphasize that, although theoretical in nature, our work uses only basic and well known mathematical concepts, namely functions and their basic operations.

Appendix

Proof Proof of Proposition 1

Let A and B be two nodes of $\mathcal {C}$,and suppose there is a cycle on A consisting of two functions:f : A → B and g : B → A.Suppose that there is some element a in A such thatg ∘ f(a) = a^′and a≠a^′.This implies that a^′depends on a, a contradiction to our assumption that the nodes of$\mathcal {C}$represent sets of independent values. Therefore the only possibility to have a cycle on A is wheng ∘ f(a) = a for all a in A; in other words the only possible cycle on A isι_A,where ι_Ais the identity function on A. □

Proof Proof: Proposition 2

Observe first that, as ans_Qis a function with source B, the query Q^″is well formed, that is, its grouping function g and its measuring functionans_Qhave the same source (namely B), and op is an operation on the target ofans_Q.

Let c ∈ C.It follows from well known properties of functions that:

ifg^− 1(c) = {b₁,…,b_k}then$(g \circ f)^{-1}(c) = f^{-1}(b_{1}) \bigcup {\ldots } \bigcup f^{-1}(b_{k})$

From our definition of answer, we have:

$ans_{Q^{\prime }}(c) = red(m/(g \circ f)^{-1}(c), op)$

As op is a distributive operation and the family{f^− 1(b₁),…,f^− 1(b_k)}is a partition of (g ∘ f)^− 1(c),we have:

$$\begin{array}{@{}rcl@{}} ans_{Q^{\prime}}(c) &=& red(m/(g \circ f)^{-1}(c), op) \\ &=& op(red(m/f^{-1}(b_{1}), op), \ldots, red(m/f^{-1}(b_{k}), op)) \\ &=& op(ans_{Q}(b_{1}), \ldots, ans_{Q}(b_{k})) [because red(m/f^{-1}(b_{i}), op))= ans_{Q^{\prime}}(b_{i})\\ &=& red(m/g^{-1}(c), op) \\ &=& ans_{Q^{\prime\prime}}(c) \end{array} $$

Therefore $ans_{Q^{\prime }}(c) = ans_{Q^{\prime \prime }}(c)$for all c in C and this concludes the proof. □

Proof Proof of Proposition 3.1

Suppose first that there is a function h : A → B such that g = h ∘ f.Then for all a, a^′in A such that f(a) = f(a^′)we have: g(a) = h(f(a)) = h(f(a^′)) = g(a^′).Therefore f ≤ g.Suppose next that f ≤ g.It follows that π_f ≤ π_g.Consider now any b in the range of f and define:h(b) = g(f^− 1(b)).As f ≤ g,the block f^− 1(b)of π_fis included in some block of π_g,say g^− 1(c),where c is in the range of g. It follows that:g(f^− 1(b)) = c,therefore h is a well defined function over the rangeof f . Moreover, from the definition of h we have:h(f(a)) = g(f^− 1(b)) = g(b).Therefore h ∘ f = g.Finally, suppose there is a function h^′ : A → B such that h^′≠h and h^′∘ f = g = h ∘ f.Now, for any b in the range of f there is a in A such thatf(a) = b.Therefore h^′(b) = h^′(f(a)) = (h^′∘ f)(a) = g = (h ∘ f)(a).It follows that: h^′/range(f) = h/range(f)and this completes the proof. □