Continuously maintaining approximate quantile summaries over large uncertain datasets

Highlights

• We define quantile over uncertain datasets in terms of probabilistic cardinality. We develop a novel algorithm, namely uGK, to compute approximate quantile summaries over uncertain datasets incrementally. We theoretically analysis the complexity of the uGK algorithm, and experimentally verify the efficiency of our algorithm. Using only little space, our uGK algorithm obtains summaries that can support any quantile query within a given error.

Abstract

Quantile summarization is a useful tool for management of massive datasets in the rapidly growing number of applications, and its importance is further enhanced with uncertainty in the data being explored. In this paper, we focus on the problem of computing approximate quantile summaries over large uncertain datasets. On the basis of GK [14] algorithm, we propose a novel online algorithm namely uGK. Using only little space, the proposed uGK algorithm maintains a small set of tuples, each of which contains a point value and the “count” of uncertain elements that are not larger than this value, and supports any quantile query within a given error. Experimental evaluation on both synthetic and real-life datasets illustrates the effectiveness of our uGK algorithm.

Introduction

Over recent years massive datasets generated in the presence of uncertainty have been increasingly common in numerous applications, e.g., sensor networks, environmental monitoring, moving object management, data cleaning, and integration. The uncertainty in these applications results from unreliable data transferring, imprecise measurement, repeated sampling, privacy protection, and so forth [18], [28]. The applications have created the demand for efficiently processing and managing massive uncertain datasets, which has gradually become the first-class issue in modern database systems [6], [8]. To this end, the first and fundamental operation is using effective data reduction techniques to compress large amounts of uncertain data down to summaries with capturing important portraits of the original one [8], [50]. Similar to their deterministic counterpart, such summaries provide the foundation for answering query processing, query plan and optimization, and statistical data analysis over uncertain data [6], [8], [50].

Among various data reduction techniques [1], quantile summarization is the one that is able to efficiently characterize the distribution of a real world dataset. Informally, a quantile is the element at a specified position of a sorted data sequence. Recently, most studies have focused on estimating approximate quantiles, in which error bound is used to improve space and time efficiency. In addition to the fields above, approximate quantile is also very useful to data mining, database parallelization, and management of data streams [14]. Hence, it is interesting and important to compute approximate quantile summaries on massive uncertain datasets. An estimate of such summaries over massive datasets with uncertainty can provide the distribution function induced by such datasets with reasonable precision approximation, supporting aforementioned uses. Some more concrete examples are as follows.

• In sensor networks, sensors are commonly deployed in an ad-hoc fashion to monitor various physical environments, such as temperature, sound, light intensity, and so forth [19], [40]; for energy conservation, individual sensor nodes are usually designed to transmit the distribution of the uncertain sensor values to base stations, thereby users can pose sophisticated queries or analysis into the sensor data [15], [19].

• In data stream classification applications, take credit card fraud detection for example, private information of customer such as age, address, and occupation may be masked by imprecise values in publishing for the purpose of data mining; the distribution of an uncertain data stream can be used as sufficient information for constructing classification models, such as very fast decision trees [11].

• In management of uncertain datasets, the approximation of the distribution of the datasets can be used to implement a query optimization or aggregate query processing, such as estimating cardinality on two uncertain attributes [6], [41].

Despite the practical importance above, there have been very few works proposed to compute summary of uncertain data (we review related work in next section). Existing important works include computing histogram-based, wavelet-based, and essential aggressive-based summaries on probabilistic data [6], [8], [20], [47]. These works only consider categorical or static data; however, uncertain numerical data, which is frequently generated in an incremental or stream fashion [11], [32], is ubiquitous in many application domains. Take the applications above for example, the attributes of interest, such as temperature, sound, age, and income, are generated continuously with uncertainty represented by continuous probability distributions, e.g., uniform or Gaussian distributions [10], [28]. In our previous work [32], to classify uncertain data streams, we gave a somewhat simplistic solution to approximate incrementally the Gaussian distribution of these streams. The solution, however, does not bound the error between obtained distribution and the real one.

Motivated by the above, we in this paper consider the problem of incrementally computing approximate quantile summaries over a large dataset with numerical uncertainty. Generally, the definition of quantile is related to an ordered sequence; however, data elements with uncertainty cannot be sorted. Thus, instead of defining quantile according to the sorted data, we introduce quantile over uncertain data based on probabilistic cardinality. On the basis of GK algorithm [14], we propose an algorithm, namely uGK, for computing quantile summaries over uncertain data online. The proposed uGK algorithm uses similar data structure to store the summary and has similar space complexity to those of GK. In computing process, it makes only a single scan on uncertain data and requires very little space, but obtains a summary that can approximate the real observed distribution within a given error. Our main contributions can be summarized as follows:

• We give the definition of quantile over uncertain datasets based on probabilistic cardinality, and state the problem of computing quantiles over such datasets.

• On the basis of GK algorithm, we develop uGK algorithm to compute quantile summaries over uncertain datasets incrementally.

• We theoretically analyze space and time complexity of our uGK algorithm.

• We conduct a comprehensive experimental study on both real and synthetic datasets, illustrating the effectiveness of our proposed uGK algorithms.

In the rest of this paper, we firstly discuss related works in Section 2. Formal problem definition is described in Section 3. We discuss the details of our uGK algorithm in Section 4. Later, in Section 5 we analyze the time and space complexity of our uGK algorithm. Finally, the experimental study is presented in Section 6, and we conclude our work in Section 7.