Dynamic adaptive data structures for monitoring data streams

Abstract

The monitoring of data streams is a very important issue in many different areas. Aspects such as accuracy, the speed of response, the use of memory and the adaptability to the changing nature of data may vary in importance depending on the situation. Examples such as Web page access monitoring, approximate aggregation in relational queries or IP message routing are clear examples of a varied range of those needs.

There are different data structures that deal with this problem such as the counting bloom filters, the spectral bloom filters and the dynamic count filters. Those data structures range from static to complex dynamic representations of the data stream that keep an approximate count of the number of occurrences for each data value.

In this paper, we focus on three main aspects. First, we analyze the problem in perspective and review the existing static and dynamic solutions. Second, we propose and analyze in depth a simple yet powerful partitioning strategy that reinforces the advantages of the methods proposed up to now solving most of their drawbacks. Finally, using real executions and mathematical models, we evaluate the existing methods alone and in combination with our partitioning strategy. We show that with our partitioning strategy, it is possible to reduce the memory requirements and average response time, improving the adaptiveness to changing data characteristics and leaving the accuracy of the partitioned dynamic data structures intact.

Introduction

Monitoring streamed data is very important in many different scenarios. Examples of such cases are: telecom companies or Internet service providers that need to collect exact or approximate statistics of the access to the services they provide; the large amount of streamed data collected in retail-chain transaction processes that has to be monitored for statistical records; and, in the database management context, cases where it is necessary to approximate aggregate answers to user queries over data stream windows [4], [16], [20], [19], or to estimate the data frequency peaks or join sizes [2], [3].

Thus, the massive streams of data generated by these applications, sometimes, need to be summarized through compact and fast data structures to support queries and mining. One of the typical uses of these summaries is to provide the number of occurrences per data item in the input stream, such that we may provide fast approximate answers to different types of aggregate or join size queries. Moreover, the data structures used with this purpose in mind have to be able to adapt to the heavy-tailed distributions that these streams usually present.

As a whole, this is a complex problem and several data structures have been proposed to keep approximate counts of items using the smallest amount of memory possible and a fast per-item response time. The data structures proposed in this area are the counting bloom filters (CBF) [17], and their dynamic extensions: the spectral bloom filters (SBF) [10] and the dynamic count filters (DCF) [1]. CBF are a counter-oriented extension of bloom filters [6], where each presence bit of a bloom filter is substituted by a fixed size bit-counter. CBF may saturate and fail in their mission of keeping an accurate track of counts with skewed data. As an alternative, the SBF were designed to dynamically adapt the size of the counters to the characteristics of the data. SBF are composed of variable-sized counters that allow for real adaptiveness but fail to provide fast access times because of the indexing structures they require. Finally, DCF allow for dynamic environments with unlimited size counters and fast access times. However, they waste a large percentage of the memory they allocate in the presence of data skew, and fail to provide fast average response times because of the heavy reconstruction phases that they require.

In this paper, apart from reviewing and analyzing all the data structures mentioned above thoroughly, we propose a partitioning strategy that, once applied to a dynamic approach such as SBF and DCF, solves all the problems mentioned above. On one hand, it helps to make SBF and DCF insensitive to changes in the characteristics of the data for the following reasons. First, it minimizes the amount of memory required at any time without limiting the counting possibilities of the methods, facilitating the use of dynamic methods in hardware implementations where memory may be a restriction. Second, it reduces the average response time because it simplifies the painful reconstruction phases of the previous non-partitioned methods, reducing the worst case response times, and adapting better to high data streamed frequencies. Third, it assures and, under some circumstances, it improves the accuracy of previous approaches, making it possible to use it in environments where accuracy is important but memory is again a restriction.

On the other hand, under stress situations, like changes in the intensity of the data flux, our partitioning strategy allows for fast response and high accuracy. The degradation shown by SBF in such situations, is reduced significantly by our partitioned strategy combined with DCF.

In order to make this paper self-contained and to perform a thorough analysis of the methods studied and proposed, we make the following contributions:

• We analyze the streamed data monitoring problem in perspective, reviewing and understanding the previous literature on the topic, and in particular CBF, SBF and DCF. The analysis allows us to understand the benefits and drawbacks of these structures.

• We propose a generic partitioning strategy that can be easily adapted to dynamic approaches in general at a low cost, significantly improving their characteristics as mentioned above. Our partitioning strategy tackles the drawbacks of SBF and DCF with clear benefits in all the aspects mentioned above.

• We present mathematical models that allow for two different analysis: (i) a memory comparison of the four approaches and (ii) a comparison of the complexity of the insert/delete/query/rebuild operations for each of the data structures. The models that we propose are very helpful in order to show the most interesting features of each data structure and to understand the results obtained through our real tests.

• We evaluate all the strategies analyzed in the paper, including the partitioned and non-partitioned SBF and DCF, which is the first profound evaluation of such monitoring data structures. We do that in a set of very different scenarios to give a complete view of their characteristics.

• As a general view of our contributions, our results show that our partitioning strategy is robust because it satisfies the constraints imposed by data stream environments, improving significantly in terms of memory space, response time, accuracy and versatility compared to the use of SBF and DCF alone.

This paper is organized as follows: In Section 2 we enumerate the different structures used for monitoring data streams and explain a set of scenarios in which the techniques proposed and evaluated in this paper can be used. In Section 3 we describe the counter-based filter structures proposed in the literature. In Section 4, we describe the partitioning strategy that we propose, and its application to previous strategies. Later, in Section 5, we seek the optimal number of partitions for PSBF and PDCF. In Section 6 we model and compare PSBF, PDCF, SBF, and DCF. In Section 7, we present the experimental results for the different scenarios and, finally, we conclude in Section 8.