A fast and progressive algorithm for skyline queries with totally- and partially-ordered domains

Abstract

We devise a skyline algorithm that can efficiently mitigate the enormous overhead of processing millions of tuples on totally- and partially-ordered domains (henceforth, TODs and PODs). With massive datasets, existing techniques spend a significant amount of time on a dominance comparison because of both a large number of skyline points and the unprogressive method of skyline computing with PODs. (If data has high dimensionality, the situation is undoubtedly aggravated.) The progressiveness property turns out to be the key feature for solving all remaining problems. This article presents a FAST-SKY algorithm that deals successfully with these two obstacles and improves skyline query processing time strikingly, even with high-dimensional data. Progressive skyline evaluation with PODs is guaranteed by new index structures and topological sorting order. A stratification technique is adopted to index data on PODs, and we propose two new index structures: stratified R-trees (SR-trees) for low-dimensional data and stratified MinMax treaps (SM-treaps) for high-dimensional data. A fast dominance comparison is achieved by using a reporting query instead of a dominance query, and a dimensionality reduction technique. Experimental results suggest that in general cases (anti-correlated and uniform distributions) FAST-SKY is orders of magnitude faster than existing algorithms.

Introduction

The database research community has recently given considerable attention to skyline computation, particularly for progressive algorithms that can immediately return intermediate skyline results. Using the common definition in the literature, given a set of objects $p_1\text{,}{p}_2\text{,}\dots \text{,}{p}_N$, the skyline operator returns all objects $p_i$ such that $p_i$ is not dominated by another object $p_j$. The underlying query type in skyline computation is known to be a dominance query, which requires a series of pairwise comparisons.

Since the skyline operator was formally introduced in Börzsönyi et al. (2001), many researchers have devised novel algorithms for skyline queries in various environments (see Balke et al., 2004, Papadias et al., 2005, Chan et al., 2005, Sacharidis et al., 2009, Kossmann et al., 2002, Pei et al., 2006, Wang et al., 2007, Lee et al., 2007, Lian and Chen, 2008, Vlachou et al., 2008). Most skyline evaluation methods developed recently are aimed at efficient computation with TODs, and we have seen remarkable progress in that area. On the other hand, skyline computation with TODs and PODs has remained in a premature state since it was first tackled by Chan et al. (2005) until recent work by Sacharidis et al. (2009) advances the performance of skyline processing with PODs using topological sort.

The performance of skyline computation is mainly affected by two activities. The first one, which has already been researched extensively, is the IO overhead of accessing data. Many researchers think that reducing the number of IO accesses to a disk would have a great benefit, and lots of excellent algorithms showed substantial improvements. However, the IO overhead becomes less meaningful as data increases to millions of records. Rather, the CPU cost of carrying out a series of dominance checks with existing skyline points becomes the major factor in performance overhead, which is the second activity. Existing skyline techniques do not fully take the second type of overhead into consideration, and as a result, leave much room for further enhancement. In addition, if data has high dimensionality, the entire overhead grows nontrivially; in particular, skyline processing with high-dimensional data on PODs has not yet been researched thoroughly.

The main focus of this paper is on developing a fast and progressive algorithm for skyline queries with TODs and PODs. We propose the FAST-SKY algorithm that has the progressiveness property with TODs and PODs, and as a result, FAST-SKY can use a reporting query in dominance comparisons to tremendous effect. In order to achieve IO-optimality¹ and progressiveness (Tan et al., 2001) in skyline computation with PODs, we adopt a stratification technique and design two new index structures, each of which has merits and demerits. The stratification that is applied to PODs through domain classification, which partitions data into disjoint sets according to a value from POD, enables the FAST-SKY algorithm to perform IO-optimal and progressive skyline evaluation.

The new index structures are stratified R-trees (SR-trees) and stratified MinMax treaps (SM-treaps). Due to the stratification, both index structures are not concerned with attributes in PODs, only attributes in TODs. An SR-tree considers full d-dimensionality of data, while an SM-treap sees only two dimensions, min and max coordinates, when indexing data. An SR-tree has an advantage in that it supports all types of progressive evaluations such as arbitrary dimensionality. The drawback, however, is the severe performance degradation of the R-tree that indexes high-dimensional data $(d>10)$. To overcome the performance deterioration, an SM-treap enforces dimensionality reduction on multidimensional data in such a way that data is represented by two coordinates. It then inserts data into a treap with min (key) and max (priority) values. The merits of an SM-treap are twofold. An SM-treap can index high-dimensional data and support progressive computation, and unlike the R-tree-based algorithm (i.e., BBS (Papadias et al., 2005)), it does not require a heap space to buffer skyline candidates because the preorder traversal of an SM-treap guarantees progressive processing. Eliminating the heap overhead by guaranteeing progressive processing, which was discussed in ZBtree (Lee et al., 2007) using data on an integer domain (TODs only), contributes to a substantial performance gain in our problem domain as well (real TODs and PODs). To the best of our knowledge, this is the first work on space efficiency and the progressiveness property of an SM-treap for high-dimensional data in a real domain.

For an efficient dominance comparison, we use the well-known property in orthogonal range searching that if an intermediate skyline set is an insert-only² set, then a dominance check can be realized by using a reporting query, instead of a dominance query. In other words, if skyline computation is progressive, then a dominance check can be optimized directly by the above property. Because of the progressiveness property, FAST-SKY can also exploit a reporting query. To deal with high-dimensional skyline points as well, we reduce all intermediate skyline points in 2-dimensional space by using a dimensionality reduction technique; we adopt and modify the iMinMax technique (Yu et al., 2004). Performing a dominance check using a reporting query on the reduced dimensions improves the entire query processing time significantly.

The main contributions of this paper follow:

• We address the problem of skyline computation with TODs and PODs even in high-dimensional space. We also note that the CPU cost of a dominance test becomes the major factor in performance overhead as datasets grow over millions of records.

• For progressive skyline evaluation with PODs, we design two new index structures: SR-trees and SM-treaps, each of which has strengths and weaknesses when used for different purposes. In particular, SM-treaps resolve progressive skyline processing with PODs in high dimensional space efficiently.

• We achieve a fast dominance test by using a reporting query on reduced dimensions. An iMax based B+-tree with novel pruning techniques decreases the cost of dominance tests.

The rest of the paper is organized as follows: Section 2 reviews the related work in further detail and discusses its advantages and limitations. Section 3 introduces fundamental definitions. Section 4 gives an in-depth explanation of the FAST-SKY algorithm, and Section 5 experimentally evaluates FAST-SKY and compares it to SDC+ in a variety of settings. Finally, Section 6 concludes the article and describes directions for future work.