A near-optimal similarity join algorithm and performance evaluation

Abstract

Similarity join, a basic operation for multi-media databases, amounts to combinations of all pairs of points, with the distance between each pair bounded by a given parameter ε. In this paper, properties of index-based join algorithms are studied and a highly efficient and near-optimal similarity join algorithm is proposed. Our algorithm utilizes the Breadth-First strategy, and guides the join computation and I/O access through the cache content. In contrast with many other proposed join algorithms, our algorithm is advantageous due to the essential independence of the ordering strategies and the minimal cache capacity requirement. As a result, a more precise plan for the sequence of join computations and I/O access can be realized. Generally, processing and accessing each page can be done with only one attempt. Qualitative and quantitative analysis of the performance of the algorithm is provided. Although only R-tree (a common index structure) based similarity join is discussed in this paper, the idea can be generalized to implement other join algorithms without substantial difficulties. Experiments based on our analysis indicate that the new algorithm yields superior performances across a wide range of dimensions and sizes of databases.

Introduction

The application of similarity join, a basic operation for database queries, ranges widely from reporting “similar” pairs in a multimedia database to speeding up data mining processes. In fact, similarity join can be considered as certain combinations of a series of similarity queries.

In order to retrieve data objects efficiently, similarity query strategies are usually required to transform data objects into multi-dimensional points (feature vector), based on the index structure. Accordingly, similarity query amounts to the query of multi-dimensional feature points in index structure. Similarity join combines multi-dimensional point sets so that the resulted sets contain all “close” pairs of points. Although many index-based join algorithms have appeared [5], [6], [11], index-based approaches are thought recently [5] to be inefficient, particularly in high-dimensional case. For example, as pointed out in [5], a serious optimization conflict may exist between CPU time and I/O time, that is, while fine-grained index structures seem to be efficient for CPU, it tends to deteriorate the I/O performance.

In this paper, we first investigate the properties of the index-based similarity join. The comparison of the features of several widely used join algorithms––R-tree spatial join (RSJ) [6], breadth first R-tree join (BFRJ) [11] and multi-page index similarity join (MISJ) [5]––shows that, to achieve local and global optimization simultaneously, the index-based join needs to be computed by levels (breadth-first), and join computations and I/O access should be guided by cache content (The word “cache” always represents the application cache in this paper) so that one may plan the sequence of I/O access and join computations a priori. However, as opposed to many reported results, it is pointed out that the index-based join algorithm is in itself independent of the ordering strategies, such as Plane-Sweep, space-filling curves. Next, a near-optimal similarity join (NOSJ) algorithm is proposed. The algorithm guides the join computations and I/O access through the cache content, nearly minimizes the capacity of the demanded cache and I/O access times, and efficiently utilizes the locality of the cache. Theoretical analysis and experiments show that, generally, processing and accessing each page can be completed with only one attempt. Moreover, how the performance of the NOSJ algorithm can be optimized regarding the CPU and the I/O costs is discussed. Considering its superior performance, we base our experiments for NOSJ on R-tree. Our experiments show that NOSJ yields superior performances across a wide range of dimensions and sizes of databases.

This paper is organized as follows. In Section 2 related work and backgrounds are discussed. Our NOSJ algorithm is described in Section 3. In Section 4 of this paper, we optimize the NOSJ algorithm from two aspects: CPU cost and I/O cost. After evaluating the NOSJ experimentally in Section 5, we conclude this paper in Section 6.