PPP-Codes for Large-Scale Similarity Searching

Abstract

Many current applications need to organize data with respect to mutual similarity between data objects. A typical general strategy to retrieve objects similar to a given sample is to access and then refine a candidate set of objects. We propose an indexing and search technique that can significantly reduce the candidate set size by combination of several space partitionings. Specifically, we propose a mapping of objects from a generic metric space onto main memory codes using several pivot spaces; our search algorithm first ranks objects within each pivot space and then aggregates these rankings producing a candidate set reduced by two orders of magnitude while keeping the same answer quality. Our approach is designed to well exploit contemporary HW: (1) larger main memories allow us to use rich and fast index, (2) multi-core CPUs well suit our parallel search algorithm, and (3) SSD disks without mechanical seeks enable efficient selective retrieval of candidate objects. The gain of the significant candidate set reduction is paid by the overhead of the candidate ranking algorithm and thus our approach is more advantageous for datasets with expensive candidate set refinement, i.e. large data objects or expensive similarity function. On real-life datasets, the search time speedup achieved by our approach is by factor of two to five.

Appendices

Appendix I: Correctness of Algorithm 2

Lemma 1

If d maintains Eq. (8) then Algorithm 2 GetNextIDs(q, j) returns $\mathrm {ID}$s of objects with the lowest j-th ranking $\psi _q^j(x)$, $j\in \varLambda$ that were not returned so far.

Proof

The algorithm returns $\mathrm {ID}$s from Q containing nodes and $\mathrm {ID}$s from the j-th PPP-Tree. Because every node and $\mathrm {ID}$ is inserted into Q maximally once, the algorithm always returns something, unless all $\mathrm {ID}$s were returned. Q is ordered by $d(q,\langle i_1,\ldots ,i_{l'}\rangle )$ where $\langle i_1,\ldots ,i_{l'}\rangle$ is either path to a node or it is equal to $\varPi _x^j(1..l)$ for $\mathrm {ID}_x$ (recall that $d(q,\varPi _x^j(1..l))$ generate $\psi _q^j(x)$). Let $\mathrm {ID}_x$ be returned by the algorithm; we prove the lemma by contradiction. Let us assume that there exists : $d(q,\varPi _y^j(1..l))<d(q,\varPi _x^j(1..l))$ and $\mathrm {ID}_y$ was not returned by the algorithm. If $\mathrm {ID}_y$ is in Q then it must be ahead of $\mathrm {ID}_x$ (contradiction). Thus, $\mathrm {ID}_y$ is not in Q, but Q must contain a node with path $\langle i_1,\ldots ,i_{l'}\rangle$, $l'\le l$ such that $\langle i_1,\ldots ,i_{l'}\rangle =\varPi _y^j(1..l')$, because Q initially contains root of j-th PPP-Tree and then recursively all child nodes are inserted into Q (line 8). Because of (8), $d(q,\langle i_1,\ldots ,i_{l'}\rangle )\le d(q,\varPi _y^j(1..l))<d(q,\varPi _x^j(1..l))$ which is in contradiction with the fact that $\mathrm {ID}_x$ was on top of Q.

Appendix II: Optimizations of Algorithm 2

Complexity of the GetNextIDs routine is $O(|Q|\cdot \log |Q|)$ and the length of Q depends on “tightness” of Eq. (8). We propose the following optimizations for the $d_{\varDelta }$ distance.

Optimization 1. Distance $d_{\varDelta }(q,\varPi (1..l'))$ between and PP prefix on level $l'<l$ is corrected so that it returns the minimum theoretical distance to PPP-Codes on level l with prefix $\varPi (1..l')$:

$$\begin{aligned} d_{\varDelta }'(q,\varPi (1..l')) = d_{\varDelta }(q,\varPi (1..l') \oplus \varPi _q(1..l\!-\!l')). \end{aligned}$$

Notation $\oplus \, \varPi _q(1..l\!-\!l'))$ is concatenation of the pivot indexes closest to the query. This addition does not break the condition (8) but, in our test cases, it resulted to reduction of the queue length by factor of 0.4–0.7.

Optimization 2. This optimization is relatively trivial: A leaf of the PPP-Tree at level $l'<l$ can keep $\mathrm {ID}$s with the same PP suffix together as $\langle \varPi (l'\!+\!1..l);\mathrm {ID}_{x_1},\ldots ,\mathrm {ID}_{x_m}\rangle$ (see Sect. 4.1 for the original proposal); the list of $\mathrm {ID}$s can be further optimized e.g. using delta encoding . This results in index memory reduction and, especially, in a slight reduction of the Q size, because such entry is inserted to Q only once.

Optimization 3. Another important cost component of the GetNextIDs algorithm are distances $d(q,\langle i_1,\ldots ,i_{l'},i_{l'+1}\rangle )$ evaluated for each item added into Q (lines 8 and 12). If the formula of distance d is a sum of independent values for each level from 1 to $l'\!+\!1$ (as the $d_{\varDelta }$ distance $(4^{\prime })$) then value of $d(q,\langle i_1,\ldots ,i_{l'},i_{l'+1}\rangle )$ can be calculated as a sum of distance of its parent node $dist =d(q, \langle i_1,\ldots ,i_{l'}\rangle )$ plus the addend for level $l'+1$. In this case, the distances are calculated stepwise and no calculations are repeated.