Exploiting block co-occurrence to control block sizes for entity resolution

Abstract

The problem of identifying duplicated entities in a dataset has gained increasing importance during the last decades. Due to the large size of the datasets, this problem can be very costly to be solved due to its intrinsic quadratic complexity. Both researchers and practitioners have developed a variety of techniques aiming to speed up a solution to this problem. One of these techniques is called blocking, an indexing technique that splits the dataset into a set of blocks, such that each block contains entities that share a common property evaluated by a blocking key function. In order to improve the efficacy of the blocking technique, multiple blocking keys may be used, and thus, a set of blocking results is generated. In this paper, we investigate how to control the size of the blocks generated by the use of multiple blocking keys and maintain reasonable quality results, which is measured by the quality of the produced blocks. By controlling the size of the blocks, we can reduce the overall cost of solving an entity resolution problem and facilitate the execution of a variety of tasks (e.g., real-time and privacy-preserving entity resolution). For doing so, we propose many heuristics which exploit the co-occurrence of entities among the generated blocks for pruning, splitting and merging blocks. The experimental results we carry out using four datasets confirm the adequacy of the proposed heuristics for generating block sizes within a predefined range threshold as well as maintaining reasonable blocking quality results.

Appendices

Employed blocking keys

Let S be a string, $F_n(S)$ be the first n letters of S, nB(S) be the first n bigrams [4] of S and nT(S) be the first n trigrams [4] of S. Using this notation, in Table 13 we present the blocking key functions which have been used in order to index the datasets in the conducted experiments (Sect. 5). Since the entities in the CoraATDV dataset do not present a predefined schema, we have split the entities by white space. Each part of the split result is denoted as SplitLine[k], such that k is the index of the resulting array.

Table 13 Employed blocking key functions

Fixed window size for SN

Let $S_{min}$ and $S_{max}$ be two size parameters, D be a dataset, $\mathcal {F}$ be a set of blocking key functions and $\mathcal {B}$ be the blocking collection generated by indexing D using $\mathcal {F}$. Taking into account the problem presented in Definition 6, the pruned blocking collection $\mathcal {B}_{max}'$ that yields to the maximum aggregated cardinality is composed by ($|D|\cdot |\mathcal {F}|$ div $S_{max}$) blocks containing $S_{max}$ entities and a block containing ($|D|\cdot |\mathcal {F}|$ mod $S_{max}$) entities. Following Definition (1), we can calculate the aggregated cardinality generated by $||\mathcal {B}'_{max}||$ as shown in Eq. (5).

$$\begin{aligned} ||\mathcal {B}'_{max}||&= |\mathcal {F}| \cdot 2^{-1} \cdot \Big ( (|D| \text { div } S_{max}) \cdot S_{max} \cdot (S_{max}-1) \nonumber \\&\quad +\cdot (|D| \text { mod } S_{max}) \cdot ((|D| \text { mod } S_{max}) - 1) \Big ) \end{aligned}$$

(5)

Therefore, by employing a sorted neighborhood (SN) approach with a fixed window size equal to w over the blocks in $\mathcal {B}$, the number of generated comparisons cannot exceed $||\mathcal {B}'_{max}||$. Since the SN algorithm is executed $|\mathcal {F}|$ times (i.e., for each employed blocking key function), the number of comparisons generated by employing the SN method for each blocking key function cannot exceed $\frac{||\mathcal {B}'_{max}||}{|\mathcal {F}|}$. Thus, we need to configure a fixed window size (w) that generates at most $\frac{||\mathcal {B}'_{max}||}{|\mathcal {F}|}$ comparisons.

Since the number of comparisons generated by the SN algorithm can be theoretically estimated [5], we can calculate the maximum window size allowed ($w_{max}$) based on Eq. (6).

$$\begin{aligned}&(w-1) \cdot (|D|-\frac{w}{2}) = \frac{||\mathcal {B}'_{max}||}{|\mathcal {F}|} \nonumber \\&w \cdot |D| - \frac{w^2}{2} - |D| + \frac{w}{2} = \frac{||\mathcal {B}'_{max}||}{|\mathcal {F}|} \nonumber \\&\quad - \frac{w^2}{2} + \frac{(2 \cdot w \cdot |D|) + w}{2} - \frac{||\mathcal {B}'_{max}||}{|\mathcal {F}|} - |D| = 0 \nonumber \\&w^2 - (2 \cdot |D| + 1) \cdot w + 2 \cdot \Big (\frac{||\mathcal {B}'_{max}||}{|\mathcal {F}|} + |D| \Big ) = 0 \end{aligned}$$

(6)

From Eq. (6), we conclude that $w_{max}$ can be calculated as shown in Eq. (7), i.e., the maximum value of w that generates at most $\frac{||\mathcal {B}'_{max}||}{|\mathcal {F}|}$ comparisons between entities.

$$\begin{aligned} w_{max} = \frac{(2 \cdot |D| + 1) + \sqrt{(-2 \cdot |D| + 1)^2 - 4 \cdot 2 \cdot \Big (\frac{||\mathcal {B}'_{max}||}{|\mathcal {F}|} + |D|\Big )}}{2} \end{aligned}$$

(7)

Note that, since our approach can perform shrink and exclude operations over the input blocking collection $\mathcal {B}$, we cannot calculate the minimum number of comparisons to be generated by the pruned blocking collection $\mathcal {B}'$. For this reason, for each dataset, we employ the SN method using two variations of the $w_{max}$ (following Eq. (7)) result.

Fig. 10

Distribution of block sizes on CoraATDV dataset

Block size distribution

We present the distribution of block sizes in the input blocking collection of two datasets (Cora and DBLPM4) employed in the experiments and in the pruned blocking collection generated by one of the proposed algorithms (lECP—see Table 8), in Figs. 10 and 11.

Fig. 11

Distribution of block sizes on DBLPM4 dataset