Towards the efficient parallelization of multi-pass adaptive blocking for entity matching

Highlights

• A new approach for the Entity Matching task parallelization is proposed.

• The idea relies on performing a MapReduce-based multi-pass adaptive blocking strategy.

• The proposed approach shows significant superior performance efficiency.

Abstract

Modern parallel computing programming models, such as MapReduce (MR), have proven to be powerful tools for efficient parallel execution of data-intensive tasks such as Entity Matching (EM) in the era of Big Data. For this reason, studies about challenges and possible solutions of how EM can benefit from this well-known cloud computing programming model have become an important demand nowadays. Furthermore, the effectiveness and scalability of MR-based implementations for EM depend on how well the workload distribution is balanced among all reduce tasks. In this article, we investigate how MapReduce can be used to perform efficient (load balanced) parallel EM using a variation of the multi-pass Sorted Neighborhood Method (SNM) that uses a varying size (adaptive) window. We propose Multi-pass MapReduce Duplicate Count Strategy (MultiMR-DCS++), a MR-based approach for multi-pass adaptive SNM, aiming to increase even more the performance of the SNM. The evaluation results based on real-world datasets and cluster infrastructure show that our approach increases the performance of MapReduce-based SNM regarding the EM execution time and detection quality.

Introduction

Cloud computing has become an important resource for efficiently processing data and computationally intensive application tasks in the era of Big Data  [11]. Extensive powerful distributed hardware and service infrastructures capable of processing millions of these tasks are available around the world. Aiming to make efficient use of such cluster environments, many programming models have been created to deal with a vast amount of data. In this context, MapReduce (MR)  [6], a well-known programming model for parallel processing on cluster infrastructures, has given to the data management community a powerful “chainsaw” to tackle Big Data problems. Its simplicity, flexibility, fault tolerance and capability for being a scalable parallel shared-nothing data-processing make MR an excellent resource for the efficient workload distribution of data-intensive tasks.

In this article, we investigate the MR-based parallelization of data-processing for the complex problem of Entity Matching (EM) (also known as entity resolution, deduplication, record linkage, or reference reconciliation), i.e., the task of identifying entities referring to the same real-world object  [14]. Given the pairwise-comparison nature of the problem, EM is a data-intensive and performance critical task that demands studies on how it can benefit from cloud computing. The task is a fundamental problem in every information integration and data cleansing application  [11], e.g., to find duplicate costumers or to match product descriptions on enterprise datasets. It is also essential for other types of applications, such as web pages deduplication  [3], plagiarism detection  [5] and click fraud detection  [20].

Detecting such similar pairs is challenging nowadays. Besides the need of applying matching techniques on the Cartesian product of all input entities which leads to a computational cost in the order of $O\left(n^2\right)$, there is an increasing trend of applications being expected to deal with vast amounts of data that usually do not fit in the main memory of one machine. This means that the application of such approach is ineffective for Big Data datasets. One way to minimize the workload caused by the Cartesian product execution and to maintain the match quality is to reduce the search space by applying blocking techniques  [11]. Such techniques work by partitioning the input data into blocks of similar entities and restricting the EM process to entities that belong to the same block. For instance, it is sufficient to compare entities of the same manufacturer when matching product offers.

The Sorted Neighborhood Method (SNM)  [8] is one of the most popular blocking approaches. It sorts all entities using an appropriate blocking key, e.g., the first three letters of the entity name, and only compares entities within a predefined (and fixed) distance window $w$. SNM thus reduces the execution complexity to $O(n\cdot w)$ for the actual matching. Fig. 1 shows an execution example of SNM for a window size $w=3$. The input set $\mathrm{Products Source}$ consists of $n=9$ entities (from Camera Samsung DV150 to IPod Nano 16G Original) represented by letters $L$ (from $A$ to $I$). All the entities are sorted according to their blocking key $K$ ($\mathrm{Cam}$, $\mathrm{Iph}$, or $\mathrm{IPo}$) which in turn is composed by the first three letters of the product name. Initially, the window includes the first three entities ($A,D,B$) and generates three pairs of comparisons $[(A-D),(A-B),(D-B)]$. After that, the window is slided down (one entity) to cover the entities $D$, $B$, $E$ and two more pairs of comparisons are generated $[(D-E),(B-E)]$. The sliding process is repeated until the window reaches the last three entities ($C,G,I$). In this process, the number of comparisons generated is $(n-w/2)\cdot (w-1)$.

However, the SNM presents a critical performance disadvantage due to the fixed and difficult to configure window size: if it is selected too small, some duplicates might be missed (e.g., in Fig. 1, the similar pair $B-H$ is not detected). On the other hand, a too large window leads to unnecessary comparisons (e.g., in Fig. 1, the computation of the pair $A-B$ is unnecessary). Note that if effectiveness is more relevant than the execution time, the ideal window size should be equal to the size of the largest duplicate sequence in the dataset. Thus, it is common to request the intervention of a data specialist to solve this tradeoff (small/large window size). To overcome this disadvantage, the authors of  [7] proposed an efficient (not parallel) SNM variation denoted as Duplicate Count Strategy (DCS) that follows the idea of increasing the window size in regions of high similarity and decreasing it in regions of low similarity. They also proved that their improved variant of DCS, known as DCS++, overcomes the performance of traditional SNM by obtaining at least the same matching results with a significant reduction in the number of entity comparisons.

Besides the difficulty in configuring the window size, there is the challenge of choosing a ideal/proper/effective blocking key, which may appear especially when dealing with dirty (inaccurate, incomplete or erroneous) input data. In this case, it may not be sufficient to use a single blocking key to find all duplicates. Even using an adaptive window size, if the blocking is not ideal, it is quite common that the similarity value between distant entities remains considerable. To overcome this problem, the SNM also has a multi-pass variant  [8], in which multiple blocking keys (e.g., using multiple or the combination of entity attributes) are generated. For each generated blocking key, a new window slide is performed over the set of entities sorted according to the respective blocking key. The multi-pass variant is an important resource for the cases in which the effectiveness of the similarity detection is essential.

Even with the significant advances in the SNM design, EM remains a critical task in terms of performance when applied to large datasets. Thus, this work proposes a MR-based approach capable of combining the efficiency gain achieved by the DCS++ method with the benefit of efficient parallelization of data-intensive tasks in cluster infrastructures. Thereby, we can decrease even more the execution time of EM tasks performed with the multi-pass SNM. In this sense, we make the following contributions:

• We propose the multi-pass MapReduce-based Duplicate Count Strategy (MultiMR-DCS++), a MR-based approach that provides an efficient parallelization of the multi-pass DCS++ method  [7] by using multiple MR jobs and applying a tailored data replication during data redistribution to allow the resizing of the adaptive window. The approach also addresses the data skewness problem with an automatic data partitioning strategy that is combined with MultiMR-DCS++ to provide a satisfactory load balancing across the available nodes.

• We evaluate the MultiMR-DCS++ (adaptive window) against the MR-based multi-pass SNM state of the art approach RepSN   [12] (fixed window) and show that our approach provides a better performance by diminishing the overall EM execution time. The evaluation is performed on a real cluster environment and uses real-world data.

In our previous work  [19], we proposed an approach for the single-pass MapReduce-based Duplicate Count Strategy. The solution provides an efficient parallelization of the DCS++ method  [7] by using multiple MR jobs. However, the solution presented in  [19] does not support the performance of a multi-pass DCS++ variant within the same MR jobs. This means that, to perform the multi-pass DCS++, the MR process must be repeated for each pass. In this new study, we present a more sophisticated model in terms of robustness and extensibility that addresses both the single- and multi-pass MR-DCS++ without the need of the MR process serialization.

This article is structured as follows. Section  2 introduces the EM performance problem and explains how this problem can be treated with the MapReduce programming paradigm. Section  3 discusses related work. Section  4 describes our load balanced multi-pass adaptive windowing approach for EM using MapReduce. Section  5 presents the performed experiments and evaluation. Finally, Section  6 concludes the article and provides suggestions for future work.