Abstract
Assuring high quality of data stored in information systems (ISs) is challenging and it is one of concerns of companies. Typically, data stored in ISs are not free from errors, which include among others wrong and missing values as well as duplicates. Data deduplication has received a lot of attention from the research community. The research efforts have resulted in a state-of-the-art data deduplication pipeline, supported by software tools and algorithms. One of the tasks in the pipeline consists in reducing the complexity of records comparisons. This task is known as blocking. Multiple algorithms for blocking have been proposed and one of them is the sorted neighborhood method. In this paper, we focus on tuning and evaluating the method on a real data set composed of 5.5M of customer records. To the best of our knowledge, this is the largest real data set being used in research. The findings reported in this paper come from a R &D project run for a big company in a financial sector.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alamuri, M., Surampudi, B.R., Negi, A.: A survey of distance/similarity measures for categorical data. In: International Joint Conference on Neural Networks (IJCNN), pp. 1907–1914. IEEE (2014)
Andrzejewski, W., Bębel, B., Boiński, P., Sienkiewicz, M., Wrembel, R.: Text similarity measures in a data deduplication pipeline for customers records. In: International Workshop on Design, Optimization, Languages and Analytical Processing of Big Data DOLAP, co-located with EDBT/ICDT. CEUR Workshop Proceedings, CEUR-WS.org (2023, to appear)
Baxter, R., Christen, P.: A comparison of fast blocking methods for record linkage. In: ACM SIGKDD Workshop on Data Cleaning, Record Linkage, and Object Consolidation (2003)
Bilenko, M., Kamath, B., Mooney, R.J.: Adaptive blocking: learning to scale up record linkage. In: The IEEE International Conference on Data Mining (ICDM), pp. 87–96. IEEE Computer Society (2006)
Boiński, P., Sienkiewicz, M., Bębel, B., Wrembel, R., Gałęzowski, D., Graniszewski, W.: On customer data deduplication: lessons learned from a R &D project in the financial sector. In: Workshops of the EDBT/ICDT 2022 Joint Conference. CEUR Workshop Proceedings, vol. 3135. CEUR-WS.org (2022)
Boiński, P., Sienkiewicz, M., Wrembel, R., Bębel, B., Andrzejewski, W.: Text similarity measures in a data deduplication pipeline for customers records. In: ACM/SIGAPP Symposium on Applied Computing SAC. ACM (2023, to appear)
Boriah, S., Chandola, V., Kumar, V.: Similarity measures for categorical data: a comparative evaluation. In: SIAM International Conference on Data Mining (SDM), pp. 243–254. SIAM (2008)
Cao, Y., Chen, Z., Zhu, J., Yue, P., Lin, C., Yu, Y.: Leveraging unlabeled data to scale blocking for record linkage. In: International Joint Conference on Artificial Intelligence IJCAI, pp. 2211–2217 (2011)
Christen, P.: A comparison of personal name matching: techniques and practical issues. In: International Conference on Data Mining (ICDM), pp. 290–294. IEEE Computer Society (2006)
Christen, P.: Data Matching - Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. DCSA, Springer (2012). https://doi.org/10.1007/978-3-642-31164-2
Christen, P.: A survey of indexing techniques for scalable record linkage and deduplication. IEEE Trans. Knowl. Data Eng. 24(9), 1537–1555 (2012)
Christophides, V., Efthymiou, V., Palpanas, T., Papadakis, G., Stefanidis, K.: An overview of end-to-end entity resolution for big data. ACM Comput. Surv. 53(6), 127:1–127:42 (2021)
Colyer, A.: The morning paper on An overview of end-to-end entity resolution for big data (2020). https://blog.acolyer.org/2020/12/14/entity-resolution/
Elmagarmid, A.K., Ipeirotis, P.G., Verykios, V.S.: Duplicate record detection: a survey. IEEE Trans. Knowl. Data Eng. 19(1), 1–16 (2007)
Kejriwal, M.: Sorted neighborhood for the semantic web. In: AAAI Conference on Artificial Intelligence, pp. 4174–4175. AAAI Press (2015)
Kejriwal, M., Miranker, D.P.: An unsupervised algorithm for learning blocking schemes. In: IEEE International Conference on Data Mining, pp. 340–349. IEEE Computer Society (2013)
Köpcke, H., Rahm, E.: Frameworks for entity matching: a comparison. Data Knowl. Eng. 69(2), 197–210 (2010)
Li, G., Wu, Q., Tu, D., Sun, S.: A sorted neighborhood approach for detecting duplicated regions in image forgeries based on DWT and SVD. In: IEEE International Conference on Multimedia and Expo ICME, pp. 1750–1753. IEEE Computer Society (2007)
Naumann, F.: Similarity Measures. Hasso Plattner Institute (2013)
Papadakis, G., Skoutas, D., Thanos, E., Palpanas, T.: Blocking and filtering techniques for entity resolution: a survey. ACM Comput. Surv. 53(2), 31:1–31:42 (2020)
Papadakis, G., Tsekouras, L., Thanos, E., Giannakopoulos, G., Palpanas, T., Koubarakis, M.: Domain- and structure-agnostic end-to-end entity resolution with JedAI. SIGMOD Rec. 48(4), 30–36 (2019)
Powell, M.J.D.: An efficient method for finding the minimum of a function of several variables without calculating derivatives. Comput. J. 7(2), 155–162 (1964)
Puhlmann, S., Weis, M., Naumann, F.: XML duplicate detection using sorted neighborhoods. In: Ioannidis, Y., et al. (eds.) EDBT 2006. LNCS, vol. 3896, pp. 773–791. Springer, Heidelberg (2006). https://doi.org/10.1007/11687238_46
Ramadan, B., Christen, P., Liang, H., Gayler, R.W.: Dynamic sorted neighborhood indexing for real-time entity resolution. ACM J. Data Inf. Qual. 6(4), 15:1–15:29 (2015)
Sienkiewicz, M., Wrembel, R.: Managing data in a big financial institution: conclusions from a R &D project. In: Workshops of the EDBT/ICDT 2021 Joint Conference. CEUR Workshop Proceedings, vol. 2841. CEUR-WS.org (2021)
de Souza Silva, L., Murai, F., da Silva, A.P.C., Moro, M.M.: Automatic identification of best attributes for indexing in data deduplication. In: Mendelzon, A. (ed.) International Workshop on Foundations of Data Management. CEUR Workshop Proceedings, vol. 2100. CEUR-WS.org (2018)
Vatsalan, D., Christen, P.: Sorted nearest neighborhood clustering for efficient private blocking. In: Pei, J., Tseng, V.S., Cao, L., Motoda, H., Xu, G. (eds.) PAKDD 2013. LNCS (LNAI), vol. 7819, pp. 341–352. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37456-2_29
Yan, S., Lee, D., Kan, M., Giles, C.L.: Adaptive sorted neighborhood methods for efficient record linkage. In: ACM/IEEE Joint Conference on Digital Libraries JCDL, pp. 185–194. ACM (2007)
Acknowledgements
The project is supported by the grant from the National Center for Research and Development no. POIR.01.01.01-00-0287/19.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Boiński, P., Andrzejewski, W., Bębel, B., Wrembel, R. (2023). On Tuning the Sorted Neighborhood Method for Record Comparisons in a Data Deduplication Pipeline. In: Strauss, C., Amagasa, T., Kotsis, G., Tjoa, A.M., Khalil, I. (eds) Database and Expert Systems Applications. DEXA 2023. Lecture Notes in Computer Science, vol 14146. Springer, Cham. https://doi.org/10.1007/978-3-031-39847-6_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-39847-6_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-39846-9
Online ISBN: 978-3-031-39847-6
eBook Packages: Computer ScienceComputer Science (R0)