Jing Ao
ABSTRACT
As a key data-integration step, entity matching (EM) identifies tuples referring to the same real-world entities in disparate data sources. In many cases, the EM quality can be improved by repairing incorrect values in the data; at the same time, it is well known that the time costs of data cleaning by human experts could be prohibitive. In this paper, we focus on the time-consuming human-in-the-loop data-cleaning problem for relational EM, by recommending to human experts a time-efficient order in which values of attributes could be cleaned in the given data. Our proposed domain-independent cleaning framework aims to save human users' time, by guiding them in cleaning the EM inputs in an attribute order that is as conducive to maximizing EM accuracy as possible within a given constraint on the time they spend on cleaning. In guiding the cleaning process, our attribute-recommendation methods discover and take advantage of information provided by the data, and also use feedback from the EM engine. Our preliminary experimental results suggest that the proposed approach leads to measurable speedup, for a variety of time constraints, in the improvement of EM accuracy over the baseline approach, in which domain experts choose the sequence in which to clean the attributes of the inputs.
- 2019. OpenRefine(GoogleRefine). http://openrefine.orgGoogle Scholar
- 2019. Trifacta. https://www.trifacta.com/Google Scholar
- Ahmad Assadi, Tova Milo, and Slava Novgorodov. 2018. Cleaning Data with Constraints and Experts. In Proceedings of the 21st International Workshop on the Web and Databases (WebDB'18). ACM, New York, NY, USA, 1:1--1:6. Google ScholarDigital Library
- Moria Bergman, Tova Milo, Slava Novgorodov, and Wang-Chiew Tan. 2015. Query-Oriented Data Cleaning with Oracles. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD '15). ACM, New York, NY, USA, 1199--1214. Google ScholarDigital Library
- Gavin Brown, Adam Pocock, Ming-Jie Zhao, and Mikel Luján. 2012. Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection. J. Mach. Learn. Res. 13 (Jan. 2012), 27--66. http://dl.acm.org/citation.cfm?id=2188387 Google ScholarDigital Library
- Peter Christen. 2012. Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. Springer, New York, NY, USA. Google ScholarDigital Library
- Xu Chu, Ihab F. Ilyas, Sanjay Krishnan, and Jiannan Wang. 2016. Data Cleaning: Overview and Emerging Challenges. In Proceedings of the 2016 International Conference on Management of Data (SIGMOD '16). ACM, New York, NY, USA, 2201--2206. Google ScholarDigital Library
- Sanjib Das, AnHai Doan, Paul Suganthan G. C., Chaitanya Gokhale, and Pradap Konda. 2019. The Magellan Data Repository. https://sites.google.com/site/anhaidgroup/projects/data.Google Scholar
- AnHai Doan. 2018. Human-in-the-Loop Data Analysis: A Personal Perspective. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (HILDA'18). ACM, New York, NY, USA, 1:1--1:6. Google ScholarDigital Library
- AnHai Doan, Adel Ardalan, Jeffrey Ballard, Sanjib Das, Yash Govind, Pradap Konda, Han Li, Sidharth Mudgal, Erik Paulson, G. C. Paul Suganthan, and Haojun Zhang. 2017. Human-in-the-Loop Challenges for Entity Matching: A Midterm Report. In Proceedings of the 2Nd Workshop on Human-In-the-Loop Data Analytics (HILDA'17). ACM, New York, NY, USA, 12:1--12:6. Google ScholarDigital Library
- AnHai Doan, Alon Halevy, and Zachary Ives. 2012. Principles of Data Integration (1st ed.). Morgan Kaufmann Publishers, San Francisco, CA, USA. Google ScholarDigital Library
- AnHai Doan, Pradap Konda, Paul Suganthan G C, Adel Ardalan, Jeffrey R Ballard, Sanjib Das, Yash Govind, Han Li, Philip Martinkus, Sidharth Mudgal, Erik Paulson, and Haojun Zhang. 2018. Toward a System Building Agenda for Data Integration (and Data Science). IEEE Data Eng. Bull. 41, 2 (2018), 35--46. http://sites.computer.org/debull/A18june/p35.pdfGoogle Scholar
- Xin Luna Dong and Divesh Srivastava. 2013. Big Data Integration. In 2013 IEEE 29th International Conference on Data Engineering (ICDE). IEEE, Piscataway, NJ, USA, 1245--1248. Google ScholarDigital Library
- Xin Luna Dong and Divesh Srivastava. 2015. Big Data Integration. Morgan & Claypool Publishers, San Rafael, CA, USA.Google Scholar
- Ahmed K Elmagarmid, Panagiotis G Ipeirotis, and Vassilios S Verykios. 2007. Duplicate Record Detection: A Survey. IEEE Transactions on Knowledge and Data Engineering 19, 1 (Jan 2007), 1--16. Google ScholarDigital Library
- Wenfei Fan and Floris Geerts. 2012. Foundations of Data Quality Management. Morgan & Claypool Publishers, San Rafael, CA, USA. Google ScholarDigital Library
- Jian He, Enzo Veltri, Donatello Santoro, Guoliang Li, Giansalvatore Mecca, Paolo Papotti, and Nan Tang. 2016. Interactive and Deterministic Data Cleaning. In Proceedings of the 2016 International Conference on Management of Data (SIGMOD '16). ACM, New York, NY, USA, 893--907. Google ScholarDigital Library
- Ihab F. Ilyas and Xu Chu. 2015. Trends in Cleaning Relational Data: Consistency and Deduplication. Now Publishers, Hanover, MA, USA. Google ScholarDigital Library
- George H. John, Ron Kohavi, and Karl Pfleger. 1994. Irrelevant Features and the Subset Selection Problem. In Machine Learning, Proceedings of the Eleventh International Conference. Morgan Kaufmann Publishers, San Francisco, CA, USA, 121--129. Google ScholarDigital Library
- Pradap Konda, Sanjib Das, Paul Suganthan G. C., AnHai Doan, Adel Ardalan, Jeffrey R. Ballard, Han Li, Fatemah Panahi, Haojun Zhang, Jeff Naughton, Shishir Prasad, Ganesh Krishnan, Rohit Deep, and Vijay Raghavendra. 2016. Magellan: Toward Building Entity Matching Management Systems. Proc. VLDB Endow. 9, 12 (Aug. 2016), 1197--1208. Google ScholarDigital Library
- Sanjay Krishnan, Daniel Haas, Michael J. Franklin, and Eugene Wu. 2016. Towards Reliable Interactive Data Cleaning: A User Survey and Recommendations. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (HILDA '16). ACM, New York, NY, USA, 9:1--9:5. Google ScholarDigital Library
- Sanjay Krishnan, Jiannan Wang, Eugene Wu, Michael J. Franklin, and Ken Goldberg. 2016. ActiveClean: Interactive Data Cleaning for Statistical Modeling. Proc. VLDB Endow. 9, 12 (Aug. 2016), 948--959. Google ScholarDigital Library
- Guoliang Li. 2017. Human-in-the-loop Data Integration. Proc. VLDB Endow. 10, 12 (Aug. 2017), 2006--2017. Google ScholarDigital Library
- Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino, Jiliang Tang, and Huan Liu. 2016. Feature Selection: A Data Perspective. (2016). arXiv:1601.07996 http://arxiv.org/abs/1601.07996 Google ScholarDigital Library
- Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino, Jiliang Tang, and Huan Liu. 2017. Feature Selection: A Data Perspective. ACM Comput. Surv. 50, 6 (Dec. 2017), 94:1--94:45. Google ScholarDigital Library
- Patrick Emmanuel Meyer, Colas Schretter, and Gianluca Bontempi. 2008. Information-theoretic Feature Selection in Microarray Data Using Variable Complementarity. IEEE Journal of Selected Topics in Signal Processing 2, 3 (June 2008), 261--274.Google ScholarCross Ref
- Protiva Rahman, Courtney Hebert, and Arnab Nandi. 2018. ICARUS: Minimizing Human Effort in Iterative Data Completion. Proc. VLDB Endow. 11, 13 (Sept. 2018), 2263--2276. Google ScholarDigital Library
- Jiannan Wang, Sanjay Krishnan, Michael J. Franklin, Ken Goldberg, Tim Kraska, and Tova Milo. 2014. A Sample-and-clean Frame work for Fast and Accurate Query Processing on Dirty Data. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (SIGMOD '14). ACM, New York, NY, USA, 469--480. Google ScholarDigital Library
- Mohamed Yakout, Ahmed K. Elmagarmid, Jennifer Neville, Mourad Ouzzani, and Ihab F. Ilyas. 2011. Guided Data Repair. Proc. VLDB Endow. 4, 5 (Feb. 2011), 279--289. Google ScholarDigital Library
- Howard Hua Yang and John Moody. 1999. Data Visualization and Feature Selection: New Algorithms for Nongaussian Data. In Proceedings of the 12th International Conference on Neural Information Processing Systems (NIPS'99). MIT Press, Cambridge, MA, USA, 687--693. http://dl.acm.org/citation.cfm?id=3009657.3009755 Google ScholarDigital Library
- Lei Yu and Huan Liu. 2004. Efficient Feature Selection via Analysis of Relevance and Redundancy. J. Mach. Learn. Res. 5 (Dec. 2004), 1205--1224. http://dl.acm.org/citation.cfm?id=1005332.1044700 Google ScholarDigital Library
Index Terms
- Effective and Efficient Data Cleaning for Entity Matching
Recommendations
Data Cleaning: Overview and Emerging Challenges
SIGMOD '16: Proceedings of the 2016 International Conference on Management of DataDetecting and repairing dirty data is one of the perennial challenges in data analytics, and failure to do so can result in inaccurate analytics and unreliable decisions. Over the past few years, there has been a surge of interest from both industry and ...
A grammar-based entity representation framework for data cleaning
SIGMOD '09: Proceedings of the 2009 ACM SIGMOD International Conference on Management of dataFundamental to data cleaning is the need to account for multiple data representations. We propose a formal framework that can be used to reason about and manipulate data representations. The framework is declarative and combines elements of a generative ...
A Comparative Study of Data Cleaning Tools
In the information era, data is crucial in decision making. Most data sets contain impurities that need to be weeded out before any meaningful decision can be made from the data. Hence, data cleaning is essential and often takes more than 80 percent ...
Comments