Pranav Subramaniam
ABSTRACT
Join discovery is a crucial part of exploration on data lakes. It often involves finding joinable tables that are semantically relevant. However, data lakes often contain numeric tables with unreliable column headers, and ID columns whose text names have been lost. Finding semantically relevant joins over numeric tables is a challenge. State-of-the-art describes join discovery using semantic similarity, but do not consider purely numeric tables. In this paper, we describe a system, NumJoin that includes two novel approaches for discovering joinable tables in a data lake: one that maps tables to knowledge graphs, and another that leverages numeric types and distributions. We demonstrate the effectiveness of NumJoin on a large data lake, including transportation data and finance data.
- Alex Bogatu, Alvaro A. A. Fernandes, Norman W. Paton, and Nikolaos Konstantinou. 2020. Dataset Discovery in Data Lakes. In 2020 IEEE 36th International Conference on Data Engineering (ICDE). 709--720. https://doi.org/10.1109/ICDE48307.2020.00067Google Scholar
- Raul Castro Fernandez, Essam Mansour, Abdulhakim A. Qahtan, Ahmed Elmagarmid, Ihab Ilyas, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, and Nan Tang. 2018. Seeping Semantics: Linking Datasets Using Word Embeddings for Data Discovery. In 2018 IEEE 34th International Conference on Data Engineering (ICDE). 989--1000. https://doi.org/10.1109/ICDE.2018.00093Google Scholar
- Nadiia Chepurko, Ryan Marcus, Emanuel Zgraggen, Raul Castro Fernandez, Tim Kraska, and David Karger. 2020. ARDA: Automatic Relational Data Augmentation for Machine Learning. Proc. VLDB Endow., Vol. 13, 9 (jun 2020), 1373--1387. https://doi.org/10.14778/3397230.3397235Google ScholarDigital Library
- Yuyang Dong, Kunihiro Takeoka, Chuan Xiao, and Masafumi Oyamada. 2020. Efficient Joinable Table Discovery in Data Lakes: A High-Dimensional Similarity-Based Approach. CoRR, Vol. abs/2010.13273 (2020). showeprint[arXiv]2010.13273 https://arxiv.org/abs/2010.13273Google Scholar
- Yuyang Dong, Chuan Xiao, Takuma Nozawa, Masafumi Enomoto, and Masafumi Oyamada. 2022. DeepJoin: Joinable Table Discovery with Pre-trained Language Models. https://doi.org/10.48550/ARXIV.2212.07588Google Scholar
- Mahdi Esmailoghli, Jorge-Arnulfo Quiané -Ruiz, and Ziawasch Abedjan. 2021a. MATE: Multi-Attribute Table Extraction. CoRR, Vol. abs/2110.00318 (2021). showeprint[arXiv]2110.00318 https://arxiv.org/abs/2110.00318Google Scholar
- Mahdi Esmailoghli, Jorge-Arnulfo Quiané-Ruiz, and Ziawasch Abedjan. 2021b. COCOA: COrrelation COefficient-Aware Data Augmentation. In EDBT.Google Scholar
- Grace Fan, Jin Wang, Yuliang Li, Dan Zhang, and Renée Miller. 2022. Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning. https://doi.org/10.48550/ARXIV.2210.01922Google Scholar
- Catherine Faron, Chiara Ghidini, Ahmad Alobaid, Emilia Kacprzak, Oscar Corcho, Catherina Faron, and Chiara Ghidini. 2021. Typology-Based Semantic Labeling of Numeric Tabular Data. Semant. Web, Vol. 12, 1 (jan 2021), 5--20. https://doi.org/10.3233/SW-200397Google ScholarDigital Library
- Aamod Khatiwada, Grace Fan, Roee Shraga, Zixuan Chen, Wolfgang Gatterbauer, Renée J. Miller, and Mirek Riedewald. 2022. SANTOS: Relationship-based Semantic Table Union Search. https://doi.org/10.48550/ARXIV.2209.13589Google Scholar
- Udayan Khurana and Sainyam Galhotra. 2021. Semantic Concept Annotation for Tabular Data. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (Virtual Event, Queensland, Australia) (CIKM '21). Association for Computing Machinery, New York, NY, USA, 844--853. https://doi.org/10.1145/3459637.3482295Google ScholarDigital Library
- Udayan Khurana, Kavitha Srinivas, and Horst Samulowitz. 2022. A Survey on Semantics in Automated Data Science. https://doi.org/10.48550/ARXIV.2205.08018Google Scholar
- Peng Li, Xiang Cheng, Xu Chu, Yeye He, and Surajit Chaudhuri. 2021. Auto-FuzzyJoin: Auto-Program Fuzzy Similarity Joins Without Labeled Examples. In Proceedings of the 2021 International Conference on Management of Data (Virtual Event, China) (SIGMOD '21). Association for Computing Machinery, New York, NY, USA, 1064--1076. https://doi.org/10.1145/3448016.3452824Google ScholarDigital Library
- Dan Ofer. 2019. DBPedia Classes: Hierarchical Taxonomy of Wikipedia article classes. https://www.kaggle.com/datasets/danofer/dbpedia-classesGoogle Scholar
- Sahaana Suri, Ihab F. Ilyas, Christopher Ré, and Theodoros Rekatsinas. 2021. Ember: No-Code Context Enrichment via Similarity-Based Keyless Joins. Proc. VLDB Endow., Vol. 15, 3 (nov 2021), 699--712. https://doi.org/10.14778/3494124.3494149Google ScholarDigital Library
- Kunihiro Takeoka, Masafumi Oyamada, Shinji Nakadai, and Takeshi Okadome. 2019. Meimei: An Efficient Probabilistic Approach for Semantically Annotating Tables. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 01 (Jul. 2019), 281--288. https://doi.org/10.1609/aaai.v33i01.3301281Google ScholarDigital Library
- Erkang Zhu, Dong Deng, Fatemeh Nargesian, and Renée J. Miller. 2019. JOSIE: Overlap Set Similarity Search for Finding Joinable Tables in Data Lakes. In Proceedings of the 2019 International Conference on Management of Data (Amsterdam, Netherlands) (SIGMOD '19). Association for Computing Machinery, New York, NY, USA, 847--864. https://doi.org/10.1145/3299869.3300065Google ScholarDigital Library
- Erkang Zhu, Yeye He, and Surajit Chaudhuri. 2017. Auto-Join: Joining Tables by Leveraging Transformations. Proc. VLDB Endow., Vol. 10, 10 (jun 2017), 1034--1045. https://doi.org/10.14778/3115404.3115409Google ScholarDigital Library
- Erkang Zhu, Fatemeh Nargesian, Ken Q. Pu, and Renée J. Miller. 2016. LSH Ensemble: Internet-Scale Domain Search. Proc. VLDB Endow., Vol. 9, 12 (aug 2016), 1185--1196. https://doi.org/10.14778/2994509.2994534Google ScholarDigital Library
Index Terms
- NumJoin: Discovering Numeric Joinable Tables with Semantically Related Columns
Recommendations
Mining Association Rules on Related Numeric Attributes
PAKDD '99: Proceedings of the Third Pacific-Asia Conference on Methodologies for Knowledge Discovery and Data MiningIn practical applications, some property is represented by a pair of related attributes. For example, blood pressure, temperature changes etc. The existing data mining approaches for association rules can not tackle those cases, because they treat every ...
Discovering Numeric Association Rules via Evolutionary Algorithm
PAKDD '02: Proceedings of the 6th Pacific-Asia Conference on Advances in Knowledge Discovery and Data MiningAssociation rules are one of the most used tools to discover relationships among attributes in a database. Nowadays, there are many efficient techniques to obtain these rules, although most of them require that the values of the attributes be discrete. ...
Discovering associations with numeric variables
KDD '01: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data miningThis paper further develops Aumann and Lindell's [3] proposal for a variant of association rules for which the consequent is a numeric variable. It is argued that these rules can discover useful interactions with numeric data that cannot be discovered ...
Comments