Henning Köhler
ABSTRACT
We investigate the problem of creating and analyzing samples of relational databases to find relationships between string-valued attributes. Our focus is on identifying attribute pairs whose value sets overlap, a pre-condition for typical joins over such attributes. However, real-world data sets are often 'dirty', especially when integrating data from different sources. To deal with this issue, we propose new similarity measures between sets of strings, which not only consider set based similarity, but also similarity between strings instances. To make the measures effective, we develop efficient algorithms for distributed sample creation and similarity computation. Test results show that for dirty data our measures are more accurate for measuring value overlap than existing sample-based methods, but we also observe that there is a clear tradeoff between accuracy and speed. This motivates a two-stage filtering approach, with both measures operating on the same samples.
- A. Y. Halevy, A. Rajaraman, and J. J. Ordille, "Data integration: The teenage years," in VLDB, 2006, pp. 9--16. Google Scholar
Digital Library
- E. Rahm and P. A. Bernstein, A survey of approaches to automatic schema matching, VLDB J., vol. 10, no. 4, pp. 334--350, 2001. Google ScholarDigital Library
- M. J. Franklin, A. Y. Halevy, and D. Maier, From databases to dataspaces: a new abstraction for information management, SIGMOD Record, vol. 34, no. 4, pp. 27--33, 2005. Google ScholarDigital Library
- A. Y. Halevy, M. J. Franklin, and D. Maier, Principles of dataspace systems, in PODS, 2006, pp. 1--9. Google ScholarDigital Library
- T. Dasu, T. Johnson, S. Muthukrishnan, and V. Shkapenyuk, Mining database structure; or, how to build a data quality browser, in SIGMOD, 2002, pp. 240--251. Google ScholarDigital Library
- A. Broder, On the resemblance and containment of documents, in SEQUENCES: Proceedings of the Compression and Complexity of Sequences. IEEE Computer Society, 1997, p. 21. Google ScholarDigital Library
- A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig, Syntactic clustering of the web, Computer Networks, vol. 29, no. 8-13, pp. 1157--1166, 1997. Google ScholarDigital Library
- U. Manber, Finding similar files in a large file system, in USENIX Winter, 1994, pp. 1--10. Google ScholarDigital Library
- A. Z. Broder, Identifying and filtering near-duplicate documents, in CPM, 2000, pp. 1--10. Google ScholarDigital Library
- C. E. Shannon, A Mathematical Theory of Communication. CSLI Publications, 1948. Google ScholarDigital Library
- C. Li, B. Wang, and X. Yang, Vgram: Improving performance of approximate queries on string collections using variable-length grams, in VLDB, 2007, pp. 303--314. Google ScholarDigital Library
- J. Bauckmann, U. Leser, F. Naumann, and V. Tietz, Efficiently detecting inclusion dependencies, in ICDE, 2007, pp. 1448--1450.Google ScholarCross Ref
- A. K. Elmagarmid, P. G. Ipeirotis, and V. S. Verykios, Duplicate record detection: A survey, IEEE Trans. Knowl. Data Eng., vol. 19, no. 1, pp. 1--16, 2007. Google ScholarDigital Library
- D. Barbar'a, W. Dumouchel, C. Faloutsos, P. J. Haas, J. M. Hellerstein, Y. Ioannidis, H. V. Jagadish, T. Johnson, R. Ng, V. Poosala, K. A. Ross, and K. C. Sevcik, The New Jersey data reduction report, IEEE Data Engineering Bulletin, vol. 20, pp. 3--45, 1997.Google Scholar
- F. Olken and D. Rotem, Random sampling from databases - a survey, Statistics and Computing, vol. 5, pp. 25--42, 1994.Google ScholarCross Ref
- R. J. Miller, L. M. Haas, and M. A. Hernáandez, Schema mapping as query discovery, in VLDB, 2000, pp. 77--88. Google ScholarDigital Library
- R. Dhamankar, Y. Lee, A. Doan, A. Y. Halevy, and P. Domingos, iMAP: Discovering complex mappings between database schemas, in SIGMOD, 2004, pp. 383--394. Google ScholarDigital Library
- B. T. Dai, N. Koudas, D. Srivastava, A. K. H. Tung, and S. Venkatasubramanian, Validating multi-column schema matchings by type, in ICDE, 2008, pp. 120--129. Google ScholarDigital Library
- W. W. Cohen, Integration of heterogeneous databases without common domains using queries based on textual similarity, in SIGMOD, 1998, pp. 201--212. Google ScholarDigital Library
- L. Gravano, P. G. Ipeirotis, H. V. Jagadish, N. Koudas, S. Muthukrishnan, and D. Srivastava, Approximate string joins in a database (almost) for free, in VLDB, 2001, pp. 491--500. Google ScholarDigital Library
- F. Olken and D. Rotem, Simple random sampling from relational databases, in VLDB, 1986, pp. 160--169. Google ScholarDigital Library
- S. Ganguly, P. B. Gibbons, Y. Matias, and A. Silberschatz, Bifocal sampling for skew-resistant join size estimation, in SIGMOD, 1996, pp. 271--281. Google ScholarDigital Library
- S. Chaudhuri, R. Motwani, and V. R. Narasayya, On random sampling over joins, in SIGMOD, 1999, pp. 263--274. Google ScholarDigital Library
- S. Acharya, P. B. Gibbons, and V. Poosala, Congressional samples for approximate answering of group-by queries, in SIGMOD, 2000, pp. 487--498. Google ScholarDigital Library
- S. Chaudhuri, G. Das, and U. Srivastava, Effective use of block-level sampling in statistics estimation, in SIGMOD Conf., 2004, pp. 287--298. Google ScholarDigital Library
- P. J. Haas and C. Koenig, A bi-level Bernoulli scheme for database sampling, in SIGMOD, 2004, pp. 275--286. Google ScholarDigital Library
- J. Gryz, J. Guo, L. Liu, and C. Zuzarte, Query sampling in DB2 universal database, in SIGMOD, 2004, pp. 839--843. Google ScholarDigital Library
Index Terms
- Sampling dirty data for matching attributes
Recommendations
Schema matching based on position of attribute in query statement
Attribute-level schema matching is a critical step in numerous database applications, such as DataSpaces, Ontology Merging and Schema Integration. There exist many researches on this topic, however, they all ignore evidences about the positions of ...
Schema matching based on SQL statements
AbstractSchema matching is a critical step in numerous database applications such as web data sources integrating, data warehouse loading and information exchanging among several authorities. In this paper, we propose to exploit the similarities of the ...
Feature-preserved sampling over streaming data
In this article, we explore a novel sampling model, called feature preserved sampling (FPS) that sequentially generates a high-quality sample over sliding windows. The sampling quality we consider refers to the degree of consistency between the sample ...
Comments