Abstract
Many web databases can be seen as providing partial and overlapping information about entities in the world. To answer queries effectively, we need to integrate the information about the individual entities that are fragmented over multiple sources. At first blush this is just the inverse of traditional database normalization problem—rather than go from a universal relation to normalized tables, we want to reconstruct the universal relation given the tables (sources). The standard way of reconstructing the entities will involve joining the tables. Unfortunately, because of the autonomous and decentralized way in which the sources are populated, they often do not have Primary Key–Foreign Key relations. While tables may share attributes, naive joins over these shared attributes can result in reconstruction of many spurious entities thus seriously compromising precision. Our system, SmartInt is aimed at addressing the problem of data integration in such scenarios. Given a query, our system uses the Approximate Functional Dependencies (AFDs) to piece together a tree of relevant tables to answer it. The result tuples produced by our system are able to strike a favorable balance between precision and recall.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
In other words, web data sources can be seen as resulting from an ad hoc normalization followed by the attribute name change.
Presently we give equal weight to all the attributes in the system, this can be generalized to account for attributes with different levels of importance.
Cumulative Confidence is defined as product of the confidences of all the dependencies in a chain.
At first blush, pruning highly specific AFDs seems to hurt the precision, but in the current set of experiements specificity based pruning reduced the total running time and did not effect the accuracy.
References
Agrawal, R., & Srikant, R. (1994). Fast algorithms for mining association rules in large databases. VLDB.
Balmin, A., Hristidis, V., & Papakonstantinou, Y. (2004). Objectrank: Authority-based keyword search in databases. In: VLDB.
Bhalotia, G., Hulgeri, A., Nakhe, C., Chakrabarti, S., Sudarshan, S., & Bombay, I. (2002). Keyword searching and browsing in databases using banks. ICDE.
DeMichiel, L. (1989). Resolving database incompatibility: An approach to performing relational operations over mismatched domains. IEEE Transactions on Knowledge and Data Engineering, 1(4), 485–493.
Gummadi, R., Khulbe, A., Kalavagattu, A., Salvi, S., & Kambhampati, S. (2010). Smartint: A system for answering queries over web databases using attribute dependencies. ICDE (Demo).
Gummadi, R., Khulbe, A., Kalavagattu, A., Salvi, S., & Kambhampati, S. (2011). SmartInt: Using mined attribute dependencies to integrate fragmented web databases. WWW (Poster).
Halevy, A. Y. (2001). Answering queries using views: A survey. The VLDB Journal, 10(4), 270–294.
Hristidis V., & Papakonstantinou, Y. (2002). Discover: Keyword search in relational databases. In: VLDB.
Huhtala, Y., Kärkkäinen, J., Porkka, P., & Toivonen, H. (1999). TANE: An efficient algorithm for discovering functional and approximate dependencies. The Computer Journal, 42(2), 100–111.
Ilyas, I. F., Markl, V., Haas, P., Brown, P., & Aboulnaga, A. (2004). Cords: Automatic discovery of correlations and soft functional dependencies. In SIGMOD.
Kalavagattu, A. (2008). Mining approximate functional dependencies as condensed representations of association rules. Master’s thesis, Arizona State University.
Kambhampati, S., Lambrecht, E., Nambiar, U., Nie, Z., & Senthil, G. (2004). Optimizing recursive information gathering plans in emerac. Journal of Intelligent Information Systems, 22, 119–153.
Larson, J., Navathe, S., & Elmasri, R. (1989). A theory of attributed equivalence in databases with application to schema integration. IEEE Transaction on Software Engineering, 15, 258–274.
Lenzerini, M. (2002). Data integration: A theoretical perspective. In PODS (pp. 233–246).
Li, W.-S., & Clifton, C. (1995). Semint: A system prototype for semantic integration in heterogeneous databases. In SIGMOD.
Lim, E.-P., Srivastava, J., Prabhakar, S., & Richardson, J. (1993). Entity identification in database integration. In Proc. ICDE (pp. 294–301).
Melnik, S., Garcia-Molina, H., & Rahm, E. (2002). Similarity flooding: A versatile graph matching algorithm and its application to schema matching. In ICDE.
Nambiar U., & Kambhampati, S. (2006). Answering imprecise queries over autonomous web databases. In ICDE (p. 45).
Oan, A., Domingos, P., & Halevy, A. Y. (2003). Learning to match the schemas of data sources: A multistrategy approach. Machine Learning, 50(3), 279–301.
Sayyadian, M., LeKhac, H., Doan, A., & Gravano, L. (2007). Efficient keyword search across heterogeneous relational databases. ICDE.
Wolf, G., Khatri, H., Chokshi, B., Fan, J., Chen, Y., & Kambhampati, S. (2007). Query processing over incomplete autonomous databases. In VLDB.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research is supported in part by the NSF grant IIS-0738317, the ONR grant N000140910032 and two Google research awards. We thank Raju Balakrishnan and Sushovan De for helpful feedback on the previous drafts. Earlier versions of this work were presented as a 4-page demo paper at ICDE 2010 (Gummadi et al. 2010) and a 2-page poster paper at WWW 2011 (Gummadi et al. 2011).
Rights and permissions
About this article
Cite this article
Gummadi, R., Khulbe, A., Kalavagattu, A. et al. SMARTINT: using mined attribute dependencies to integrate fragmented web databases. J Intell Inf Syst 38, 575–599 (2012). https://doi.org/10.1007/s10844-011-0169-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10844-011-0169-0