Active learning in keyword search-based data integration

Abstract

The problem of scaling up data integration, such that new sources can be quickly utilized as they are discovered, remains elusive: Global schemas for integrated data are difficult to develop and expand, and schema and record matching techniques are limited by the fact that data and metadata are often under-specified and must be disambiguated by data experts. One promising approach is to avoid using a global schema, and instead to develop keyword search-based data integration—where the system lazily discovers associations enabling it to join together matches to keywords, and return ranked results. The user is expected to understand the data domain and provide feedback about answers’ quality. The system generalizes such feedback to learn how to correctly integrate data. A major open challenge is that under this model, the user only sees and offers feedback on a few “top-$k$” results: This result set must be carefully selected to include answers of high relevance and answers that are highly informative when feedback is given on them. Existing systems merely focus on predicting relevance, by composing the scores of various schema and record matching algorithms. In this paper, we show how to predict the uncertainty associated with a query result’s score, as well as how informative feedback is on a given result. We build upon these foundations to develop an active learning approach to keyword search-based data integration, and we validate the effectiveness of our solution over real data from several very different domains.

Appendix: Incremental update on source discovery

In addition to finding informative query answers and learning from user’s feedback, another challenge in keyword search-based data integration is to incrementally update the underlying model when we add new data sources [42]. This involves not only updating the base data in the form of the schema graph, but also updating any materialized views that were formulated through keyword search. We wish to automatically combine new data sources into the existing schema graph and to predict edge costs in order to discover query trees to generate potentially useful new results for existing keyword search-based views.

In more detail, within the Q System, each keyword query can be saved as a view whose results can be revisited over time. For each view, we seek to only add new alignment edges that can potentially affect the results in the view, upon new data sources are connected. Formally, suppose we have $G = (V, E)$ as the existing schema graph and $G' = (V', E')$. We are also tied to a fixed view derived from a keyword search query $Q = \{K_i\}$. The goal of automatic incremental update is to derive a probability distribution over edge costs for each pair of attribute nodes $(v, v')$ where $v\in V$ and $v'\in V'$. Notice that a naive way of performing such computation for all possible pairs requires examining $\varOmega (|V||V'|)$ pairs, which is an undesirable quadratic term that does not scale well as the number of schema graph nodes becomes large. Ideally, we need a strategy to compute only that subset of possible joins that indeed produces results affecting the top-$k$ answers of the existing view.

Our information need-based strategy adopts a pruning approach and limits our search space to only a subset of possible pairs. Let $C_{max}$ be the maximum expected tree cost (relevance) among all top-$k$ trees in a fixed view. We also set a threshold $\tau > C_{max}$ (but not too large). Building upon [42], we say that a new attribute node $A$ is feasible if and only if there exists a keyword node $K_i\in Q$ such that the minimum expected cost from $K_i$ to $A$ is less than $\tau$.

We formalize this in Algorithm 4, which identifies all feasible attribute nodes by using BFS. The algorithm starts with all the keyword nodes as seeds, and iteratively expands new regions. When a node is to be expanded, we check if its expected distance to the keywords is greater than $\tau$. If this happens, the algorithm will stop further expansion from the node. After we obtain all feasible nodes, we will align each of them with every node in $V$ to compute the costs.