Lightweight integration of IR and DB for scalable hybrid search with integrated ranking support

Abstract

The Web contains a large amount of documents and an increasing quantity of structured data in the form of RDF triples. Many of these triples are annotations associated with documents. While structured queries constitute the principal means to retrieve structured data, keyword queries are typically used for document retrieval. Clearly, a form of hybrid search that seamlessly integrates these formalisms to query both textual and structured data can address more complex information needs. However, hybrid search on the large scale Web environment faces several challenges. First, there is a need for repositories that can store and index a large amount of semantic data as well as textual data in documents, and manage them in an integrated way. Second, methods for hybrid query answering are needed to exploit the data from such an integrated repository. These methods should be fast and scalable, and in particular, they shall support flexible ranking schemes to return not all but only the most relevant results. In this paper, we present CE², an integrated solution that leverages mature information retrieval and database technologies to support large scale hybrid search. For scalable and integrated management of data, CE² integrates off-the-shelf database solutions with inverted indexes. Efficient hybrid query processing is supported through novel data structures and algorithms which allow advanced ranking schemes to be tightly integrated. Furthermore, a concrete ranking scheme is proposed to take features from both textual and structured data into account. Experiments conducted on DBpedia and Wikipedia show that CE² can provide good performance in terms of both effectiveness and efficiency.

Introduction

Recently, we have seen a strong increase in the availability of structured data on the Web, RDF in particular. This structured data might be associated with documents in the form of annotations or might be embedded in documents. The Web as it is today can be considered as a large repository of interlinked documents, structured data and annotations. Examples of publicly available datasets on the Web that contain these different types of data are Wikipedia¹ (textual data), DBPedia² (annotations and domain-independent structured data) and DBLP³ (annotations and structured data in the bibliographic domain).

Currently, keyword search is commonly supported by commercial search engines for the retrieval of documents. Beyond document retrieval, structured data on the Web can support other retrieval scenarios. Instead of documents, a search engine might return exact answers to the users as the result of a complex question, represented as a structured query. With the increase of annotations available on the Web, there is a potential to support more expressive document retrieval to address more complex needs. As discussed in [33], annotations can be seen as an additional layer of information on top of the documents, which can be exploited to answer more complex queries. In addition, annotations in form of RDFa⁴ and Microformats⁵ (data embedded in Web documents) are quite popular and have been used in search engines like Yahoo to improve retrieval accuracy [28].

Keyword queries and structured queries are the two principal means to find resources. More specialized systems such as Digital Library applications go further and employ hybrid queries combining both the advantages of structured queries and keyword queries. This type of queries is suitable for hybrid search [6], a paradigm that allows querying over textual and structured data in an integrated way. With hybrid search, the user can ask for documents or structured data, using both keywords and structural constraints. For example, a user can ask for pieces of data with descriptions containing a given keyword, e.g., “Find Turing Award Winners working at IBM that are associated with documents containing Algorithm”.

Hybrid search on a Web scale environment however brings several challenges. It requires the capacity to store and index a large amount of textual and structured data. In order to answer user queries against this data, it requires scalable solutions for integrated processing of hybrid queries so that results can be returned within a reasonable amount of time. Another crucial aspect is ranking because given the volume and the heterogeneity of Web resources, it is likely that a query results in a large number of candidates, which may differ in many aspects, including quality and recentness. Ranking is thus needed to help users focusing on the most relevant results.

In this paper, we elaborate on infrastructure components that are necessary to support large scale hybrid search. This work specifically addresses the above challenges and the main contributions are listed as follows:

• We describe a unified framework to represent and to query documents and graph-structured data in an integrated way.

• We leverage mature information retrieval and database technologies to build a repository, which can scale over a large amount of documents and graph-structured data.

• We propose a novel data structure called Occurrence Probability Table (OPT) and on top of the data structure, a set of algorithms for hybrid query processing that allows a flexible integration of advanced ranking schemes.

• We elaborate on a concrete ranking scheme which propagates and aggregates scores along a data structure called answer tree.

The repository and the hybrid query engine implementing our approach are embedded into an integrated solution called CE². We have conducted experiments with CE² on RDF data contained in DBpedia and on documents from Wikipedia. Results show that CE² supports effective ranking and scales to millions of documents and RDF triples.

The rest of this paper is organized as follows: Section 2 presents a formal model of hybrid search including definitions of resources, queries, answers and ranking. Section 3 introduces the architecture of CE². In Section 4 and Section 5, we elaborate on data storage and hybrid query processing in details. We show our experimental results in Section 6. Related work is presented in Section 7 and conclusions in Section 8.