Extracting Information from Semi-structured Web Documents

Abstract

The World Wide Web has nowen tered its mature age. It not only hosts and serves large amounts of pages but also offers large amounts of information potentially useful for individuals and businesses. Modern decision support can no more be effective without timely and accurate access to this unprecedented source of data. However, unlike in a database, the structure of data available on the Web is not known a priori and its understanding seems to require human intervention. Yet the conjunction of rules for interpreting layout and simple domain knowledge enables in many cases the automatic extraction of such data. In such cases we say that data is semi-structured. In this paper, we present a framework in which we try to address the problem of extracting semi-structured data. This framework combines a syntactical extraction strategy with a set of mapping rules, heuristics and simple domain knowledge, which maps a syntactical structure identified in Web documents to a conceptual/ semantic structure. We present and analyse one instance of this framework in which a syntactical extraction strategy exploits the HTML structure of Web documents using a Tree Alignment algorithm with a novel combination of heuristics to detect repeated patterns and infer rules to extract relevant records. Then, by the use of domain knowledge, we refine the extraction rules such that not only are they able to extract data, but they also construe meaning to the extracted results.