A Fast Method for Web Template Extraction via a Multi-sequence Alignment Approach

Abstract

The increased richness of the page contents and the diffusion of content management systems are responsible for the impressive changes happened in the last decade in a typical Web site layout. In fact, most of the Web sites are endowed with a template which gives them a uniform graphical and functional structure. Templates, by themselves, do not change the informative content of the pages, but they are typically designed to enhance the usability by uniformly organizing the contents following a standardized arrangement of functional blocks and by providing navigation tools, like menus or banners. However, the additional information provided by the template can worsen the performances of many algorithms for automatic Web processing. In fact, templates are designed for human users and provide redundant information that is marginally correlated with the main contents of a given page. These additional parts act as a noise source for many automated tasks such as web crawling, indexing, page classification and clustering. Hence, a preprocessing step to detect and strip the parts related to the template is needed to extract only the specific contents of each page. The critical part for the automation of this process is the accurate detection of the template, given a minimal set of pages from a given site.

The template consists in parts of the HTML tag structure that are shared by all the pages from the site, and its detection is made difficult by the variable parts intermixed with them. We propose an algorithm for template extraction that is based on the alignment of the HTML sequences of a set of pages. This approach is quite fast since it exploits efficient alignment algorithms proposed in bioinformatics and does not require complex tree matching or visual layout analysis. The algorithm aligns the HTML tag streams from pairs of pages and extracts a set of candidate templates that are merged following a binary tree consensus schema to increase the algorithm precision. The experimental evaluation shows that 16 sample pages are enough to extract the site template with good accuracy. The effects of the template stripping on a clustering task are also investigated, showing that the clustering quality can be effectively improved.