Abstract
The increased richness of the page contents and the diffusion of content management systems are responsible for the impressive changes happened in the last decade in a typical Web site layout. In fact, most of the Web sites are endowed with a template which gives them a uniform graphical and functional structure. Templates, by themselves, do not change the informative content of the pages, but they are typically designed to enhance the usability by uniformly organizing the contents following a standardized arrangement of functional blocks and by providing navigation tools, like menus or banners. However, the additional information provided by the template can worsen the performances of many algorithms for automatic Web processing. In fact, templates are designed for human users and provide redundant information that is marginally correlated with the main contents of a given page. These additional parts act as a noise source for many automated tasks such as web crawling, indexing, page classification and clustering. Hence, a preprocessing step to detect and strip the parts related to the template is needed to extract only the specific contents of each page. The critical part for the automation of this process is the accurate detection of the template, given a minimal set of pages from a given site.
The template consists in parts of the HTML tag structure that are shared by all the pages from the site, and its detection is made difficult by the variable parts intermixed with them. We propose an algorithm for template extraction that is based on the alignment of the HTML sequences of a set of pages. This approach is quite fast since it exploits efficient alignment algorithms proposed in bioinformatics and does not require complex tree matching or visual layout analysis. The algorithm aligns the HTML tag streams from pairs of pages and extracts a set of candidate templates that are merged following a binary tree consensus schema to increase the algorithm precision. The experimental evaluation shows that 16 sample pages are enough to extract the site template with good accuracy. The effects of the template stripping on a clustering task are also investigated, showing that the clustering quality can be effectively improved.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Bar-Yossef, Z., Rajagopalan, S.: Template detection via data mining and its applications. In: Proceedings of the 11th International Conference on World Wide Web, WWW 2002, pp. 580–591. ACM, New York (2002)
Chakrabarti, D., Kumar, R., Punera, K.: Page-level template detection via isotonic smoothing. In: Proceedings of the 16th international conference on World Wide Web, WWW 2007, pp. 61–70. ACM, New York (2007)
Cover, T.M., Thomas, J.A.: Elements of information theory. John Wiley & Sons, New York (1991)
Debnath, S., Mitra, P., Giles, C.L.: Automatic extraction of informative blocks from webpages. In: Proceedings of the 2005 ACM Symposium on Applied Computing, SAC 2005, pp. 1722–1726. ACM, New York (2005)
Geraci, F., Pellegrini, M., Maggini, M., Sebastiani, F.: Cluster generation and labelling for web snippets: A fast, accurate hierarchical solution. Internet Matematics 3(4), 413–443 (2008)
Gonzalez, T.F.: Clustering to minimize the maximum intercluster distance. Theoretical Computer Science 38(2/3), 293–306 (1985)
Kovacevic, M., Diligenti, M., Gori, M., Milutinovic, V.: Recognition of common areas in a web page using visual information: a possible application in a page classification. In: Proceedings of the IEEE International Conference on Data Mining, ICDM 2002 (2002)
Needlemana, S.B., Wunscha, C.D.: A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology 48(3), 443–453 (1970)
Song, R., Liu, H., Wen, J.R., Ma, W.Y.: Learning block importance models for web pages. In: Proceedings of the 13th International Conference on World Wide Web, WWW 2004, pp. 203–211. ACM, New York (2004)
Strehl, A.: Relationship-based Clustering and Cluster Ensembles for High-dimensional Data Mining. Ph.D. thesis, University of Texas, Austin, US (2002)
Wong, T.L., Lam, W.: Adapting web information extraction knowledge via mining site-invariant and site-dependent features. ACM Trans. Internet Technol. 7 (February 2007)
Yi, L., Liu, B., Li, X.: Eliminating noisy information in web pages for data mining. In: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 296–305 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Geraci, F., Maggini, M. (2013). A Fast Method for Web Template Extraction via a Multi-sequence Alignment Approach. In: Fred, A., Dietz, J.L.G., Liu, K., Filipe, J. (eds) Knowledge Discovery, Knowledge Engineering and Knowledge Management. IC3K 2011. Communications in Computer and Information Science, vol 348. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37186-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-37186-8_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-37185-1
Online ISBN: 978-3-642-37186-8
eBook Packages: Computer ScienceComputer Science (R0)