Abstract
Duplicate webpages can affect the user experience of search engine. This paper proposed webpage deletion algorithm based on hierarchical filtering according to the features of duplicate webpage. The webpage feature extraction is divided into three layers, which are paragraphs, sentences and words. The webpage features are formed by layer filtering redundant information. In the sentence layer paragraph sentences are extracted according to the sentence semantics, while in the word layer the sentences are denoised filtering based on statistics of the part of speech in them. This algorithm improves the noise immunity and the original coverage of the feature extraction. The experiments show that the proposed method can accurately filter out duplicate webpage.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Ukkonen, E.: Constructing Suffix Trees On-Line in Linear Time. In: Proceedings of the IFIP 12th World Computer Congress on Algorithms, Software, Architecture, pp. 484–492 (1992)
Broder, A.: Identifying and Filtering Near-Duplicate Documents. In: Giancarlo, R., Sankoff, D. (eds.) CPM 2000. LNCS, vol. 1848, pp. 1–10. Springer, Heidelberg (2000)
Zhang, G., Liu, T., Zheng, S.F.: Fast Deletion Algorithm for Large Scale Duplicated Web Pages. The First Student Computational Linguistics Seminar 2002 (2002)
Elhadi, M., Al-Tobi, A.: Webpage Duplicate Detection Using Combined POS and Sequence Alignment Algorithm. In: 2009 World Congress on Computer Science and Information Engineering, pp. 630–634 (2009)
Theobald, M., Siddharth, J., Paecke, A.: SpotSigs: Robust and Efficient Near Duplicate Detection in Large Web Collections. In: Proceeding of the 31st Annual International ACM SIGIR Conference on Resesrch and Development in Information Retrieval, pp. 563–570 (2008)
Chowdhury, A., Frieder, O., Grossman, D., et al.: Collection Statistics for Fast Duplicate Document Detection. ACM Transactions on Information System 20(2), 171–191 (2002)
Meng, W.C., Liu, L.C., Dai, T.: A Modified Approach to Keyword Extraction Based on Word-similarity. In: Proceedings of the Sixteenth International Florida Artificial Intelligence Research Society Conference, pp. 392–396 (2003)
Matsuo, Y., Ishizuka, M.: Keyword Extraction From a Single Document Using Word Co-occurrence Statistical Information. International Journal on Artificial Intelligence Tools 13(1), 157–169 (2004)
Luo, J., Chen, L.: Research on Fast Text Classifier Based on New Keywords Extraction Method. Application Research of Computers 4, 32–34 (2006)
Cheng, L.L., He, P.L., Sun, Y.H.: Study on Chinese Keyword Extraction Algorithm Based on Naive Bayes Model. Journal of Computer Applications 25(12), 2780–2782 (2005)
Fang, J., Guo, L., Wang, X.D.: Semantically Improved Automatic Keyphrase Extraction. Computer Science 35(6), 148–150 (2008)
Liu, S.W., Zhang, Y., Xia, Y.M.: Webpage Deletion Algorithm Based on HTML Mark and Long Sentence Retrieve. Microcomputer Applications 25(8), 30–32 (2009)
Fan, Y., Zheng, J.H.: Research on Elimination of Similar Web Pages Computer. Engineering and Application 45(12), 141–143 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chen, X., Wang, W., Man, D., Xuan, S. (2012). A Webpage Deletion Algorithm Based on Hierarchical Filtering. In: Wang, F.L., Lei, J., Gong, Z., Luo, X. (eds) Web Information Systems and Mining. WISM 2012. Lecture Notes in Computer Science, vol 7529. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33469-6_68
Download citation
DOI: https://doi.org/10.1007/978-3-642-33469-6_68
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33468-9
Online ISBN: 978-3-642-33469-6
eBook Packages: Computer ScienceComputer Science (R0)