Abstract
Syntactically different URLs could represent the same web page on the World Wide Web, and duplicate representation for web pages causes web applications to handle a large amount of same web pages unnecessarily. In the standard communities, there are on-going efforts to define the URL normalization that helps eliminate duplicate URLs. On the other hand, there are research efforts to extend the standard URL normalization methods to reduce false negatives further while allowing false positives on a limited level. This paper presents a method that evaluates the effectiveness of a URL normalization method in terms of page loss/gain/change and the URL reduction. Over 94 million URLs were extracted from web pages for our experiment and interesting statistical results are reported in this paper.
This work was supported by grant number KRF-2004-005-D00172 from the Korea Science and Engineering Foundation.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Burner, M.: Crawling Towards Eternity: Building an Archive of the World Wide Web. Web Techniques Magazine 2(5), 37–40 (1997)
Heydon, A., Najork, M.: Mercator: A Scalable, Extensible Web Crawler. International Journal of WWW 2(4), 219–229 (1999)
Kim, S.J., Lee, S.H.: Implementation of a Web Robot and Statistics on the Korean Web. LNCS, vol. 2713, pp. 341–350. Springer, Heidelberg (2003)
Shkapenyuk, V., Suel, T.: Design and Implementation of a High-performance Distributed Web Crawler. In: Proceedings of 18th Data Engineering Conference, pp. 357–368 (2002)
Berners-Lee, T., Fielding, R., Masinter, L.: Uniform Resource Identifiers (URI): Generic Syntax (2004), http://gbiv.com/protocols/uri/rev-2002/rfc2396bis.html
Lee, S.H., Kim, S.J.: On URL Normalization. In: Proceedings of the Workshop on Ubiquitous Web Systems and Intelligence (2005) (to appear)
Netcraft: Web Server Survey (2004), http://news.netcraft.com/archives/web_server_survey.html
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lee, S.H., Kim, S.J., Jeong, H.S. (2005). How to Evaluate the Effectiveness of URL Normalizations. In: Shimojo, S., Ichii, S., Ling, TW., Song, KH. (eds) Web and Communication Technologies and Internet-Related Social Issues - HSI 2005. HSI 2005. Lecture Notes in Computer Science, vol 3597. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11527725_24
Download citation
DOI: https://doi.org/10.1007/11527725_24
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-27830-6
Online ISBN: 978-3-540-31808-8
eBook Packages: Computer ScienceComputer Science (R0)