Skip to main content
SpringerLink
Log in

Reliable Evaluations of URL Normalization

  • Conference paper
Computational Science and Its Applications - ICCSA 2006 (ICCSA 2006)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 3984))

Included in the following conference series:

Abstract

URL normalization is a process of transforming URL strings into canonical form. Through this process, duplicate URL representations for web pages can be reduced significantly. There are a number of normalization methods. In this paper, we describe four metrics for evaluating normalization methods. The reliability and consistency of a URL is also considered in our evaluation. With the metrics proposed, we evaluate seven normalization methods. The evaluation results on over 25 million URLs, extracted from the web, are reported in this paper.

This work was supported by Korea Research Foundation Grant (KRF-2004-005-D00172).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Berners-Lee, T., Fielding, R., Masinter, L.: Uniform Resource Identifiers (URI): Generic Syntax (2005), http://gbiv.com/protocols/uri/rfc/rfc2396.html

  2. Burner, M.: Crawling Towards Eternity: Building an Archive of the World Wide Web. Web Techniques Magazine 2(5), 37–40 (1997)

    Google Scholar 

  3. Heydon, A., Najork, M.: Mercator: A Scalable, Extensible Web Crawler. International Journal of WWW 2(4), 219–229 (1999)

    Google Scholar 

  4. Kim, S.J., Lee, S.H.: Implementation of a Web Robot and Statistics on the Korean Web. In: Chung, C.-W., Kim, C.-k., Kim, W., Ling, T.-W., Song, K.-H. (eds.) HSI 2003. LNCS, vol. 2713, pp. 341–350. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  5. Kim, S.-J., Lee, S.-H.: An empirical study on the change of web pages. In: Zhang, Y., Tanaka, K., Yu, J.X., Wang, S., Li, M. (eds.) APWeb 2005. LNCS, vol. 3399, pp. 632–642. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  6. Lee, S.-H., Kim, S.-J., Hong, S.H.: On URL normalization. In: Gervasi, O., Gavrilova, M.L., Kumar, V., Laganá, A., Lee, H.P., Mun, Y., Taniar, D., Tan, C.J.K. (eds.) ICCSA 2005. LNCS, vol. 3481, pp. 1076–1085. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  7. Shkapenyuk, V., Suel, T.: Design and Implementation of a High-performance Distributed Web Crawler. In: Proceedings of 18th Data Engineering Conference, pp. 357–368 (2002)

    Google Scholar 

  8. Netcraft: Web Server Survey (2004), http://news.netcraft.com/archives/web_server_survey.html

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Seoul National University, Seoul, Korea

    Sung Jin Kim

  2. School of Computing, Soongsil University, Seoul, Korea

    Hyo Sook Jeong & Sang Ho Lee

Authors
  1. Sung Jin Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hyo Sook Jeong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sang Ho Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of Calgary,, 2500 University Drive N.W., T2N 1N4, Calgary, AB, Canada

    Marina L. Gavrilova

  2. Department of Mathematics and Computer Science, University of Perugia, via Vanvitelli, 1, I-06123, Perugia, Italy

    Osvaldo Gervasi

  3. William Norris Professor, Head of the Computer Science and Engineering Department, University of Minnesota, USA

    Vipin Kumar

  4. OptimaNumerics Ltd.,, Cathedral House 23-31 Waring Street, BT1 2DX, Belfast, UK

    C. J. Kenneth Tan

  5. Clayton School of IT, Monash University, 3800, Clayton, Australia

    David Taniar

  6. Department of Chemistry, University of Perugia, Via Elce di Sotto, 8, I-06123, Perugia, Italy

    Antonio Laganá

  7. School of Computing, Soongsil University, Seoul, Korea

    Youngsong Mun

  8. School of Information and Communication Engineering, Sungkyunkwan University, Korea

    Hyunseung Choo

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Kim, S.J., Jeong, H.S., Lee, S.H. (2006). Reliable Evaluations of URL Normalization. In: Gavrilova, M.L., et al. Computational Science and Its Applications - ICCSA 2006. ICCSA 2006. Lecture Notes in Computer Science, vol 3984. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11751649_67

Download citation

  • DOI: https://doi.org/10.1007/11751649_67

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-34079-9

  • Online ISBN: 978-3-540-34080-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation