Skip to main content
SpringerLink
Log in

Fixing the Threshold for Effective Detection of Near Duplicate Web Documents in Web Crawling

  • Conference paper
Advanced Data Mining and Applications (ADMA 2010)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 6440))

Included in the following conference series:

Abstract

The drastic development of the WWW in recent times has made the concept of Web Crawling receive remarkable significance. The voluminous amounts of web documents swarming the web have posed huge challenges to web search engines making their results less relevant to the users. The presence of duplicate and near duplicate web documents in abundance has created additional overheads for the search engines critically affecting their performance and quality which have to be removed to provide users with the relevant results for their queries. In this paper, we have presented a novel and efficient approach for the detection of near duplicate web pages in web crawling where the keywords are extracted from the crawled pages and the similarity score between two pages is calculated. The documents having similarity score greater than a threshold value are considered as near duplicates. In this paper we have fixed the threshold value.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Narayana, V.A., Premchand, P., Govardhan, A.: A Novel and Efficient Approach For Near Duplicate Page Detection in Web crawling. In: IEEE International Advance Computing Conference, Patiala, pp. 1492–1496 (2009)

    Google Scholar 

  2. Pant, G., Srinivasan, P., Menczer, F.: Crawling the Web. In: Web Dynamics: Adapting to Change in Content, Size, Topology and Use. Springer, Heidelberg (2004)

    Google Scholar 

  3. Balamurugan, S., Rajkumar, N.: Design and Implementation of a New Model Web Crawler with Enhanced Reliability. Proceedings of World Academy of Science, Engineering and Technology 32 (2008) ISSN 2070-3740

    Google Scholar 

  4. Menczer, F., Pant, G., Srinivasan, P., Ruiz, M.E.: Evaluating topic-driven web crawlers. In: Proc. 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 241–249. ACM, New Orleans (2001)

    Google Scholar 

  5. Broder, A.Z., Najork, M., Wiener, J.L.: Efficient URL caching for World Wide Web crawling. In: International Conference on World Wide Web, pp. 679–689. ACM, Budapest (2003)

    Google Scholar 

  6. Chakrabarti, S.: Mining the Web: Discovering Knowledge from Hypertext Data. Morgan Kaufmann, San Francisco (2002)

    Google Scholar 

  7. Cho, J., Garcia-Molina, H., Page, L.: Efficient crawling through URL ordering. Computer Networks and ISDN Systems 30(1-7), 161–172 (1998)

    Article  Google Scholar 

  8. Charikar, M.: Similarity estimation techniques from rounding algorithms. In: Proc. 34th Annual Symposium on Theory of Computing (STOC 2002), pp. 380–388. ACM, Montreal (2002)

    Google Scholar 

  9. Cho, J., Shivakumar, N., Garcia-Molina, H.: Finding replicated web collections. ACM SIGMOD Record 29(2), 355–366 (2000)

    Article  Google Scholar 

  10. Conrad, J.G., Guo, X.S., Schriber, C.P.: Online duplicate document detection: signature reliability in a dynamic retrieval environment. In: CIKM, pp. 443–452. ACM, New Orleans (2003)

    Google Scholar 

  11. Pandey, S., Olston, C.: User-centric Web crawling. In: Proceedings of the 14th International Conference on World Wide Web, pp. 401–411. ACM, Chiba (2005)

    Chapter  Google Scholar 

  12. Xiao, C., Wang, W., Lin, X.M., Xu Yu, J.: Efficient Similarity Joins for Near Duplicate Detection. In: Proceeding of the 17th International Conference on World Wide Web, pp. 131–140. ACM, Beijing (2008)

    Chapter  Google Scholar 

  13. Henzinger, M.: Finding near-duplicate web pages: a large-scale evaluation of algorithms. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 284–291. ACM, Seattle (2006)

    Google Scholar 

  14. Castillo, C.: Effective web crawling. ACM SIGIR Forum 39(1), 55–56 (2005)

    Article  MathSciNet  Google Scholar 

  15. Manku, G.S., Jain, A., Sarma, A.D.: Detecting near-duplicates for web crawling. In: Proceedings of the 16th International Conference on World Wide Web, pp. 141–150. ACM, Banff (2007)

    Google Scholar 

  16. Gibson, D., Kumar, R., Tomkins, A.: Discovering large dense subgraphs in massive graphs. In: VLDB, pp. 721–732. ACM, Trondheim (2005)

    Google Scholar 

  17. Spertus, E., Sahami, M., Buyukkokten, O.: Evaluating similarity measures: a large-scale study in the orkut social network. In: KDD, pp. 678–684. ACM, Chicago (2005)

    Google Scholar 

  18. Singh, A., Srivatsa, M., Liu, L., Miller, T.: Apoidea: A Decentralized Peer-to-Peer Architecture for Crawling the World Wide Web. In: Proceedings of the SIGIR 2003 Workshop on Distributed Information Retrieval. LNCS, pp. 126–130. ACM, Toronto (2003)

    Google Scholar 

  19. Lovins, J.B.: Development of a stemming algorithm. Mechanical Translation and Computational Linguistics 11, 22–31 (1968)

    Google Scholar 

  20. Bacchin, M., Ferro, N., Melucci, M.: Experiments to evaluate a statistical stemming algorithm. In: Peters, C., Braschler, M., Gonzalo, J. (eds.) CLEF 2002. LNCS, vol. 2785, pp. 161–168. Springer, Heidelberg (2003)

    Google Scholar 

  21. Brin, S., Davis, J., Garcia-Molina, H.: Copy detection mechanisms for digital documents. ACM SIGMOD Record 24(2), 398–409 (1995)

    Article  Google Scholar 

  22. Broder, A.Z., Glassman, S.C., Manasse, M.S., Zweig, G.: Syntactic clustering of the web. In: Proceedings of WWW6 1997, pp. 391–404. Elsevier Science, Santa Clara (1997)

    Google Scholar 

  23. Conrad, J., Schriber, C.P.: Online duplicate document detection: signature reliability in a dynamic retrieval environment. In: Proceedings of the Twelfth International Conference on Information and Knowledge Management, pp. 443–452. ACM, New Orleans (2003)

    Google Scholar 

  24. Metzler, D., Bernstein, Y., Bruce Croft, W.: Similarity Measures for Tracking Information Flow. In: Proceedings of the Fourteenth International Conference on Information and Knowledge Management, CIKM 2005, pp. 517–524. ACM, Bremen (2005)

    Google Scholar 

  25. Yang, H., Callan, J.: Near-duplicate detection for eRulemaking. In: Proceedings of the 2005 National conference on Digital Government Research, pp. 78–86. Digital Government Society of North America, Atlanta (2005)

    Google Scholar 

  26. Yang, H., Callan, J., Shulman, S.: Next steps in near-duplicate detection for eRulemaking. In: Proceedings of the 2006 International Conference on Digital Government Research, pp. 239–248. ACM, San Diego (2006)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, CMR College of Engineering & Technology, Hyderabad, India

    V. A. Narayana

  2. Department of Computer Science & Engineering, University College of Engineering, Osmania University, Hyderabad, AP, India

    P. Premchand

  3. JNTUH College of Engineering, Nachupally (Kondagattu), Karimnagar Dist, AP, India

    A. Govardhan

Authors
  1. V. A. Narayana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. Premchand
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. Govardhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Faculty of Engineering and Information Technology, University of Technology Sydney, 2007, Sydney, NSW, Australia

    Longbing Cao

  2. College of Computer Science, Chongqing University, 400030, Chongqing, China

    Yong Feng

  3. College of Computer Science, Chongqing University , 400030, Chongqing, China

    Jiang Zhong

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Narayana, V.A., Premchand, P., Govardhan, A. (2010). Fixing the Threshold for Effective Detection of Near Duplicate Web Documents in Web Crawling. In: Cao, L., Feng, Y., Zhong, J. (eds) Advanced Data Mining and Applications. ADMA 2010. Lecture Notes in Computer Science(), vol 6440. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17316-5_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-17316-5_16

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-17315-8

  • Online ISBN: 978-3-642-17316-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation