Abstract
World Wide Web (WWW) is a biggest place of information repository in the Universe. A common man often seeks the assistance of Web for gathering information to enrich and enhance the knowledge of his interest to become an expert in his/her field. More often than not, search engines come in handy to provide information to the user. The nightmare of the search engines relies on the relevancy of the result-set presented to the user. To provide more relevant results, most of the search engines will have Web crawler in its armory as an important component to index the Web pages. Web crawlers (also called Web Spiders or Robots) are programs used to download documents from the internet. A focused crawler is a specialized crawler which will search for and index the Web page of a particular topic, thus reducing the amount of network traffic and download. This paper determines and identified a set of factors to determine the relevancy of Web documents and introduces a Contextual metadata framework to summarize the captured relevancy data that can be used to categorize and sort results and in essence to improve the quality of the result-set presented to the end user.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Lawrence, S., Giles, C.: Accessibility of information on the web. Comput. J. Nat. 400(6740), 107–109 (1999)
Qin, J., Zhou, Y., Chau, M.: Building domain-specific web collections for scientific digital libraries: a meta search enhanced focused crawling method. Digital Libraries, 2004. Proceedings of the 2004 Joint ACM/IEEE Conference on. IEEE (2004)
Liu, B.: Web data mining (from Chaps. 6, 7, 8). pp. 183–235, 237–270, 273–31. Springer, Berlin (2007)
Cho, J., Garcia-Molina, H., Page, L.: Efficient crawling through URL ordering. Comp. Netw. ISDN Syst. 30(1–7):161–172 (1998)
Shkapenyuk, V., Suel, T.: Design and implementation of a high-performance distributed web crawler. Data Engineering, 2002. Proceedings. 18th International Conference on. IEEE (2002)
Boldi, P., Codenotti, B., Santini, M., Vigna, S.: Ubicrawler: a scalable fully distributed web crawler (2002)
Yuan, X., Harms, J.: An efficient scheme to remove crawler traffic from the internet. In: Proceedings of the 11th International Conferences On Computer Communications and Networks, pp. 90–95. IEEE, New York (2002)
Chakrabarti, S.: Mining the Web (from Chaps. 2, 3). Giga-Pedia, pp. 17–77
Chakrabarti, S., van den Berg, M., Dom, B.: Focused crawling: a new approach to topic-specific web resource discovery. Comput. Netw. 31(11), 1623–1640 (1999)
Faniel, I.M., Yakel, E.,: Significant properties as contextual metadata. J. Libr. Metadata 11(3–4), 155–165
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer India
About this paper
Cite this paper
Rajesh, L., Shanthi, V., Varadhan, V. (2015). Enhanced Web Crawler Design to Classify Web Documents Using Contextual Metadata. In: Das, K., Deep, K., Pant, M., Bansal, J., Nagar, A. (eds) Proceedings of Fourth International Conference on Soft Computing for Problem Solving. Advances in Intelligent Systems and Computing, vol 336. Springer, New Delhi. https://doi.org/10.1007/978-81-322-2220-0_42
Download citation
DOI: https://doi.org/10.1007/978-81-322-2220-0_42
Published:
Publisher Name: Springer, New Delhi
Print ISBN: 978-81-322-2219-4
Online ISBN: 978-81-322-2220-0
eBook Packages: EngineeringEngineering (R0)