Abstract
Web crawlers have been used for nearly a decade as a search engine component to create and update large collections of documents. Typically the crawler and the rest of the search engine are not closely integrated. If the purpose of a search engine is to have as large a collection as possible to serve the general Web community, a close integration may not be necessary. However, if the search engine caters to a specific community with shared focused interests, it can take advantage of such an integration. In this paper we investigate a tightly coupled system in which the crawler and the search engine engage in a symbiotic relationship. The crawler feeds the search engine and the search engine in turn helps the crawler to better its performance. We show that the symbiosis can help the system learn about a community’s interests and serve such a community with better focus.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Arasu, A., Cho, J., Garcia-Molina, H., Paepcke, A., Raghavan, S.: Searching the Web. ACM Transactions on Internet Technology 1(1) (2001)
Attardi, G., Gullí, A., Sebastiani, F.: Automatic Web page categorization by link and context analysis. In: Proceedings of THAI 1999, 1st European Symposium on Telematics, Hypermedia and Artificial Intelligence (1999)
Balabanović, M., Shoham, Y.: Content-based, collaborative recommendation. Communications of the ACM 40(3), 67–72 (1997)
Bradshaw, S., Hammond, K.: Automatically indexing documents: Content vs. reference. In: Sixth International Conference on Intelligent User Interfaces, San Francisco, CA, January 14-17 (2002)
Bradshaw, S.: Reference Directed Indexing: Indexing Scientific Literature in the Context of its Use. PhD thesis, Northwestern University (2002)
Brin, S., Page, L.: The anatomy of a large-scale hypertextual Web search engine. Computer Networks 30(1-7), 107–117 (1998)
Chakrabarti, S., Dom, B., Raghavan, P., Rajagopalan, S., Gibson, D., Kleinberg, J.: Automatic resource compilation by analyzing hyperlink structure and associated text. Computer Networks 30(1-7), 65–74 (1998)
Chakrabarti, S., van den Berg, M., Dom, B.: Focused crawling: A new approach to topic-specific Web resource discovery. Computer Networks 31(11-16), 1623–1640 (1999)
Craswell, N., Hawking, D., Robertson, S.: Effective site finding using link anchor information. In: Proc. 24th Annual Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval (2001)
Giles, C.L., Bollacker, K.D., Lawrence, S.: Citeseer: an automatic citation indexing system. In: Proceedings of the third ACM conference on Digital libraries (1998)
Hersovici, M., Jacovi, M., Maarek, Y.S., Pelleg, D., Shtalhaim, M., Ur, S.: The shark-search algorithm — An application: Tailored Web site mapping. In: WWW7 (1998)
McCallum, A.K., Nigam, K., Rennie, J., Seymore, K.: Automating the construction of internet portals with machine learning. Information Retrieval 3(2), 127–163 (2000)
Menczer, F., Belew, R.K.: Adaptive retrieval agents: Internalizing local context and scaling up to the Web. Machine Learning 39(2-3), 203–242 (2000)
Menczer, F., Pant, G., Ruiz, M., Srinivasan, P.: Evaluating topic-driven Web crawlers. In: Proc. 24th Annual Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval (2001)
Menczer, F., Pant, G., Srinivasan, P.: Topical web crawlers: Evaluating adaptive algorithms. To appear in ACM Trans. on Internet Technologies (2003), http://dollar.biz.uiowa.edu/~fil/Papers/TOIT.pdf
Menczer, F., Street, W.N., Vishwakarma, N., Monge, A., Jakobsson, M.: IntelliShopper: A proactive, personal, private shopping assistant. In: Proc. 1st ACM Int. Joint Conf. on Autonomous Agents and MultiAgent Systems, AAMAS (2002)
Pant, G., Srinivasan, P., Menczer, F.: Exploration versus exploitation in topic driven crawlers. In: WWW 2002 Workshop on Web Dynamics (2002)
Porter, M.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980)
Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, New York (1983)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Pant, G., Bradshaw, S., Menczer, F. (2003). Search Engine-Crawler Symbiosis: Adapting to Community Interests. In: Koch, T., Sølvberg, I.T. (eds) Research and Advanced Technology for Digital Libraries. ECDL 2003. Lecture Notes in Computer Science, vol 2769. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-45175-4_21
Download citation
DOI: https://doi.org/10.1007/978-3-540-45175-4_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40726-3
Online ISBN: 978-3-540-45175-4
eBook Packages: Springer Book Archive