Abstract
The focused crawler is a topic-driven document-collecting crawler that was suggested as a promising alternative of maintaining up-to-date Web document indices in search engines. A major problem inherent in previous focused crawlers is the liability of missing highly relevant documents that are linked from off-topic documents. This problem mainly originated from the lack of consideration of structural information in a document. Traditional weighting method such as TFIDF employed in document classification can lead to this problem.
In order to improve the performance of focused crawlers, this paper proposes a scheme of locality-based document segmentation to determine the relevance of a document to a specific topic. We segment a document into a set of sub-documents using contextual features around the hyperlinks. This information is used to determine whether the crawler would fetch the documents that are linked from hyperlinks in an off-topic document.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
De Bra, P., et al.: Information Retrieval in Distributed Hypertexts. In: Proc. 4th Int’l. Conf. Intelligent Multimedia Information Retrieval System and Management, Center of High Int’l. Studies of Documentary Information Retrieval, pp. 481–491 (1994)
Hersovici, M., et al.: The SharkSearch Algorithm-An Application: Tailored Web Site Mapping. In: Proc. 8th Int’l. World Wide Web Conf., pp. 213–225 (1998)
Cho, J., et al.: Efficient Crawling through URL Ordering. Computer Networks and ISDN Systems 30, 161–172 (1998)
Chakrabarti, S., Van den Berg, M.H., Dom, B.E.: Focused Crawling: A New Approach to Topic-Specific Web Resource Discovery. Computer Networks 31(11-16), 1623–1640 (1999)
Diligenti, M., et al.: Focused Crawling Using Context Graphs. In: Proc. 26’th Int’l. Conf. Very Large Data Bases, pp. 527–534. Morgan Kaufmann, San Francisco (2000)
McCallum, A., et al.: Building Domain-Specific Search Engines with Machine Learning Techniques. In: Proc. AAAI Symp. Intelligent Agents in Cyberspace, pp. 28–39. AAAI Press, Menlo Park (1999)
Mitchell, T.M.: Machine Learning, pp. 154–199. McGraw-Hill, New York (1997)
Yang, Y., Pedersen, J.P.: A Comparative Study on Feature Selection in Text Categorization. In: Proceedings of the Fourteenth International Conference on Machine Learning, pp. 412–420 (1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Yang, J., Kang, J., Choi, J. (2005). A Focused Crawler with Document Segmentation. In: Gallagher, M., Hogan, J.P., Maire, F. (eds) Intelligent Data Engineering and Automated Learning - IDEAL 2005. IDEAL 2005. Lecture Notes in Computer Science, vol 3578. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11508069_13
Download citation
DOI: https://doi.org/10.1007/11508069_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26972-4
Online ISBN: 978-3-540-31693-0
eBook Packages: Computer ScienceComputer Science (R0)