Adaptive Focused Crawling Using Online Learning

Abstract

Focused crawlers aim to automatically discover online content resources relevant to a domain of interest by automatically navigating through the Web link structure and selecting which hyperlinks to follow based on an estimation of their relevance to the topic of interest; to this end, classifier-guided approaches are typically employed for identifying hyperlinks having the higher likelihood of leading to relevant content. However, the training data used for building these classifiers might not be entirely representative of the domain of interest, or the domain of interest might change over time. To meet these challenges, this work proposes a novel adaptive focused crawling framework that allows the classifiers that underlie the hyperlink selection policy to be adapted based on the evidence they encounter during their crawls. Our framework uses two different approaches to retrain its models: (i) Interactive Adaptation, where a user manually evaluates the discovered resources, and (ii) Automatic Adaptation, where the framework uses the already trained classifiers to assess the relevance of newly discovered resources. The evaluation experiments in the domain of Islamic extremism indicate the effectiveness of online learning in focused crawling.