Skip to main content
SpringerLink
Log in

Focused crawling for the hidden web

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

A constantly growing amount of high-quality information resides in databases and is guarded behind forms that users fill out and submit. The Hidden Web comprises all these information sources that conventional web crawlers are incapable of discovering. In order to excavate and make available meaningful data from the Hidden Web, previous work has focused on developing query generation techniques that aim at downloading all the content of a given Hidden Web site with the minimum cost. However, there are circumstances where only a specific part of such a site might be of interest. For example, a politics portal should not have to waste bandwidth or processing power to retrieve sports articles just because they are residing in databases also containing documents relevant to politics. In cases like this one, we need to make the best use of our resources in downloading only the portion of the Hidden Web site that we are interested in. We investigate how we can build a focused Hidden Web crawler that can autonomously extract topic-specific pages from the Hidden Web by searching only the subset that is related to the corresponding area. In this regard, we present an approach that progresses iteratively and analyzes the returned results in order to extract terms that capture the essence of the topic we are interested in. We propose a number of different crawling policies and we experimentally evaluate them with data from four popular sites. Our approach is able to download most of the content in search in all cases, using a significantly smaller number of queries compared to existing approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17

Similar content being viewed by others

Notes

  1. www.informatik.uni-trier.de/∼ley/db/

  2. www.imdb.com/?

  3. http://www.dmoz.org

  4. http://stackexchange.com/

  5. The New York Times Annotated Corpus, Linguistic Data Consortium, Philadelphia, http://catalog.ldc.upenn.edu/LDC2008T19

  6. http://istc-bigdata.org/index.php/our-research-data-sets/

References

  1. Álvarez, M., Raposo, J., Pan, A., Cacheda, F., Bellas, F., Carneiro, V.: Deepbot: A focused crawler for accessing hidden web content. In: Proceedings of the 3rd International Workshop on Data Enginering Issues in E-commerce and Services (EC), pp. 18–25, San Diego (2007)

  2. Barbosa, L., Freire, J.: Siphoning hidden-web data through keyword-based interfaces. In: SBBD, pp. 309–321. Distrito Federal, Brasil (2004)

  3. Barbosa, L., Freire, J.: Searching for hidden-web databases. In: Proceedings of the 8th International WebDB, pp. 1–6, Baltimore (2005)

  4. Barbosa, L., Freire, J.: An adaptive crawler for locating hidden-web entry points. In: Proceedings of the 16th International Conference on World Wide Web (WWW), pp. 441–450. Banff, Canada (2007)

  5. Bergholz, A., Chidlovskii, B.: Crawling for domain-specific hidden web resources. In: Proceedings of the 4th International Conference on Web Information Systems Engineering (WISE), pp. 125–133, Roma (2003)

  6. Bergman, M.K.: The deep web. surfacing hidden value. J. Electron. Publ. 7(1), 1–17 (2001)

    Article  Google Scholar 

  7. Cafarella, M.J., Madhavan, J., Halevy, A.: Web-scale extraction of structured data. SIGMOD Rec. 37(4), 55–61 (2009)

    Article  Google Scholar 

  8. Chakrabarti, S., van den Berg, M., Dom, B.: Focused crawling: A new approach to topic-specific web resource discovery. In: In Proceedings of the 8th International Conference on World Wide Web (WWW), pp. 1623–1640, Toronto (1999)

  9. Diligenti, M., Coetzee, F., Lawrence, S., Giles, C.L., Gori, M.: Focused crawling using context graphs. In: Proceedings of the 26th International Conference on Very Large Data Bases (VLDB), pp. 527–534, Cairo (2000)

  10. Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian network classifiers. Mach. Learn. 29(2–3), 131–163 (1997)

    Article  MATH  Google Scholar 

  11. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: An update. SIGKDD Explor. Newsl. 11(1), 10–18 (2009)

    Article  Google Scholar 

  12. He, B., Patel, M., Zhang, Z., Chang, K.C.-C.: Accessing the deep web: A survey. Communications of the ACM 50(5), 94–101 (2007)

  13. Ipeirotis P.G., Gravano, L.: Distributed search over the hidden web: Hierarchical database sampling and selection. In: Proceedings of the 28th International Conference on Very Large Data Bases (VLDB), pp. 394–405, Hong Kong (2002)

  14. Ipeirotis, P.G., Gravano, L., Sahami, M.: Probe, count, and classify: Categorizing hidden web databases. SIGMOD Rec. 30, 67–78 (2001)

    Article  Google Scholar 

  15. Liakos P., Ntoulas, A.: Topic-sensitive hidden-web crawling. In: Proceedings of the 13th International Conference on Web Information Systems Engineering (WISE), pp. 538–551, Paphos (2012)

  16. Lim, T.-S., Loh, W.-Y., Shih, Y.-S.: A comparison of prediction accuracy, complexity, and training time of old, thirty-three algorithms, new classification. Mach. Learn. 40(3), 203–228 (2000)

    Article  MATH  Google Scholar 

  17. Lu, J., Wang, Y., Liang, J., Chen, J., Liu, J.: An approach to deep web crawling by sampling. In: Proceedings of the 2008 IEEE / WIC / ACM International Conference on Web Intelligence, (WI), pp. 718–724, New SouthWales (2008)

  18. Madhavan, J., Ko, D., Kot, Ł., Ganapathy, V., Rasmussen, A., Halevy, A.: Google’s deep web crawl. Proc. VLDB Endow. 1(2), 1241–1252 (2008)

    Article  Google Scholar 

  19. McCandless, M., Hatcher, E., Gospodnetic, O.: Lucene in Action, 2nd. Manning Publications Co., Greenwich (2010)

    Google Scholar 

  20. Noh, S., Choi, Y., Seo, H., Choi, K., Jung, G.: An intelligent topic-specific crawler using degree of relevance. In: IDEAL, volume 3177 of Lecture Notes in Computer Science, pp. 491–498 (2004)

  21. Ntoulas, A., Zerfos, P., Cho, J.: Downloading textual hidden web content through keyword queries. In: Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL), pp. 100–109, Denver (2005)

  22. Platt, J.C.: Advances in Kernel Methods. Chapter Fast Training of Support Vector Machines Using Sequential Minimal Optimization, pp. 185–208. MIT Press, Cambridge (1999)

    Google Scholar 

  23. Raghavan, S., Garcia-Molina, H.: Crawling the hidden web. In: Proceedings of the 27th International Conference on Very Large Data Bases (VLDB), p. 2001, Roma

  24. Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill Inc., New York (1986)

    MATH  Google Scholar 

  25. Schonhofen, P.: Identifying document topics using the wikipedia category network. In: Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pp. 456–462, Hong Kong (2006)

  26. Wang, Y., Lu, J., Chen, J.: Crawling deep web using a new set covering algorithm. In: Proceedings of the 5th International Conference on Advanced Data Mining and Applications (ADMA), pp. 326–337, Beijing (2009)

  27. Wu, P., Wen, J.-R., Liu, H., Ma, W.-Y. : Query selection techniques for efficient crawling of structured web sources, p. 47, Atlanta (2006)

  28. Wu, W., Yu, C., Doan, A., Meng, W.: An interactive clustering-based approach to integrating source query interfaces on the deep web. In: Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data, pp. 95–106, Paris (2004)

  29. Yang, Y., Bansal, N., Dakka, W., Ipeirotis, P., Koudas, N., Papadias D.: Query by document. In: Proceedings of the 2nd ACM International Conference on Web Search and Data Mining (WSDM), pp. 34–43, Barcelona (2009)

  30. Zhang, Z, He, B., Chang, K. C.-C.: Understanding web query interfaces: Best-effort parsing with hidden syntax. In: Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data, pp. 107–118, Paris (2004)

Download references

Acknowledgements

This work has been partially supported by SocWeb and Sucre FP7 EU projects. A preliminary version of the work appeared in the Proc. of the 13th Int. Conf. on Web Information Systems Engineering [15].

Author information

Authors and Affiliations

  1. Universtiy of Athens, 15784, Athens, Greece

    Panagiotis Liakos, Alexandros Ntoulas & Alex Delis

  2. Zynga, San Fransisco, CA, 94103, USA

    Alexandros Ntoulas

  3. University of Pittsburgh, Pittsburgh, PA, 15260, USA

    Alexandros Labrinidis

Authors
  1. Panagiotis Liakos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexandros Ntoulas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexandros Labrinidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alex Delis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Panagiotis Liakos.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liakos, P., Ntoulas, A., Labrinidis, A. et al. Focused crawling for the hidden web. World Wide Web 19, 605–631 (2016). https://doi.org/10.1007/s11280-015-0349-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-015-0349-x

Keywords

Search

Navigation