Abstract
Web information is growing at an explosive rate. The crawling ability of the single-machine crawler becomes the bottleneck, so distributed web crawling techniques become the focus of research. However, the existing distributed web crawler systems have some shortcomings. Thread management for solving thread synchronization and resource competition is usually designed by using pure multi-thread asynchronous methods. But the execution of this mechanism observably reduces the performance. Moreover, the deduplication algorithms lead to low efficiency in dealing with large data sets or the problem of occupying large storage space. Therefore, we propose and implement a distributed web crawler system based on Apache Flink, which combines and integrates the Mesos/Marathon framework. It can make full use of the computing resources of the cluster and significantly improve the efficiency of the web crawler system. Taking the data of Netease news pages as an example, the experimental results show that the distributed crawler proposed has higher execution efficiency and reliability.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Cho, J., Garcia-Molina, H.: Parallel crawlers. In: 11th International Conference Proceedings on World Wide Web, pp. 124–135. Association for Computing Machinery, Honolulu (2002)
Boldi, P., Codenotti, B., Santini, M., Vigna, S.: UbiCrawler: a scalable fully distributed web crawler. Softw.-Pract. Exp. 34(8), 711–726 (2004)
Cambazoglu, B.B., Karaca, E., Kucukyilmaz, T., Turk, A., Aykanat, C.: Architecture of a grid-enabled web search engine. Inf. Process. Manag. 43(3), 609–623 (2007)
Singh, A., Srivatsa, M., Liu, L., Miller, T.: Apoidea: a decentralized peer-to-peer architecture for crawling the world wide web. In: Callan, J., Crestani, F., Sanderson, M. (eds.) SIGIR 2003 Workshop on Distributed Information Retrieval 2003, LNCS, vol. 2924, pp. 126–142. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24610-7_10
Loo, B.T., Krishnamurthy, S., Cooper, O.: Distributed Web Crawling over DHTs. University of California at Berkely, Berkely (2004)
Heydon, A., Najork, M.: Mercator: a scalable, extensible web crawler. World Wide Web-Internet Web Inf. Syst. 2(4), 219–229 (1999)
Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. In: the19th ACM Symposium Proceedings on Operating Systems Principles, pp. 29–43. Association for Computing Machinery, Lake George (2003)
Achsan, H.T.Y., Wibowo, W.C.: A fast distributed focused-web crawling. Proc. Eng. 69, 492–499 (2014)
Huang, Q., Li, Q., Yan, Z., Fu, H.: A novel incremental parallel web crawler based on focused crawling. J. Comput. Inf. Syst. 9(6), 2461–2469 (2013)
Su, L., Wang, F.: Web crawler model of fetching data speedily based on hadoop distributed system. In: IEEE International Conference Proceedings on Software Engineering and Service Science, pp. 927–931. IEEE Computer Society, Beijing (2016)
Yang, Y., Yang, J.: Design and implementation of a multi-area distributed crawler based on Skipnet-YL network. In: Pacific-Asia Conference Proceedings on Circuits Communications and System, pp. 4–6. IEEE Computer Society, Wuhan (2011)
Friedman, E., Tzoumas, K.: Introduction to Apache Flink: Stream Processing for Real Time and Beyond. O’Reilly Media, Sebastopol (2016)
Deshpande, T.: Learning Apache Flink. Packt Publishing, Birmingham (2017)
Gen, L.: Research and Optimization of Web Crawler System under Distributed Environment. Beijing University of Posts and Telecommunications, Beijing (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Ye, F., Jing, Z., Huang, Q., Hu, C., Chen, Y. (2018). The Research and Implementation of a Distributed Crawler System Based on Apache Flink. In: Hu, T., Wang, F., Li, H., Wang, Q. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2018. Lecture Notes in Computer Science(), vol 11338. Springer, Cham. https://doi.org/10.1007/978-3-030-05234-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-05234-8_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-05233-1
Online ISBN: 978-3-030-05234-8
eBook Packages: Computer ScienceComputer Science (R0)