Abstract
Topical crawlers are becoming important tools to support applications such as specialized Web portals, online searching, and competitive intelligence. A topic driven crawler chooses the best URLs to pursue during web crawling. It is difficult to evaluate what URLs downloaded are the best. This paper presents some important metrics and an evaluation function for ranking URLs about pages relevance. We also discuss an approach to evaluate the function based on GA. GA evolving process can discover the best combination of the metrics’ weights. Avoiding misleading the result by a single topic, this paper presents a method which characterization of the metrics’ combination be extracted by mining frequent patterns. Extracting features adopts a novel FP-tree structure and FP-growth mining method based on FP-tree without candidate generation. The experiment shows that the performance is exciting, especially about a popular topic.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Pinkerton, B.: Finding What People Want:Experiences with the WebCrawler. In: Proceedings of the 2nd International World Wide Web Conference, Chicago, IL, USA (1994)
De Bra, R., Post, D.J.: Information Retrieval in the World-Wide Web: Making Client-based Searching Feasible. In: Proceedings of the First International World-Wide Web conference, Geneva (1994)
Hersovici, M., Jacovi, M., Maarek, Y.S., Pelleg, D., Shtalhaim, M., Ur, S.: The shark-search algorithm-An application:Tailored Web site mapping. In: Proc. 7th Intl. World-Wide Web Conference (1998)
Cho, J., Garcia-Molina, H., Page, L.: Efficient Crawling Through URL Ordering. In: Proceedings of 7th World Wide Web Conference (1998)
Menczer, F., Belew, R.: Adaptive retrieval agents: internalizing local context and scaling up to the web. Machine Learning 39(2–3), 203–242 (2000)
Pant, G., Menczer, F.: Topical Crawling for Business Intelligence. In: Koch, T., Sølvberg, I.T. (eds.) ECDL 2003. LNCS, vol. 2769, pp. 233–244. Springer, Heidelberg (2003)
Holland, J.H.: Adaptation in Natural and Artificial Systems. The University of Michigan Press (1975)
Goldberg, D.E.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, New York (1989)
Johnson, J., Tsioutsiouliklis, K., Giles, C.L.: Evolving strategies for focused Web crawling. In: Proceedings of the Twentieth International Conference on Machine Learning (ICML 2003), Washington DC (2003)
Xu, B.W., Zhang, W.F.: Search Engine and Information Retrieval Technology, pp. 147–150. Tsinghua university press, BeiJing (2001)
Zhou, C.G., Liang, Y.C.: Computational Intelligence. Jilin university press, Changchun (2001)
Herrera, F., Lozano, M., Verdegay, J.L.: Tackling Real Coded Genetic Algorithms: Operators and Tools for Behavioural Analysis. Artificial Intelligence Review, vol. 12, pp. 265–319. Kluwer Academic Publishers, Dordrecht (1998); Printed in the Netherlands
Han, J., Pei, J., Yin, Y.: Mining Frequent Patterns without Candidate Generation. In: SIGMOD Conference, pp. 1–12 (2000)
Han, J., Kamber, M.: Data Mining:Concepts and Techniques. Higher Education Press (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Peng, T., Zuo, W., Liu, Y. (2005). Characterization of Evaluation Metrics in Topical Web Crawling Based on Genetic Algorithm. In: Wang, L., Chen, K., Ong, Y.S. (eds) Advances in Natural Computation. ICNC 2005. Lecture Notes in Computer Science, vol 3611. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11539117_98
Download citation
DOI: https://doi.org/10.1007/11539117_98
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-28325-6
Online ISBN: 978-3-540-31858-3
eBook Packages: Computer ScienceComputer Science (R0)