Abstract
According to the characteristics of duplicate web pages in the meta search engine, a duplicate web pages detection algorithm is proposed based on a web page URL, title and abstract, and according to their different characteristics, different similarity computing method is proposed, firstly, the page URL is standardization processed in the algorithm, and then for the title detection, the algorithm improves the title string fuzzy matching algorithm and calculate the similarity based on the word frequency of each items in the query, for the abstract judgment, similarity computing is in accordance with the sentences of the abstract, for each sentence the algorithm gives three weights, and calculates the weights of similarity on base of each summary statement, the effect of the algorithm is obvious, it has been verified by experiment that the algorithm is superior to the traditional algorithm in the precision and recall rate.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Smyth, B., Boydell, O.: Meta search engine. US (2010)
Lawrence, S., Giles, L.: The NECI meta search engine. In: World Wide Web Conference Series
Lawrence, S., Giles, C.L.: Inquirus, the NECI meta search engine. Comput. Netw. ISDN Syst. 30, 95–105 (1998)
Zhao-Hui, X.U., Zhao, S.M., Yan, F.L., Qin, J.: An improved DSC removing duplicated webpages algorithm based on feature vector. Sci. Technol. Eng. (2013)
Guo-Rong, S.U., Yang, Y.X., Deng, J.S.: An algorithm of removing duplicate URL. J. Guangxi Normal Univ. (2010)
Qiang, S., Cheng, G.: Comparison and analysis of hash algorithm based on flows. J. Nanjing Normal Univ. (2008)
Rathinasabapathy, R., Bhaskaran, R.: Performance comparison of hashing algorithm with Apriori. In: International Conference on Advances in Computing, Control, & Telecommunication Technologies, pp. 729–733
Ye, F., Liu, J., Liu, B., Chai, K.: Duplicate page detection algorithm based on the field characteristic clustering. In: Luo, X., Cao, Y., Yang, B., Liu, J., Ye, F. (eds.) ICWL 2010. LNCS, vol. 6537, pp. 75–84. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20539-2_9
Yang, J.W.: A Chinese web page clustering algorithm based on the suffix tree. In: Conference on Web Information System and Applications, pp. 817–822
YuJun, Y., YiMei, Y.: Text information hiding algorithm based on dot-matrix character code. Comput. Syst. Appl. 19, 231–233 (2010)
Acknowledgments
This work was funded by the National Natural Science Foundation of China under Grant (No. 61772152 and No. 61502037), the Basic Research Project (No. JCKY2016206B001, JCKY2014206C002 and JCKY2017604C010), and the Technical Foundation Project (No. JSQB2017206C002).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, H., He, M., Zhou, L., Li, Z., Zhan, H., Wang, R. (2018). Remove-Duplicate Algorithm Based on Meta Search Result. In: Sun, X., Pan, Z., Bertino, E. (eds) Cloud Computing and Security. ICCCS 2018. Lecture Notes in Computer Science(), vol 11064. Springer, Cham. https://doi.org/10.1007/978-3-030-00009-7_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-00009-7_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00008-0
Online ISBN: 978-3-030-00009-7
eBook Packages: Computer ScienceComputer Science (R0)