Abstract
Security management center needs to detect and delete many similar records of the device status information to reduce the data redundancy before analyzing the status of the supervised device. Most similarity record detection algorithms are based on the “sort-merge” model. Detection algorithms usually sort data set with keywords before detection of similar data. Existing methods of generating keywords tend to have the following problems: the keywords is not accurate, or multiple keywords are generated for sorting of multiple keywords. The paper proposes a method of synthesizing keywords by multiple encoding fields, and it is verified that this method can significantly optimize the performance of algorithm through experiment. We also compare the performance of each common detection algorithm through experiment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dhivyabharathi, G.V., Kumaresan, S.: A survey on duplicate record detection in real world data. In: International Conference on Advanced Computing and Communication Systems, pp. 1–5 (2016)
Guo, W.: Improved SNM algorithm based on length filtering and effective weights. Comput. Eng. Appl. (2014)
Hernandez, M., Stolfo, S.: Real- world data is dirty: data cleansing and the merge/purge problem. Data Mining Knowl. Discov. 2(1), 9–37 (1998)
Kolb, L., Thor, A., Rahm, E.: Multi-pass sorted neighborhood blocking with MapReduce. Comput. Sci. – Res. Dev. 27(1), 45–63 (2012)
Hernandez, M., Stolfo, S.: The merge/purge problem for large databases. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, San Jose, California, pp. 127–138 (1995)
Shankar V., Rao, C.V.G.: A density based priority queue strategy to evaluate iceberg queries efficiently using compressed bitmap indices. Int. J. Comput. Appl. 67(21), 39–44 (2013)
Monge, A., Elkan, C.: An efficient domain independent algorithm for detecting approximately duplicate database records. In: Proceedings of the SIGMOD Workshop on Data Mining and Knowledge Discovery, Tucson, Arizona, pp. 23–29 (1997)
Minton, S.N., Nanjo, C., Knoblock, C.A., et al.: A heterogeneous field matching method for record linkage. In: Proceeding of the 5th IEEE International Conference on Data Mining, Houston, Texas, USA, pp. 314–321 (2005)
Smith, T.F., Waterman, M.S.: Identification of common molecular subsequences. J. Mol. Biol. 147(1), 195–197 (1981)
Levenshtein, I.V.: Binary codes capable of correcting spurious insertions and deletions of ones. Probl. Inf. Trans. 1, 8–17 (1965)
Rehman, M., Esichaikul, V.: Duplicate record detection for database cleansing. In: Proceedings of the 2nd International Conference on Machine Vision, Dubai, United Arab Emirates, pp. 333–338 (2009)
Acknowledgments
This work is supported by the National Key Research and Development Program of China (2016YFB0800303).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Liu, Z., Fang, L., Yin, L., Guo, Y., Li, F. (2017). Research on Similarity Record Detection of Device Status Information Based on Multiple Encoding Field. In: Wang, G., Atiquzzaman, M., Yan, Z., Choo, KK. (eds) Security, Privacy, and Anonymity in Computation, Communication, and Storage. SpaCCS 2017. Lecture Notes in Computer Science(), vol 10658. Springer, Cham. https://doi.org/10.1007/978-3-319-72395-2_6
Download citation
DOI: https://doi.org/10.1007/978-3-319-72395-2_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-72394-5
Online ISBN: 978-3-319-72395-2
eBook Packages: Computer ScienceComputer Science (R0)