Abstract
The Open Archives Initiative (OAI) is making feasible to build high level services such as a federated search service that harvests metadata from different data providers using the OAI protocol for metadata harvesting (OAI-PMH) and provides a unified search interface. There are numerous challenges to build and maintain a federation service, and one of them is managing duplicates. Detecting exact duplicates where two records have identical set of metadata fields is straight-forward. The problem arises when two or more records differ slightly due to data entry errors, for example. Many duplicate detection algorithms exist, but are computationally intensive for large federated digital library. In this paper, we propose an efficient duplication detection algorithm for a large federated digital library like Arc.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sung., S.Y., Li., Z., Peng, S.: A Fast Filtering Scheme for Large Database Cleansing. In: Proceedings 11th international conference on Inform & knowledge management. ACM, New York (2002)
Hernandez, M., Stolfo, S.: The merge/purge problem for large databases. In: Proceedings of the ACM SIGMOD International Conference on Management of Data (May 1995)
ARC - A Cross Archive Search Service ODU, Digital Library Research Group
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Khan, H.M., Maly, K., Zubair, M. (2005). Similarity and Duplicate Detection System for an OAI Compliant Federated Digital Library. In: Rauber, A., Christodoulakis, S., Tjoa, A.M. (eds) Research and Advanced Technology for Digital Libraries. ECDL 2005. Lecture Notes in Computer Science, vol 3652. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11551362_68
Download citation
DOI: https://doi.org/10.1007/11551362_68
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-28767-4
Online ISBN: 978-3-540-31931-3
eBook Packages: Computer ScienceComputer Science (R0)