Skip to main content
SpringerLink
Log in

A Supervised Machine Learning Approach for Duplicate Detection over Gazetteer Records

  • Conference paper
Book cover GeoSpatial Semantics (GeoS 2011)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 6631))

Included in the following conference series:

Abstract

This paper presents a novel approach for detecting duplicate records in the context of digital gazetteers, using state-of-the-art machine learning techniques. It reports a thorough evaluation of alternative machine learning approaches designed for the task of classifying pairs of gazetteer records as either duplicates or not, built by using support vector machines or alternating decision trees with different combinations of similarity scores for the feature vectors. Experimental results show that using feature vectors that combine multiple similarity scores, derived from place names, semantic relationships, place types and geospatial footprints, leads to an increase in accuracy. The paper also discusses how the proposed duplicate detection approach can scale to large collections, through the usage of filtering or blocking techniques.

This work was partially supported by the Fundação para a Ciência e a Tecnologia (FCT), through project grant PTDC/EIA-EIA/109840/2009 (SInteliGIS).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Arasu, A., Ganti, V., Kaushik, R.: Efficient exact set-similarity joins. In: Proceedings of the 32nd International Conference on Very Large Data Bases (2006)

    Google Scholar 

  2. Bayardo, R.J., Ma, Y., Srikant, R.: Scaling up all pairs similarity search. In: Proceeding of the 16th International Conference on World Wide Web (2007)

    Google Scholar 

  3. Beeri, C., Kanza, Y., Safra, E., Sagiv, Y.: Object fusion in geographic information systems. In: Proceedings of the 30th International Conference on Very Large Data Bases (2004)

    Google Scholar 

  4. Bernstein, A., Kaufmann, E., Kiefer, C., Bürki, C.: Simpack: A generic java library for similiarity measures in ontologies (2005) (working paper)

    Google Scholar 

  5. Bilenko, M., Kamath, B., Mooney, R.J.: Adaptive blocking: Learning to scale up record linkage and clustering. In: Proceedings of the 6th IEEE International Conference on Data Mining (2006)

    Google Scholar 

  6. Bilenko, M., Mooney, R.J.: On evaluation and training-set construction for duplicate detection. In: Proceedings of the KDD 2003 Workshop on Data Cleaning, Record Linkage, and Object Consolidation (2003)

    Google Scholar 

  7. Bilenko, M., Mooney, R.J.: Adaptive duplicate detection using learnable string similarity measures. In: Proceedings of the 9th ACM Conference on Knowledge Discovery and Data Mining (2006)

    Google Scholar 

  8. Cohen, W., Ravikumar, P., Fienberg, S.: A comparison of string distance metrics for name-matching tasks. In: Proceedings of 9th ACM Conference on Knowledge Discovery and Data Mining (2003)

    Google Scholar 

  9. Cohen, W.W., Richman, J.: Learning to match and cluster large high-dimensional datasets for data integration. In: Proceedings of 8th ACM Conference on Knowledge Discovery and Data Mining (2002)

    Google Scholar 

  10. Davis, C., Salles, E.: Approximate string matching for geographic names and personal names. In: Proceedings of the 9th Brazilian Symposium on GeoInformatics (2007)

    Google Scholar 

  11. Elmagarmid, A.K., Ipeirotis, P.G., Verykios, V.S.: Duplicate record detection: A survey. IEEE Transactions on Knowledge and Data Engineering 19(1) (2007)

    Google Scholar 

  12. Freund, Y., Mason, L.: The alternating decision tree learning algorithm. In: Proceedings of the 16th International Conference on Machine Learning (1999)

    Google Scholar 

  13. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. SIGKDD Explorations Newsletter 11 (2009)

    Google Scholar 

  14. Hastings, J., Hill, L.L.: Treatment of duplicates in the alexandria digital library gazetteer. In: Proceedings of the 2002 GeoScience Conference (2002)

    Google Scholar 

  15. Hastings, J.T.: Automated conflation of digital gazetteer data. International Journal Geographic Information Science 22(10) (2008)

    Google Scholar 

  16. Hernandez, M.A., Stolfo, S.J.: The merge/purge problem for large databases. In: Proceedings of the 1995 ACM Conference on Management of Data (1995)

    Google Scholar 

  17. Hill, L.L.: Core elements of digital gazetteers: Placenames, categories, and footprints. In: Proceedings of the 4th European Conference on Research and Advanced Technology for Digital Libraries (2000)

    Google Scholar 

  18. Hill, L.L.: Georeferencing: The Geographic Associations of Information. The MIT Press, Cambridge (2006)

    Google Scholar 

  19. Joachims, T.: Making large-scale SVM learning practical. In: Scholkopf, B., Burges, C.J.C., Smola, A.J. (eds.) Advances in Kernel Methods - Support Vector Learning. The MIT Press, Cambridge (1999)

    Google Scholar 

  20. Kang, H., Sehgal, V., Getoor, L.: Geoddupe: A novel interface for interactive entity resolution in geospatial data. In: Proceeding of the 11th IEEE International Conference on Information Visualisation (2007)

    Google Scholar 

  21. Lawrence, P.: The double metaphone search algorithm. C/C++ Users Journal 18(6) (2000)

    Google Scholar 

  22. Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10 (1966)

    Google Scholar 

  23. Lin, D.: An information-theoretic definition of similarity. In: Proceedings of the 15th International Conference on Machine Learning (1998)

    Google Scholar 

  24. McCallum, A.K., Nigam, K., Ungar, L.: Efficient clustering of high-dimensional datasets with application to reference matching. In: Proceedings of 6th ACM Conference on Knowledge Discovery and Data Mining (2000)

    Google Scholar 

  25. Moguerza, J.M., Muñoz, A.: Support vector machines with applications. Statistical Science 21(3) (2006)

    Google Scholar 

  26. Monge, A.E., Elkan, C.: The field matching problem: Algorithms and applications. In: Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (1996)

    Google Scholar 

  27. Naumann, F., Herschel, M., Ozsu, M.T.: An Introduction to Duplicate Detection. Morgan & Claypool Publishers (2010)

    Google Scholar 

  28. Pasula, H., Marthi, B., Milch, B., Russell, S., Shpitser, I.: Identity uncertainty and citation matching. In: Proceedings of the 7th Annual Conference on Neural Information Processing Systems (2003)

    Google Scholar 

  29. Resnik, P.: Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language. Journal of Artificial Intelligence Research 11 (1999)

    Google Scholar 

  30. Safavian, S.R., Landgrebe, D.: A survey of decision tree classifier methodology. IEEE Transactions on Systems, Man and Cybernetics 21(3) (1991)

    Google Scholar 

  31. Samal, A., Seth, S., Cueto, K.: A feature-based approach to conflation of geospatial sources. International Journal of Geographical Information Science 18 (2004)

    Google Scholar 

  32. Sarawagi, S., Bhamidipaty, A.: Interactive deduplication using active learning. In: Proceedings of 8th ACM Conference on Knowledge Discovery and Data Mining (2002)

    Google Scholar 

  33. Schwarz, P., Deng, Y., Rice, J.E.: Finding similar objects using a taxonomy: A pragmatic approach. In: Proceedings of the 5th International Conference on Ontologies, Databases and Applications of Semantics (2006)

    Google Scholar 

  34. Sehgal, V., Getoor, L., Viechnicki, P.D.: Entity resolution in geospatial data integration. In: Proceedings of the 14th International Symposium on Advances on Geographical Information Systems (2006)

    Google Scholar 

  35. Tejada, S., Knoblock, C.A., Minton, S.: Learning domain-independent string transformation weights for high accuracy object identification. In: Proceedings of 8th ACM Conference on Knowledge Discovery and Data Mining (2002)

    Google Scholar 

  36. Winkler, W.E.: Methods for record linkage and bayesian networks. Technical report, Statistical Research Division, U.S. Census Bureau (2002)

    Google Scholar 

  37. Winkler, W.E.: Overview of record linkage and current research directions. Technical report, Statistical Research Division, U.S. Census Bureau (2006)

    Google Scholar 

  38. Witten, I.H., Frank, R.: Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Francisco (2000)

    Google Scholar 

  39. Xiao, C., Wang, W., Lin, X., Yu, J.X.: Efficient similarity joins for near duplicate detection. In: Proceeding of the 17th International Conference on World Wide Web (2008)

    Google Scholar 

  40. Zheng, Y., Fen, X., Xie, X., Peng, S., Fu, J.: Detecting nearly duplicated records in location datasets. In: Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Instituto Superior Técnico, INESC-ID, Av. Professor Cavaco Silva, 2744-016, Porto Salvo, Portugal

    Bruno Martins

Authors
  1. Bruno Martins
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Naval Academy Research Institute, 29240, Brest Cedex 9, France

    Christophe Claramunt

  2. Centro de Investigacion en Computacion, Instituto Politecnico Nacional, 07738, Mexico City, Mexico

    Sergei Levashkin

  3. School of Computer Science and Informatics, University College Dublin, Dublin, Ireland

    Michela Bertolotto

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Martins, B. (2011). A Supervised Machine Learning Approach for Duplicate Detection over Gazetteer Records. In: Claramunt, C., Levashkin, S., Bertolotto, M. (eds) GeoSpatial Semantics. GeoS 2011. Lecture Notes in Computer Science, vol 6631. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20630-6_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-20630-6_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-20629-0

  • Online ISBN: 978-3-642-20630-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation