Abstract
Data cleaning deals with the detection and removal of errors and inconsistencies in data, gathered from distributed sources. This process is essential for drawing correct conclusions from data in decision support systems. Eliminating fuzzy duplicate records is a fundamental part of the data cleaning process. The vagueness and uncertainty involved in detecting fuzzy duplicates make it a niche, for applying fuzzy reasoning. Although uncertainty alg ebras like fuzzy logic are known, their applicability to the problem of duplicate elimination has remained unexplored and unclear, until today. In this paper, a novel and flexible fuzzy expert system for detection and elimination of fuzzy duplicates in the process of data cleaning is devised, which circumvents the repetitive and inconvenient task of hard-coding. Some of the crucial advantages of this approach are its flexibility, ease of use, extendibility, fast development time and efficient run time, when used in various information systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bilenko, M., Mooney, R.J.: Adaptive Duplicate Detection Using Learnable String Similarity Measures. In: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2003), Washington, DC (August 2003)
Cohen, W., Ravikumar, P., Fienberg, S.: A Comparison of String Distance Metrics for Name-Matching Tasks. In: IIWeb Workshop 2003 (2003)
Galhardas, H., Florescu, D., et al.: Declarative Data Cleaning: Language, Model, and Algorithms. In: Proc. of the 27th VLDB Conference (2001)
Haidarian Shahri, H., Barforush, A.A.: Data Mining for Removing Fuzzy Duplicates Using Fuzzy Inference. In: 23rd International Conference of the North American Fuzzy Information Processing Society (NAFIPS 2004), Banff, Alberta, Canada, June 27-30 (2004)
Hernandez, M.A., Stolfo, S.J.: Real-world Data is Dirty: Data Cleansing and the Merge/Purge Problem. Data Mining and Knowledge Discovery 2(1), 9–37 (1998)
Low, W.L., Lee, M.L., Ling, T.W.: A Knowledge-based Approach for Duplicate Elimination in Data Cleaning. Information Systems 26, 585–606 (2001)
Mamdani, E.H.: Advances in Linguistic Synthesis of Fuzzy Controllers. Int. J. Man Machine Studies 8 (1976)
Monge, A.E., Elkan, P.C.: An Efficient Domain-independent Algorithm for Detecting Approximately Duplicate Database Records. In: Proceedings of the SIGMOD 1997 Workshop on Data Mining and Knowledge Discovery (May 1997)
Rahm, E., Do, H.H.: Data Cleaning: Problems and Current Approaches. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, Special Issue on Data Cleaning 23(4) (December 2000)
Raman, V., Hellerstein, J.M.: Potter’s Wheel: An Interactive Data Cleaning System. In: Proc. of the 27th VLDB Conference (2001)
Winkler, W.E.: The State of Record Linkage and Current Research Problems. Statistics of Income Division, Internal Revenue Service Publication R99/04 (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Shahri, H.H., Barforush, A.A. (2004). A Flexible Fuzzy Expert System for Fuzzy Duplicate Elimination in Data Cleaning. In: Galindo, F., Takizawa, M., Traunmüller, R. (eds) Database and Expert Systems Applications. DEXA 2004. Lecture Notes in Computer Science, vol 3180. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30075-5_16
Download citation
DOI: https://doi.org/10.1007/978-3-540-30075-5_16
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-22936-0
Online ISBN: 978-3-540-30075-5
eBook Packages: Springer Book Archive