Welcome to the InfoSci Platform
Reference Hub9
The Use of Smart Tokens in Cleaning Integrated Warehouse Data

The Use of Smart Tokens in Cleaning Integrated Warehouse Data

Christie I. Ezeife, Timothy E. Ohanekwu
Copyright: © 2005 |Volume: 1 |Issue: 2 |Pages: 22
ISSN: 1548-3924|EISSN: 1548-3932|ISSN: 1548-3924|EISBN13: 9781615202164|EISSN: 1548-3924|DOI: 10.4018/jdwm.2005040101
Cite Article Cite Article

MLA

Ezeife, Christie I., and Timothy E. Ohanekwu. "The Use of Smart Tokens in Cleaning Integrated Warehouse Data." IJDWM vol.1, no.2 2005: pp.1-22. http://doi.org/10.4018/jdwm.2005040101

APA

Ezeife, C. I. & Ohanekwu, T. E. (2005). The Use of Smart Tokens in Cleaning Integrated Warehouse Data. International Journal of Data Warehousing and Mining (IJDWM), 1(2), 1-22. http://doi.org/10.4018/jdwm.2005040101

Chicago

Ezeife, Christie I., and Timothy E. Ohanekwu. "The Use of Smart Tokens in Cleaning Integrated Warehouse Data," International Journal of Data Warehousing and Mining (IJDWM) 1, no.2: 1-22. http://doi.org/10.4018/jdwm.2005040101

Export Reference

Mendeley
Favorite Full-Issue Download

Abstract

Identifying integrated records that represent the same real-world object in numerous ways is just one form of data disparity (dirt) to be resolved in a data warehouse. Data cleaning is a complex process, which uses multidisciplinary techniques to resolve conflicts in data drawn from different data sources. There is a need for initial cleaning at the time a data warehouse is built, and incremental cleaning whenever new records are brought into the data warehouse during refreshing. Existing work on data cleaning have used pre-specified record match thresholds and multiple scanning of records to determine matching records in integrated data. Little attention has been paid to incremental matching of records. Determining optimal record match score threshold in a domain is hard. Also, direct long record string comparison is highly inefficient and intolerant to typing errors. Thus, this article proposes two algorithms, the first of which uses smart tokens defined from integrated records to match and identify duplicate records during initial warehouse cleaning. The second algorithm uses these tokens for fast, incremental cleaning during warehouse refreshing. Every attribute value forms either a special token like birth date or an ordinary token, which can be alphabetic, numeric, or alphanumeric. Rules are applied for forming tokens belonging to each of these four classes. These tokens are sorted and used for record match. The tokens also form very good warehouse identifiers for future faster incremental warehouse cleaning. This approach eliminates the need for match threshold and multiple passes at data. Experiments show that using tokens for record comparison produces a far better result than using the entire or greater part of a record.

Request Access

You do not own this content. Please login to recommend this title to your institution's librarian or purchase it from the IGI Global bookstore.