On solving the continuous data editing problem

doi:10.1016/0305-0548(96)81769-2

Computers & Operations Research

Volume 23, Issue 3, March 1996, Pages 263-273

https://doi.org/10.1016/0305-0548(96)81769-2 Get rights and content

Abstract

The data editing problem is concerned with identifying the most likely source of errors in computerized data bases. Given a record that is known to fail one or more ological consistency edits, the objective is to determine the minimum (possibly weighted) number of fields that could be changed in order to correct the record. While this problem can easily be formulated as a pure fixed-charge problem, it can be extremely difficult to solve under certain data conditions. In this paper we show how a number of structural characteristics in this problem can be exploited to dramatically reduce the computational time required to solve particularly difficult data edition problems.

References (19)

G.E. Liepins
Can automatic data editing by justified? One person's opinion
P.G. McKeown et al.
A computational study of using preprocessing and stronger formulations to solve large general fixed-charge problems
Computers Op. Res.
(1990)
K.R. MacLeod et al.
AXIS: A framework for combining structured algorithms and knowledge-based systems
Computers Op. Res.
(1993)
Associated Press
Court Debates Computer Goofs
Roanoke Times & World News
(1994)
R.J. Freund et al.
A procedure for automatic edit and imputation
J. Am. Statist. Assoc.
(1967)
J.I. Naus et al.
A probabilistic model for identifying errors in data editing
J. Am. Statist. Assoc.
(1972)
I.P. Fellegi et al.
A systematic approach to automatic edit and imputation
J. Am. Statist. Assoc.
(1976)
B. Greenburg
Developing an edit system for industry statistics
P.G. McKeown
A mathematical programming approach to editing of continuous survey data
SIAM J. Scient. Stat. Comput.
(1984)

There are more references available in the full text version of this article.

Cited by (10)

A branch-and-cut algorithm for the continuous error localization problem in data cleaning
2007, Computers and Operations Research
Citation Excerpt :
Once again, the master problem is enriched with cuts generated by the subproblem from a non-feasible solution of the master problem. However, in contrast to previous approaches [1,11,6,7] these new inequalities are not set-covering cuts. At first glance, using this alternative is not apparently an advantage since it usually requires the solving of a long sequence of smaller problems.
Data collected by statistical agencies may contain mistakes made during the acquisition, transcription and coding process. Thus, before using all this information to infer statistical properties, the statistical agencies must check the consistency of their information. To this end, each record has to be tested on a set of consistency rules. Therefore, if one of these records does not meet all the consistency rules, it must be established which fields have to be modified in order to make the new record valid with respect to that set of consistency rules. Among all the possible solutions, statistical agencies are interested in finding one in which the number of fields that should be modified is minimum, thus leading to a combinatorial optimization problem known as the Error Localization Problem. This article approaches the optimization problem of finding the smallest set of fields whose values must be changed in order to satisfy a given set of consistency rules. With this purpose in mind an Integer Linear Programming model is proposed for the particular case in which the fields are continuous values and the consistency rules are given by linear inequalities. This model is solved through a branch-and-cut approach based on a Benders’ decomposition. The new proposal is compared to others previously published in the literature and tested on benchmark instances. The overall performance of these new algorithms succeeded in solving to optimality difficult instances with up to 100 fields in a record in about 1 min of a personal computer.
A heuristic approach for the continuous error localization problem in data cleaning
2007, Computers and Operations Research
The Error Localization Problem concerns finding the minimum number of fields in a record such that by modifying the values in these fields the new record satisfies a given set of rules. This problem is of great interest to statistical agencies in as far as cleaning microdata is concerned. It has been shown to be $NP$ -hard, and exact methods in literature only succeed in solving small instances. This article presents a new heuristic algorithm based on a descending search approach to obtain near-optimal solutions. Some procedures of this descending search make use of Farkas’ Lemma in Linear Programming to drastically reduce the search space in one of the proposed neighborhoods. Computational experience on randomly generated instances shows that the approach can deal with instances of up to 1000 fields and 400 edits.
Discrete models for data imputation
2004, Discrete Applied Mathematics
The paper is concerned with the problem of automatic detection and correction of inconsistent or out of range data in a general process of statistical data collection. The proposed approach is able to deal with hierarchical data containing both qualitative and quantitative values. As customary, erroneous data records are detected by formulating a set of rules. Erroneous records should then be corrected, by modifying as less as possible the erroneous data, while causing minimum perturbation to the original frequency distributions of the data. Such process is called imputation. By encoding the rules with linear inequalities, we convert imputation problems into integer linear programming problems. The proposed procedure is tested on a real-world case of census. Results are extremely encouraging both from the computational and from the data quality point of view.
A generalized Fellegi-Holt paradigm for automatic error localization
2016, Survey Methodology
Automatic editing with hard and soft edits
2012, Survey Methodology
Handbook of Statistical Data Editing and Imputation
2011, Handbook of Statistical Data Editing and Imputation

View all citing articles on Scopus

^†: Cliff T. Ragsdale is an Assistant Professor of Management Science at Virginia Polytechnic Institute and State University. He received his B.A. and M.B.A. degrees from the University of Central Florida and holds a Ph.D. in Management Science and Information Technology from the University of Georgia. Dr Ragsdale's primary research interests are in the areas of applied statistics, optimization, and artificial intelligence. His research has appeared in Decision Sciences, Naval Research Logistics, Computers & Operations Research, OMEGA, Operational Research Letters, Financial Services Review, and other publications. He is also author of the book Spreadsheet Modeling and Decision Analysis: A Practical Introduction to Management Science, published recently by Course Technology, Inc.

^§: Patrick G. McKeown is a Professor in the Department of Management at the University of Georgia. He received his Ph.D. from the University of North Carolina at Chapel Hill and his M.S. and B.S. degrees from the Georgia Institute of Technology. Dr McKeown's primary research interests are in the areas of linear and integer programming, and algorithm development. He has authored numerous books in the areas of management science, computer programming, and information systems and technology. His research has appeared in several journals including Operations Research, Management Science, Naval Research Logistics Quarterly, Computers & Operations Research, and the SIAM Journal of Scientific and Statistical Computing.

View full text

On solving the continuous data editing problem

Abstract

Computers Op. Res.

Computers Op. Res.

Court Debates Computer Goofs

Roanoke Times & World News

A procedure for automatic edit and imputation

J. Am. Statist. Assoc.

A probabilistic model for identifying errors in data editing

J. Am. Statist. Assoc.

A systematic approach to automatic edit and imputation

J. Am. Statist. Assoc.

Developing an edit system for industry statistics

A mathematical programming approach to editing of continuous survey data

SIAM J. Scient. Stat. Comput.