skip to main content
10.1145/2983323.2983679acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

Qualitative Cleaning of Uncertain Data

Published: 24 October 2016 Publication History

Abstract

We propose a new view on data cleaning: Not data itself but the degrees of uncertainty attributed to data are dirty. Applying possibility theory, tuples are assigned degrees of possibility with which they occur, and constraints are assigned degrees of certainty that say to which tuples they apply. Classical data cleaning modifies some minimal set of tuples. Instead, we marginally reduce their degrees of possibility. This reduction leads to a new qualitative version of the vertex cover problem. Qualitative vertex cover can be mapped to a linear-weighted constraint satisfaction problem. However, any off-the-shelf solver cannot solve the problem more efficiently than classical vertex cover. Instead, we utilize the degrees of possibility and certainty to develop a dedicated algorithm that is fixed parameter tractable in the size of the qualitative vertex cover. Experiments show that our algorithm is faster than solvers for the classical vertex cover problem by several orders of magnitude, and performance improves with higher numbers of uncertainty degrees.

References

[1]
M. Arenas, L. E. Bertossi, and J. Chomicki. Consistent query answers in inconsistent databases. In SIGMOD, pages 68--79, 1999.
[2]
L. E. Bertossi, S. Kolahi, and L. V. S. Lakshmanan. Data cleaning and query answering with matching dependencies and matching functions. Theory Comput. Syst., 52(3):441--482, 2013.
[3]
G. Beskales, I. F. Ilyas, L. Golab, and A. Galiullin. On the relative trust between inconsistent data and inaccurate constraints. In ICDE, pages 541--552, 2013.
[4]
A. Chalamalla, I. F. Ilyas, M. Ouzzani, and P. Papotti. Descriptive and prescriptive data cleaning. In SIGMOD, pages 445--456, 2014.
[5]
X. Chu, I. F. Ilyas, S. Krishnan, and J. Wang. Data cleaning: Overview and emerging challenges. In SIGMOD, pages 2201--2206, 2016.
[6]
M. C. Cooper, S. de Givry, M. Sanchez, T. Schiex, M. Zytnicki, and T. Werner. Soft arc consistency revisited. Artif. Intell., 174(7--8):449--478, 2010.
[7]
P. Domingos and D. Lowd. Markov Logic: An Interface Layer for Artificial Intelligence. Morgan & Claypool Publishers, 2009.
[8]
R. G. Downey and M. R. Fellows. Fundamentals of Parameterized Complexity. Springer, 2013.
[9]
D. Dubois and H. Prade. Fuzzy set and possibility theory-based methods in artificial intelligence. Artif. Intell., 148(1--2):1--9, 2003.
[10]
D. Dubois and H. Prade. Practical methods for constructing possibility distributions. Int. J. Intell. Syst., 31(3):215--239, 2016.
[11]
D. Dubois, H. Prade, and P. Smets. "not impossible" vs. "guaranteed possible" in fusion and revision. In ECSQARU, pages 522--531, 2001.
[12]
T. Eiter and G. Gottlob. On the complexity of propositional knowledge base revision, updates, and counterfactuals. Artif. Intell., 57(2--3):227--270, 1992.
[13]
V. Ganti and A. D. Sarma. Data Cleaning: A Practical Perspective. Morgan & Claypool Publishers, 2013.
[14]
P. Gärdenfors and H. Rott. Belief revision. In Handbook of Logic in Artificial Intelligence and Logic Programming, volume 4, page 35--132. Oxford University Press, 1995.
[15]
N. Hall, H. Köhler, S. Link, H. Prade, and X. Zhou. Cardinality constraints on qualitatively uncertain data. Data Knowl. Eng., 99:126--150, 2015.
[16]
M. Hannula, J. Kontinen, and S. Link. On independence atoms and keys. In CIKM, pages 1229--1238, 2014.
[17]
M. Hannula, J. Kontinen, and S. Link. On the finite and general implication problems of independence atoms and keys. J. Comput. Syst. Sci., 82(5):856--877, 2016.
[18]
R. M. Karp. Reducibility among combinatorial problems. In Complexity of Computer Computations, pages 85--103, 1972.
[19]
H. Köhler, U. Leck, S. Link, and H. Prade. Logical foundations of possibilistic keys. In JELIA, pages 181--195, 2014.
[20]
H. Köhler, U. Leck, S. Link, and X. Zhou. Possible and certain keys for SQL. VLDB J., 25(4):571--596, 2016.
[21]
H. Köhler and S. Link. SQL schema design: Foundations, normal forms, and normalization. In SIGMOD, pages 267--279, 2016.
[22]
H. Köhler, S. Link, and X. Zhou. Possible and certain SQL keys. PVLDB, 8(11):1118--1129, 2015.
[23]
H. Köhler, S. Link, and X. Zhou. Discovering meaningful certain keys from incomplete and inconsistent relations. IEEE Data Eng. Bull., 39(2):21--37, 2016.
[24]
P. Koutris and J. Wijsen. Consistent query answering for primary keys. SIGMOD Record, 45(1):15--22, 2016.
[25]
P. Liberatore. The complexity of belief update. Artif. Intell., 119(1--2):141--190, 2000.
[26]
S. Link and H. Prade. Possibilistic functional dependencies and their relationship to possibility theory. IEEE Trans. Fuzzy Systems, 24(3):757--763, 2016.
[27]
L. Mo, R. Cheng, X. Li, D. W. Cheung, and X. S. Yang. Cleaning uncertain data for top-k queries. In ICDE, pages 134--145, 2013.
[28]
N. Prokoshyna, J. Szlichta, F. Chiang, R. J. Miller, and D. Srivastava. Combining quantitative and logical data cleaning. PVLDB, 9(4):300--311, 2015.
[29]
D. Suciu, D. Olteanu, C. Ré, and C. Koch. Probabilistic Databases. Morgan & Claypool Publishers, 2011.
[30]
M. W. Vincent, J. Liu, H. Liu, and S. Link. Technical correspondence: "Differential dependencies: Reasoning and discovery" revisited. ACM Trans. Database Syst., 40(2):14, 2015.
[31]
J. Wang and N. Tang. Towards dependable data repairing with fixing rules. In SIGMOD, pages 457--468, 2014.
[32]
G. J. Woeginger. Exact algorithms for NP-hard problems: A survey. In Combinatorial Optimization, pages 185--208, 2001.

Cited By

View all
  • (2025)Gray Learning From Non-IID Data With Out-of-Distribution SamplesIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.333047536:1(1396-1409)Online publication date: Jan-2025
  • (2023)Entity integrity management under data volume, variety and veracityKnowledge and Information Systems10.1007/s10115-022-01814-165:7(2895-2934)Online publication date: 25-Jan-2023
  • (2022)Possibilistic Data CleaningIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2021.306231834:12(5939-5950)Online publication date: 1-Dec-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management
October 2016
2566 pages
ISBN:9781450340731
DOI:10.1145/2983323
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. database repair
  2. possibility theory
  3. vertex cover

Qualifiers

  • Research-article

Conference

CIKM'16
Sponsor:
CIKM'16: ACM Conference on Information and Knowledge Management
October 24 - 28, 2016
Indiana, Indianapolis, USA

Acceptance Rates

CIKM '16 Paper Acceptance Rate 160 of 701 submissions, 23%;
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)14
  • Downloads (Last 6 weeks)0
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Gray Learning From Non-IID Data With Out-of-Distribution SamplesIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.333047536:1(1396-1409)Online publication date: Jan-2025
  • (2023)Entity integrity management under data volume, variety and veracityKnowledge and Information Systems10.1007/s10115-022-01814-165:7(2895-2934)Online publication date: 25-Jan-2023
  • (2022)Possibilistic Data CleaningIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2021.306231834:12(5939-5950)Online publication date: 1-Dec-2022
  • (2018)Handling Uncertainty in Relational Databases with Possibility Theory - A Survey of Different ModelingsScalable Uncertainty Management10.1007/978-3-030-00461-3_30(396-404)Online publication date: 11-Sep-2018
  • (2017)Probabilistic KeysIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2016.263334229:3(670-682)Online publication date: 1-Mar-2017

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media