Skip to main content
SpringerLink
Log in

Sampling from repairs of conditional functional dependency violations

  • Regular Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

Violations of functional dependencies (FDs) and conditional functional dependencies (CFDs) are common in practice, often indicating deviations from the intended data semantics. These violations arise in many contexts such as data integration and Web data extraction. Resolving these violations is challenging for a variety of reasons, one of them being the exponential number of possible repairs. Most of the previous work has tackled this problem by producing a single repair that is nearly optimal with respect to some metric. In this paper, we propose a novel data cleaning approach that is not limited to finding a single repair, namely sampling from the space of possible repairs. We give several motivating scenarios where sampling from the space of CFD repairs is desirable, we propose a new class of useful repairs, and we present an algorithm that randomly samples from this space in an efficient way. We also show how to restrict the space of repairs based on constraints that reflect the accuracy of different parts of the database. We experimentally evaluate our algorithms against previous approaches to show the utility and efficiency of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Notes

  1. http://archive.ics.uci.edu/ml/datasets/Census-Income+(KDD).

References

  1. UIS data generator, http://www.cs.utexas.edu/users/ml/riddle

  2. Abiteboul, S., Hull, R., Vianu, V.: Foundations of Databases. Addison-Wesley, Reading (1995)

    MATH  Google Scholar 

  3. Afrati, F.N., Kolaitis, P.G.: Repair checking in inconsistent databases: algorithms and complexity. In: ICDT, pp. 31–41 (2009)

  4. Arenas, M., Bertossi, L.E., Chomicki, J.: Consistent query answers in inconsistent databases. In: PODS, pp. 68–79 (1999)

  5. Baxter, R., Christen, P., Churches, T.: A comparison of fast blocking methods for record linkage. In ACM SIGKDD (2003)

  6. Benjelloun, O., Sarma, A.D., Halevy, A.Y., Widom, J.: ULDBs: Databases with uncertainty and lineage. In: VLDB, pp. 953–964 (2006)

  7. Beskales, G., Ilyas, I.F., Golab, L.: Sampling the repairs of functional dependency violations under hard constraints. PVLDB 3(1), 197–207 (2010)

    Google Scholar 

  8. Bohannon, P., Fan, W., Geerts, F., Jia, X., Kementsietsidis, A.: Conditional functional dependencies for data cleaning. In: ICDE, pp. 746–755 (2007)

  9. Bohannon, P., Flaster, M., Fan, W., Rastogi, R.: A cost-based model and effective heuristic for repairing constraints by value modification. In: SIGMOD, pp. 143–154 (2005)

  10. Chomicki, J., Marcinkowski, J.: Minimal-change integrity maintenance using tuple deletions. Inf. Comput. 197(1/2), 90–121 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  11. Chomicki, J., Marcinkowski, J., Staworko, S.: Computing consistent query answers using conflict hypergraphs. In: CIKM, pp. 417–426 (2004)

  12. Cong, G., Fan, W., Geerts, F., Jia, X., Ma, S.: Improving data quality: consistency and accuracy. In: VLDB, pp. 315–326 (2007)

  13. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 2nd edn. MIT Press and McGraw-Hill, Cambridge and New York (2001)

    MATH  Google Scholar 

  14. Yuan, Y.C.: Multiple imputation for missing data: Concepts and new development. In: The 25th Annual SAS Users Group International Conference (2002)

  15. Eckerson, W.W.: Data quality and the bottom line: Achieving business success through a commitment to high quality data. TDWI Report Series (2002)

  16. Elmagarmid, A.K., Ipeirotis, P.G., Verykios, V.S.: Duplicate record detection: a survey. IEEE Trans. Knowl. Data Eng. 19(1), 1–16 (2007)

    Article  Google Scholar 

  17. Fan, W., Geerts, F., Jia, X., Kementsietsidis, A.: Conditional functional dependencies for capturing data inconsistencies. ACM Trans. Database Syst. 33(2), 6:1–6:48 (2008)

    Google Scholar 

  18. Fan, W., Li, J., Ma, S., Tang, N., Yu, W.: Towards certain fixes with editing rules and master data. PVLDB 3(1), 173–184 (2010)

    Google Scholar 

  19. Fan, W., Li, J., Ma, S., Tang, N., Yu, W.: Towards certain fixes with editing rules and master data. VLDB J. 21(2), 213–238 (2012)

    Article  Google Scholar 

  20. Fuxman, A., Miller, R.J.: First-order query rewriting for inconsistent databases. J. Comput. Syst. Sci. 73(4), 610–635 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  21. Greco, S., Molinaro, C.: Approximate probabilistic query answering over inconsistent databases. In: ER, pp. 311–325 (2008)

  22. Jampani, R., Xu, F., Wu, M., Perez, L.L., Jermaine, C.M., Haas, P.J.: MCDB: a monte carlo approach to managing uncertain data. In: SIGMOD Conference, pp. 687–700 (2008)

  23. Kolahi, S., Lakshmanan, L.V.S.: On approximating optimum repairs for functional dependency violations. In: ICDT, pp. 53–62 (2009)

  24. Lopatenko, A., Bertossi, L.E.: Complexity of consistent query answering in databases under cardinality-based and incremental repair semantics. In: ICDT, pp. 179–193 (2007)

  25. McCallum, A., Nigam, K., Ungar, L.H.: Efficient clustering of high-dimensional data sets with application to reference matching. In: KDD, pp. 169–178 (2000)

  26. Müller, H., Freytag, J.C.: Problems, Methods and Challenges in Comprehensive Data Cleansing. Technical Report HUB-IB-164, Humboldt-Universität zu Berlin, Institut für Informatik (2003)

  27. Mulry, M.H., Bean, S.L., Bauder, D.M., Wagner, D., Mule, T., Petroni., R.J.: Evaluation of estimates of census duplication using administrative records information. J. Off. Stat. 22(4), 655–679 (2006)

    Google Scholar 

  28. Sen, P., Deshpande, A.: Representing and querying correlated tuples in probabilistic databases. In: ICDE, pp. 596–605 (2007)

  29. Tarjan, R.E.: Efficiency of a good but not linear set union algorithm. J. ACM 22(2), 215–225 (1975)

    Article  MATH  MathSciNet  Google Scholar 

  30. Wijsen, J.: Condensed representation of database repairs for consistent query answering. In: ICDT, pp. 375–390 (2003)

  31. Wijsen, J.: Database repairing using updates. ACM Trans. Database Syst. 30(3), 722–768 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Qatar Computing Research Institute, Doha, Qatar

    George Beskales & Ihab F. Ilyas

  2. University of Waterloo, Waterloo, Canada

    Lukasz Golab & Artur Galiullin

Authors
  1. George Beskales
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ihab F. Ilyas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lukasz Golab
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Artur Galiullin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to George Beskales.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Beskales, G., Ilyas, I.F., Golab, L. et al. Sampling from repairs of conditional functional dependency violations. The VLDB Journal 23, 103–128 (2014). https://doi.org/10.1007/s00778-013-0316-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-013-0316-z

Keywords

Search

Navigation