Skip to main content
SpringerLink
Log in

Discovering Functional Dependencies in Vertically Distributed Big Data

  • Conference paper
  • First Online:
Book cover Web Information Systems Engineering – WISE 2015 (WISE 2015)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9419))

Included in the following conference series:

Abstract

The issue of discovering FDs has received a great deal of attention in the database research community. However, as the problem is exponential in the number of attributes, existing approaches can only be applied on small centralized datasets. It is challenging to discover FDs from big data, especially if data is distributed. We present a new algorithm DFDD for discovering all functional dependencies in parallel in vertically distributed big data following a breadth-first traversal strategy of the attribute lattice that combines efficient pruning. We verify experimentally that our approach can process distributed big datasets and it is scalable with the number of cluster nodes and the size of datasets.

This work was supported in part by National Basic Research Program 973 of China (No. 2012CB316203), Natural Science Foundation of China (Nos. 61033007, 61272121, 61332006, 61472321), National High Technology Research and Development Program 863 of China (No. 2012AA011004), Basic Research Fund of Northwestern Polytechnical University (No. 3102014JSJ0005, 3102014JSJ0013).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Codd, E.F.: Further normalization of the data base model. Technical report 909, IBM (1971)

    Google Scholar 

  2. Yao, H., Hamilton, H.J.: Mining functional dependencies from data. Data Min. Knowl. Disc. 16(2), 197–219 (2008)

    Article  MathSciNet  Google Scholar 

  3. Maier, D.: The Theory of Relational Databases. Computer Science Press, Rockville (1983)

    MATH  Google Scholar 

  4. Huhtala, Y., Karkkainen, J., Porkka, P., Toivonen, H.: TANE: an efficient algorithm for discovering functional and approximate dependencies. Comput. J. 42(2), 100–111 (1999)

    Article  MATH  Google Scholar 

  5. Li, W., Li, Z., Chen, Q., Jiang, T., Liu, H., Pan, W.: Functional dependencies discovering in distributed big data. J. Comput. Res. Dev. 52(2), 282–294 (2015)

    Google Scholar 

  6. Özsu, M.T., Valduriez, P.: Principles of Distributed Database Systems, 2nd edn. Prentice-Hall, Upper Saddle River (1999)

    Google Scholar 

  7. Novelli, N., Cicchetti, R.: FUN: an efficient algorithm for mining functional and embedded dependencies. In: Van den Bussche, J., Vianu, V. (eds.) ICDT 2001. LNCS, vol. 1973, pp. 189–203. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  8. Lopes, S., Petit, J.-M., Lakhal, L.: Efficient discovery of functional dependencies and Armstrong relations. In: Zaniolo, C., Grust, T., Scholl, M.H., Lockemann, P.C. (eds.) EDBT 2000. LNCS, vol. 1777, pp. 350–364. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  9. United States Department of Transportation. http://apps.bts.gov/xml/ontimesummarystatistics

Download references

Author information

Authors and Affiliations

  1. Northwestern Polytechnical University, Xi’an, 710072, Shaanxi, China

    Weibang Li, Zhanhuai Li, Qun Chen, Tao Jiang & Hailong Liu

Authors
  1. Weibang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhanhuai Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qun Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hailong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weibang Li .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Jianyong Wang

  2. Poznan University of Economics, Poznan, Poland

    Wojciech Cellary

  3. Florida Atlantic University, Boca Raton, Florida, USA

    Dingding Wang

  4. Victoria University, Melbourne, Victoria, Australia

    Hua Wang

  5. Florida International University, Miami, Florida, Florida, USA

    Shu-Ching Chen

  6. Florida International University, Miami, Florida, USA

    Tao Li

  7. Victoria University, Melbourne, Victoria, Australia

    Yanchun Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Li, W., Li, Z., Chen, Q., Jiang, T., Liu, H. (2015). Discovering Functional Dependencies in Vertically Distributed Big Data. In: Wang, J., et al. Web Information Systems Engineering – WISE 2015. WISE 2015. Lecture Notes in Computer Science(), vol 9419. Springer, Cham. https://doi.org/10.1007/978-3-319-26187-4_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-26187-4_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-26186-7

  • Online ISBN: 978-3-319-26187-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation