Skip to main content
SpringerLink
Log in
Book cover

International Conference on Big Data Analytics and Knowledge Discovery

DaWaK 2017: Big Data Analytics and Knowledge Discovery pp 19–32Cite as

Optimal Task Ordering in Chain Data Flows: Exploring the Practicality of Non-scalable Solutions

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10440))

Abstract

Modern data flows generalize traditional Extract-Transform-Load and data integration workflows in order to enable end-to-end data processing and analytics. The more complex they become, the more pressing the need for automated optimization solutions. Optimizing data flows comes in several forms, among which, optimal task ordering is one of the most challenging ones. We take a practical approach; motivated by real-world examples, such as those captured by the TPC-DI benchmark, we argue that exhaustive non-scalable solutions are indeed a valid choice for chain flows. Our contribution is that we thoroughly discuss the three main directions for exhaustive enumeration of task ordering alternatives, namely backtracking, dynamic programming and topological sorting, and we provide concrete evidence up to which size and level of flexibility of chain flows they can be applied.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    An abstract of these ideas, without considering TPC-DI, have appeared in [11] in less than a page.

  2. 2.

    http://www.essi.upc.edu/dtim/blog/post/tpc-di-etls-using-pdi-aka-kettle.

References

  1. Agrawal, K., Benoit, A., Dufossé, F., Robert, Y.: Mapping filtering streaming applications. Algorithmica 62(1–2), 258–308 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  2. Burge, J., Munagala, K., Srivastava, U.: Ordering pipelined query operators with precedence constraints. Technical report 2005–40, Stanford InfoLab (2005)

    Google Scholar 

  3. Chaudhuri, S.: An overview of query optimization in relational systems. In: PODS, pp. 34–43 (1998)

    Google Scholar 

  4. Chaudhuri, S., Dayal, U., Narasayya, V.: An overview of business intelligence technology. Commun. ACM 54, 88–98 (2011)

    Article  Google Scholar 

  5. Florescu, D., Levy, A., Manolescu, I., Suciu, D.: Query optimization in the presence of limited access patterns. In: SIGMOD, pp. 311–322. ACM (1999)

    Google Scholar 

  6. Grehant, X., Demeure, I., Jarp, S.: A survey of task mapping on production grids. ACM Comput. Surv. 45(3), 37:1–37:25 (2013)

    Article  MATH  Google Scholar 

  7. Halasipuram, R., Deshpande, P.M., Padmanabhan, S.: Determining essential statistics for cost based optimization of an ETL workflow. In: EDBT, pp. 307–318 (2014)

    Google Scholar 

  8. Hueske, F., Peters, M., Sax, M., Rheinländer, A., Bergmann, R., Krettek, A., Tzoumas, K.: Opening the black boxes in data flow optimization. PVLDB 5(11), 1256–1267 (2012)

    Google Scholar 

  9. Ioannidis, Y.E.: Query optimization. ACM Comput. Surv. 28(1), 121–123 (1996)

    Article  Google Scholar 

  10. Jovanovic, P., Romero, O., Abelló, A.: A unified view of data-intensive flows in business intelligence systems: a survey. T. Large-Scale Data Knowl. Centered Syst. 29, 66–107 (2016)

    Google Scholar 

  11. Kougka, G., Gounaris, A.: Optimization of data-intensive flows: is it needed? Is it solved? In: DOLAP, pp. 95–98 (2014)

    Google Scholar 

  12. Kougka, G., Gounaris, A., Simitsis, A.: The many faces of data-centric workflow optimization: a survey. CoRR, abs/1701.07723 (2017)

    Google Scholar 

  13. Ogasawara, E.S., de Oliveira, D., Valduriez, P., Dias, J., Porto, F., Mattoso, M.: An algebraic approach for data-centric scientific workflows. PVLDB 4, 1328–1339 (2011)

    Google Scholar 

  14. Poess, M., Rabl, T., Caufield, B.: TPC-DI: the first industry benchmark for data integration. PVLDB 7(13), 1367–1378 (2014)

    Google Scholar 

  15. Selinger, P.G., Astrahan, M.M., Chamberlin, D.D., Lorie, R.A., Price, T.G.: Access path selection in a relational database management system. In: SIGMOD, pp. 23–34 (1979)

    Google Scholar 

  16. Simitsis, A., Vassiliadis, P., Sellis, T.K.: State-space optimization of ETL workflows. IEEE Trans. Knowl. Data Eng. 17(10), 1404–1419 (2005)

    Article  Google Scholar 

  17. Simitsis, A., Wilkinson, K., Castellanos, M., Dayal, U.: Optimizing analytic data flows for multiple execution engines. In: SIGMOD, pp. 829–840 (2012)

    Google Scholar 

  18. Srivastava, U., Munagala, K., Widom, J., Motwani, R.: Query optimization over web services. In: Proceedings of the 32nd International Conference on Very Large Data Bases VLDB, pp. 355–366 (2006)

    Google Scholar 

  19. Varol, Y.L., Rotem, D.: An algorithm to generate all topological sorting arrangements. Comput. J. 24(1), 83–84 (1981)

    Article  MATH  Google Scholar 

  20. Yerneni, R., Li, C., Ullman, J., Garcia-Molina, H.: Optimizing large join queries in mediation systems. In: Beeri, C., Buneman, P. (eds.) ICDT 1999. LNCS, vol. 1540, pp. 348–364. Springer, Heidelberg (1999). doi:10.1007/3-540-49257-7_22

    Chapter  Google Scholar 

  21. Zinn, D., Bowers, S., McPhillips, T., Ludäscher, B.: Scientific workflow design with data assembly lines. In: Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, WORKS 2009, pp. 14:1–14:10. ACM (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Informatics, Aristotle University of Thessaloniki, Thessaloniki, Greece

    Georgia Kougka & Anastasios Gounaris

Authors
  1. Georgia Kougka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anastasios Gounaris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anastasios Gounaris .

Editor information

Editors and Affiliations

  1. LIAS/ISAE-ENSMA, Chasseneuil, France

    Ladjel Bellatreche

  2. University of Texas at Arlington, Arlington, Texas, USA

    Sharma Chakravarthy

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Kougka, G., Gounaris, A. (2017). Optimal Task Ordering in Chain Data Flows: Exploring the Practicality of Non-scalable Solutions. In: Bellatreche, L., Chakravarthy, S. (eds) Big Data Analytics and Knowledge Discovery. DaWaK 2017. Lecture Notes in Computer Science(), vol 10440. Springer, Cham. https://doi.org/10.1007/978-3-319-64283-3_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-64283-3_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-64282-6

  • Online ISBN: 978-3-319-64283-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation