Skip to main content
SpringerLink
Log in

Algorithms and Architecture for Managing Evolving ETL Workflows

  • Conference paper
  • First Online:
New Trends in Databases and Information Systems (ADBIS 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1064))

Included in the following conference series:

Abstract

ETL processes are responsible for extracting, transforming and loading data from data sources into a data warehouse. Currently, managing ETL workflows has some challenges. First, each ETL tool has its own model for specifying ETL processes. This makes it is difficult to specify ETL processes that are beyond the capabilities of a chosen tool or switch between ETL tools without having to redesign the entire ETL workflow again. Second, a change in structure of a data source leads to ETL workflows that can no longer be executed and yields errors.

Therefore, we propose a logical model for ETL processes that makes it feasible to (semi-)automatically repair ETL workflows. Our first approach is to specify ETL processes using Relational Algebra extended with update operations. This way, ETL processes can be automatically translated into SQL queries to be executed into any relational database management system. Later, we will consider expressing ETL tasks by means of an Extensible Markup Language (XML) and other programming languages. We also propose the Extended Evolving-ETL (E3TL) framework in which we will develop algorithms for (semi-) automatic repair of ETL workflows upon data source schema changes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Curino, C., Moon, H.J., Tanca, L., Zaniolo, C.: Schema evolution in Wikipedia: toward a web information system benchmark. In: Proceedings of the 10\(^{\rm th}\) International Conference on Enterprise Information Systems, ICEIS 2008. pp. 323–332. Citeseer, Barcelona, Spain (2008)

    Google Scholar 

  2. El Akkaoui, Z., Zimányi, E.: Defining ETL worfklows using BPMN and BPEL. In: Proceedings of the ACM 12th International Workshop on Data Warehousing and OLAP, DOLAP 2009. pp. 41–48. ACM, Hong Kong, China (2009)

    Google Scholar 

  3. Muñoz, L., Mazón, J.-N., Pardillo, J., Trujillo, J.: Modelling ETL processes of data warehouses with uml activity diagrams. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM 2008. LNCS, vol. 5333, pp. 44–53. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88875-8_21

    Chapter  Google Scholar 

  4. Papastefanatos, G., Vassiliadis, P., Simitsis, A., Sellis, T., Vassiliou, Y.: Rule-based management of schema changes at ETL sources. In: Grundspenkis, J., Kirikova, M., Manolopoulos, Y., Novickis, L. (eds.) ADBIS 2009. LNCS, vol. 5968, pp. 55–62. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12082-4_8

    Chapter  Google Scholar 

  5. Poess, M., Rabl, T., Jacobsen, H., Caufield, B.: TPC-DI: The first industry benchmark for data integration. Proc. of the VLDB Endowment 7(13), 1367–1378 (2014)

    Google Scholar 

  6. Santos, V., Belo, O.: Using relational algebra on the specification of real world ETL processes. In: Proceedings of the 13\(^{\rm th}\) IEEE International Conference on Dependable, Autonomic and Secure Computing, DASC 2015. pp. 861–866. IEEE, Liverpool, UK (2015)

    Google Scholar 

  7. Trujillo, J., Luján-Mora, S.: A UML based approach for modeling ETL processes in data warehouses. In: Song, I.-Y., Liddle, S.W., Ling, T.-W., Scheuermann, P. (eds.) ER 2003. LNCS, vol. 2813, pp. 307–320. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-39648-2_25

    Chapter  Google Scholar 

  8. Vassiliadis, P., Simitsis, A., Skiadopoulos, S.: Conceptual modeling for ETL processes. In: Proceedings of the 5\(^{\rm th}\) ACM International workshop on Data Warehousing and OLAP, DOLAP 2002. pp. 14–21. ACM, McLean, Virginia, USA (2002)

    Google Scholar 

  9. Vassiliadis, P., Simitsis, A., Skiadopoulos, S.: Modeling ETL activities as graphs. In: Proceedings of the 4\(^{\rm th}\) International Workshop on Design and Management of Data Warehouses, DMDW 2002. pp. 52–61. CEUR-WS.org, Toronto, Canada (2002)

    Google Scholar 

  10. Wojciechowski, A.: ETL workflow reparation by means of case-based reasoning. Inf. Syst. Front. 20(1), 21–43 (2018)

    Google Scholar 

Download references

Acknowledgements

The work of Judith Awiti is supported by the European Commission through the Erasmus Mundus Joint Doctorate project Information Technologies for Business Intelligence-Doctoral College (IT4BI-DC).

Author information

Authors and Affiliations

  1. Université Libre de Bruxelles, Brussels, Belgium

    Judith Awiti

Authors
  1. Judith Awiti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Judith Awiti .

Editor information

Editors and Affiliations

  1. University of Maribor, Maribor, Slovenia

    Tatjana Welzer

  2. Alpen Adria University Klagenfurt, Klagenfurt am Wörthersee, Austria

    Johann Eder

  3. University of Maribor, Maribor, Slovenia

    Vili Podgorelec

  4. Poznan University of Technology, Poznan, Poland

    Robert Wrembel

  5. University of Novi Sad, Novi Sad, Serbia

    Mirjana Ivanović

  6. Free University of Bozen-Bolzano, Bolzano, Italy

    Johann Gamper

  7. Poznań University of Technology, Poznan, Poland

    Mikoƚaj Morzy

  8. University of Thessaly, Lamia, Greece

    Theodoros Tzouramanis

  9. Université Lumière Lyon 2, Lyon, France

    Jérôme Darmont

  10. University of Maribor, Maribor, Slovenia

    Aida Kamišalić Latifić

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Awiti, J. (2019). Algorithms and Architecture for Managing Evolving ETL Workflows. In: Welzer, T., et al. New Trends in Databases and Information Systems. ADBIS 2019. Communications in Computer and Information Science, vol 1064. Springer, Cham. https://doi.org/10.1007/978-3-030-30278-8_51

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30278-8_51

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30277-1

  • Online ISBN: 978-3-030-30278-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation