Abstract
Online or real-time BI has remained elusive despite significant efforts by academic and industrial research. Some of the most prominent problems in accomplishing faster turnaround are related to the data ingest. The process of extracting data from source systems, transforming, and loading (ETL) it is often bottlenecked by architectural choices and fragmentation of the processing chain.
In this paper, we present a vision for a resource-efficient infrastructure that enables just-in-time processing with regards to data ingest. At the heart of our approach are (1) the converting of compute intensive parts of the ETL process into in-database processing and (2) the activating of the process on demand via a system of flexible views.
Our approach avoids processing of data that is not being accessed any time soon, scales effectively with the database system and avoids administration and management overhead.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Agrawal, D., Abbadi, A., Singh, A., Yurek, T.: Efficient View Maintenance at Data Warehouses. SIGMOD Record 26(2), 417–427 (1997)
Andzic, J., Fiore, V., Sisto, L.: Extraction, Transformation, and Loading Processes. In: Wrembel, R., Koncilia, C. (eds.) Data Warehouses and OLAP: Concepts, Architectures and Solutions, pp. 88–110. IGI Global (2007)
Buchmann, A., Koldehofe, B.: Complex Event Processing. Information Technology 51(5), 241–242 (2009)
Bruckner, R.M., List, B., Schiefer, J.: Striving Towards Near Real-time Data Integration for Data Warehouses. In: Kambayashi, Y., Winiwarter, W., Arikawa, M. (eds.) DaWaK 2002. LNCS, vol. 2454, pp. 317–326. Springer, Heidelberg (2002)
Chaudhuri, S., Dayal, U., Narasayya, V.: An Overview of Business Intelligence Technology. Communications of the ACM 54(8), 88–98 (2011)
Colby, L.S., Griffin, T., Libkin, L., Mumick, I.S., Trickey, H.: Algorithms for Deferred View Maintenance. SIGMOD Record 25(2), 469–480 (1996)
Greenplum Inc.: Technical Documentation, http://www.greenplum.com
Gupta, A., Mumick, I.S.: Materialized Views: Techniques, Implementations, and Application, 1st edn. MIT Press (1999)
Kimball, R., Caserta, J.: The Data Warehouse ETL Toolkit. John Wiley & Sons Inc. (2004)
Microsoft Corporation. Microsoft SQL Server, Technical Documentation, http://www.microsoft.com/sql
Oracle Corporation: Technet. Technical Documentation, http://www.oracle.com/technetwork/indexes/documentation/index.html
Quass, D., Widom, J.: On-Line Warehouse View Maintenance. In: Proc. of ACM SIGMOD Int. Conf. on Management of Data, pp. 393–404 (1997)
Vassiliadis, P., Karagiannis, A., Tziovara, V., Simitsis, A.: Towards a Benchmark for ETL Workflows. In: Proc. of Int. Workshop on Quality in Databases (QDB), pp. 49–60 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Freudenreich, T., Furtado, P., Koncilia, C., Thiele, M., Waas, F., Wrembel, R. (2013). An On-Demand ELT Architecture for Real-Time BI. In: Castellanos, M., Dayal, U., Rundensteiner, E.A. (eds) Enabling Real-Time Business Intelligence. BIRTE 2012. Lecture Notes in Business Information Processing, vol 154. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39872-8_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-39872-8_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39871-1
Online ISBN: 978-3-642-39872-8
eBook Packages: Computer ScienceComputer Science (R0)