Skip to main content
Springer Nature Link
Log in

Integrating Cluster-Based Main-Memory Accelerators in Relational Data Warehouse Systems

  • Schwerpunktbeitrag
  • Published:
Datenbank-Spektrum Aims and scope Submit manuscript

Abstract

Today, data warehouse systems are faced with challenges for providing nearly realtime response times even for complex analytical queries on enormous data volumes. Highly scalable computing clusters in combination with parallel in-memory processing of compressed data are valuable techniques to address these challenges. In this paper, we give an overview on core techniques of the IBM Smart Analytics Optimizer—an accelerator engine for IBM’s mainframe database system DB2 for z/OS. We particularly discuss aspects of a seamless integration between the two worlds and describe techniques exploiting features of modern hardware such as parallel processing, cache utilization, and SIMD. We describe issues encountered during the development and evaluation of our system and outline current research activities for solving them.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Performance evaluations in the context of ISAOpt are still in progress.

References

  1. Abadi D, Madden S, Ferreira M (2006) Integrating compression and execution in column-oriented database systems. In: SIGMOD ’06, proceedings of the 2006 ACM SIGMOD international conference on management of data. ACM, New York, pp 671–682 doi:10.1145/1142473.1142548

    Chapter  Google Scholar 

  2. Beier F Parallel and non-disruptive reorganization of distributed in-memory database tables. Master’s thesis, Ilmenau University of Technology (2010)

  3. Beier F, Stolze K, Sattler KU (2011, to appear) Online reorganization in read optimized MMDBS. In: Proceedings of the ACM SIGMOD international conference on management of data (SIGMOD), June 12–16

  4. Bodner T (2010) Evaluation and implementation of specialized dictionaries for the IBM smart analytics optimizer. Bachelor thesis, Duale Hochschule Baden-Württemberg, Stuttgart

  5. Bretschneider S (2009) Erstellung eines Werkzeugs zur automatischen Definition optimierter Data-Marts auf Grundlage vorhandener Abfragen und Abfragestatistiken. Master’s thesis, Hochschule für Wirtschaft und Recht Berlin

  6. Deutsch P (1996) DEFLATE compressed data format specification version 1.3

  7. Goldstein J, Ramakrishnan R, Shaft U (1998) Compressing relations and indexes. In: ICDE ’98, proceedings of the fourteenth international conference on data engineering. IEEE Comput. Soc., Washington, pp 370–379

    Chapter  Google Scholar 

  8. Graefe G, Shapiro LD (1991) Data compression and database performance. In: Proc ACM/IEEE-CS symp on applied computing, pp 22–27

    Google Scholar 

  9. Holloway AL, DeWitt DJ (2008) Read-optimized databases, in depth. VLDB J 1(1):502–513

    Google Scholar 

  10. The IBM financial markets industry models: greater insight for greater value (2007). http://www.ibm.com/software/data/industry-models/financial-data/

  11. IBM Corp (2009) General parallel file system—administration and programming reference, Version 3 Release 3. http://publib.boulder.ibm.com/epubs/pdf/a2322213.pdf

  12. IBM Corp (2010) DB2 Version 9.1 for z/OS. http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2

  13. Johnson R, Raman V, Sidle R, Swart G (2008) Row-wise parallel predicate evaluation. VLDB J 1(1):622–634

    Google Scholar 

  14. Lehner W (2003) Datenbanktechnologie für Data-Warehouse-Systeme. Konzepte und Methoden. Dpunkt Verlag, Heidelberg

    Google Scholar 

  15. Plattner H (2011) SanssouciDB: an in-memory database for processing enterprise workloads. In: Datenbanksysteme für Business, Technologie und Web (BTW). Lecture notes in informatics, pp 2–21

    Google Scholar 

  16. Raman V, Swart G, Qiao L, Reiss F, Dialani V, Kossmann D, Narang I, Sidle R (2008) Constant-time query processing. In: ICDE ’08, proceedings of the 2008 IEEE 24th international conference on data engineering. IEEE Computer Society, Washington, pp 60–69

    Chapter  Google Scholar 

  17. Roth MA, Van Horn SJ (1993) Database compression. SIGMOD Rec 22(3):31–39

    Article  Google Scholar 

  18. TPC: TPC benchmark DS. Standard. Transaction Processing Performance Council (2007)

  19. Ziv J, Lempel A (1977) A universal algorithm for sequential data compression. IEEE Trans Inf Theory 23:337–343

    Article  MathSciNet  MATH  Google Scholar 

  20. Zukowski M, Boncz PA, Nes N, Heman S (2005) MonetDB/X100—a DBMS in the CPU cache. IEEE Data Eng Bull 28(2):17–22

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IBM Germany Research & Development, Schönaicher Str. 220, 71032, Böblingen, Germany

    Knut Stolze & Oliver Koeth

  2. Database & Information Systems Group, Ilmenau University of Technology, P.O. Box 100 565, 98684, Ilmenau, Germany

    Felix Beier & Kai-Uwe Sattler

Authors
  1. Knut Stolze
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Felix Beier
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Oliver Koeth
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Kai-Uwe Sattler
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Knut Stolze.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stolze, K., Beier, F., Koeth, O. et al. Integrating Cluster-Based Main-Memory Accelerators in Relational Data Warehouse Systems. Datenbank Spektrum 11, 101–110 (2011). https://doi.org/10.1007/s13222-011-0056-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13222-011-0056-4

Keywords