Abstract
With the advent of SQL column stores, compression has gained renewed interest and drawn considerable attention from both academia and industry. Unlike row stores, column stores use lightweight compression methods and, generally, compression granularity is at entire column level. In this paper we outline and explore an alternative compression strategy for column stores that works at a different granularity and adapts itself to data, on-the-fly, using a compression planner. The approach yields good compression ratios, facilitates compression during bulk data load and also mitigates some issues that arise from having to maintain global meta-data on compression. We describe its implementation in analytics database dbX, a cloud agnostic, columnar MPP SQL product and present experimental results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadi, D., Boncz, P.A., Harizopoulos, S., Idreos, S., Madden, S.: The design and implementation of modern column-oriented database systems. Found. Trends Databases 5(3), 197–280 (2013)
Abadi, D.J., Madden, S., Ferreira, M.: Integrating compression and execution in column-oriented database systems. In: SIGMOD 2006, pp. 671–682. ACM (2006)
Abadi, D.J., Madden, S., Hachem, N.: Column-stores vs. row-stores: how different are they really? In: SIGMOD 2008, pp. 967–980. ACM (2008)
Baklarz, G.: DB2 compression estimation tool. Technical report, IBM Corporation, Canada, October 2016
Binnig, C., Hildenbrand, S., Färber, F.: Dictionary-based order-preserving string compression for main memory column stores. In: SIGMOD 2009, pp. 283–296. ACM (2009)
Copeland, G.P., Khoshafian, S.: A decomposition storage model. In: SIGMOD, pp. 268–279. ACM (1985)
Damme, P., Habich, D., Hildebrandt, J., Lehner, W.: Lightweight data compression algorithms: an experimental survey. In: Proceedings of 20th EDBT, pp. 72–83 (2017)
Flajolet, P., Fusy, E., Gandouet, O., Mennier, F.: HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm. In: Analysis of Algorithms 2007 (AofA07), pp. 127–146 (2007)
Goldstein, J., Ramakrishnan, R., Shaft, U.: Compressing relations and indexes. In: Proceedings of 14th ICDE, pp. 370–379. IEEE (1998)
Graefe, G., Shapiro, L.: Data compression and database performance. In: Proceedings of ACM/IEEE-CS Symposium on Applied Computing, pp. 22–27. IEEE (1991)
Hovestadt, M., Kao, O., Kliem, A., Warneke, D.: Evaluating adaptive compression to mitigate the effects of shared I/O in clouds. In: 25th IEEE IPDPS, pp. 1042–1051. IEEE (2011)
Idreos, S., Groffen, F., Nes, N., Manegold, S., Mullender, K.S., Kersten, M.L.: MonetDB: two decades of research in column-oriented database architectures. IEEE Data Eng. Bull. 35(1), 40–45 (2012)
Iyer, B.R., Wilhite, D.: Data compression support in databases. In: Proceedings of 20th VLDB, pp. 695–704 (1994)
Krintz, C., Sucu, S.: Adaptive on-the-fly compression. IEEE Trans. Parallel Distrib. Syst. 17(1), 15–24 (2006)
Krueger, J., Grund, M., Tinnefeld, C., Plattner, H., Zeier, A., Faerber, F.: Optimizing write performance for read optimized databases. In: Kitagawa, H., Ishikawa, Y., Li, Q., Watanabe, C. (eds.) DASFAA 2010. LNCS, vol. 5982, pp. 291–305. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12098-5_23
Lang, H., Mühlbauer, T., Funke, F., Boncz, P.A., Neumann, T., Kemper, A.: Data blocks: hybrid OLTP and OLAP on compressed storage using both vectorization and compilation. In: SIGMOD 2016, pp. 311–326. ACM (2016)
Lelewer, D.A., Hirschberg, D.S.: Data compression. ACM Comput. Surv. 19(3), 261–296 (1987)
Poess, M., Potapov, D.: Data compression in oracle. In: Proceedings of 29th VLDB, pp. 761–770 (2003)
Raman, V., et al.: DB2 with BLU acceleration: so much more than just a column store. PVLDB 6(11), 1080–1091 (2013)
Sridhar, K.T., Sakkeer, M.A.: Optimizing database load and extract for big data era. In: Bhowmick, S.S., Dyreson, C.E., Jensen, C.S., Lee, M.L., Muliantara, A., Thalheim, B. (eds.) DASFAA 2014. LNCS, vol. 8422, pp. 503–512. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-05813-9_34
Stonebraker, M., et al.: C-store: a column-oriented DBMS. In: Proceedings of the 31st International Conference on Very Large Data Bases, pp. 553–564. VLDB Endowment (2005)
TeraData: compression types supported by teradata database, release 16.0 (2016). https://info.teradata.com
Viglas, S.: Just-in-time compilation for SQL query processing. PVLDB 6(11), 1190–1191 (2013)
Wust, J., Krüger, J., Gund, M., Hartmann, U., Plattner, H.: Sparse dictionaries for in-memory column stores. In: Proceedings of 4th DBKDA, pp. 25–33. IARIA (2012)
Zukowski, M., Héman, S., Nes, N., Boncz, P.A.: Super-scalar RAM-CPU cache compression. In: Proceedings of 22nd ICDE, pp. 59–71. IEEE (2006)
Acknowledgement
We thank M.A. Sakkeer for his feedback in using modules of compression and decompression in bulk loading and query execution; Dipanjan Deb for operational cloud support and help with AWS runs.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Sridhar, K.T., Johnson, J. (2018). Entropy Aware Adaptive Compression for SQL Column Stores. In: Kozielski, S., Mrozek, D., Kasprowski, P., Małysiak-Mrozek, B., Kostrzewa, D. (eds) Beyond Databases, Architectures and Structures. Facing the Challenges of Data Proliferation and Growing Variety. BDAS 2018. Communications in Computer and Information Science, vol 928. Springer, Cham. https://doi.org/10.1007/978-3-319-99987-6_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-99987-6_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99986-9
Online ISBN: 978-3-319-99987-6
eBook Packages: Computer ScienceComputer Science (R0)