Abstract
Modern large-scale applications are generating staggering amounts of data. In an effort to summarize and index these data sets, databases often use bitmap indices. These indices have become widely adopted due to their dual properties of (1) being able to leverage fast bit-wise operations for query processing and (2) compressibility. Today, two pervasive bitmap compression schemes employ a variation of run-length encoding, aligned over bytes (BBC) and words (WAH), respectively. While BBC typically offers high compression ratios, WAH can achieve faster query processing, but often at the cost of space. Recent work has further shown that reordering the rows of a bitmap can dramatically increase compression. However, these sorted bitmaps often display patterns of changing run-lengths that are not optimal for a byte nor a word alignment. We present a general framework to facilitate a variable length compression scheme. Given a bitmap, our algorithm is able to use different encoding lengths for compression on a per-column basis. We further present an algorithm that efficiently processes queries when encoding lengths share a common integer factor. Our empirical study shows that in the best case our approach can out-compress BBC by 30% and WAH by 70%, for real data sets. Furthermore, we report a query processing speedup of 1.6× over BBC and 1.25× over WAH. We will also show that these numbers drastically improve in our synthetic, uncorrelated data sets.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Abadi, D., Madden, S., Ferreira, M.: Integrating compression and execution in column-oriented database systems. In: ACM SIGMOD International Conference on Management of Data, pp. 671–682 (2006)
Antoshenkov, G.: Byte-aligned bitmap compression. In: DCC 1995: Proceedings of the Conference on Data Compression, p. 476. IEEE Computer Society, USA (1995)
Apaydin, T., Tosun, A.Ş., Ferhatosmanoglu, H.: Analysis of basic data reordering techniques. In: Ludäscher, B., Mamoulis, N. (eds.) SSDBM 2008. LNCS, vol. 5069, pp. 517–524. Springer, Heidelberg (2008)
Brisaboa, N.R., Ladra, S., Navarro, G.: Directly addressable variable-length codes. In: Karlgren, J., Tarhio, J., Hyyrö, H. (eds.) SPIRE 2009. LNCS, vol. 5721, pp. 122–130. Springer, Heidelberg (2009)
Chan, C.-Y., Ioannidis, Y.E.: An efficient bitmap encoding scheme for selection queries. In: Proceedings of the 1999 ACM SIGMOD International Conference on Management of data SIGMOD 1999, pp. 215–226. ACM, New York (1999)
Deliege, F., Pederson, T.: Position list word aligned hybrid: Optimizing space and performance for compressed bitmaps. In: Proceedings of the 2010 International Conference on Extending Database Technology (EDBT 2010), pp. 228–239 (2010)
Donno, F., Litmaath, M.: Data management in wlcg and egee. worldwide lhc computing grid. Technical Report CERN-IT-Note-2008-002, CERN, Geneva (February 2008)
Elias, P.: Universal codeword sets and representations of the integers. IEEE Transactions on Information Theory, 21(2), 194–203 (1975)
Golomb, S.W.: Run-Length Encodings. IEEE Transactions on Information Theory 12(3), 399–401 (1966)
Kaser, O., Lemire, D., Aouiche, K.: Histogram-aware sorting for enhanced word-aligned compression in bitmap indexes. In: ACM 11th International Workshop on Data Warehousing and OLAP, pp. 1–8 (2008)
Lemire, D., Kaser, O.: Reordering columns for smaller indexes. Information Sciences 181 (2011)
Lemire, D., Kaser, O., Aouiche, K.: Sorting improves word-aligned bitmap indexes. Data and Knowledge Engineering 69, 3–28 (2010)
Moffat, A., Zobel, J.: Parameterised compression for sparse bitmaps. In: SIGIR, pp. 274–285 (1992)
Moffat, A., Zobel, J.: Self-indexing inverted files for fast text retrieval. ACM Transactions on Information Systems 14, 349–379 (1996)
O’Neil, P.E.: Model 204 architecture and performance. In: Gawlick, D., Reuter, A., Haynie, M. (eds.) HPTS. LNCS, vol. 359, pp. 40–59. Springer, Heidelberg (1989)
Pinar, A., Tao, T., Ferhatosmanoglu, H.: Compressing bitmap indices by data reorganization. In: Proceedings of the 2005 International Conference on Data Engineering (ICDE 2005), pp. 310–321 (2005)
Sinha, R.R., Winslett, M.: Multi-resolution bitmap indexes for scientific data. ACM Trans. Database Syst 32 ( August 2007)
Wong, H.K.T., Fen Liu, H., Olken, F., Rotem, D., Wong, L.: Bit transposed files. In: Proceedings of VLDB 1985, pp. 448–457 (1985)
Wu, K., Otoo, E., Shoshani, A.: An efficient compression scheme for bitmap indices. ACM Transactions on Database Systems (2004)
Wu, K., Otoo, E.J., Shoshani, A.: Compressing bitmap indexes for faster search operations. In: Proceedings of the 2002 International Conference on Scientific and Statistical Database Management Conference (SSDBM 2002), pp. 99–108 (2002)
Wu, K., Otoo, E.J., Shoshani, A., Nordberg, H.: Notes on design and implementation of compressed bit vectors. Technical Report LBNL/PUB-3161, Lawrence Berkeley National Laboratory (2001)
Zaki, M.J., Wang, J.T.L.: Special issue on bionformatics and biological data management. Information Systems 28, 241–367 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Corrales, F., Chiu, D., Sawin, J. (2011). Variable Length Compression for Bitmap Indices. In: Hameurlain, A., Liddle, S.W., Schewe, KD., Zhou, X. (eds) Database and Expert Systems Applications. DEXA 2011. Lecture Notes in Computer Science, vol 6861. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23091-2_32
Download citation
DOI: https://doi.org/10.1007/978-3-642-23091-2_32
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-23090-5
Online ISBN: 978-3-642-23091-2
eBook Packages: Computer ScienceComputer Science (R0)