skip to main content
10.1145/2808194.2809483acmconferencesArticle/Chapter ViewAbstractPublication PagesictirConference Proceedingsconference-collections
short-paper

Optimal Packing in Simple-Family Codecs

Published: 27 September 2015 Publication History

Abstract

The Simple family of codecs is popular for encoding postings lists for a search engine because they are both space effective and time efficient at decoding. These algorithms pack as many integers into a codeword as possible before moving on to the next codeword. This technique is known as left-greedy. This contribution proves that left-greedy is not optimal and then goes on to introduce a dynamic programming solution to find the optimal packing. Experiments on .gov2 and INEX Wikipedia 2009 show that although this is an interesting theoretical result, left-greedy is empirically near optimal in effectiveness and efficiency.

References

[1]
Anh, V. N., A. Moffat, Inverted Index Compression Using Word-Aligned Binary Codes. Information Retrieval, 8(1):151--166, 2005.
[2]
Anh, V. N., A. Moffat, Simplified Similarity Scoring Using Term Ranks. In SIGIR 2005, pp. 226--233.
[3]
Anh, V. N., A. Moffat, Index Compression Using 64-bit words. Software Practice & Experience, 40(2):131--147, 2010.
[4]
Dean, J. Challenges in Building Large-Scale Information Retrieval Systems. In WSDM 2009.
[5]
Elias, P., Universal Codeword Sets and the Representation of the Integers. IEEE Transactions on Information Theory, 21(2):194--203, 1975.
[6]
Golomb, S. W., Run-length Encodings. IEEE Transactions on Information Theory, 12(3):399--401, 1966.
[7]
Lemire, D., L. Boytsov, Decoding Billions of Integers per Second Through Vectorization. Software: Practice & Experience, To Appear.
[8]
Lemire, D., L. Boytsov, N. Kurz, SIMD Compression and the Intersection of Sorted Integers. CoRR abs/1401.6399, 2014.
[9]
Persin, M., J. Zobel, R. Sacks-Davis, Filtered Document retrieval with Frequency-Sorted Indexes. J. Am. Soc. Inf. Sci., 47(10):749--764, 1996.
[10]
Robertson, S. E., S. Walker, S. Jones, M. M. Beaulieu, M. Gatford. Okapi at TREC-3. In TREC-3, pp. 109--126, 1994.
[11]
Scholer, F., H. E. Williams, J. Yiannis, J. Zobel. Compression of Inverted Indexes for Fast Query Evaluation. In SIGIR 2002, pp. 222--229
[12]
Silvestri, F., R. Venturini, VSEncoding: Efficient Coding and Fast Decoding of Integer Lists via Dynamic Programming, In CIKM 2010, 1219--1228.
[13]
Trotman, A., Compression, SIMD, and Postings Lists, In ADCS 2014, pp. 50--57.
[14]
Trotman, A., X. Jia, M. Crane, Towards an Efficient and Effective Search Engine, In SIGIR 2012 Workshop on Open Source Information Retrieval. pp. 40--47.
[15]
Zhang, J., X. Long, T. Suel, Performance of Compressed Inverted List Caching in Search Engines, In WWW 2008, pp. 387--396.
[16]
Zukowski, M., S. Heman, N. Nes, P. Boncz, Super-Scalar RAM-CPU Cache Compression, In ICDE 2006.

Cited By

View all
  • (2017)ANS-Based Index CompressionProceedings of the 2017 ACM on Conference on Information and Knowledge Management10.1145/3132847.3132888(677-686)Online publication date: 6-Nov-2017

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICTIR '15: Proceedings of the 2015 International Conference on The Theory of Information Retrieval
September 2015
402 pages
ISBN:9781450338332
DOI:10.1145/2808194
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2015

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. compression
  2. inverted files
  3. procrastination

Qualifiers

  • Short-paper

Conference

ICTIR '15
Sponsor:

Acceptance Rates

ICTIR '15 Paper Acceptance Rate 29 of 57 submissions, 51%;
Overall Acceptance Rate 235 of 527 submissions, 45%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 30 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2017)ANS-Based Index CompressionProceedings of the 2017 ACM on Conference on Information and Knowledge Management10.1145/3132847.3132888(677-686)Online publication date: 6-Nov-2017

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media