A Nearly Memory-Optimal Data Structure for Sets and Mappings

Abstract

A novel, very memory-efficient hash table structure for representing a set of bit vectors — such as the set of reachable states of a system — is presented. Let the length of the bit vectors be w. There is an information-theoretic lower bound on the average memory consumption of any data structure that is capable of representing a set of at most n such bit vectors. We prove that, except in extreme cases, this bound is within 10% of nw – nlog₂ n+n. In many cases this is much smaller than what typical implementations of hash tables, binary trees, etc., require, because they already use nw bits for representing just the payload. We give both theoretical and experimental evidence that our data structure can attain an estimated performance of 1.07nw – 1.05nlog₂ n + 6.12n, which is close to the lower bound and much better than nw for many useful values of n and w. We show how the data structure can be extended to mappings while retaining its good performance. Furthermore, our data structure is not unduly slow.