Entropy of Weight Distributions of Small-Bias Spaces and Pseudobinomiality

Abstract

A classical bound in information theory asserts that small $L_1$-distance between probability distributions implies small difference in Shannon entropy, but the converse need not be true. We show that if a probability distribution on $\{0,1\}^n$ has small-bias, then the converse holds for its weight distribution in the proximity of the binomial distribution. Namely, we argue that if a probability distribution $\mu$ on $\{0,1\}^n$ is $\delta$-biased, then $\Vert \overline{\mu }-bin_n \Vert _1^2\le (2\ln {2})(n \delta +H(bin_n)-H(\overline{\mu }))$, where $\overline{\mu }$ is the weight distribution of $\mu$ and $bin_n$ is the binomial distribution on $\{0,\ldots , n\}$. We use this relation to study the notion of pseudobinomiality: we call a probability distributions on $\{0,1\}^n$ pseudobinomial if the weight distribution of each of its restrictions and translations is close to the binomial distribution. We show that, for spaces with small bias, the pseudobinomiality error in the $L_1$-sense is equivalent to that in the entropy-difference-sense. We also study the notion of average case pseudobinomiality and the resulting questions on the pseudobinomiality of sums of independent small-bias spaces.