A nearly-optimal Fano-based coding algorithm

Abstract

Statistical coding techniques have been used for a long time in lossless data compression, using methods such as Huffman’s algorithm, arithmetic coding, Shannon’s method, Fano’s method, etc. Most of these methods can be implemented either statically or adaptively. In this paper, we show that although Fano coding is sub-optimal, it is possible to generate static Fano-based encoding schemes which are arbitrarily close to the optimal, i.e. those generated by Huffman’s algorithm. By taking advantage of the properties of the encoding schemes generated by this method, and the concept of “code word arrangement”, we present an enhanced version of the static Fano’s method, namely Fano⁺. We formally analyze Fano⁺ by presenting some properties of the Fano tree, and the theory of list rearrangements. Our enhanced algorithm achieves compression ratios arbitrarily close to those of Huffman’s algorithm on files of the Calgary corpus and the Canterbury corpus.

Introduction

Huffman’s algorithm is a well-known encoding method that generates an optimal prefix encoding scheme, in the sense that the average code word length is minimum. As opposed to this, Fano’s method has not been used so much because it generates prefix encoding schemes that can be sub-optimal.

In this paper, we present an enhancement of the traditional Fano’s method by which encoding schemes, which are arbitrarily close to the optimum, can be easily constructed.

We assume that we are given a source alphabet, $\text{S}\text{={s}{}_1\text{,…,s}{}_{\mathrm{m}}\text{}}$, whose probabilities of occurrence are $\text{P}\text{=[p}{}_1\text{,…,p}{}_{\mathrm{m}}\text{]}$, and a code alphabet, $\text{A}\text{={a}{}_1\text{,…,a}{}_{\mathrm{r}}\text{}}$. We intend to generate an encoding scheme, {s_i→w_i}, in such a way that $\text{ℓ}\text{̄}\text{=∑}{}_{\mathrm{i=1}}{}^{\mathrm{m}}\text{p}{}_{\mathrm{i}}\text{ℓ}{}_{\mathrm{i}}$ is minimized, where ℓ_i is the length of w_i. Although, we specifically consider the binary code alphabet, the generalization to the r-ary case is not too intricate.

Lossless encoding methods used to solve this problem include Huffman’s algorithm (Huffman, 1952), Shannon’s method (Shannon & Weaver, 1949), arithmetic coding (Sayood, 2000), Fano’s method (Hankerson, Harris, & Johnson, 1998), etc. Adaptive versions of these methods have been proposed, and can be found in (Faller, 1973; Gallager, 1978; Hankerson et al., 1998; Knuth, 1985; Rueda, 2002; Sayood, 2000). Our survey is necessarily brief as this is a well-reputed field.

We assume that the source is memoryless or zeroth-order, which means that the occurrence of the next symbol is independent of any other symbol that has occurred previously. Higher-order models include Markov models (Hankerson et al., 1998), dictionary techniques (Ziv & Lempel, 1977; Ziv & Lempel, 1978), prediction with partial matching (Witten, Moffat, & Bell, 1999), grammar based compression (Kieffer & Yang, 2000), etc., and the techniques introduced here are also readily applicable for such “structure” models.

Huffman’s algorithm proceeds by generating the so called Huffman tree, by recursively merging symbols (nodes) into a new conceptual symbol which constitutes an internal node of the tree. In this way, Huffman’s algorithm generates the tree in a bottom-up fashion. As opposed to this, Fano’s method proceeds by generating a coding tree as well, but it proceeds in a top-down fashion. At each step, the list of symbols is partitioned into two (or more, if the output alphabet is non-binary) new sub-lists, generating two or more new nodes in the corresponding coding tree. Although Fano’s method, typically, generates a sub-optimal encoding scheme,³ the loss in compression ratio with respect to Huffman’s algorithm can be relatively small, but can also be quite significant if the optimality of the partitioning is small.

The binary alphabet version of Fano’s method proceeds by partitioning the list of symbols into two new sub-lists in such a way that the sums of probabilities of these two new sub-lists are as close to being equal as possible. This procedure is recursively applied to the new sub-lists until two atomic sub-lists with a single symbol are obtained, and simultaneously a bit is appended to the code words of each symbol in these sub-lists. As a result of this, if we start with the probabilities of occurrence in a decreasing order, it can be seen that, occasionally, symbols with higher probabilities are assigned to longer code words than those with lower probabilities. This condition is not desirable; we attempt to rectify this condition in Fano⁺, a superior Fano-based scheme which we develop in this paper.

On the other hand, after constructing a coding tree (such as a Huffman tree or a Fano tree), an encoding scheme is generated from that tree by labeling the branches with the code alphabet (typically, binary) symbols. Given a tree constructed from a source alphabet of m symbols, 2^m−1 different encoding schemes can be generated. Of these, only one of them is of a family of codes known as canonical codes, in which the code words are arranged in a lexicographical order (Witten et al., 1999). Canonical codes are desired because they allow extremely fast decoding, and require approximately half of the space used by a decoding tree. These codes are generated by Fano coding in a natural way.

In this paper, we introduce Fano⁺, an enhanced version of the static Fano coding approach utilizing the concept which we called “code word arrangement”, and which is based on a fundamental property of two lists arranged in an increasing and decreasing order respectively (Hardy, Littlewood, & Póyla, 1959). In our context, these lists are the code words (in terms of their lengths) and their probabilities respectively. This paper formally details the encoding scheme generation algorithm, the partitioning procedures suitable for Fano⁺, and a rigorous analysis of their respective properties.

We finally discuss some empirical results obtained from running the static Huffman’s algorithm, the static version of the traditional Fano coding, and Fano⁺, on real life data. Our empirical results show that the compression ratios achieved by Fano⁺ are comparable to those of other optimal encoding methods such as Huffman’s algorithm. Although we use the zeroth-order statistical model, other structure/statistical models such as higher-order models, dictionary models, etc., can also be used in conjunction with Fano⁺ to achieve compression ratios that are close to those attained by most well known compression schemes.