Information Theory

Related Concepts

Cryptology (Classical); Shannon’s Model

The entropy function H(X) is a measure of the uncertainty of X, in formula

$$H(X) = -{\sum \limits_{a:{p}_{X}(a)>0}}{p}_{X}(a) \cdot {\textrm{ log}}_{\textrm{ 2}}\;{p}_{X}(a),$$

where ${p}_{\textrm{ X}}(a) = \textrm{ Pr}[x = A]$ denotes the probability that random variable X takes on value a. The interpretation is that with probability ${p}_{X}(a)$, X can be described by ${\log}_{2}\ {p}_{X}(a)$ bits of information.

The conditional entropy or equivocation (Shannon, 1949) H(X ∕ Y ) denotes the uncertainty of X provided Y is known:

$$H\left (\left. X\right \vert Y \right ) = -\sum \limits_{a,b:\,{p}_{\left. X\right \vert Y }\left (\left. a\right \vert b\right )>0}{p}_{X,Y }\left (a,b\right ) {\cdot \log }_{2}\,{p}_{\left. X\right \vert Y }\left (\left. a\right \vert b\right )$$

where ${p}_{X,Y }(a,b) {= }_{\textrm{ def}}\textrm{ Pr}[(X = a) \wedge (Y = b)]$ and ${p}_{X\vert Y }(a\vert b)$ obeys Bayes’ rule for conditional probabilities:

$$\begin{array}{l} {p}_{X,Y }\left (a,b\right ) = {p}_{Y }\left (b\right ) \cdot {p}_{X\left /\right. Y }\left (\left. a\right \vert b\right )\!,\,\textrm{ thus} \\ {-\log }_{2}\,{p}_{X,Y }\left (a,b\right ) = {-\log }_{2}\,{p}_{Y }\left (b\right ) {-\log }_{2}\,{p}_{\left. X\right \vert Y }\left (\left. a\right \vert b\right )\!\end{array}$$

The basic relation on conditional entropy follows from this:

$$H\left (X,Y \right ) = H\left (\left. X\right \vert Y \right ) + H\left (Y \right )$$

In particular, it is to be noted that the entropy is additive if and only if X and Y are independent:

$$H(X,Y ) = H(X) + H(Y ),$$

in analogy to the additive entropy of thermodynamical systems.

The redundancy of a text is that part (expressed in bits) that does not carry information. In common English, the redundancy is roughly 3.5 [bit/char], the information is roughly 1.2 [bit/char], redundancy and information sum up to $4.7 {=\log }_{2}$ 26 [bit/char].

Three possible properties of a cryptosystem are now described using this terminology. A cryptosystem is of Vernam type if H(K) = H(C), where H(K) is the entropy of the key K and H(C) is the entropy of the ciphertext C. A cryptosystem has independent key if the plaintext P and keytext K are mutually independent: H(P) = H(P | K) and H(K) = H(K | P) (“knowledge of the keytext does not change the uncertainty of the plaintext, and knowledge of the plaintext does not change the uncertainty of the keytext”).

A cryptosystem is called perfect if plaintext and ciphertext are mutually independent: H(P) = H(P | C) and H(C) = H(C | P) (“knowledge of the ciphertext does not change the uncertainty of the plaintext, and knowledge of the plaintext does not change the uncertainty of the ciphertext”). This means that the security of the system depends entirely on the key; perfect cryptosystems correspond to holocryptic keytexts (Key), which are extremely difficult to achieve in practice.

Shannon’s Main Theorem

In a cryptosystem, where the key character is uniquely determined by the plaintext character and the ciphertext character (“ciphertext and plaintext together allow no uncertainty on the keytext”), any two of the following three properties of the cryptosystem imply the third one:

Vernamtype, independentkey, perfect.

The unicity distance for a given language, a given cryptosystem, and a given cryptanalytic procedure of attack is the minimal length of the plaintext such that decryption is unique. Example: let Z be the cardinality of keytext space, assume simple substitution (Substitutions and Permutations) and an attack by letter frequency. Then for English with an alphabet of 26 letters, the unicity distance U is given by:

• (1) $U \approx { \frac{1} {0.53}\log }_{2}Z$ for decryption with single-letter frequencies

• (2) $U \approx { \frac{1} {1.2}\log }_{2}Z$ for decryption with bigram frequencies

• (3) $U \approx { \frac{1} {1.5}\log }_{2}Z$ for decryption with trigram frequencies

• (w) $U \approx { \frac{1} {2.1}\log }_{2}Z$ for decryption with word frequencies

• (∗ ) $U \approx { \frac{1} {3.5}\log }_{2}Z$ for decryption using all grammatical and semantical rules

For simple substitution with Z = 26!, one has ${\log}_{2}Z \approx 88.38$. This leads to the values 167, 74, 59, 42, and 25 for the unicity distance, which are confirmed by practical experience.

For bigram substitution with Z = 676!, there is ${\log}_{2}Z \approx 5,385.76$ and $U \approx 1,530$ for decryption using all grammatical and semantical rules.

The situation is rather similar for German, French, Italian, Russian, and related Indo-European languages.

For holocryptic sequences of key elements, the unicity distance is infinite.