Minimum Description Length Principle

Synonyms

Information theory; MDL; Minimum encoding inference

Definition

The original “general” minimum description length (MDL) principle for estimation of statistical properties in observed data y ⁿ, or the modelf(y ⁿ; θ, k), represented by parameters θ = θ ₁, …, θ _k , can be stated thus,

• “Find the model with which observed data and the model can be encoded with shortest code length”:

  $$\min _{\theta ,k}\;[\,\log 1/f({y}^{n};\theta ,k) + L(\theta ,k)],$$

where L(θ, k) denotes the code length for the parameters.

The principle is very general and produces a model defined by the estimated parameters. It leaves the selection of L(θ, k) open, and in complex applications the code length can be calculated by visualizing a coding process. The only requirement is that the data must be decodable.

Motivation and Background

The MDL principle is based on the fact that it is not possible to compress data well without taking advantage of the regular features in them. Hence, estimation and data...