MDL/Bayesian Criteria Based on Universal Coding/Measure

Abstract

In the minimum description length (MDL) and Bayesian criteria, we construct description length of data z ⁿ = z ₁ ⋯ z _n of length n such that the length divided by n almost converges to its entropy rate as n → ∞, assuming z _i is in a finite set A. In model selection, if we knew the true probability P of z ⁿ ∈ A ⁿ, we would choose a model F such that the posterior probability of F given z ⁿ is maximized. But, in many situations, we use Q:A ⁿ → [0,1] such that $\sum_{z^n\in A^n}Q(z^n)\leq 1$ rather than P because only data z ⁿ are available. In this paper, we consider an extension such that each of the attributes in data can be either discrete or continuous. The main issue is what Q is qualified to be an alternative to P in the generalized situations. We propose the condition in terms of the Radon-Nikodym derivative of P with respect to Q, and give the procedure of constructing Q in the general setting. As a result, we obtain the MDL/Bayesian criteria in a general sense.