On the sample complexity of various learning strategies in the probabilistic PAC learning paradigms

Abstract

Recently, in the context of learning probability distributions or stochastic rules, learning strategies which take into account both the simplicity of the hypothesized model and how well it explains the data have been shown to be effective. There are several strategies which fall into this general category, such as the minimum description length (MDL) principle and Occam's Razor. In this paper, we give au intuitive account of the reason why hypotheses obtained by such strategies may exhibit fast convergence to the true or optimal model as the sample size increases. We do so using the notion of ‘uniform convergence.’ We then investigate how we might apply the ‘uniform convergence method’ to estimate the convergence rates of these strategies, using the well-known Kullback-Leibler divergence as the distance measure between probabilistic information sources. In the process of doing so, we show that in fact for proving fast convergence with respect to the Kullback-Leibler divergence by the uniform convergence method, it is convenient to modify the MDL principle. We thus propose a new principle of statistical estimation, which we call ‘NIC(a new information criterion),’ motivated primarily by the goal of proving fast convergence to the true model.