Minimizing α -Information for Generalization and Interpretation

Abstract.

In this paper we propose a minimum α -information method to maximize and minimize information contained in hidden units. The method aims to interpret internal representations and to improve generalization performance. The α -information minimization forces hidden units to have maximum information or minimum information, depending on the importance of hidden units. To interpret internal representation, we have only to see a small number of hidden units with maximum information, ignoring hidden units with minimum information. In addition, by minimizing the α -information, unnecessary information can be eliminated, leading to better generalization. First, α -information was applied to a simple data compression problem in which four characters can be compressed into one hidden unit. Then we applied α -information minimization to one of the most challenging topics in neural networks, namely, the description and understanding of natural languages. As an example to demonstrate the acquisition of descriptive adequacy by neural networks, we dealt with the inference of well-formedness of an artificial language close to English. Experimental results confirmed that explicit internal representations can be obtained and generalization can be significantly improved.