Using Images to Compare Two Constructive Network Techniques

Abstract

A common theme in answering the question of how distinct concepts are represented in neural networks is to associate meaning to the activity of an individual, or set of neurons. While such models may provide an explanation of how reasoning may occur, they rarely suggest how structures come to exist, often relying on some form of the innateness conjecture. This paper addresses the issue of concept formation by using the principle of structural risk minimization during network construction. This paper describes two related methods for incrementally constructing networks from smaller network architectures. The methods are applied to an image prediction problem that provides a sufficiently difficult learning task, while allowing both quantitive and qualitative analysis of performance. Experiments suggest that incrementally introducing structure during learning in a principled manner can assist in overcoming the bias/variance problem as, for the given problem, the methods show no sign of over-fitting after prolonged training.