Tests of different regularization terms in small networks

Abstract

Several regularization terms, some of them widely applied to neural networks, such as weight decay and weight elimination, and some others new, are tested when applied to networks with a small number of connections handling continuous variables. These networks are found when using additive algorithms that work by adding processors. First the different methods and their rationale is presented. Then, results are shown, first for curve fitting problems. Since the network constructive algorithm is being used for system modeling, results are also shown for a toy problem that includes recurrency buildup, in order to test the influence of the regularization terms in this process. The results show that this terms can be of help in order to detect unnecessary connections. No clear winner has been found among the presented terms in these tests.