Capacity of structured multilayer networks with shared weights

Abstract

The capacity or Vapnik-Chervonenkis dimension of a feedforward neural architecture is the maximum number of input patterns that can be mapped correctly to fixed arbitrary outputs. So far it is known that the upper bound for the capacity of two-layer feedforward architectures with independent weights depends on the number of connections in the neural architecture [1].

In this paper we focus on the capacity of multilayer feedforward networks structured by shared weights. We show that these structured architectures can be transformed into equivalent conventional multilayer feed-forward architectures. Known estimations for the capacity are extended to achieve upper bounds for the capacity of these general multi-layer feedforward architectures. As a result an upper bound for the capacity of structured architectures is derived that increases with the number of independent network parameters. This means that weight sharing in a fixed neural architecture leads to a significant reduction of the upper bound of the capacity.