Learning by back-propagation: Computing in a systolic way

Abstract

In this paper we present a systolic algorithm for back-propagation, a supervised, iteratived, gradient-descent, connectionist learning rule. The algorithm works on feedforward networks where connections can skip layers and fully exploits spatial and training parallelism, which are inherent to back-propagation. Spatial parallelism arises during the propagation of activity—forward—and error—backward—for a particular input-output. On the other hand, when this computation is carried out simultaneously for all input-output pairs, training parallelism is obtained. In the spatial dimension, a single systolic ring carries out sequentially the three main steps of the learning rule—forward, backward and weight increments update. Furthermore, the same pattern of matrix delivery is used in both the forward and the backward passes. In this manner, the algorithm preserves the similarity of the forward and backward passes in the original model. The resulting systolic algorithm is dual with respect to the pattern of matrix delivery—either columns or rows.