Learning efficiency improvement of back-propagation algorithm by error saturation prevention method

Abstract

Back-propagation (BP) algorithm is currently the most widely used learning algorithm in artificial neural networks. With proper selection of feed-forward neural network architecture, it is capable of approximating most problems with high accuracy and generalization ability. However, the slow convergence is a serious problem when using this well-known BP learning algorithm in many applications. As a result, many researchers take effort to improve the learning efficiency of BP algorithm by various enhancements. In this research, we consider that the error saturation (ES) condition, which is caused by the use of gradient descent method, will greatly slow down the learning speed of BP algorithm. Thus, in this paper, we will analyze the causes of the ES condition in the output layer. An error saturation prevention (ESP) function is then proposed to prevent the nodes in the output layer from the ES condition. We also apply this method to the nodes in hidden layers to adjust the learning terms. By the proposed methods, we can not only improve the learning efficiency by the ES condition prevention but also maintain the semantic meaning of the energy function. Finally, some simulations are given to show the workings of our proposed method.

Introduction

The back-propagation (BP) algorithm [25] is a widely used learning algorithm in artificial neural networks [11], [12], [14], [18]. It works well for many problems (e.g., classification or pattern recognition, etc.) [2], [3], [9], [25]. However, it suffers two critical drawbacks from the use of gradient-descent method: one is the learning process often traps into local minimum and another is its slow learning speed. As a result, there are many researches on them [4], [5], [15], especially for the learning efficiency improvement by preventing the premature saturation (PS) phenomenon [16], [30]. The PS means that outputs of the artificial neural networks temporarily trap into high error level during the early learning stage. In the learning issues of PS phenomenon, researchers have designed many usable modifications to solve this phenomenon [7], [17], [31]. However, the proposed methods are limited to many assumptions and seem to be complex (will be detailed in the next subsection). In this study, the error saturation (ES) condition [17] is considered as the main cause of PS phenomenon. The ES condition means that the nodes in each layer of artificial neural network models have outputs near the extreme value 0 or 1, but with obvious differences between the desired and actual outputs. Consequently, the learning term (signal) will be very small for the increment of weights or others parameters [23]. Therefore, we will use the error saturation prevention (ESP) method to overcome the PS phenomenon and thus the learning convergence will speed up and the local minimum problem will be relieved. Besides, we keep the semantic meaning of used mean-square-error (MSE) function to rationalize the evaluation of error criterion.

The slow learning speed of conventional BP algorithm for training feed-forward multi-layer neural network models is due to the fact that back propagation is a gradient descent method [25]. Many researchers have taken efforts on the efficiency improvement issue of BP algorithm and we will summarize them as below.

In the research of Lee et al. [16], [17], they analyzed the slow learning convergence caused by the PS phenomenon. Under this situation, the learning term will be too small to lead the training weights adaptation [17], [30], [31]. In their presentation, they found that the PS phenomenon during early epochs of learning procedure is caused by inappropriate setting of initial weights. Also, the probability of PS can be derived by the maximum value of initial weights, the number of nodes in each layer, and the maximum slope of the sigmoid activation function. After deriving the PS probability function, they avoid the PS phenomenon by properly setting initial weights. However, they derived the PS probability equation based on the assumption that the initial weights and thresholds both are in uniform distribution. Besides, some limitations must be set during learning. In real-world applications, however, these uniform distribution requirement and limitations of learning parameters may not be easily achieved.

Vetela and Reifman [30], [31] analyzed the causes of PS by dividing the learning process into three stages: beginning of saturation stage, saturation plateau stage and complete recovery stage. They proposed four necessary conditions for the occurrence of PS phenomenon by gradient of weights, the momentum terms, and extreme error condition. After constructing the PS mechanism and necessary conditions, they prevent anyone of the four conditions to be satisfied and thus avoid the slow learning condition. Actually, they think that the momentum term plays the leading role in the occurrence of PS phenomenon. Therefore, they improve the convergence speed by properly setting momentum term in learning process. This method is efficient to prevent the PS phenomenon in the early learning stage. However, they also suffer from some limitations. For constructing the mechanism of PS phenomenon, they must design a scenario which describes the relationship between weight summation value and momentum term. The scenario assumes that the weight summation value is too small to contribute to learning process due to large quantity of momentum term in the early learning stage. By this scenario, they constructed the mechanism and necessary conditions of PS phenomenon.

Ng et al. [22], Oh [23], and Ooyen and Nienhuis [24] analyzed the causes of slow learning PS phenomenon by the gradient data of activation function. Oh [23] found that if the actual value of output nodes is under the ES condition, the learning term will be too small to improve the weights learning. By this reason, he proposed a modified energy function to overcome the drawbacks of the ES conditions. The proposed energy function will make the learning term reasonable regardless of the distribution of actual output value. Similarly, Ng et al. [22] modified the energy function to scale up the partial derivatives of the activation function and proposed a new weight evolution algorithm based on the modified energy function. Also, Ooyen and Nienhuis [24] improve the learning convergence by a new energy function based on the cross entropy (CE) algorithm. These methods could prevent the occurrence of PS phenomenon, but the used energy functions might be meaningless.