Multi-layer feed-forward neural networks are commonly used in supervised learning, for which data training is required. One popular way to check whether the training is completed is to monitor the mean square error. It is expected that the learning is completed when the mean square error is less than or equal to an error threshold, which is usually a very small positive real number (e.g., 0.001). However, this terminating condition does not always work very effectively. This paper proposes a new terminating condition to identify the convergence of the learning process in multi-layer feed-forward neural networks. The new termination condition is called Threshold of Output Differences (TOD), which is the difference between an ouput value and its corresponding desired (target) output value to identify the convergence of the learning process. It proposes that the learning is completed when the difference for each output is less than a threshold. The performance investigation showed that the convergence rate of a learning algorithm with this new terminating condition is generally faster than the original one. Moreover, the classification rate (generalization) of a learning algorithm with TOD is usually better than the original one.