Split and merge algorithm for deep learning and its application for additional classes

Highlights

• We present a novel GA based feature extractor for network optimal initialization.

• Our method selects more dominant feature extractor in merge phase using GA.

• Results show improvements recognition performance in comparison with DBNs.

• We also suggest a new approach for retraining additional classes as its application.

• Our approach for retraining can add output classes at lower error rate than DBNs.

Abstract

In this paper, we propose a novel split training and merge algorithm for deep learning. The proposed algorithm improves recognition accuracy and suggests a new approach for retraining. The algorithm is motivated by the genetic algorithm (GA) and is composed of two procedures. The first procedure initializes two individual networks using deep belief networks (DBNs), and the second procedure merges the two networks using the GA. Biases and weights of the network that is trained using DBNs are represented as a matrix between each layer, and each row of this matrix is used as a chromosome in the merge procedure. To evaluate the performance, we conduct two set of experiments. The first set is to recognize accuracy of the proposed algorithm, and the second set is for a new retraining approach. The results show that the proposed algorithm has a lower average error rate (6.84 ± 4.57%) than the DBNs, and it can add classes at a lower average error rate (9.06 ± 6.17% and 10.17 ± 4.51%) without pre-training using the restrict Boltzmann machines (RBMs) for existing classes data.

Introduction

Back propagation (BP) is a typical learning method for deep neural networks (DNNs), and is used in conjunction with an optimization method such as gradient descent [17]. However, it has problems such as local minima, slow convergence speed as the number of hidden layers increase, and overfitting [9]. To overcome these problems, Hinton proposed deep belief networks (DBNs) with a pre-training method that uses restrict Boltzmann machines (RBMs) [7].

DBNs consist of a number of RBM layers trained in a greedy layer-wise manner [15]. RBMs are composed of one visible layer and one hidden layer. RBMs have a restricted condition where the neurons of the same layer are not connected to each other as they are connected in Boltzmann machines (BMs) [4], [18]. Because of such constraint, the training time in RBMs is reduced compared with BMs.

RBMs are trained using an unsupervised learning method for pre-training DBNs. A joint probability between the visible layer and the hidden layer is defined using an energy function in RBMs [11]. This probability distribution is used to obtain weights and biases between the visible layer and the hidden layer via the steepest descent method. To obtain the expected model values, this process starts from any random state of the visible units and takes a long time to perform the alternating Gibbs sampling [6]. However, Hinton assigned the states of the visible units to a training vector and performed alternating Gibbs sampling only once to achieve a faster learning procedure. This one-step contrastive divergence algorithm (CD-1) [8] generated satisfactory training results and was used for RBMs training.

DBNs solved the problems of the conventional DNNs by applying the RBMs pre-training to DNNs for network initialization and increased recognition performance. Hence, recently several approaches have been developed for deep learning that start with DBNs [12], [20].

Previous learning algorithms often require substantial efforts for hand-designing features. However, the RBMs can automatically learn features representations using an unsupervised learning method. After RBMs pre-training for network initialization, dominant features are selected in learning procedure. This process is important to networks performance for several reasons [3], which are as follows:

• The features used for each layer’s input determine the search space to be explored in learning phase. The irrelevant and redundant features make search space larger, increase the learning time, and reduce the recognition performance.

• If the selected feature does not contain all the information needed to determine pattern for distinguish classes, the recognition performance can be unsatisfactory.

• Improperly selected irrelevant and redundant features may make the learning process ineffective.

therefore, to select more dominant features than the RBMs and improve performance of DBNs, we propose a split and merge algorithm for deep learning based on a genetic algorithm (GA). In the past, a study about combining the neural networks with GA have been proposed to improve classification accuracy [1]. However, this method is different from our proposed algorithm; our proposed algorithm use the GA as a feature extractor for neural network initial weights optimization, while this method use the GA as a post-processor for image segmentation after image processing using the neural networks.

Main idea of the proposed algorithm is that extractor for more dominant features can be selected in the merge phase using the GA after the split network training. The proposed algorithm is comprised of two procedures: split and merge. In the split process, the entire training data set is divided into two training data sets. These data sets are trained by DBNs. In the merge process, the two trained networks are merged using the GA.

In the merge phase, the proposed algorithm uses the existing trained information of networks, i.e., weights and biases. Thus, the training data or classes are added to the networks while the original DBNs algorithm is used to pre-train the entire data composed of original and additional data. Then, the proposed algorithm is created with new networks by only pre-training the additional data and using the existing network information. After pre-training the additional data, the result is merged with the existing network using the GA. Consequently, the proposed algorithm will be able to suggest a new approach for retraining.

Recently, there have been several studies on deep neural networks (DNNs). However, most proposed methods are dedicated to enhance the recognition rate for a single complete training set [16]. The goal of the current study is to evaluate a novel split training and merge algorithm designed to improve DBNs recognition performance and suggest a new approach for retraining additional data or classes.