A new efficient training strategy for deep neural networks by hybridization of artificial bee colony and limited–memory BFGS optimization algorithms

Abstract

Working up with deep learning techniques requires profound understanding of the mechanisms underlying the optimization of the internal parameters of complex structures. The major factor limiting this understanding is that there exist only a few optimization methods such as gradient descent and Limited–memory Broyden–Fletcher–Goldfarb–Shannon (L-BFGS) to find the best local minima of the problem space for these complex structures such as deep neural network (DNN). Therefore, in this paper, we represent a new training approach named hybrid artificial bee colony based training strategy (HABCbTS) to tune the parameters of a DNN structure, which includes one or more autoencoder layers cascaded to a softmax classification layer. In this strategy, a derivative-free optimization algorithm “ABC” is combined with a derivative-based algorithm “L-BFGS” to construct “HABC”, which is used in the HABCbTS. Detailed simulation results supported by statistical analysis show that the proposed training strategy results in better classification performance compared to the DNN classifier trained with the L-BFGS, ABC and modified ABC. The obtained classification results are also compared with the state-of-the-art classifiers, including MLP, SVM, KNN, DT and NB on 15 data sets with different dimensions and sizes.

Introduction

Deep neural networks (DNNs) were first proposed in early 1980s [1] and in 2002 [2]. However, due to training difficulties in deep architectures, DNNs could not receive the attention that they deserve until the years 2006 and 2007 [3], [4], [5]. Nowadays, DNNs are employed in a wide variety of application areas, such as speech processing [6], pattern recognition [7], classification [8], medical applications [9], [10], [11], etc. [12]. The main reason of this popularity is the attractive properties of DNNs such as good classification performance for complex data sets, applicability to large size data (also for big data), well-established design criteria for complex systems and so on [13], [14]. Besides, the most significant property of DNNs is the capability of producing most convenient features in classification of data sets. In practice, however, it has been proved that the training of a DNN structure is a challenging task [15].

In the literature, there are many techniques to optimize a function [16], [17] or to train different types of network based models [18], however, only a few of these techniques are suitable to find the local minima of a DNN [19]. Derivative-based optimization algorithms including gradient descent (GD), stochastic gradient descent (SGD), limited memory Broyden–Fletcher–Goldfarb–Shannon (L-BFGS) and conjugate gradient (CG) algorithms are generally used to find the optimum parameter values of the DNNs and variants [19]. Although the GD is usually preferred due to its simplicity and speed benefits, it can produce good results only if the initialization of the parameters are close to an optimum solution [3]. Besides, the SGD has better optimization performance than the GD because the SGD needs fewer training samples to optimize the DNN. However, the SGD has several tuning parameters such as learning rate, convergence criteria, etc. Moreover, the L-BFGS and CG utilize approximate second-order derivative information with the goal of dealing with nonlinearity and ill-conditioning. Therefore, the L-BFGS and CG have relatively better performances than the SGD and GD, while the L-BFGS being slightly better than the CG for training of the DNN [19].

The above-mentioned optimization algorithms have more or less some drawbacks such as getting trapped at a local minima, slow convergence rates, complex mathematical backgrounds, and instability. Furthermore, the performance of these algorithms rely on one or more tuning parameters. These parameters are heuristically chosen and externally supplied by the user, since there is no analytical method to determine their optimal values.

In addition to the derivative-based optimization algorithms discussed above, a lot of derivative-free optimization algorithms such as genetic algorithm and artificial bee colony (ABC) algorithm have also been presented to solve optimization problems like training neural network (NN) [20], [21], [22], [23]. Although derivative-free optimization algorithms are highly competitive or sometimes superior to the derivative-based optimization algorithms for the optimization of the complex problems and models, especially for NNs, their performances are usually not very satisfactory to find the optimum values of the weights of DNN structures due to high dimensionality. Compared to the other derivative-free optimization algorithms, the ABC is a highly efficient optimization algorithm to train the DNNs because the ABC has fewer tuning parameters, convergence speed benefits, and better performance than many other derivative-free optimization algorithms [24].

In this paper, in order to overcome the above-mentioned problems, the combination of a derivative-based optimization algorithm “L-BFGS” and a derivative-free optimization algorithm “ABC” is proposed to optimize DNN structures consisting of one or more autoencoders with a softmax classification layer. The proposed hybrid optimization strategy is named Hybrid Artificial Bee Colony based Training Strategy (HABCbTS) to train the DNN. The HABCbTS offers the ability of L-BFGS’s exploring local minima of search space and the capability of ABC’s discovering new possible solution candidates, which are far away from current point where the algorithm is trapped. The HABCbTS is tested with benchmark data sets. Simulation results supported with statistical analysis show that the HABCbTS is not only better than the DNN trained with the ABC but also better than the DNN trained with the L-BFGS. To the best of our knowledge, our method is the first one to study the optimization of the DNN based structures with a hybrid optimization method.

The rest of the paper is organized as follows: In Section 2, we briefly overview the deep learning techniques and formulate the autoencoder, stacked autoencoder and softmax classification unit. In Section 3, we describe the proposed strategy for training of the DNN. In Section 4, experimental results are presented and some aspects of the proposed strategy are discussed by interpreting its time complexity and statistical analysis. In Section 5, the paper is concluded.