A new EM-based training algorithm for RBF networks

Abstract

In this paper, we propose a new Expectation–Maximization (EM) algorithm which speeds up the training of feedforward networks with local activation functions such as the Radial Basis Function (RBF) network. In previously proposed approaches, at each E-step the residual is decomposed equally among the units or proportionally to the weights of the output layer. However, these approaches tend to slow down the training of networks with local activation units. To overcome this drawback in this paper we use a new E-step which applies a soft decomposition of the residual among the units. In particular, the decoupling variables are estimated as the posterior probability of a component given an input–output pattern. This adaptive decomposition takes into account the local nature of the activation function and, by allowing the RBF units to focus on different subregions of the input space, the convergence is improved. The proposed EM training algorithm has been applied to the nonlinear modeling of a MESFET transistor.

Introduction

Radial Basis Function (RBF) networks have become one of the most popular feedforward neural networks with applications in regression, classification and function approximation problems (Bishop, 1997, Haykin, 1994). The RBF network approximates nonlinear mappings by weighted sums of Gaussian kernels. Therefore, an RBF learning algorithm must estimate the centers of the units, their variances and the weights of the output layer. Typically, the learning process is separated into two steps: first, a nonlinear optimization procedure to select the centers and the variances and, second, a linear optimization step to fix the output weights. To simplify the nonlinear optimization step, the variances are usually fixed in advance and the centers are selected at random (Broomhead & Lowe, 1988) or applying a clustering algorithm (Moody & Darken, 1989).

Other approaches try to solve the global nonlinear optimization problem using supervised (gradient-based) procedures to estimate the network parameters, which minimize the mean square error (MSE) between the desired output and the output of the network (Karayanis, 1997, Lowe, 1989, Santamarı́a et al., 1999). However, gradient descent techniques tend to be computationally complex and suffer from local minima.

As an alternative to global optimization procedures, a general and powerful method such as the Expectation–Maximization (EM) algorithm (Dempster, Laird, &, Rubin, 1977) can be applied to obtain maximum likelihood (ML) estimates of the network parameters. In the neural networks literature, the EM algorithm has been applied in a number of problems: supervised/nonsupervised learning, classification/function approximation, etc. Here we concentrate on its application to supervised learning in function approximation problems. In this context, Jordan and Jacobs (1994) proposed to use the EM algorithm to train the mixture of experts architecture for regression problems. The EM algorithm has been also applied to estimate the input/output joint pdf, modeled through a Gaussian mixture model, and then estimating the regressor as the conditional pdf (Ghahramani & Jordan, 1994). In both cases the missing data select the most likely member of the mixture given the observations, and then each member is trained independently.

More recently, the EM algorithm has been applied for efficient training of feedforward and recurrent networks (Ma and Ji, 1998, Ma et al., 1997). The work in Ma et al. (1997) connects to the previous work of Feder and Weinstein (1988) for estimating superimposed signals in noise. In both methods, the E-step reduces to decompose at each iteration the total residual into G components (G being the number of neurons). In Feder and Weinstein (1988), the variables used to decompose the residual can be arbitrary values, as long as they sum one, but constant over the function domain: for instance, they propose to decompose the residual into G equal components. In Ma et al. (1997), the residual is decomposed proportionally to the weights of the output layer. Both approaches work well for feedforward networks with global activation functions such as the MLP, but tend to be rather slow for networks with local activation functions since each individual unit is forced to approximate regions far away from the domain of the activation unit.

To overcome this drawback we propose in this paper a new EM algorithm, specific for RBF networks, which aims to accelerate its convergence. We perform a soft decomposition of the residual, taking into account the locality of the basis functions. Different examples show that this modification speeds up the convergence in comparison with previous EM approaches.

The paper is organized as follows. In Section 2, the main features of the EM algorithm are presented. In Section 3, we present some EM-based approaches for the training of feedforward neural networks. In Section 4, the EM algorithm is applied to train an RBF network taking advantage of the local nature of its activation function. Simulation results are provided in Section 5 to validate the proposed algorithm. In Section 6, we apply this algorithm to the small-signal modeling of a MESFET transistor to reproduce the intermodulation behavior. To conclude the paper, in Section 7, the main conclusions are exposed.