Natural Gradient and Multiclass NLDA Networks

Abstract

Natural gradient has been recently introduced as a method to improve the convergence of Multilayer Perceptron (MLP) training [1] as well as that of other neural network type algorithms. The key idea is to recast the training process as a problem in quasi maximum log—likelihood estimation of a certain semipara-metric probabilistic model. This allows the natural introduction of a riemannian metric tensor G in the probabilistic model space. Once G is computed, the “natural” gradient in this setting is $c G\left( W \right)^{ - 1} \nabla _W e\left( {X,y:W} \right)$ , rather than the ordinary euclidean gradient $\nabla _W e\left( {X,y;W} \right)$ . Here e(X,y; W) denotes an error function associated to a concrete pattern (X, y) and weight set W. For instance, in MLP training, e(X,y;W) = (y – F(X,W))²/2, with F the MLP transfer function. Viewing (y – F(X, W))²/2 as the log—likelihood of a probability density, the metric tensor is

$$ G\left( W \right) = \smallint \smallint \frac{{\partial \log p}} {{\partial W}}\left( {\frac{{\partial \log p}} {{\partial W}}} \right)^t p\left( {X,y;W} \right)dXdy. $$

G(W) is also known as the Fisher Information matrix, as it gives the variance of the Cramer—Rao bound for the optimal W estimator. In this work we shall consider a natural gradient—like training for Non Linear Discriminant Analysis (NLDA) networks, a non—linear extension of Fisher’s well known Linear Discriminant Analysis introduced in [6] (more details below). Instead of following an approach along the previous lines, we observe that (1) can be viewed as the covariance $G\left( W \right) = E\left[ {\nabla _W e\left( {X,y;W} \right)\nabla _W e\left( {X,y;W} \right)^t } \right]$ of the random vector $\nabla _W \left( {X,y;W} \right)$ .

With partial support of TIC 01-572 and CAM 02-18