Recurrent neural networks can be trained to be maximum a posteriori probability classifiers

doi:10.1016/0893-6080(94)00059-U

Neural Networks

Volume 8, Issue 1, 1995, Pages 25-29

https://doi.org/10.1016/0893-6080(94)00059-U Get rights and content

Abstract

This paper proves that supervised learning algorithms used to train recurrent neural networks have an equilibrium point when the network implements a maximum a posteriori probability (MAP) classifier. The result holds as a limit when the size of the training set goes to infinity. The result is general, because it stems as a property of cost minimizing algorithms, but to prove it we implicitly assume that the network we are training has enough computing power to actually implement the MAP classifier. This assumption can be satisfied using a universal dynamic system approximator We refer our discussion to Block Feedback Neural Networks (B_FNs) and show that they actually have the universal approximation property.

References (7)

E.K. Blum et al.
Approximation theory and feedforward networks
Neural Networks
(1991)
K. Hornik
Approximation capabilities of multilayer feedforward networks
Neural Networks
(1991)
J.B. Hampshire et al.
Equivalence proofs for multi-layer perceptron classifiers and the Bayesian discriminant function

There are more references available in the full text version of this article.

Cited by (17)

A new boosting algorithm for improved time-series forecasting with recurrent neural networks
2008, Information Fusion
Ensemble methods for classification and regression have focused a great deal of attention in recent years. They have shown, both theoretically and empirically, that they are able to perform substantially better than single models in a wide range of tasks. We have adapted an ensemble method to the problem of predicting future values of time series using recurrent neural networks (RNNs) as base learners. The improvement is made by combining a large number of RNNs, each of which is generated by training on a different set of examples. This algorithm is based on the boosting algorithm where difficult points of the time series are concentrated on during the learning process however, unlike the original algorithm, we introduce a new parameter for tuning the boosting influence on available examples. We test our boosting algorithm for RNNs on single-step-ahead and multi-step-ahead prediction problems. The results are then compared to other regression methods, including those of different local approaches. The overall results obtained through our ensemble method are more accurate than those obtained through the standard method, backpropagation through time, on these datasets and perform significantly better even when long-range dependencies play an important role.
Hybrid NN/HMM acoustic modeling techniques for distributed speech recognition
2006, Speech Communication
Distributed speech recognition (DSR) where the recognizer is split up into two parts and connected via a transmission channel offers new perspectives for improving the speech recognition performance in mobile environments. In this work, we present the integration of hybrid acoustic models using tied-posteriors in a distributed environment. A comparison with standard Gaussian models is performed on the AURORA2 task and the WSJ0 task. Word-based HMMs and phoneme-based HMMs are trained for distributed and non-distributed recognition using either MFCC or RASTA-PLP features. The results show that hybrid modeling techniques can outperform standard continuous systems on this task. Especially the tied-posteriors approach is shown to be usable for DSR in a very flexible way since the client can be modified without a change at the server site and vice versa.
Approaches to fractional land cover and continuous field mapping: A comparative assessment over the BOREAS study region
2004, Remote Sensing of Environment
Subpixel land cover mapping involves the estimation of surface properties using sensors whose spatial sampling is coarse enough to produce mixtures of the properties within each pixel. This study evaluates five algorithms for mapping subpixel land cover fractions and continuous fields of vegetation properties within the BOREAS study area. The algorithms include a conventional “hard”, per-pixel classifier, a neural network, a clustering/look-up-table approach, multivariate regression, and linear least squares inversion. A land cover map prepared using a Landsat TM mosaic was adopted as the source of fine scale calibration and validation data. Coarse scale mixtures of five basic land cover classes and continuous vegetation fields, both corresponding to the field of view of SPOT-VEGETATION imagery (1.15-km pixel size), were synthesised from the TM mosaic using a modelled point spread function. Two measures of land cover distribution were used, fractions of fine scale land cover categories and continuous fields of vegetation structural characteristics. The subpixel algorithms were applied using both proximate (<100 km) and distant (>400 km) separation between training and validation regions. “Hard” classification performed poorly in estimating proportions or continuous fields. The neural network, look-up-table and multivariate regression algorithms produced good matches of spatial patterns and regional land cover composition for the proximate treatment. However, all three methods exhibited substantial biases with the distant treatment due to the characteristics of the training data. Linear least squares inversion offers a relatively unbiased but less precise alternative for subpixel proportion and fraction mapping as it avoids calibration to the a priori distribution of land cover in the training data. In general, a combination of multivariate regression for proximate training data and linear least squares inversion for distant training data resulted in woody fraction estimates within 20% of the Landsat TM classification-based estimates.
Learning long-term dependencies by the selective addition of time-delayed connections to recurrent neural networks
2002, Neurocomputing
Recurrent neural networks possess interesting universal approximation capabilities, making them good candidates for time-series modeling. Unfortunately, long-term dependencies are difficult to learn if gradient descent algorithms are employed. We support the view that it is easier for these algorithms to find good solutions if time-delayed connections are included in the recurrent networks. The algorithm we present here allows one to choose the right locations and delays for such connections. As we show on several univariate benchmarks and one multivariate real-world problem, this algorithm produces very good results while keeping the total number of connections in the recurrent network to a minimum.
Bayesian learning for recurrent neural networks [2]
2001, Neurocomputing
A view-based neurocomputational system for relational map-making and navigation in visual environments
1995, Robotics and Autonomous Systems
Artificial navigation systems stand to benefit greatly from learning maps of visual environments, but traditional map-making techniques are inadequate in several respects. This paper describes an adaptive, view-based, relational map-making system for navigating within a 3D environment defined by a spatially distributed set of visual landmarks. Inspired by an analogy to learning aspect graphs of 3D objects, the system comprises two neurocomputational architectures that emulate cognitive mapping in the rat hippocampus. The first architecture performs unsupervised place learning by combining the “What” with the “Where”, namely through conjunctions of landmark identity, pose, and egocentric gaze direction within a local, restricted sensory view of the environment. The second associatively learns action consequences by incorporating the “When”, namely through conjunctions of learned places and coarsely coded robot motions. Together, these networks form a map reminiscent of a partially observable Markov decision process, and consequently provide an ideal neural substrate for prediction, environment recognition, route planning, and exploration. Preliminary results from real-time implementations on a mobile robot called MAVIN (the Mobile Adaptive VIsual Navigator) demonstrate the potential for these capabilities.

View all citing articles on Scopus

View full text

Contributed articleRecurrent neural networks can be trained to be maximum a posteriori probability classifiers

Abstract

Neural Networks

Neural Networks

Equivalence proofs for multi-layer perceptron classifiers and the Bayesian discriminant function

Contributed article
Recurrent neural networks can be trained to be maximum a posteriori probability classifiers