Abstract
This chapter introduces a probabilistic interpretation of artificial neural networks (ANNs), moving the focus from posterior probabilities to probability density functions (pdfs). Parametric and non-parametric neural-based algorithms for unsupervised estimation of pdfs, relying on maximum-likelihood or on the Parzen Window techniques, are reviewed. The approaches may overcome the limitations of traditional statistical estimation methods, possibly leading to improved pdf models. Two paradigms for combining ANNs and hidden Markov models (HMMs) for sequence recognition are then discussed. These models rely on (i) an ANN that estimates state-posteriors over a maximum-a-posteriori criterion, or on (ii) a connectionist estimation of emission pdfs, featuring global optimization of HMM and ANN parameters over a maximumlikelihood criterion. Finally, the chapter faces the problem of the classification of graphs (structured data), by presenting a connectionist probabilistic model for the posterior probability of classes given a labeled graphical pattern. In all cases, empirical evidence and theoretical arguments underline the fact that plausible probabilistic interpretations of ANNs are viable and may lead to improved statistical classifiers, not only in the statical but also in the sequential and structured pattern recognition setups.
Keywords
- Hide Markov Model
- Speech Recognition
- Recurrent Neural Network
- Probabilistic Interpretation
- Inductive Logic Programming
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2), 157–166 (1994); Special Issue on Recurrent Neural Networks (March 1994)
Bengio, Y.: Neural Networks for Speech and Sequence Recognition. International Thomson Computer Press, London (1996)
Bengio, Y., De Mori, R., Flammia, G., Kompe, R.: Global optimization of a neural network-hidden Markov model hybrid. IEEE Transactions on Neural Networks 3(2), 252–259 (1992)
Besag, J.: Spatial Interaction and the Statistical Analysis of Lattice Systems. Journal of the Royal Statistical Society 36, 192–236 (1974)
Besag, J.: Statistical Analysis of Non-Lattice Data. The Statistician 24, 179–195 (1975)
Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995)
Bourlard, H., Morgan, N.: Continuous speech recognition by connectionist statistical methods. IEEE Trans. on Neural Networks 4(6), 893–909 (1993)
Bourlard, H., Morgan, N.: Connectionist Speech Recognition. A Hybrid Approach. The Kluwer international series in engineering and computer science, vol. 247. Kluwer Academic Publishers, Boston (1994)
Bourlard, H., Morgan, N.: Connectionist Speech Recognition. A Hybrid Approach, p. 117. Kluwer Academic Publishers, Boston (1994)
Bourlard, H., Wellekens, C.: Links between hidden Markov models and multilayer perceptrons. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 1167–1178 (1990)
Bridle, J.S.: Alphanets: a recurrent ‘neural’ network architecture with a hidden Markov model interpretation. Speech Communication 9(1), 83–92 (1990)
Buntine, W.L.: Chain Graphs for Learning. In: UAI 1995: Proceedings of the Eleventh Annual Conference on Uncertainty in Artificial Intelligence, pp. 46–54. Morgan Kaufmann, San Francisco (1995)
Duda, R.O., Hart, P.E.: Pattern Classification and Scene Analysis. Wiley, New York (1973)
Elman, J.L.: Finding structure in time. Cognitive Science 14, 179–211 (1990)
Franzini, M.A., Lee, K.F., Waibel, A.: Connectionist Viterbi training: a new hybrid method for continuous speech recognition. In: International Conference on Acoustics, Speech and Signal Processing, Albuquerque, NM, pp. 425–428 (1990)
Gonsalves, C.M.: Comparison Of Search-based And Kernel-based Methods For Graph-based Relational Learning. University of Texas at Arlington (August 2005)
Gori, M., Bengio, Y., De Mori, R.: BPS: A learning algorithm for capturing the dynamical nature of speech. In: Proceedings of the International Joint Conference on Neural Networks, Washington D.C, pp. 643–644. IEEE, New York (1989)
Gori, M., Monfardini, G., Scarselli, F.: A new model for learning in graph domains. In: Proc. of IJCNN 2005 (August 2005)
Haffner, P., Franzini, M., Waibel, A.: Integrating time alignment and neural networks for high performance continuous speech recognition. In: International Conference on Acoustics, Speech and Signal Processing, Toronto, pp. 105–108 (1991)
Hammer, B., Micheli, A., Sperduti, A.: Universal approximation capability of cascade correlation for structures. Neural Computation 17(5), 1109–1159 (2005)
Haykin, S.: Neural Networks (A Comprehensive Foundation). Macmillan, New York (1994)
Hertz, J., Krogh, A., Palmer, R.: Introduction to the Theory of Neural Computation. Addison-Wesley, Reading (1991)
Huang, X.D., Ariki, Y., Jack, M.: Hidden Markov Models for Speech Recognition. Edinburgh University Press, Edinburgh (1990)
Jordan, M.I. (ed.): Learning in Graphical Models. MIT Press, Cambridge (1999)
Jordan, M.I.: Serial order: A parallel, distributed processing approach. In: Elman, J.L., Rumelhart, D.E. (eds.) Advances in Connectionist Theory: Speech. Lawrence Erlbaum, Hillsdale (1989)
Kindermann, R., Snell, J.L.: Markov Random Fields and Their Applications. American Mathematical Society, Providence (1980)
Lavrac, N., Dzeroski, S.: Inductive Logic Programming: Techniques and Applications. Ellis Horwood, New York (1994)
Levin, E.: Word recognition using hidden control neural architecture. In: International Conference on Acoustics, Speech and Signal Processing, Albuquerque, NM, pp. 433–436 (1990)
Liu, D.C., Nocedal, J.: On the Limited Memory BFGS Method for Large Scale Optimization. Mathematical Programming 45, 503–528 (1989)
Minsky, M.L., Papert, S.A.: Perceptrons. MIT Press, Cambridge (1969)
Morgan, N., Bourlard, H.: Continuous speech recognition using multilayer perceptrons with hidden Markov models. In: International Conference on Acoustics, Speech and Signal Processing, Albuquerque, NM, pp. 413–416 (1990)
Mozer, M.C.: Neural net architectures for temporal sequence processing. In: Weigend, A., Gershenfeld, N. (eds.) Predicting the future and understanding the past, pp. 243–264. Addison-Wesley, Redwood City (1993)
Neal, R.M.: Connectionist Learning of Belief Networks. Artificial Intelligence 56(1), 71–113 (1992)
Neapolitan, R.E.: Learning Bayesian Networks. Prentice-Hall, Upper Saddle River (2004)
Niles, L.T., Silverman, H.F.: Combining hidden Markov models and neural network classifiers. In: International Conference on Acoustics, Speech and Signal Processing, Albuquerque, NM, pp. 417–420 (1990)
Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, San Francisco (1988)
Pearlmutter, B.A.: Learning state space trajectories in recurrent neural networks. Neural Computation 1, 263–269 (1989)
Pérez, P.: Markov Random Fields and Images. CWI Quarterly 11, 413–437 (1998)
Pineda, F.J.: Recurrent back-propagation and the dynamical approach to adaptive neural computation. Neural Computation 1, 161–172 (1989)
Rabiner, L., Juang, B.H.: Fundamentals of Speech Recognition. Prentice-Hall, Englewood Cliffs (1993)
Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 77(2), 257–286 (1989)
Richardson, M., Domingos, P.: Markov Logic Networks. Machine Learning 62, 107–136 (2006)
Robinson, R.W.: Counting Unlabeled Acyclic Digraphs. In: Little, C.H.C. (ed.) Combinatorial Mathematics V. LNM, vol. 622, pp. 28–43. Springer, New York (1977)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Rumelhart, D.E., McClelland, J.L. (eds.) Parallel Distributed Processing, ch. 8, vol. 1, pp. 318–362. MIT Press, Cambridge (1986)
Sato, M.: A real time learning algorithm for recurrent analog neural networks. Biological Cybernetics 62, 237–241 (1990)
Sperduti, A., Starita, A.: Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks 8(3), 714–735 (1997)
Spirtes, P., Glymour, C., Scheines, R.: Causation, Prediction, and Search, 2nd edn. MIT Press, Cambridge (2001); Original work published 1993 by Springer-Verlag
Tebelskis, J., Waibel, A., Petek, B., Schmidbauer, O.: Continuous speech recognition using linked predictive networks. In: Lippman, R.P., Moody, R., Touretzky, D.S. (eds.) Advances in Neural Information Processing Systems, Denver, CO, vol. 3, pp. 199–205. Morgan Kaufmann, San Mateo (1991)
Trentin, E.: Networks with trainable amplitude of activation functions. Neural Networks 14, 471–493 (2001)
Trentin, E., Di Iorio, E.: A Simple and Effective Neural Model for the Classification of Structured Patterns. In: Apolloni, B., Howlett, R.J., Jain, L. (eds.) KES 2007, Part I. LNCS (LNAI), vol. 4692, pp. 9–16. Springer, Heidelberg (2007)
Trentin, E., Di Iorio, E.: Classification of Molecular Structures Made Easy. In: 2008 International Joint Conference on Neural Networks, pp. 3241–3246 (2008)
Trentin, E., Gori, M.: Robust combination of neural networks and hidden Markov models for speech recognition. IEEE Transactions on Neural Networks 14(6) (November 2003)
Trentin, E., Gori, M.: Inversion-Based Nonlinear Adaptation of Noisy Acoustic Parameters for a Neural/HMM Speech Recognizer. Neurocomputing 70, 398–408 (2006)
Trentin, E., Matassoni, M., Gori, M.: Evaluation on the Aurora 2 Database of Acoustic Models that are less Noise-sensitive. In: Proceedings of Eurospeech 2003 (September 2003)
Trentin, E.: Simple and Effective Connectionist Nonparametric Estimation of Probability Density Functions. In: Schwenker, F., Marinai, S. (eds.) ANNPR 2006. LNCS (LNAI), vol. 4087, pp. 1–10. Springer, Heidelberg (2006)
Waibel, A.: Modular construction of time-delay neural networks for speech recognition. Neural Computation 1, 39–46 (1989)
Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, K.: Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing 37, 328–339 (1989)
Werbos, P.J.: Generalization of backpropagation with application to a recurrent gas market model. Neural Networks 1, 339–356 (1988)
Williams, R.J., Zipser, D.: Experimental analysis of the real-time recurrent learning algorithm. Connection Science 1, 87–111 (1989)
Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1, 270–280 (1989)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Trentin, E., Freno, A. (2009). Probabilistic Interpretation of Neural Networks for the Classification of Vectors, Sequences and Graphs. In: Bianchini, M., Maggini, M., Scarselli, F., Jain, L.C. (eds) Innovations in Neural Information Paradigms and Applications. Studies in Computational Intelligence, vol 247. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04003-0_7
Download citation
DOI: https://doi.org/10.1007/978-3-642-04003-0_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04002-3
Online ISBN: 978-3-642-04003-0
eBook Packages: EngineeringEngineering (R0)