Abstract
Forward Decoding Kernel Machines (FDKM) combine large-margin classifiers with Hidden Markov Models (HMM) for Maximum a Posteriori (MAP) adaptive sequence estimation. State transitions in the sequence are conditioned on observed data using a kernel-based probability model, and forward decoding of the state transition probabilities with the sum-product algorithm directly produces the MAP sequence. The parameters in the probabilistic model are trained using a recursive scheme that maximizes a lower bound on the regularized cross-entropy. The recursion performs an expectation step on the outgoing state of the transition probability model, using the posterior probabilities produced by the previous maximization step. Similar to Expectation-Maximization (EM), the FDKM recursion deals effectively with noisy and partially labeled data.
We also introduce a multi-class support vector machine for sparse conditional probability regression, GiniSVM based on a quadratic formulation of entropy. Experiments with benchmark classification data show that GiniSVM generalizes better than other multi-class SVM techniques. In conjunction with FDKM, GiniSVM produces a sparse kernel expansion of state transition probabilities, with drastically fewer non-zero coefficients than kernel logistic regression. Preliminary evaluation of FDKM with GiniSVM on a subset of the TIMIT speech database reveals significant improvements in phoneme recognition accuracy over other SVM and HMM techniques.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
L. Rabiner and B-H Juang, Fundamentals of Speech Recognition, Englewood Cliffs, NJ: Prentice-Hall, 1993.
Robinson, A. J., “An application of recurrent nets to phone probability estimation,” IEEE Transactions on Neural Networks, vol. 5,No. 2,March 1994.
Bengio, Y., “Learning long-term dependencies with gradient descent is difficult,” IEEE T. Neural Networks, vol. 5, pp. 157–166, 1994.
Boser, B., Guyon, I. and Vapnik, V., “A training algorithm for optimal margin classifier,” in Proceedings of the Fifth Annual ACM Workshop on Computational Learning Theory, pp 144–52, 1992.
Vapnik, V. The Nature of Statistical Learning Theory, New York: Springer-Verlag, 1995.
Girosi, F., Jones, M. and Poggio, T. “Regularization Theory and Neural Networks Architectures,” Neural Computation, vol. 7, pp 219–269, 1995
Clark, P. and Moreno, M. J. “On the use of Support Vector Machines for Phonetic Classification,” IEEE Conf. Proc., 1999.
Laderfoged, P. A Course in Phonetics, New York, Harcourt Brace Jovanovich, 2nd ed., 1982.
Chakrabartty, S. and Cauwenberghs, G. “Sequence Estimation and Channel Equalization using Forward Decoding Kernel Machines,” IEEE Int. Conf. Acoustics and Signal Proc. (ICASSP’2002), Orlando FL, 2002.
Bourlard, H. and Morgan, N., Connectionist Speech Recognition: A Hybrid Approach, Kluwer Academic, 1994.
Bahl, L. R., Cocke J., Jelinek F. and Raviv J. “Optimal decoding of linear codes for minimizing symbol error rate,” IEEE Transactions on Inform. Theory, vol. IT–20, pp. 284–287, 1974.
Jaakkola, T. and Haussler, D. “Probabilistic kernel regression models,” Proceedings of Seventh International Workshop on Artificial Intelligence and Statistics, 1999.
Schölkopf, B., Burges, C. and Smola, A., Eds., Advances in Kernel Methods-Support Vector Learning, MIT Press, Cambridge, 1998.
Wahba, G. Support Vector Machine, Reproducing Kernel Hilbert Spaces and Randomized GACV, Technical Report 984, Department of Statistics, University of Wisconsin, Madison WI.
Zhu, J and Hastie, T., “Kernel Logistic Regression and Import Vector Machine,” Adv. IEEE Neural Information Processing Systems (NIPS’2001), Cambridge, MA: MIT Press, 2002.
Platt, J., “Probabilities for SV Machines,” Adv. Large Margin Classifiers, Smola, Bartlett et al., Eds., Cambridge MA: MIT Press, 1999.
Breiman, L. Friedman, J. H. et al. Classification and Regression Trees, Wadsworth and Brooks, Pacific Grove, CA, 1984.
Huber, P. J., “Robust Estimation of Location Parameter,” Annals of Mathematical Statistics, vol. 35, March 1964.
Platt, J. “Fast Training of Support Vector Machine using Sequential Minimal Optimization,” Adv. Kernel Methods, Scholkopf, Burgeset al., Eds., Cambridge MA: MIT Press, 1999.
Cauwenberghs, G. and Poggio, T., “Incremental and Decremental Support Vector Machine Learning,” Adv. IEEE Neural Information Processing Systems (NIPS’2000), Cambridge MA: MIT Press, 2001.
Chakrabartty, S. and Cauwenberghs, G. “Sequential Minimal Optimization for Kernel Probabilistic Regression,” Research Note, Center for Language and Speech Processing, The Johns Hopkins University, MD, 2002.
Weston, J. and Watkins, C., “Multi-Class Support Vector Machines,” Technical Report CSD-TR-9800-04, Department of Computer Science, Royal Holloway, University of London, May 1998.
Fisher, W., Doddington G. et al The DARPA Speech Recognition Research Database: Specifications and Status. Proceedings DARPA speech recognition workshop, pp. 93–99, 1986.
Lee, K. F. and Hon, H.W, “Speaker-Independent phone recognition using hidden markov models,” IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 37, pp. 1641–1648, 1989.
Fosler-Lussier, E. Greenberg, S. Morgan, N., “Incorporating contextual phonetics into automatic speech recognition,” Proc. XIVth Int. Cong. Phon. Sci., 1999.
Wald, A. Sequential Analysis, Wiley, New York, 1947.
Chakrabartty, S., Singh, G. and Cauwenberghs, G. “Hybrid Support vector Machine/Hidden Markov Model Approach for Continuous Speech recognition,” Proc. IEEE Midwest Symp. Circuits and Systems (MWSCAS’2000), Lansing, MI, Aug. 2000.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chakrabartty, S., Cauwenberghs, G. (2002). Forward Decoding Kernel Machines: A Hybrid HMM/SVM Approach to Sequence Recognition. In: Lee, SW., Verri, A. (eds) Pattern Recognition with Support Vector Machines. SVM 2002. Lecture Notes in Computer Science, vol 2388. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45665-1_22
Download citation
DOI: https://doi.org/10.1007/3-540-45665-1_22
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44016-1
Online ISBN: 978-3-540-45665-0
eBook Packages: Springer Book Archive