Abstract
In this article we extend the (recently published) unsupervised information theoretic vector quantization approach based on the Cauchy–Schwarz-divergence for matching data and prototype densities to supervised learning and classification. In particular, first we generalize the unsupervised method to more general metrics instead of the Euclidean, as it was used in the original algorithm. Thereafter, we extend the model to a supervised learning method resulting in a fuzzy classification algorithm. Thereby, we allow fuzzy labels for both, data and prototypes. Finally, we transfer the idea of relevance learning for metric adaptation known from learning vector quantization to the new approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Haykin, S.: Neural Networks - A Comprehensive Foundation. IEEE Press, New York (1994)
Kohonen, T.: Self-Organizing Maps. Springer Series in Information Sciences, vol. 30. Springer, Heidelberg (1995), 2nd extended edn. (1997)
Oja, E., Lampinen, J.: Unsupervised learning for feature extraction. In: Zurada, J.M., Marks II, R.J., Robinson, C.J. (eds.) Computational Intelligence Imitating Life, pp. 13–22. IEEE Press, Los Alamitos (1994)
Brause, R.: Neuronale Netze, 2nd edn. B. G. Teubner, Stuttgart (1995)
Deco, G., Obradovic, D.: An Information-Theoretic Approach to Neural Computing. Springer, Berlin (1997)
Jain, A.K., Duin, R.P.W., Mao, J.: Statistical pattern recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 4–37 (2000)
Kapur, J.N.: Measures of Information and their Application. Wiley, New Delhi (1994)
Principe, J.C., Fischer III, J.W., Xu, D.: Information theoretic learning. In: Haykin, S. (ed.) Unsupervised Adaptive Filtering, Wiley, New York (2000)
Zador, P.L.: Asymptotic quantization error of continuous signals and the quantization dimension. IEEE Transaction on Information Theory (28), 149–159 (1982)
Van Hulle, M.M.: Faithful Representations and Topographic Maps. Wiley Series and Adaptive Learning Systems for Signal Processing, Communications, and Control. Wiley & Sons, New York (2000)
Villmann, T., Claussen, J.-C.: Magnification control in self-organizing maps and neural gas. Neural Computation 18(2), 446–469 (2006)
Van Hulle, M.M.: Joint entropy maximization in kernel-based topographic maps. Neural Computation 14(8), 1887–1906 (2002)
Kullback, S., Leibler, R.A.: On information and sufficiency. Annals of Mathematical Statistics 22, 79–86 (1951)
Shannon, C.E.: A mathematical theory of communication. Bell System Technical Journal 27, 379–432 (1948)
Renyi, A.: On measures of entropy and information. In: Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, University of California Press (1961)
Lehn-Schiler, T., Hegde, A., Erdogmus, D., Principe, J.C.: Vector quantization using information theoretic concepts. Natural Computing 4(1), 39–51 (2005)
Renyi, A.: Probability Theory. North-Holland Publishing Company, Amsterdam (1970)
Jenssen, R.: An Information Theoretic Approach to Machine Learning, PhD thesis, University of Troms, Department of Physics (2005)
Seo, S., Obermayer, K.: Soft learning vector quantization. Neural Computation 15, 1589–1604 (2003)
Sato, A., Yamada, K.: Generalized learning vector quantization. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Advances in Neural Information Processing Systems 8. Proceedings of the 1995 Conference, pp. 423–429. MIT Press, Cambridge (1996)
Seo, S., Bode, M., Obermayer, K.: Soft nearest prototype classification. IEEE Transaction on Neural Networks 14, 390–398 (2003)
Torkkola, K.: Feature extraction by non-parametric mutual information maximization. Journal of Machine Learning Research 3, 1415–1438 (2003)
Villmann, T., Schleif, F.-M., Hammer, B.: Comparison of relevance learning vector quantization with other metric adaptive classification methods. Neural Networks 19 (in press, 2006)
Silverman, B.W.: Density Estimation for Statistics and Data Analysis. Chapman & Hall, Boca Raton (1986)
Blake, C.L., Merz, C.J.: UCI repository of machine learning databases. University of California, Department of Information and Computer Science, Irvine, CA (1998), available at: http://www.ics.uci.edu/~mlearn/MLRepository.html
Torkkola, K., Campbell, W.M.: Mutual information in learning feature transformations. In: Proc. Of International Conference on Machine Learning ICML 2000, Stanford, CA (2000)
Hammer, B., Strickert, M., Villmann, T.: Supervised neural gas with general similarity measure. Neural Processing Letters 21(1), 21–44 (2005)
Verleysen, M., François, D.: Computational Intelligence and Bioinspired Systems. In: Cabestany, J., Prieto, A., Hernández, F.S. (eds.) Proceedings of the 8th International Work-Conference on Artificial Neural Networks 2005 (IWANN), Barcelona (2005)
Hammer, B., Villmann, T.: Generalized relevance learning vector quantization. Neural Networks 15(8-9), 1059–1068 (2002)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Villmann, T., Hammer, B., Schleif, F.M., Geweniger, T., Fischer, T., Cottrell, M. (2006). Prototype Based Classification Using Information Theoretic Learning. In: King, I., Wang, J., Chan, LW., Wang, D. (eds) Neural Information Processing. ICONIP 2006. Lecture Notes in Computer Science, vol 4233. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11893257_5
Download citation
DOI: https://doi.org/10.1007/11893257_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-46481-5
Online ISBN: 978-3-540-46482-2
eBook Packages: Computer ScienceComputer Science (R0)