Abstract
We propose a new feature extraction method called Margin Maximizing Discriminant Analysis (MMDA) which seeks to extract features suitable for classification tasks. MMDA is based on the principle that an ideal feature should convey the maximum information about the class labels and it should depend only on the geometry of the optimal decision boundary and not on those parts of the distribution of the input data that do not participate in shaping this boundary. Further, distinct feature components should convey unrelated information about the data. Two feature extraction methods are proposed for calculating the parameters of such a projection that are shown to yield equivalent results. The kernel mapping idea is used to derive non-linear versions. Experiments with several real-world, publicly available data sets demonstrate that the new method yields competitive results.
Chapter PDF
Similar content being viewed by others
Keywords
- Feature Extraction
- Linear Discriminant Analysis
- Input Pattern
- Feature Extraction Method
- Quadratic Programming Problem
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Aizerman, M., Braverman, E., Rozonoer, L.: Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control 25, 821–837 (1964)
Baudat, G., Anouar, F.: Generalized discriminant analysis using a kernel approach. Neural Computation 12(10), 2385–2404 (2000)
Bennett, K.P., Campbell, C.: Support vector machines: Hype or hallelujah? SIGKDD Explorations 2(2), 1–13 (2000)
Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998)
Boser, B.E., Guyon, I., Vapnik, V.: A training algorithm for optimal margin classifiers. In: Computational Learing Theory, pp. 144–152 (1992)
Fukumizu, K., Bach, F.R., Jordan, M.I.: Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. Journal of Machine Learning Research 5, 73–99 (2004)
Fukunaga, K.: Statistical Pattern Recognition. Acad. Press, NY (1989)
Herbrich, R., Graepel, T.: A PAC-bayesian margin bound for linear classifiers: Why SVMs work. In: Advances in Neural Information Processing Systems, vol. 13, pp. 224–230. MIT Press, Cambridge (2000)
Jolliffe, I.J.: Principal Component Analysis. Springer, NY (1986)
Kocsor, A., Kovács, K.: Kernel Springy Discriminant Analysis and Its Application to a Phonological Awareness Teaching System. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2002. LNCS (LNAI), vol. 2448, pp. 325–328. Springer, Heidelberg (2002)
Kocsor, A., Tóth, L.: Kernel-Based Feature Extraction with a Speech Technology Application. IEEE Trans. Signal Processing. 52(8) (2004)
Li, H., Jiang, T., Zhang, K.: Efficient and robust feature extraction by maximum margin criterion. In: Advances in Neural Information Processing Systems, Vancouver, Canada, vol. 16, pp. 97–104 (2003)
Mercer, J.: Functions of positive and negative type and their connection with the theory of integral equations. Phil. Trans. Roy. Soc. London, A 209, 415–446 (1909)
Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Müller, K.-R.: Fisher discriminant analysis with kernels. In: Hu, Y.-H., Larsen, J., Wilson, E., Douglas, S. (eds.) Neural Networks for Signal Processing IX, pp. 41–48. IEEE, Los Alamitos (1999)
Poggio, T.: On optimal nonlinear associative recall. Biol. Cyber. 19, 201–209 (1975)
Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo (1993)
Roth, V., Steinhage, V.: Nonlinear discriminant analysis using kernel functions. In: Adv. in Neural Information Processing Systems NIPS, vol. 12, pp. 568–574 (1999)
Schölkopf, B., Smola, A., Müller, K.: Kernel principal component analysis. In: Adv. in Kernel Methods - SV Learning, pp. 327–352. MIT Press, Cambr. (1999)
Vapnik, V.: The Nature of Statistical Learning Theory. Springer, NY (1995)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kocsor, A., Kovács, K., Szepesvári, C. (2004). Margin Maximizing Discriminant Analysis. In: Boulicaut, JF., Esposito, F., Giannotti, F., Pedreschi, D. (eds) Machine Learning: ECML 2004. ECML 2004. Lecture Notes in Computer Science(), vol 3201. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30115-8_23
Download citation
DOI: https://doi.org/10.1007/978-3-540-30115-8_23
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23105-9
Online ISBN: 978-3-540-30115-8
eBook Packages: Springer Book Archive