Abstract
Fisher discriminant analysis gives the unsatisfactory results if points in the same class have within-class multimodality and fails to produce the non-negativity of projection vectors. In this paper, we focus on the newly formulated within and between-class scatters based supervised locality preserving dimensionality reduction problem and propose an effective dimensionality reduction algorithm, namely, Multiplicative Updates based non-negative Discriminative Learning (MUNDL), which optimally seeks to obtain two non-negative embedding transformations with high preservation and discrimination powers for two data sets in different classes such that nearby sample pairs in the original space compact in the learned embedding space, under which the projections of the original data in different classes can be appropriately separated from each other. We also show that MUNDL can be easily extended to nonlinear dimensionality reduction scenarios by employing the standard kernel trick. We verify the feasibility and effectiveness of MUNDL by conducting extensive data visualization and classification experiments. Numerical results on some benchmark UCI and real-world datasets show the MUNDL method tends to capture the intrinsic local and multimodal structure characteristics of the given data and outperforms some established dimensionality reduction methods, while being much more efficient.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Amari SI (1998) Natural gradient works efficiently in learning. Neural Comput 10(2): 251–276
Bellhumer PN, Hespanha J, Kriegman D (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7): 711–720
Belkin M, Niyogi P (2001) Laplacian eigenmaps and spectral techniques for embedding and clustering. In: Advances in neural information processing system, vol 15. MIT Press, Cambridge, pp 585–591
Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6): 1373–1396
BenAbdelkader C, Griffin P (2005) A local region-based approach to gender classification from face images. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. IEEE Computer Society, San Diego, pp 52–53
Blake CL, Merz CJ (1998) UCI repository of machine learning databases. Retrieved from: http://www.ics.uci.edu/~mlearn/MLRepository.html
Borga M, Knutsson H (2001) Canonical correlation analysis in early vision processing. In: Proceedings of the 9th European symposium on artificial neural networks, Bruges, pp 309–314
Chapelle O, Schölkopf B, Zien A (2006) Semi-supervised learning. MIT Press, Cambridge
Chen SC, Zhu YL, Zhang DQ, Yang JY (2005) Feature extraction approaches based on matrix pattern: MatPCA and MatFLDA. Pattern Recogn Lett 26(8): 1157–1167
De la Torre F, Kanade T (2005) Multimodal oriented discriminant analysis. In: Proceedings of the 22nd international conference on machine learning. ACM, New York, pp 177–184
Dempster AP (1971) An overview of multivariate data analysis. J Multivar Anal 1(3): 316–346
Deng C, He XF, Han JW (2007) Isometric projection. In: Proceedings of the 22nd AAAI conference on artificial intelligence (AAAI). Vancouver, Canada, pp 528–533
Globerson A, Roweis S (2006) Metric learning by collapsing classes. In: Weiss Y, Schölkopf B, Platt J (eds) Advances in neural information processing systems, vol 18. MIT Press, Cambridge, pp 451–458
Goldberger J, Roweis S, Hinton G, Salakhutdinov R (2005) Neighbourhood components analysis. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing systems, vol 17. MIT Press, Cambridge, pp 513–520
Hardoon DR, Szedmak S, Shawe-Taylor J (2004) Canonical correlation analysis: an overview with application to learning methods. Neural Comput 16(12): 2639–2664
He XF, Niyogi P (2004) Locality preserving projections. In: Thrun S, Saul L, SchÄolkopf B (eds) Advances in neural information processing systems, vol 16. MIT Press, Cambridge, pp 585–591
Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786): 504–507
Kitamura Y, Sugiyama M (2007) Dimensionality reduction of partially labeled multimodal data. In: Proceeding of the 21st annual conference of the Japanese Society for Artificial Intelligence. Miyazaki, pp 18–22
Kivinen J, Warmuth M (1997) Exponentiated gradient versus gradient descent for linear predictors. Inf Comput 132(1): 1–63
Li H, Jiang T, Zhang K (2006) Efficient and robust feature extraction by maximum margin criterion. IEEE Trans Neural Netw 17(1): 157–165
Lin YY, Liu TL, Chen HT (2005) Semantic manifold learning for image retrieval. In: Proceedings of the ACM conference on multimedia. Singapore, pp 249–258
Martinez AM, Kak AC (2001) PCA versus LDA. IEEE Trans Pattern Anal Mach Intell 23(2): 228–233
Melzer T, Reiter M, Bischof H (2003) Appearance models based on kernel canonical correlation analysis. Pattern Recognit 36(9): 1961–1971
Mika S, Ratsch G, Weston J, Scholkopf B, Mullers KR (1999) Fisher discriminant analysis with kernels. In: Proceedings of the IEEE workshop on neural network for signal processing. Madison, WI, pp 41–48
Moghaddam B, Yang MH (2002) Gender classification with support vector machines. IEEE Trans Pattern Anal Mach Intell 24(5): 707–711
Oja E (1982) A simplified neuron model as a principal component analyzer. J Math Biol 15(3): 267–273
Roweis S, Saul L (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500): 2323–2326
Samaria FS (1994) Face recognition using hidden markov models. PhD thesis, University of Cambridge
Sha F, Saul LK, Lee DD (2003) Multiplicative updates for large margin classifiers. In: Schölkopf B, Warmuth M (eds) Proceedings of the 16th annual conference on computational learning theory (Lecture Notes in Artificial Intelligence). Washington, DC, pp 188–202
Sim T, Baker S, Bsat M (2002) The CMU pose, illumination, and expression (PIE) database. In: Proceedings of the 5th IEEE international conference on face and gesture recognition. IEEE Computer Society, Washington, DC, pp 53–54
Sugiyama M (2006) Local fisher discriminant analysis for supervised dimensionality reduction. In: Proceedings of the 23rd international conference on machine learning. Pittsburgh, pp 905–912
Sugiyama M (2007) Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. J Mach Learn Res 8: 1027–1061
Yang ZR, Laaksonen J (2007) Multiplicative updates for non-negative projections. Neurocomputing 71(13): 363–373
Yang J, Zhang D, Frangi AF, Yang JY (2004) Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans Pattern Anal Mach Intell 26(1): 131–137
Yang X, Fu H, Zha H, Barlow JL (2006) Semi-supervised nonlinear dimensionality reduction. In: Proceedings of the 23rd international conference on machine learning. Pittsburgh, pp 1065–1072
Yang ZG, Li M, Ai HZ (2006) An experimental study on automatic face gender classification. In: Proceedings of the 18th international conference on pattern recognition. IEEE Computer Society, Washington, DC, pp 1099–1102
Zelnik-Manor L, Perona P (2005) Self-tuning spectral clustering. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing systems, vol 17. MIT Press, Cambridge, pp 1601–1608
Zheng WM, Zhou XY, Zou CR, Zhao L (2006) Facial expression recognition using kernel canonical correlation analysis (KCCA). IEEE Trans Neural Netw 17(1): 233–238
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhang, Z., Jiang, M. & Ye, N. Effective multiplicative updates for non-negative discriminative learning in multimodal dimensionality reduction. Artif Intell Rev 34, 235–260 (2010). https://doi.org/10.1007/s10462-010-9172-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-010-9172-z