Abstract
In this paper, we propose a new method for kernel optimization in kernel-based dimensionality reduction techniques such as kernel principal component analysis and kernel discriminant analysis. The main idea is to use the graph embedding framework for these techniques and, therefore, by formulating a new minimization problem to simultaneously optimize the kernel parameters and the projection vectors of the chosen dimensionality reduction method. Experimental results are conducted in various datasets, varying from real-world publicly available databases for classification benchmarking to facial expressions and face recognition databases. Our proposed method outperforms other competing ones in classification performance. Moreover, our method provides a systematic way to deal with kernel parameters whose calculation was treated rather superficially so far and/or experimentally, in most of the cases.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Balasubramanian, M., Schwartz, E.: The isomap algorithm and topological stability. Science 295(5552), 7 (2002)
Belhumeur, P., Hespanha, J., Kriegman, D.: Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 711–720 (1997)
Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15(6), 1373–1396 (2003)
Bousquet, O., Herrmann, D.: On the complexity of learning the kernel matrix. In: Advances in neural information processing systems pp. 415–422 (2003)
Cai, D., He, X., Han, J., Zhang, H.J.: Orthogonal laplacianfaces for face recognition. IEEE Trans. Image Process. 15(11), 3608–3614 (2006)
Cai, D., He, X., Hu, Y., Han, J., Huang, T.: Learning a spatially smooth subspace for face recognition. In: Proceedings od IEEE Conference on Computer Vision and Pattern Recognition Machine Learning (CVPR’07) (2007)
Chen, B., Liu, H., Bao, Z.: A kernel optimization method based on the localized kernel fisher criterion. Pattern Recogn. 41(3), 1098–1109 (2008)
Chen, X., Womersley, R., Ye, J.: Minimizing the condition number of a gram matrix. The Department of Applied Mathematics, The Hong Kong Polytechnic University, Preprint (2010)
Chung, F.: Spectral Graph Theory, vol. 92. American Mathematical Society, New York (1997)
Cristianini, N., Scholkopf, B.: Support vector machines and kernel methods: the new generation of learning machines. Ai Mag. 23(3), 31 (2002)
Frank, A., Asuncion, A.: UCI machine learning repository (2010). http://archive.ics.uci.edu/ml
Fukunaga, K.: Introduction to Statistical Pattern Recognition. Academic Press, New York (1990)
Grant, M., Boyd, S.: Graph implementations for nonsmooth convex programs. In: V. Blondel, S. Boyd, H. Kimura (eds.) Recent Advances in Learning and Control, Lecture Notes in Control and Information Sciences, pp. 95–110. Springer, Berlin (2008)
Grant, M., Boyd, S.: CVX: Matlab software for disciplined convex programming, version 1.21. http://cvxr.com/cvx (2011)
He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.J.: Face recognition using laplacianfaces. IEEE Trans. Pattern Anal. Mach Intell. 27(3), 328–340 (2005)
Kanade, T., Cohn, J., Tian, Y.L.: Comprehensive database for facial expression analysis. In: Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition (FG’00), pp. 46- 53 (2000)
Lanckriet, G., Cristianini, N., Bartlett, P., Ghaoui, L., Jordan, M.: Learning the kernel matrix with semidefinite programming. J. Mach. Learn. Res. 5, 27–72 (2004)
Lanckriet, G., Deng, M., Cristianini, N., Jordan, M., Noble, W., et al.: Kernel-based data fusion and its application to protein function prediction in yeast. In: Proceedings of the Pacific Symposium on Biocomputing, vol. 9, p. 2. World Scientific Singapore (2004)
Li, J., Pan, J., Lu, Z.: Kernel optimization-based discriminant analysis for face recognition. Neural Comput. Appl. 18(6), 603–612 (2009)
Lu, Z., Pong, T.: Minimizing condition number via convex programming. Technical report, University of Washington, Seattle, WA, 2010.8 (2010)
MacDuffee, C.: The Theory of Matrices. Dover Publications, New york (2004)
Mika, S., Ratsch, G., Weston, J., Scholkopf, B., Mullers, K.: Fisher discriminant analysis with kernels. In: Neural Networks for Signal Processing IX, 1999. Proceedings of the 1999 IEEE Signal Processing Society Workshop, pp. 41–48. IEEE (1999)
Muller, K., Mika, S., Ratsch, G., Tsuda, K., Scholkopf, B.: An introduction to kernel-based learning algorithms. IEEE Trans. Neural Netw. 12(2), 181–201 (2001)
Niyogi, X.: Locality preserving projections. In: Advances in neural information processing systems 16: Proceedings of the 2003 Conference, vol. 16, p. 153. The MIT Press (2004)
Samaria, F., Harter, A.: Parameterisation of a stochastic model for human face identification. In: Proceedings of the Second IEEE Workshop on Applications of Computer Vision, pp. 138–142. IEEE (1994)
Schölkopf, B., Smola, A.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. The MIT Press, Cambridge (2002)
Schölkopf, B., Smola, A., Müller, K.: Kernel principal component analysis. In: Proceeding of the Artificial Neural Networks, pp. 583–588. Springer: Berlin (1997)
Schölkopf, B., Smola, A., Müller, K.: Kernel principal component analysis. Artificial Neural Networks? ICANN’97 pp. 583–588 (1997)
Varga, R.: Matrix Iterative Analysis, vol. 27. Springer, Berlin (2010)
Wang, F., Sun, J.: Survey on distance metric learning and dimensionality reduction in data mining. Data Mining Knowl. Discov. 29(2), 534–564 (2014)
Wang, J., Lu, H., Plataniotis, K., Lu, J.: Gaussian kernel optimization for pattern classification. Pattern Recogn. 42(7), 1237–1247 (2009)
Wold, S., Esbensen, K., Geladi, P.: Principal component analysis. Chemom. Intell. Lab. Syst. 2(1–3), 37–52 (1987)
Xiong, H., Swamy, M., Ahmad, M.: Optimizing the kernel in the empirical feature space. IEEE Trans. Neural Netw. 16(2), 460–474 (2005)
Yan, S., Xu, D., Zhang, B., Zhang, H., Yang, Q., Lin, S.: Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans. Pattern Anal. Mach. Intell. 29(1), 40–51 (2007)
Author information
Authors and Affiliations
Corresponding author
Additional information
The research leading to these results has received funding from the European Community’s Seventh Framework Programme (FP7/2007–2013) under Grant Agreement No. 248434 (MOBISERV).
Rights and permissions
About this article
Cite this article
Vretos, N., Tefas, A. & Pitas, I. A novel dimensionality reduction technique based on kernel optimization through graph embedding. SIViP 9 (Suppl 1), 3–10 (2015). https://doi.org/10.1007/s11760-015-0832-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-015-0832-y