Abstract
Recently, there has been a lot of interest in the underlying sparse representation structure in high-dimensional data such as face images. In this paper, we propose two novel efficient dimensionality reduction methods named Fast Sparsity Preserving Projections (FSPP) and Fast Fisher Sparsity Preserving Projections (FFSPP), respectively, which aim to preserve the sparse representation structure in high-dimensional data. Unlike the existing Sparsity Preserving Projections (SPP), where the sparse representation structure is learned through resolving n (the number of samples) time-consuming \( \ell^{ 1} \) norm optimization problems, FSPP constructs a dictionary through classwise PCA decompositions and learns the sparse representation structure under the constructed dictionary through matrix–vector multiplications, which is much more computationally tractable. FFSPP takes into consideration both the sparse representation structure and the discriminating efficiency by adding the Fisher constraint to the FSPP formulation to improve FSPP’s discriminating ability. Both of the proposed methods can boil down to a generalized eigenvalue problem. Experimental results on three publicly available face data sets (Yale, Extended Yale B and ORL), and a standard document collection (Reuters-21578) validate the feasibility and effectiveness of the proposed methods.
Similar content being viewed by others
Notes
Wine data set is at http://archive.ics.uci.edu/ml.
The source codes of LPP and NPE are available at http://www.zjucadcg.cn/dengcai/.
The source code of SPP is available at http://idar.lcu.edu.cn/.
ORL dataset is at http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html.
Reuters-21578 corpus is at http://www.daviddlewis.com/resources/testcollections/reuters21578/.
References
Jimenez LO, Landgrebe DA (1997) Supervised classification in high-dimensional space: geometrical, statistical, and asymptotical properties of multivariate data. IEEE Trans Syst Man Cybern C 28(1):39–54
Shang F, Jiao L, Shi J, Chai J (2011) Robust positive semidefinite L-Isomap ensemble. Pattern Recogn Lett 32(4):640–649
Gunal S, Edizkan R (2008) Subspace based feature selection for pattern recognition. Inform Sci 178(19):3716–3726
Duda RO, Hart PE, Stork DG (2000) Pattern classification, 2nd edn. Wiley, New York
He X, Niyogi P (2003) Locality preserving projections. In: Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS), pp 585–591
He X, Cai D, Yan S, Zhang H (2005) Neighborhood preserving embedding. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp 1208–1213
Qiao L, Chen S, Tan X (2010) Sparsity preserving projections with applications to face recognition. Pattern Recogn 43(1):331–341
Belhumeur P, Hepanha J, Kriegman D (1997) Eigenfaces vs. Fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720
Scholkopf B, Smola A (2002) Learning with kernels. MIT Press, Cambridge, MA
Das S, Sil S (2010) Kernel induced fuzzy clustering of image pixels with an improved differential evolution algorithm. Inform Sci 180(8):1237–1256
Maldonado S, Weber R, Basak J (2011) Simultaneous feature selection and classification using kernel-penalized support vector machines. Inform Sci 181(1):115–128
Scholkopf B, Solma A, Muller K (1999) Kernel principal component analysis. In: Proceedings of the Advances in Kernel Methods-Support Vector Learning, pp 327–352
Li J-B, Gao HJ (2011) Sparse data-dependent kernel principal component analysis based on least squares support vector machine for feature extraction and recognition. Neural Comput Appl. doi:10.1007/s00521-011-0600-z
Mika S, Ratsch G, Weston J, Scholkopf B, Muller K-R (1999) Fisher discriminant analysis with kernels. In: Proceedings of the IEEE International Workshop on Neural Networks for Signal Processing, volume IX, pp 41–48
Zhang BC, Qiao Y (2010) Face recognition based on gradient gabor feature and efficient kernel fisher analysis. Neural Comput Appl 19(4):617–623
Li J, Pan J, Chu S (2008) Kernel class-wise locality preserving projection. Inform Sci 178(7):1825–1835
Wang Z, Sun X (2008) Face recognition using kernel-based NPE. In: Proceedings of the IEEE International Conference on Computer Science and Software Engineering (CSSE), pp 802–805
Yin H, Huang W (2010) Adaptive nonlinear manifolds and their application to pattern recognition. Inform Sci 180(14):2649–2662
Shang F, Jiao L, Shi J, Gong M, Shang RH (2011) Fast density-weighted low-rank approximation spectral clustering. Data Min Knowl Discov 23(2):345–378
Tenenbaum J, Silva V, Langford J (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319–2322
Roweis S, Saul L (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326
Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6):1373–1396
Cai D, He X, Han J (2007) Isometric projection. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp 528–533
Candès E (2006) Compressive sampling. In: Proceedings of International Congress of Mathematics. Madrid, Spain, pp 1433–1452
Donoho D (2006) For most large underdetermined systems of linear equations the minimal \( \ell^{1} \)-norm solution is also the sparsest solution. Commun Pur Appl Math 59(6):797–829
Donoho D (2006) Compressed sensing. IEEE Trans Inform Theory 52(4):1289–1306
Baraniuk RG, Wakin MB (2009) Random projections of smooth manifolds. Found Comput Math 9(1):51–77
Davenport MA, Boufounos PT, Wakin MB, Baraniuk RG (2010) Signal processing with compressive measurements. IEEE J-STSP 4(2):445–460
Wright J, Ma Y, Mairal J, Sapiro G, Huang TS, Yan S (2010) Sparse representation for computer vision and pattern recognition. Proc IEEE 98(6):1031–1044
Cai J, Ji H, Liu X, Shen Z (2009) Blind motion deblurring from a single image using sparse approximation. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition(CVPR), pp 104–111
Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A (2009) Non-local sparse models for image restoration. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp 2272–2279
Yang J, Wright J, Huang T, Ma Y (2008) Image superresolution as sparse representation of raw patches. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp 1–8
Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A (2009) Supervised dictionary learning. In: Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS), pp 1033–1040
Huang K, Aviyente S (2006) Sparse representation for signal classification. In: Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS), pp 585–591
Fukunaga K (1990) Introduction to statistical pattern recognition, 2nd edn. Academic Press, San Diego
Chung F (1997) Spectral graph theory. Regional conference series in mathematics, no. 92
Wright J, Yang A, Sastry S, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):210–227
Basri R, Jacobs D (2003) Lambertian relection and linear subspaces. IEEE Trans Pattern Anal Mach Intell 25(3):218–233
Golub G, Van Loan C (1996) Matrix computations, 3rd edn. Johns Hopkins University Press, Baltimore
Cai D, He X, Han J (2007) Spectral regression: a unified approach for sparse subspace learning. In: Proceedings of the IEEE International Conference on Data Mining (ICDM), pp 73–82
Yan S, Xu D, Zhang B, Zhang H, Yang Q, Lin S (2007) Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans Pattern Anal Mach Intell 29(1):40–51
Wang F, Zhang C (2008) On discriminative semi-supervised classification. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp 720–725
Wang F, Zhang C, Li T (2009) Clustering with local and global regularizations. IEEE Trans Knowl Data En 21(12):1665–1678
Zhao H, Yuen P, Kwok J (2006) A novel incremental principal component analysis and its application for face recognition. IEEE Trans Syst Man Cybern B 36(4):873–886
Zhao H, Yuen P (2008) Incremental linear discriminant analysis for face recognition. IEEE Trans Syst Man Cybern B 38(1):210–221
Law M, Jain A (2006) Incremental nonlinear dimensionality reduction by manifold learning. IEEE Trans Pattern Anal Mach Intell 28(3):377–391
Kouropteva O, Okun O, Pietikainen M (2005) Incremental locally linear embedding. Pattern Recogn 38(10):1764–1767
Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86
He X, Yan S, Hu Y, Niyogi P, Zhang H (2005) Face recognition using laplacianfaces. IEEE Trans Pattern Anal Mach Intell 27(3):328–340
Lee K, Ho J, Kriegman D (2005) Acquiring linear subspaces for face recognition under variable lighting. IEEE Trans Pattern Anal Mach Intell 27(5):684–698
Acknowledgments
The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to significantly improve the quality of this paper. We also thank Dr. Lei Wang for valuable discussions and suggestions. This work is supported by the National Natural Science Foundation of China Nos. 61072106, 60971112, 60971128, 60970067, and 61072108; the Fundamental Research Funds for the Central Universities Nos. JY10000902001, K50510020001, and JY10000902045; and the Fund for Foreign Scholars in University Research and Teaching Programs (the 111 Project) No. B07048.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yin, F., Jiao, L.C., Shang, F. et al. Fast Fisher Sparsity Preserving Projections. Neural Comput & Applic 23, 691–705 (2013). https://doi.org/10.1007/s00521-012-0978-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-012-0978-2