Skip to main content
SpringerLink
Log in

Fast Fisher Sparsity Preserving Projections

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Recently, there has been a lot of interest in the underlying sparse representation structure in high-dimensional data such as face images. In this paper, we propose two novel efficient dimensionality reduction methods named Fast Sparsity Preserving Projections (FSPP) and Fast Fisher Sparsity Preserving Projections (FFSPP), respectively, which aim to preserve the sparse representation structure in high-dimensional data. Unlike the existing Sparsity Preserving Projections (SPP), where the sparse representation structure is learned through resolving n (the number of samples) time-consuming \( \ell^{ 1} \) norm optimization problems, FSPP constructs a dictionary through classwise PCA decompositions and learns the sparse representation structure under the constructed dictionary through matrix–vector multiplications, which is much more computationally tractable. FFSPP takes into consideration both the sparse representation structure and the discriminating efficiency by adding the Fisher constraint to the FSPP formulation to improve FSPP’s discriminating ability. Both of the proposed methods can boil down to a generalized eigenvalue problem. Experimental results on three publicly available face data sets (Yale, Extended Yale B and ORL), and a standard document collection (Reuters-21578) validate the feasibility and effectiveness of the proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Wine data set is at http://archive.ics.uci.edu/ml.

  2. The source codes of LPP and NPE are available at http://www.zjucadcg.cn/dengcai/.

  3. The source code of SPP is available at http://idar.lcu.edu.cn/.

  4. ORL dataset is at http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html.

  5. Reuters-21578 corpus is at http://www.daviddlewis.com/resources/testcollections/reuters21578/.

References

  1. Jimenez LO, Landgrebe DA (1997) Supervised classification in high-dimensional space: geometrical, statistical, and asymptotical properties of multivariate data. IEEE Trans Syst Man Cybern C 28(1):39–54

    Article  Google Scholar 

  2. Shang F, Jiao L, Shi J, Chai J (2011) Robust positive semidefinite L-Isomap ensemble. Pattern Recogn Lett 32(4):640–649

    Article  Google Scholar 

  3. Gunal S, Edizkan R (2008) Subspace based feature selection for pattern recognition. Inform Sci 178(19):3716–3726

    Article  Google Scholar 

  4. Duda RO, Hart PE, Stork DG (2000) Pattern classification, 2nd edn. Wiley, New York

    Google Scholar 

  5. He X, Niyogi P (2003) Locality preserving projections. In: Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS), pp 585–591

  6. He X, Cai D, Yan S, Zhang H (2005) Neighborhood preserving embedding. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp 1208–1213

  7. Qiao L, Chen S, Tan X (2010) Sparsity preserving projections with applications to face recognition. Pattern Recogn 43(1):331–341

    Article  MATH  Google Scholar 

  8. Belhumeur P, Hepanha J, Kriegman D (1997) Eigenfaces vs. Fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  9. Scholkopf B, Smola A (2002) Learning with kernels. MIT Press, Cambridge, MA

    Google Scholar 

  10. Das S, Sil S (2010) Kernel induced fuzzy clustering of image pixels with an improved differential evolution algorithm. Inform Sci 180(8):1237–1256

    Article  MathSciNet  Google Scholar 

  11. Maldonado S, Weber R, Basak J (2011) Simultaneous feature selection and classification using kernel-penalized support vector machines. Inform Sci 181(1):115–128

    Article  Google Scholar 

  12. Scholkopf B, Solma A, Muller K (1999) Kernel principal component analysis. In: Proceedings of the Advances in Kernel Methods-Support Vector Learning, pp 327–352

  13. Li J-B, Gao HJ (2011) Sparse data-dependent kernel principal component analysis based on least squares support vector machine for feature extraction and recognition. Neural Comput Appl. doi:10.1007/s00521-011-0600-z

    Google Scholar 

  14. Mika S, Ratsch G, Weston J, Scholkopf B, Muller K-R (1999) Fisher discriminant analysis with kernels. In: Proceedings of the IEEE International Workshop on Neural Networks for Signal Processing, volume IX, pp 41–48

  15. Zhang BC, Qiao Y (2010) Face recognition based on gradient gabor feature and efficient kernel fisher analysis. Neural Comput Appl 19(4):617–623

    Article  MathSciNet  Google Scholar 

  16. Li J, Pan J, Chu S (2008) Kernel class-wise locality preserving projection. Inform Sci 178(7):1825–1835

    Article  MATH  Google Scholar 

  17. Wang Z, Sun X (2008) Face recognition using kernel-based NPE. In: Proceedings of the IEEE International Conference on Computer Science and Software Engineering (CSSE), pp 802–805

  18. Yin H, Huang W (2010) Adaptive nonlinear manifolds and their application to pattern recognition. Inform Sci 180(14):2649–2662

    Article  MathSciNet  MATH  Google Scholar 

  19. Shang F, Jiao L, Shi J, Gong M, Shang RH (2011) Fast density-weighted low-rank approximation spectral clustering. Data Min Knowl Discov 23(2):345–378

    Article  MathSciNet  MATH  Google Scholar 

  20. Tenenbaum J, Silva V, Langford J (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319–2322

    Article  Google Scholar 

  21. Roweis S, Saul L (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326

    Article  Google Scholar 

  22. Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6):1373–1396

    Article  MATH  Google Scholar 

  23. Cai D, He X, Han J (2007) Isometric projection. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp 528–533

  24. Candès E (2006) Compressive sampling. In: Proceedings of International Congress of Mathematics. Madrid, Spain, pp 1433–1452

  25. Donoho D (2006) For most large underdetermined systems of linear equations the minimal \( \ell^{1} \)-norm solution is also the sparsest solution. Commun Pur Appl Math 59(6):797–829

    Article  MathSciNet  MATH  Google Scholar 

  26. Donoho D (2006) Compressed sensing. IEEE Trans Inform Theory 52(4):1289–1306

    Article  MathSciNet  Google Scholar 

  27. Baraniuk RG, Wakin MB (2009) Random projections of smooth manifolds. Found Comput Math 9(1):51–77

    Article  MathSciNet  MATH  Google Scholar 

  28. Davenport MA, Boufounos PT, Wakin MB, Baraniuk RG (2010) Signal processing with compressive measurements. IEEE J-STSP 4(2):445–460

    Google Scholar 

  29. Wright J, Ma Y, Mairal J, Sapiro G, Huang TS, Yan S (2010) Sparse representation for computer vision and pattern recognition. Proc IEEE 98(6):1031–1044

    Article  Google Scholar 

  30. Cai J, Ji H, Liu X, Shen Z (2009) Blind motion deblurring from a single image using sparse approximation. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition(CVPR), pp 104–111

  31. Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A (2009) Non-local sparse models for image restoration. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp 2272–2279

  32. Yang J, Wright J, Huang T, Ma Y (2008) Image superresolution as sparse representation of raw patches. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp 1–8

  33. Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A (2009) Supervised dictionary learning. In: Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS), pp 1033–1040

  34. Huang K, Aviyente S (2006) Sparse representation for signal classification. In: Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS), pp 585–591

  35. Fukunaga K (1990) Introduction to statistical pattern recognition, 2nd edn. Academic Press, San Diego

    MATH  Google Scholar 

  36. Chung F (1997) Spectral graph theory. Regional conference series in mathematics, no. 92

  37. Wright J, Yang A, Sastry S, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):210–227

    Article  Google Scholar 

  38. Basri R, Jacobs D (2003) Lambertian relection and linear subspaces. IEEE Trans Pattern Anal Mach Intell 25(3):218–233

    Article  Google Scholar 

  39. Golub G, Van Loan C (1996) Matrix computations, 3rd edn. Johns Hopkins University Press, Baltimore

    MATH  Google Scholar 

  40. Cai D, He X, Han J (2007) Spectral regression: a unified approach for sparse subspace learning. In: Proceedings of the IEEE International Conference on Data Mining (ICDM), pp 73–82

  41. Yan S, Xu D, Zhang B, Zhang H, Yang Q, Lin S (2007) Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans Pattern Anal Mach Intell 29(1):40–51

    Article  Google Scholar 

  42. Wang F, Zhang C (2008) On discriminative semi-supervised classification. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp 720–725

  43. Wang F, Zhang C, Li T (2009) Clustering with local and global regularizations. IEEE Trans Knowl Data En 21(12):1665–1678

    Article  Google Scholar 

  44. Zhao H, Yuen P, Kwok J (2006) A novel incremental principal component analysis and its application for face recognition. IEEE Trans Syst Man Cybern B 36(4):873–886

    Article  Google Scholar 

  45. Zhao H, Yuen P (2008) Incremental linear discriminant analysis for face recognition. IEEE Trans Syst Man Cybern B 38(1):210–221

    Article  MATH  Google Scholar 

  46. Law M, Jain A (2006) Incremental nonlinear dimensionality reduction by manifold learning. IEEE Trans Pattern Anal Mach Intell 28(3):377–391

    Article  Google Scholar 

  47. Kouropteva O, Okun O, Pietikainen M (2005) Incremental locally linear embedding. Pattern Recogn 38(10):1764–1767

    Article  MATH  Google Scholar 

  48. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86

    Article  Google Scholar 

  49. He X, Yan S, Hu Y, Niyogi P, Zhang H (2005) Face recognition using laplacianfaces. IEEE Trans Pattern Anal Mach Intell 27(3):328–340

    Article  Google Scholar 

  50. Lee K, Ho J, Kriegman D (2005) Acquiring linear subspaces for face recognition under variable lighting. IEEE Trans Pattern Anal Mach Intell 27(5):684–698

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to significantly improve the quality of this paper. We also thank Dr. Lei Wang for valuable discussions and suggestions. This work is supported by the National Natural Science Foundation of China Nos. 61072106, 60971112, 60971128, 60970067, and 61072108; the Fundamental Research Funds for the Central Universities Nos. JY10000902001, K50510020001, and JY10000902045; and the Fund for Foreign Scholars in University Research and Teaching Programs (the 111 Project) No. B07048.

Author information

Authors and Affiliations

  1. Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China, Institute of Intelligent Information Processing, Xidian University, Mailbox 224, No. 2 South TaiBai Road, Xi’an, 710071, China

    Fei Yin, L. C. Jiao, Fanhua Shang, Shuang Wang & Biao Hou

Authors
  1. Fei Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. L. C. Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fanhua Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shuang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Biao Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fei Yin.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Yin, F., Jiao, L.C., Shang, F. et al. Fast Fisher Sparsity Preserving Projections. Neural Comput & Applic 23, 691–705 (2013). https://doi.org/10.1007/s00521-012-0978-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-012-0978-2

Keywords

Search

Navigation