Skip to main content
SpringerLink
Log in

Sparse-Representation-Based Classification with Structure-Preserving Dimension Reduction

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Sparse-representation-based classification (SRC), which classifies data based on the sparse reconstruction error, has been a new technique in pattern recognition. However, the computation cost for sparse coding is heavy in real applications. In this paper, various dimension reduction methods are studied in the context of SRC to improve classification accuracy as well as reduce computational cost. A feature extraction method, i.e., principal component analysis, and feature selection methods, i.e., Laplacian score and Pearson correlation coefficient, are applied to the data preparation step to preserve the structure of data in the lower-dimensional space. Classification performance of SRC with structure-preserving dimension reduction (SRC–SPDR) is compared to classical classifiers such as k-nearest neighbors and support vector machines. Experimental tests with the UCI and face data sets demonstrate that SRC–SPDR is effective with relatively low computation cost

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Athitsos V, Sclaroff S. Boosting nearest neighbor classifiers for multiclass recognition. In: IEEE CVPR Workshops. 2005.

  2. Bergeaud F, Mallat S. Matching pursuit of images. In: International conference on image processing, vol. 1. 1995. p. 53–56.

  3. Bradley D, Bagnell JA. Convex coding. Tech. Report CMU-RI-TR-09-22. Pittsburgh, PA: Robotics Institute; 2009.

  4. Cambria E, Hussain A. Sentic album: content-, concept-, and context-based online personal photo management system. Cognit Comput. 2012;4(4):477–96.

    Article  Google Scholar 

  5. Candès EJ, Donoho DL. Curvelets: a surprisingly effective nonadaptive representation of objects with edges. In: Cohen A, Rabut C, Schumaker LL, editors. Curve and surface fitting. Saint-Malo: Vanderbilt University Press; 2000.

  6. Candès EJ, Tao T. Decoding by linear programming. IEEE Trans Inf Theory. 2005;51(12):4203–15.

    Article  Google Scholar 

  7. Candès and EJ, Tao T. The Dantzig selector: statistical estimation when p is much larger than n. Ann Stat. 2007;35(6):2313–51.

    Google Scholar 

  8. Chen X, Qi Y, Bai B, Lin Q, Carbonell JG. Sparse latent semantic analysis. In: SIAM international conference on data mining (SDM). 2011. p. 474–85.

  9. Cook RD, Yin X. Dimension reduction and visualization in discriminant analysis (with discussion). N Z J Stat. 2001;43(2):147–99.

    Article  Google Scholar 

  10. Cox T, Cox M. Multidimensional scaling. London: Chapman and Hall; 1994.

    Google Scholar 

  11. Dai JJ, Lieuand L, Rocke D. Dimension reduction for classification with gene expression microarray data. Statist Appl Genet Mol Biol. 2009;5(1):1–15.

    Google Scholar 

  12. Deegalla S, Boström H. Classification of microarrays with knn: comparison of dimensionality reduction methods. In: The 8th international conference on intelligent data engineering and automated, learning. 2007. p. 800–09.

  13. Elad M, Aharon M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans Image Process. 2006;15(12):3736–45.

    Article  PubMed  Google Scholar 

  14. Engan K, Aase SO, Husøy JH. Multi-frame compression: theory and design. Signal Process. 2000;80(10):2121–40.

    Article  Google Scholar 

  15. Fan J, Lv J. Sure independence screening for ultrahigh dimensional feature space. J R Stat Soc Ser B. 2008;70(5):849–911.

    Article  Google Scholar 

  16. Arbib MA. The handbook of brain theory and neural networks. Cambridge, MA: MIT Press; 1995.

  17. Bache K, Lichman M. UCI Machine Learning Repository. Irvine, CA: University of California, School of Information and Computer Science; 2013. http://archive.ics.uci.edu/ml.

  18. Gao J, Shi Q, Caetano TS. Dimensionality reduction via compressive sensing. Pattern Recognit Lett. 2012;33(9):1163–70.

    Article  Google Scholar 

  19. Georghiades AS, Belhumeur PN, Kriegman DJ. From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell. 2001;23(6):643–60.

    Article  Google Scholar 

  20. Gkioulekas IA, Zickler T. Dimensionality reduction using the sparse linear model. Adv Neural Inf Process Syst. 2011;24:271–9.

    Google Scholar 

  21. Grassi M, Cambria E, Hussain A, Piazza F. Sentic web: a new paradigm for managing social media affective information. Cognit Comput. 2011;3(3):480–9.

    Article  Google Scholar 

  22. Gregor K, LeCun Y. Learning fast approximations of sparse coding. In: International conference on machine learning (Haifa, Israel). 2010. p. 399–406.

  23. Gregor K, Szlam A, LeCun Y. Structured sparse coding via lateral inhibition. In: Advances in neural information processing systems (NIPS) 24. 2011.

  24. Guyon I, Elisseeff A. An introduction to variable and feature selection. J Mach Learn Res. 2003;3:1157–82.

    Google Scholar 

  25. He H, Chen S. Imorl: Incremental multiple objects recognition and localization. IEEE Trans. Neural Netw. 2008;19(10):1727–37.

    Article  PubMed  Google Scholar 

  26. He H, Chen S, Li K, Xu X. Incremental learning from stream data, IEEE Trans. Neural Netw Learn Syst. 2012;22(12):1901–14.

    Google Scholar 

  27. He H, Garcia EA. Learning from imbalanced data. IEEE Trans Knowl Data Eng. 2009;21(9):1263–84.

    Article  Google Scholar 

  28. He H, Ni Z, Fu J. A three-network architecture for on-line learning and optimization based on adaptive dynamic programming. Neurocomputing. 2012;78(1):3–13.

    Article  Google Scholar 

  29. He X, Cai D, Niyogi P. Laplacian score for feature selection. In: Advances in neural information processing systems 18. Cambridge, MA: MIT Press; 2005.

  30. Hu S, Yao Y, Yang Z. Mac protocol identification approach for implement smart cognitive radio. In: IEEE international conference on communications. 2012. p. 5608–12.

  31. Hu S, Yao Y, Yang Z, Zheng D. Cog-prma protocol for cr users sharing a common channel with tdma primary users. In: IEEE wireless and optical communications conference. 2011. p. 1–5.

  32. Huang J, Zhang T. The benefit of group sparsity. Ann Stat. 2010;38:1978–2004.

    Article  Google Scholar 

  33. Kim S, Koh K, Lustig M, Boyd S, Gorinevsky D. An interior-point method for large-scale l1-regularized least squares. IEEE J Sel Top Signal Process. 2007;1(4):606–17.

    Article  Google Scholar 

  34. Krause A, Cevher V. Submodular dictionary selection for sparse representation. In: International conference on machine learning (Haifa, Israel). 2010. p. 567–74.

  35. La C, Do MN. Signal reconstruction using sparse tree representation. In Proceedings of Wavelets XI at SPIE Optics and Photonics, San Diego. 2005.

  36. Labusch K, Barth E, Martinetz T. Sparse coding neural gas: learning of overcomplete data representations. Neurocomputing. 2009;72:1547–55.

    Article  Google Scholar 

  37. Lacoste-Julien S, Sha F, Michael IJ. DiscLDA: Discriminative learning for dimensionality reduction and classification. In: Advances in neural information processing systems 22. 2008.

  38. Lee H, Battle A, Raina R, Ng AY. Efficient sparse coding algorithms. In: Advances in neural information processing systems (NIPS) 19. Cambridge, MA; 2006. p. 801–8.

  39. Mairal J, Bach F, Ponce J, Sapiro G. Online dictionary learning for sparse coding. In: International conference on machine learning. 2009.

  40. Mallat S. A wavelet tour of signal processing. The sparse way. 3rd ed. New York: Academic Press; 2008.

    Google Scholar 

  41. Meinshausen N. Relaxed lasso. Comput Stat Data Anal. 2007;52:374–93.

    Article  Google Scholar 

  42. Olshausen BA, Field DJ. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature. 1996;381(6583):607–9.

    Article  CAS  PubMed  Google Scholar 

  43. Palm G. Neural associative memories and sparse coding. Neural Netw. 2013;37:165–71.

    Article  PubMed  Google Scholar 

  44. Paschou P, Ziv E, et al. PCA-correlated SNPs for structure identification in worldwide human populations. PLoS Genet. 2007;3(9):1–15.

    Article  Google Scholar 

  45. Peng H, Long F, Ding C. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell. 2005;27(8):1226–38.

    Article  PubMed  Google Scholar 

  46. Ruping S, Morik K. Support vector machines and learning about time. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 4. 2003. p. 864–7.

  47. Shaw B, Jebara T. Structure preserving embedding. In: The 26th annual international conference on machine learning, ICML. 2009. p. 937–44.

  48. Siddiqui S, Robila S, Peng J, Wang D. Sparse representations for hyperspectral data classification. In: IEEE international geoscience and remote sensing symposium, vol. 2. 2008. p. 577–80.

  49. Su L, Wang L, Chen F, Shen H, Li B, Hu D. Sparse representation of brain aging: extracting covariance patterns from structural mri. PLoS One. 2012;7(5):e6147.

    Google Scholar 

  50. Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc Ser B. 1996;58:267–88.

    Google Scholar 

  51. Turk MA, Pentland AP. Face recognition using eigenfaces. In: IEEE conference on CVPR. June 1991. p. 586–91.

  52. Wang C, Yan S, Zhang L, Zhang HJ. Multi-label sparse coding for automatic image annotation. In: IEEE conference on computer vision and pattern recognition (CVPR). 2009. p. 1643–50.

  53. Wang J, Wang L. Sparse supervised dimension reduction in high dimensional classification. Electron J Stat. 2010;4:914–31.

    Article  Google Scholar 

  54. Weinberger KQ, Packer BD, Saul LK. Nonlinear dimensionality reduction by semidefinite programming and kernel matrix factorization. In: The 10th international workshop on artificial intelligence and statistics. 2005. p. 381–8.

  55. Witten IH, Frank E. Data mining: practical machine learning tools and techniques with Java implementations. San Francisco, CA: Morgan Kaufmann Publishers Inc.; 2000.

  56. Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y. Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell. 2009;31(2):210–27.

    Article  PubMed  Google Scholar 

  57. Xu J, He H, Man H. Dcpe co-training for classification. Neurocomputing. 2012;86:75–85.

    Article  Google Scholar 

  58. Xu J, Man H. Dictionary learning based on laplacian score in sparse coding. In: Lecture notes in computer science, MLDM, vol. 6871. Springer; 2011. p. 253–64.

  59. Xu J, Yang G, Man H. Sparse representation for classification with structure preserving dimension reduction. In: The 28th international conference on machine learning (ICML) workshop, (Bellevue, WA, USA). 2011.

  60. Xu J, Yang G, Man H, He H. L1 graph based on sparse coding for feature selection. In: International symposium on neural networks (ISNN 2013). 2013. p. 594–601.

  61. Xu J, Yin Y, Man H, He H. Feature selection based on sparse imputation. In: The international joint conference on neural networks (IJCNN). 2012. p. 1–7.

  62. Yang J, Yu K, Gong Y, Huang T. Linear spatial pyramid matching using sparse coding for image classification. In: IEEE Conference on CVPR. 2009. p. 1794–801.

  63. Yang M, Zhang L. Gabor feature based sparse representation for face recognition with gabor occlusion dictionary. In: Computer Vision–ECCV 2010. Berlin, Heidelberg: Springer; 2010. p. 448–61.

  64. Zeng X, Luo S, Li Q. An associative sparse coding neural network and applications. Neurocomputing. 2010;73:684–9.

    Article  Google Scholar 

  65. Zhang L, Yang M, Feng Z, Zhang D. On the dimensionality reduction for sparse representation based face recognition. In: The 20th international conference on pattern recognition (ICPR). 2010. p. 1237–40.

  66. Zhang L, Zhu P, Hu Q, Zhang D. A linear subspace learning approach via sparse coding. In: IEEE international conference on computer vision. 2011. p. 755–61.

  67. Zheng C, Huang DS, Shang L. Feature selection in independent component subspace for microarray data classification. Neurocomputing. 2006;69(16–18):2407–10.

    Article  Google Scholar 

  68. Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc B. 2005;67(4):301–20.

    Article  Google Scholar 

  69. Zou H, Hastie T, Tibshirani R. Sparse principal component analysis. J Comput Graph Stat. 2004;15:265–86.

    Article  Google Scholar 

  70. Zylberberg J, Murphy JT, DeWeese MR. A sparse coding model with synaptically local plasticity and spiking neurons can account for the diverse shapes of v1 simple cell receptive fields. PLoS Comput Biol. 2011;7(10).

Download references

Acknowledgments

This work was supported in part by the National Science Foundation under grant ECCS 1053717, Army Research Office under grant W911NF-12-1-0378, NSF-DFG Collaborative Research on “Autonomous Learning,” a supplement grant to CNS 1117314, and Defense Advanced Research Projects Agency (DARPA) under grant FA8650-11-1-7152 and FA8650-11-1-7148.

Author information

Authors and Affiliations

  1. Operations and Information Management Department, University of Pennsylvania, Philadelphia, PA, 19104, USA

    Jin Xu

  2. Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ, 07030, USA

    Guang Yang, Yafeng Yin & Hong Man

  3. Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, RI, 02881, USA

    Haibo He

Authors
  1. Jin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yafeng Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hong Man
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haibo He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haibo He.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Xu, J., Yang, G., Yin, Y. et al. Sparse-Representation-Based Classification with Structure-Preserving Dimension Reduction. Cogn Comput 6, 608–621 (2014). https://doi.org/10.1007/s12559-014-9252-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-014-9252-5

Keywords

Search

Navigation