Skip to main content
SpringerLink
Log in

DMRAE: discriminative manifold regularized auto-encoder for sparse and robust feature learning

  • Regular Paper
  • Published:
Progress in Artificial Intelligence Aims and scope Submit manuscript

Abstract

Although the regularized over-complete auto-encoders have shown great ability to extract meaningful representation from data and reveal the underlying manifold of them, their unsupervised learning nature prevents the consideration of class distinction in the representations. The present study aimed to learn sparse, robust, and discriminative features through supervised manifold regularized auto-encoders by preserving locality on the manifold directions around each data and enhancing between-class discrimination. The combination of triplet loss manifold regularization with a novel denoising regularizer is injected to the objective function to generate features which are robust against perpendicular perturbation around data manifold and are sensitive enough to variation along the manifold. Also, the sparsity ratio of the obtained representation is adaptive based on the data distribution. The experimental results on 12 real-world classification problems show that the proposed method has better classification performance in comparison with several recently proposed relevant models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Bengio, Y., et al.: “Learning deep architectures for ai,” Foundations and trends®. Mach. Learn. 2(1), 1–127 (2009)

    MATH  Google Scholar 

  2. Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)

    Article  Google Scholar 

  3. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    MathSciNet  MATH  Google Scholar 

  4. Alain, G., Bengio, Y.: What regularized auto-encoders learn from the data-generating distribution. J. Mach. Learn. Res. 15(1), 3563–3593 (2014)

    MathSciNet  MATH  Google Scholar 

  5. Huang, R., Liu, C., Li, G., Zhou, J.: Adaptive deep supervised autoencoder based image reconstruction for face recognition. Math. Probl. Eng. 2016, (2016)

  6. Chechik, G., Sharma, V., Shalit, U., Bengio, S.: Large scale online learning of image similarity through ranking. J. Mach. Learn. Res. 11(Mar), 1109–1135 (2010)

    MathSciNet  MATH  Google Scholar 

  7. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103. ACM (2008)

  8. Bourlard, H., Kamp, Y.: Auto-association by multilayer perceptrons and singular value decomposition. Biol. Cybern. 59(4–5), 291–294 (1988)

    MathSciNet  MATH  Google Scholar 

  9. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323(6088), 533 (1986)

    MATH  Google Scholar 

  10. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification and Scene Analysis. Wiley, New York (1973)

    MATH  Google Scholar 

  11. Charte, D., Charte, F., García, S., del Jesus, M.J., Herrera, F.: A practical tutorial on autoencoders for nonlinear feature fusion: taxonomy, models, software and guidelines. Information Fusion 44, 78–96 (2018)

    Google Scholar 

  12. Rifai, S., Vincent, P., Muller, X., Glorot, X., Bengio, Y.: Contractive auto-encoders: explicit invariance during feature extraction. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 833–840 (2011)

  13. Wang, Y., Yao, H., Zhao, S.: Auto-encoder based dimensionality reduction. Neurocomputing 184, 232–242 (2016)

    Google Scholar 

  14. Japkowicz, N., Hanson, S.J., Gluck, M.A.: Nonlinear autoassociation is not equivalent to pca. Neural Comput. 12(3), 531–545 (2000)

    Google Scholar 

  15. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A.: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)

    MathSciNet  MATH  Google Scholar 

  16. Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., Glorot, X.: Higher order contractive auto-encoder. Mach. Learn. Knowl. Discov. Databases 15, 645–660 (2011)

    Google Scholar 

  17. Lee, H., Ekanadham, C., Ng, A.Y.: Sparse deep belief net model for visual area v2. In: Advances in Neural Information Processing Systems, pp. 873–880 (2008)

  18. Wright, J., Ma, Y., Mairal, J., Sapiro, G., Huang, T. S., Yan, S.: Sparse representation for computer vision and pattern recognition. In: Proceedings of the IEEE, vol. 98, pp. 1031–1044 (2010)

  19. Su, S.-Z., Liu, Z.-H., Xu, S.-P., Li, S.-Z., Ji, R.: Sparse auto-encoder based feature learning for human body detection in depth image. Signal Process. 112, 43–52 (2015)

    Google Scholar 

  20. Boureau, Y.-l., Cun, Y. L. et al.: Sparse feature learning for deep belief networks. In: Advances in Neural Information Processing Systems, pp. 1185–1192 (2008)

  21. Makhzani, A., Frey, B.: K-sparse autoencoders, arXiv preprint arXiv:1312.5663 (2013)

  22. Makhzani, A., Frey, B. J.: Winner-take-all autoencoders. In: Advances in Neural Information Processing Systems, pp. 2791–2799 (2015)

  23. Qi, Y., Wang, Y., Zheng, X., Wu, Z.: Robust feature learning by stacked autoencoder with maximum correntropy criterion. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6716–6720. IEEE (2014)

  24. Liu, W., Pokharel, P. P., Principe, J. C.: Correntropy: A localized similarity measure. In: The 2006 IEEE International Joint Conference on Neural Network Proceedings, pp. 4919–4924. IEEE (2006)

  25. Jia, K., Sun, L., Gao, S., Song, Z., Shi, B.E.: Laplacian auto-encoders: an explicit learning of nonlinear data manifold. Neurocomputing 160, 250–260 (2015)

    Google Scholar 

  26. Liu, W., Ma, T., Tao, D., You, J.: Hsae: a Hessian regularized sparse auto-encoders. Neurocomputing 187, 59–65 (2016)

    Google Scholar 

  27. Shi, Y., Lei, M., Ma, R., Niu, L.: Learning robust auto-encoders with regularizer for linearity and sparsity. IEEE Access 7, 17195–17206 (2019)

    Google Scholar 

  28. Gao, S., Zhang, Y., Jia, K., Lu, J., Zhang, Y.: Single sample face recognition via learning deep supervised autoencoders. IEEE Trans. Inf. Forensics Sec. 10(10), 2108–2118 (2015)

    Google Scholar 

  29. Xie, J., Fang, Y., Zhu, F., Wong, E.: Deepshape: deep learned shape descriptor for 3d shape matching and retrieval. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1275–1283. IEEE (2015)

  30. Liu, W., Ma, T., Xie, Q., Tao, D., Cheng, J.: Lmae: a large margin auto-encoders for classification. Signal Process. 141, 137–143 (2017)

    Google Scholar 

  31. Hu, C., Wu, X.-J., Shu, Z.-Q.: Discriminative feature learning via sparse autoencoders with label consistency constraints. Neural Process. Lett. 50, 1–13 (2018)

    Google Scholar 

  32. Du, F., Zhang, J., Ji, N., Hu, J., Zhang, C.: Discriminative representation learning with supervised auto-encoder. Neural Process. Lett. 49(2), 507–520 (2019)

    Google Scholar 

  33. Rifai, S., Dauphin, Y., Vincent, P., Bengio, Y., Muller, X.: The manifold tangent classifier. In: NIPS, vol. 271, p. 523 (2011)

  34. Arpit, D., Zhou, Y., Ngo, H., Govindaraju, V.: Why regularized auto-encoders learn sparse representation?, arXiv preprint arXiv:1505.05561 (2015)

  35. Coates, A., Lee, H., Ng, A.Y.: An analysis of single-layer networks in unsupervised feature learning. Ann Arbor 1001(48109), 2 (2010)

    Google Scholar 

  36. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249-0256 (2010)

  37. Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010, pp. 177–186. Springer (2010)

  38. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Google Scholar 

  39. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto (2009)

  40. Pulgar, F.J., Charte, F., Rivera, A.J., del Jesus, M.J.: Choosing the proper autoencoder for feature fusion based on data complexity and classifiers: analysis, tips and guidelines. Inf. Fusion 54, 44–60 (2020)

    Google Scholar 

  41. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, arXiv preprint arXiv:1708.07747 (2017)

  42. Dua, D., Graff, C.: Uci machine learning repository (2020). https://archive.ics.uci.edu/ml

  43. Jain, A.N., Dietterich, T.G., Lathrop, R.H., Chapman, D., Critchlow, R.E., Bauer, B.E., Webster, T.A., Lozano-Perez, T.: Compass: a shape-based machine learning tool for drug design. J. Comput. Aided Mol. Des. 8(6), 635–652 (1994)

    Google Scholar 

  44. Buscema, M.: Metanet*: the theory of independent judges. Subst. Use Misuse 33(2), 439–461 (1998)

    Google Scholar 

  45. Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H., et al.: Greedy layer-wise training of deep networks. Adv. Neural Inf. Process. Syst. 19, 153 (2007)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Faculty of Computer and Electrical Engineering, University of Kashan, Kashan, Iran

    Nima Farajian

  2. Artificial Intelligence Department, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran

    Peyman Adibi

Authors
  1. Nima Farajian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peyman Adibi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peyman Adibi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Farajian, N., Adibi, P. DMRAE: discriminative manifold regularized auto-encoder for sparse and robust feature learning. Prog Artif Intell 9, 263–274 (2020). https://doi.org/10.1007/s13748-020-00211-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13748-020-00211-5

Keywords

Search

Navigation