Skip to main content

Advertisement

Springer Nature Link
Log in

Kernelized Linear Autoencoder

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

This work proposes a new representation learning model called kernelized linear autoencoder. Instead of modeling non-linearity by the non-linear activation functions, we employ linear activations but account for non-linearity by the kernel trick. We propose four variants. The first one is the basic unsupervised kernelized linear autoencoder. The second one is a label consistent version for classification. The third one incorporates sparse sub-space clustering into the autoencoder formulation. Finally, we proposed a coupled version for domain adaptation problems. For each task, we compare our proposed technique with shallow and deep representation learning methods on benchmark problems. For each task, we perform significantly better than the methods compared against.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA (2010) Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11(Dec):3371–3408

    MathSciNet  MATH  Google Scholar 

  2. Bengio Y, Lamblin P, Popovici D, Larochelle H (2007) Greedy layer-wise training of deep networks. In: Advances in neural information processing systems, pp 153–160

  3. Majumdar A, Singh R, Vatsa M (2017) Face verification via class sparsity based supervised encoding. IEEE Trans Pattern Anal Mach Intell 39(6):1273–1280

    Article  Google Scholar 

  4. Chen L, Zhou M, Su W, Wu M, She J, Hirota K (2018) Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction. Inf Sci 428:49–61

    Article  MathSciNet  Google Scholar 

  5. Tian F, Gao B, Cui Q, Chen E, Liu TY (2014) Learning deep representations for graph clustering. AAAI, Menlo Park, pp 1293–1299

    Google Scholar 

  6. Peng X, Xiao S, Feng J, Yau WY, Yi Z (2016) Deep subspace clustering with sparsity prior. In: IJCAI, pp 1925–1931

  7. Chen Y, Zhang L, Yi Z (2018) Subspace clustering using a low-rank constrained autoencoder. Inf Sci 424:27–38

    Article  MathSciNet  Google Scholar 

  8. Agostinelli F, Anderson MR, Lee H (2013) Adaptive multi-column deep neural networks with application to robust image denoising. In: NIPS, pp 1493–1501.

  9. Xie J, Xu L, Chen E (2012) Image denoising and inpainting with deep neural networks. In: NIPS, pp 341–349

  10. Wang S, Ding Z, Fu Y (2016) Coupled marginalized auto-encoders for cross-domain multi-view learning. In: IJCAI, pp. 2125–2131

  11. Zeng K, Yu J, Wang R, Li C, Tao D (2017) Coupled deep autoencoder for single image super-resolution. IEEE Trans Cybern 47(1):27–37

    Article  Google Scholar 

  12. Gupta K, Biswas B, Majumdar A (2017) Motion blur removal via coupled autoencoder. In: IEEE ICIP, 2017

  13. Hornik K (1991) Approximation capabilities of multilayer feedforward networks. Neural networks 4(2):251–257

    Article  MathSciNet  Google Scholar 

  14. Grohs P, Perekrestenko D, Elbrächter D, Bölcskei H (2019) Deep neural network approximation theory. arXiv preprint arXiv:1901.02220

  15. Tan HH, Lim KH (2019) Vanishing gradient mitigation with deep learning neural network optimization. In: 2019 7th international conference on smart computing & communications (ICSCC), pp. 1–4. IEEE

  16. Kolbusz J, Rozycki P. Wilamowski BM (2017) The study of architecture MLP with linear neurons in order to eliminate the “vanishing gradient” problem. In: International conference on artificial intelligence and soft computing. Springer, Cham, pp 97–106

  17. Salman, S. and Liu, X., 2019. Overfitting mechanism and avoidance in deep neural networks. arXiv preprint arXiv:1901.06566

  18. Baudat, G. and Anouar, F., 2001, July. Kernel-based methods and function approximation. In IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No. 01CH37222) (Vol. 2, pp. 1244–1249). IEEE.

  19. Pei Y (2017) Autoencoder using kernel methoc. In: 2017 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 322–327

  20. Kampffmeyer M, Løkse S, Bianchi FM, Jenssen R,Livi L (2017) Deep kernelized autoencoders. In: Scandinavian conference on image analysis. Springer, Cham, pp 419–430

  21. Combettes PL, Pesquet JC (2011) Proximal splitting methods in signal processing. Fixed-point algorithms for inverse problems in science and engineering. Springer, New York, pp 185–212

  22. Yang M, Liu W, Luo W, Shen L (2016) Analysis-synthesis dictionary learning for universality-particularity representation based classification. AAAI, Menlo, pp 2251–2257

    Google Scholar 

  23. Gu S, Zhang L, Zuo W, Feng X (2014) Projective dictionary pair learning for pattern classification. In: NIPS, pp 793–801

  24. Van Nguyen H, Patel VM, Nasrabadi NM, Chellappa R (2012). Kernel dictionary learning. IEEE ICASSP, pp 2021–2024.

  25. Ravishankar S, Bresler Y (2013) Learning sparsifying transforms. IEEE Trans Signal Process 61(5):1072–1086

    Article  MathSciNet  Google Scholar 

  26. Maggu J, Majumdar A (2017) Kernel transform learning. Pattern Recogn Lett 98:117–122

    Article  Google Scholar 

  27. Wang, Y., Yin, W. and Zeng, J., 2015. Global convergence of ADMM in nonconvex nonsmooth optimization. arXiv preprint arXiv:1511.06324

  28. Jiang Z, Lin Z, Davis LS (2013) Label consistent K-SVD: learning a discriminative dictionary for recognition. IEEE Trans Pattern Anal Mach Intell 35(11):2651–2664

    Article  Google Scholar 

  29. Gogna A, Majumdar A, Ward R (2017) Semi-supervised stacked label consistent autoencoder for reconstruction and analysis of biomedical signals. IEEE Trans Biomed Eng 64(9):2196–2205

    Article  Google Scholar 

  30. Gogna A, Majumdar A (2019) Discriminative autoencoder for feature extraction: Application to character recognition. Neural Process Lett 49(3):1723–1735

    Article  Google Scholar 

  31. Hu C, Wu XJ, Shu ZQ (2019) Discriminative feature learning via sparse autoencoders with label consistency constraints. Neural Process Lett 50(2):1079–1091

    Article  Google Scholar 

  32. Dong X, Wu F, Jing XY, Wu S (2019) Multi-view intact discriminant space learning for image classification. Neural Process Lett 50(2):1661–1685

    Article  Google Scholar 

  33. Shan T, Jiang M (2019) Fisher Discriminative Coupled Dictionaries Learning. Neural Process Lett 50(3):2991–3008

    Article  Google Scholar 

  34. Elhamifar E, Vidal R (2013) Sparse subspace clustering: algorithm, theory, and applications. IEEE Trans Pattern Anal Mach Intell 35(11):2765–2781

    Article  Google Scholar 

  35. Liu G, Lin Z, Yan S, Sun J, Yu Y, Ma Y (2013) Robust recovery of subspace structures by low-rank representation. IEEE Trans Pattern Anal Mach Intell 35(1):171–184

    Article  Google Scholar 

  36. Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202

    Article  MathSciNet  Google Scholar 

  37. Wang S, Zhang L, Liang Y, Pan Q (2012). Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In: IEEE CVPR, pp 2216–2223

  38. Huang DA, Frank Wang YC (2013) Coupled dictionary and feature space learning with applications to cross-domain image synthesis and recognition. In: IEEE ICCV, pp 2496–2503

  39. Song M, Chen C, Bu J, Sha T (2012) Image-based facial sketch-to-photo synthesis via online coupled dictionary learning. Inf Sci 193:233–246

    Article  Google Scholar 

  40. Guo J, Guo Y, Kong X, Zhang M, He R (2016) Discriminative Analysis Dictionary Learning. AAAI, Menlo, pp 1617–1623

    Google Scholar 

  41. Chen Y, Su J (2017) Sparse embedded dictionary learning on face recognition. Pattern Recognit 64:51–59

    Article  Google Scholar 

  42. Hu J, Tan YP (2017) Nonlinear dictionary learning with application to image classification. Pattern Recognit 75:282–291

    Article  Google Scholar 

  43. Mehdipour Ghazi M, Kemal Ekenel H (2016) A comprehensive analysis of deep learning based representation for face recognition. In: IEEE conference on computer vision and pattern recognition workshops, pp 34–41

  44. Majumdar A, Singh R, Vatsa M (2016) Face verification via class sparsity based supervised encoding. IEEE Trans Pattern Anal Mach Intell 39(6):1273–1280

    Article  Google Scholar 

  45. Zhuang Y, Wang Y, Wu F, Zhang Y, Lu W (2013) Supervised coupled dictionary learning with group structures for multi-modal retrieval. AAAI, Menlo, pp 1070–1076

    Google Scholar 

  46. Rasiwasia N, Costa Pereira J, Coviello E, Doyle G, Lanckriet GR, Levy R, Vasconcelos N (2010) A new approach to cross-modal multimedia retrieval. In: ACM MM, pp 251–260

  47. Mehta J, Majumdar A (2017) RODEO: robust DE-aliasing autoencOder for real-time medical image reconstruction. Pattern Recogn 63:499–510

    Article  Google Scholar 

Download references

Acknowledgements

The author is partially supported by the Infosys Center for Artificial Intelligence at Indraprastha Institute of Information Technology.

Author information

Authors and Affiliations

  1. IIIT Delhi, Okhla Phase 3, New Delhi, India

    Angshul Majumdar

Authors
  1. Angshul Majumdar
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Angshul Majumdar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Majumdar, A. Kernelized Linear Autoencoder. Neural Process Lett 53, 1597–1614 (2021). https://doi.org/10.1007/s11063-021-10467-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-021-10467-0

Keywords