Skip to main content
SpringerLink
Log in

Diversified feature representation via deep auto-encoder ensemble through multiple activation functions

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In this paper, we propose a novel Deep Auto-Encoders Ensemble model (DAEE) through assembling multiple deep network models with different activation functions. The hidden features obtained by our proposed model have better robustness in representation than traditional variants of auto-encoders because it aggregates the diversified feature representations from multiple activation sub-networks into a more robust uniform feature representations. In order to obtain such uniform feature representations, we set the weight of each individual auto-encoder sub-network by optimizing the cost function over multiple auto-encoder sub-networks in the proposed model. Therefore, decreases the influence of individual sub-networks with improper activations and increase those with appropriate activations, to ensure the final feature representation to keep more predominant and comprehensive feature information. Extensive experiments on benchmark computer vision datasets, including MNIST, COIL-20, CIFAR-10 and SVHN, demonstrate the superiority of our proposed method among state-of-the-art auto-encoder methods, such as sparse auto-encoders (SAE), denoising auto-encoders (DAE), stacked denoising auto-encoders (SDAE) and graph regularized auto-encoders (GAE).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Mm A, Mas A, Aml A, Rs B, Yu D, A noise robust convolutional neural network for image classification, Results in Engineering https://doi.org/10.1016/j.rineng.2021.100225.

  2. Weninger F, Watanabe S, Tachioka Y, Schuller B (2014) Deep recurrent de-noising auto-encoder and blind de-reverberation for reverberated speech recognition, In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2014, pp 4623–4627

  3. Shahana P, Omman B (2015) Evaluation of features on sentimental analysis. Procedia Computer Science 46:1585–1592

    Article  Google Scholar 

  4. Segal-Rozenhaimer M, Li A, Das K, Chirayath V (2020) Cloud detection algorithm for multi-modal satellite imagery using convolutional neural-networks (cnn). Remote Sensing of Environment 237:111446

    Article  Google Scholar 

  5. Chen C, Zhuo R, Ren J (2019) Gated recurrent neural network with sentimental relations for sentiment classification. Information Sciences 502:268–278

    Article  Google Scholar 

  6. Lin D, Xu G, Xu W, Wang Y, Sun X, Fu K (2020) Scrsr: An efficient recursive convolutional neural network for fast and accurate image super-resolution. Neurocomputing 398:399–407

    Article  Google Scholar 

  7. Garcia KD, de Sá CR, Poel M, Carvalho T, Mendes-Moreira J, Cardoso JM, de Carvalho AC, Kok JN (2021) An ensemble of autonomous auto-encoders for human activity recognition. Neurocomputing 439:271–280

    Article  Google Scholar 

  8. Shen X-J, Ni C, Wang L, Zha Z-J (2021) Sliker: Sparse loss induced kernel ensemble regression. Pattern Recognition 109:107587

    Article  Google Scholar 

  9. Shen X-J, Liu S-X, Bao B-K, Pan C-H, Zha Z-J, Fan J (2020) A generalized least-squares approach regularized with graph embedding for dimensionality reduction. Pattern Recognition 98:107023

    Article  Google Scholar 

  10. Zhang Y, Lu Z, Wang S (2021) Unsupervised feature selection via transformed auto-encoder. Knowledge-Based Systems 215:106748

    Article  Google Scholar 

  11. Yu J, Liu G (2021) Extracting and inserting knowledge into stacked denoising auto-encoders. Neural Networks 137:31–42

    Article  Google Scholar 

  12. Sun W, Shao S, Zhao R, Yan R, Zhang X, Chen X (2016) A sparse auto-encoder-based deep neural network approach for induction motor faults classification. Measurement 89:171–178

    Article  Google Scholar 

  13. Bilgili E, Göknar İC, Ucan ON (2005) Cellular neural network with trapezoidal activation function. International journal of circuit theory and applications 33(5):393–417

    Article  Google Scholar 

  14. Zhang C, Woodland PC (2016) Dnn speaker adaptation using parameterised sigmoid and relu hidden activation functions, In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2016, pp 5300–5304

  15. Zhang C, Woodland PC (2016) Dnn speaker adaptation using parameterised sigmoid and relu hidden activation functions, In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2016, pp 5300–5304

  16. Ward JH Jr (1963) Hierarchical grouping to optimize an objective function. Journal of the American statistical association 58(301):236–244

    Article  MathSciNet  Google Scholar 

  17. Han J, Moraga C (1995) The influence of the sigmoid function parameters on the speed of backpropagation learning, In: International workshop on artificial neural networks, Springer, 1995, pp 195–201

  18. Bergstra J, Desjardins G, Lamblin P, Bengio Y, Quadratic polynomials learn better image features, Technical report, 1337

  19. Clevert D-A, Unterthiner T, Hochreiter S, Fast and accurate deep network learning by exponential linear units (elus), arXiv:arXiv:1511.07289

  20. Scardapane S, Scarpiniti M, Comminiello D, Uncini A (2017) Learning activation functions from data using cubic spline interpolation, In: Italian Workshop on Neural Nets, Springer, 2017, pp 73–83

  21. Goodfellow I, Warde-Farley D, Mirza M, Courville A, Bengio Y (2013) Maxout networks, In: International conference on machine learning, PMLR, 2013, pp 1319–1327

  22. Chen J, Combinatorially generated piecewise activation functions, arXiv:arXiv:1605.05216

  23. Harmon M, Klabjan D, Activation ensembles for deep neural networks, arXiv:arXiv:1702.07790

  24. Wen J, Han N, Fang X, Fei L, Yan K, Zhan S (2018) Low-rank preserving projection via graph regularized reconstruction. IEEE Transactions on Cybernetics 49(4):1279–1291

    Article  Google Scholar 

  25. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278–2324

    Article  Google Scholar 

  26. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

  27. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

  28. Gubbi J, Buyya R, Marusic S, Palaniswami M (2013) Internet of things (iot): A vision, architectural elements, and future directions. Future generation computer systems 29(7):1645–1660

    Article  Google Scholar 

  29. Zhang Y, Zhang E, Chen W (2016) Deep neural network for halftone image classification based on sparse auto-encoder. Engineering Applications of Artificial Intelligence 50:245–255

    Article  Google Scholar 

  30. Vincent P, Larochelle H, Bengio Y, Manzagol P-A (2008) Extracting and composing robust features with denoising autoencoders, in: Proceedings of the 25th international conference on Machine learning, 2008, pp 1096–1103

  31. Majumdar A (2018) Blind denoising autoencoder. IEEE transactions on neural networks and learning systems 30(1):312–317

    Article  Google Scholar 

  32. Witten IH, Frank E (2002) Data mining: practical machine learning tools and techniques with java implementations. Acm Sigmod Record 31(1):76–77

    Article  Google Scholar 

  33. Badem H, Caliskan A, Basturk A, Yuksel ME (2016) Classification of human activity by using a stacked autoencoder in medical technologies national congress (TIPTEKNO). IEEE 2016:1–4

    Google Scholar 

  34. Zhang, Y Liu R, Zhang S, Zhu M (2013) Occlusion-robust face recognition using iterative stacked denoising autoencoder, in: International Conference on Neural Information Processing, Springer, 2013, pp 352–359

  35. Budiman, A Fanany MI, Basaruddin C (2014) Stacked denoising autoencoder for feature representation learning in pose-based action recognition, in: 2014 IEEE 3rd Global Conference on Consumer Electronics (GCCE), IEEE, 2014, pp 684–688

  36. Liao Y, Wang Y, Liu Y (2016) Graph regularized auto-encoders for image representation. IEEE Transactions on Image Processing 26(6):2839–2852

    Article  MathSciNet  Google Scholar 

  37. Roweis ST (2000) Saul LK. Nonlinear dimensionality reduction by locally linear embedding, science 290(5500):2323–2326

    Google Scholar 

  38. Lu J, Tan Y-P (2009) Regularized locality preserving projections and its extensions for face recognition, IEEE Transactions on Systems, Man, and Cybernetics. Part B (Cybernetics) 40(3):958–963

    MathSciNet  Google Scholar 

  39. Zhang X, Zhu Q, Jiang Z-Y, He Y, Xu Y (2018) A novel ensemble model using plsr integrated with multiple activation functions based elm: Applications to soft sensor development. Chemometrics and Intelligent Laboratory Systems 183:147–157

    Article  Google Scholar 

  40. Zhang X-H, Zhu Q-X, He Y-L, Xu Y (2018) A novel robust ensemble model integrated extreme learning machine with multi-activation functions for energy modeling and analysis: Application to petrochemical industry. Energy 162:593–602

    Article  Google Scholar 

  41. Lauly S, Larochelle H, Khapra MM, Ravindran B, Raykar V, Saha A, et al, An autoencoder approach to learning bilingual word representations, arXiv:arXiv:1402.1454

  42. Zhang L, Zhang Q, Zhang L, Tao D, Huang X, Du B (2015) Ensemble manifold regularized sparse low-rank approximation for multiview feature embedding. Pattern Recognition 48(10):3102–3112

    Article  Google Scholar 

  43. Wang M, Hua X-S, Hong R, Tang J, Qi G-J, Song Y (2009) Unified video annotation via multigraph learning. IEEE Transactions on Circuits and Systems for Video Technology 19(5):733–746

    Article  Google Scholar 

  44. LeCun Y, Bottou L, Orr GB, Müller K-R et al (1998) Neural networks: Tricks of the trade. Springer Lecture Notes in Computer Sciences 1524(5–50):6

    Google Scholar 

  45. Jia K, Sun L, Gao S, Song Z, Shi BE (2015) Laplacian auto-encoders: An explicit learning of nonlinear data manifold. Neurocomputing 160:250–260

    Article  Google Scholar 

Download references

Acknowledgements

This work was funded in part by the National Natural Science Foundation of China (No.61572240, 61772244).

Author information

Authors and Affiliations

  1. JingJiang College, JiangSu University, JiangSu, 212113, China

    Na Qiang

  2. School of Computer Science and Communication Engineering, JiangSu University, JiangSu, 212013, China

    Xiang-Jun Shen, Chang-Bin Huang, Shengli Wu & Ernest Domanaanmwi Ganaa

  3. School of Applied Science, Tamale Technical University, Box 3ER, Tamale, Ghana

    Timothy Apasiba Abeo

  4. School of Computer Science, Jiangsu University of Science and Technology, JiangSu, 212013, China

    Shu-Cheng Huang

Authors
  1. Na Qiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiang-Jun Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chang-Bin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shengli Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Timothy Apasiba Abeo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ernest Domanaanmwi Ganaa
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Shu-Cheng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiang-Jun Shen.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was funded in part by the National Natural Science Foundation of China (No. 61572240, 61806086)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qiang, N., Shen, XJ., Huang, CB. et al. Diversified feature representation via deep auto-encoder ensemble through multiple activation functions. Appl Intell 52, 10591–10603 (2022). https://doi.org/10.1007/s10489-021-03054-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-03054-2

Keywords

Search

Navigation