Skip to main content

Advertisement

SpringerLink
Log in

Unsupervised representation learning based on the deep multi-view ensemble learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Deep networks have recently achieved great success in feature learning problem on various computer vision applications. Among different approaches in deep learning, unsupervised methods have attracted a lot of attention particularly to problems with limited training data. However, compared with supervised methods, unsupervised deep learning methods usually suffer from lower accuracy and higher computational time. To deal with these problems, we aim to restrict the number of connections between successive layers while enhancing discriminatory power using a data-driven approach. To this end, we propose a novel deep multi-view ensemble model. The structure of each layer is composed of an ensemble of encoders or decoders and mask operations. The multi-view ensemble of encoders or decoders enable the network to benefit from local complementary information and preserve local characteristics in final generated features, while mask operations determine the connections between successive layers. The experimental results on popular datasets indicate the effectiveness and validity of the method in clustering and classification tasks while the processing time is reduced.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Stacked Auto Encoder

  2. Convolutional Auto Encoder

  3. Convolutional Deep Belief Network

References

  1. Bengio Y (2013) Deep learning of representations: Looking forward, in Statistical Language and Speech Processing. Springer. p. 1-37

  2. Bengio Y, Courville A, Vincent P (2013) Representation learning: A review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828

    Article  Google Scholar 

  3. Reed S et al (2014) Learning to disentangle factors of variation with manifold interaction. In: Proceedings of the 31st International Conference on Machine Learning (ICML-14)

  4. Coates A, Ng AY, Lee H (2011) An analysis of single-layer networks in unsupervised feature learning. in International conference on artificial intelligence and statistics

  5. Bengio Y, Courville AC, Vincent P (2012) Unsupervised feature learning and deep learning: A review and new perspectives. CoRR, abs/1206.5538. 1

  6. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  7. Längkvist M, Karlsson L, Loutfi A (2014) A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recogn Lett 42:11–24

    Article  Google Scholar 

  8. Bengio Y (2009) Learning deep architectures for AI. Foundations and trends® in Machine Learning 2(1):1–127

    Article  MathSciNet  Google Scholar 

  9. Rifai S et al (2012) Disentangling factors of variation for facial expression recognition, in Computer Vision–ECCV 2012, Springer. p. 808-822

  10. Rifai S et al (2011) Contractive auto-encoders: Explicit invariance during feature extraction. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11)

  11. Zeiler MD (2013) Hierarchical convolutional deep learning in computer vision. Diss. New York University, New York

    Google Scholar 

  12. Chen G et al (2015) Combining unsupervised learning and discrimination for 3D action recognition. Signal Process 110:67–81

    Article  Google Scholar 

  13. Dosovitskiy A et al (2016) Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Trans Pattern Anal Mach Intell 38(9):1734–1747

    Article  Google Scholar 

  14. Kallenberg M et al (2016) Unsupervised deep learning applied to breast density segmentation and mammographic risk scoring. IEEE Trans Med Imaging 35(5):1322–1331

    Article  Google Scholar 

  15. Erhan D et al (2009) The difficulty of training deep architectures and the effect of unsupervised pre-training. in International Conference on artificial intelligence and statistics

  16. Erhan D et al (2010) Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research 11:625–660

    MathSciNet  MATH  Google Scholar 

  17. Raina R, Madhavan A, Ng AY (2009) Large-scale deep unsupervised learning using graphics processors. In: Proceedings of the 26th annual international conference on machine learning. ACM

  18. Le QV (2013) Building high-level features using large scale unsupervised learning. in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE

  19. Ouyang Y et al (2014) Autoencoder-based collaborative filtering. in International Conference on Neural Information Processing. Springer

  20. Ng A (2011) Sparse autoencoder. CS294A Lecture notes. 72

  21. Wu D, Sharma N, Blumenstein M (2017) Recent advances in video-based human action recognition using deep learning: A review. in Neural Networks (IJCNN), 2017 International Joint Conference on. IEEE

  22. Du Y, Wang W, Wang L (2015) Hierarchical recurrent neural network for skeleton based action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  23. Wang, L., et al. Temporal segment networks: Towards good practices for deep action recognition. in European Conference on Computer Vision. 2016. Springer

  24. LeCun Y et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  25. Krizhevsky, A. and G. Hinton, Learning multiple layers of features from tiny images. 2009

    Google Scholar 

  26. Fei-Fei L, Fergus R, Perona P (2007) Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput Vis Image Underst 106(1):59–70

    Article  Google Scholar 

  27. Griffin G, Holub A, Perona P, Caltech-256 object category dataset

  28. Zhang C et al (2017) An up-to-date comparison of state-of-the-art classification algorithms. Expert Syst Appl 82:128–150

    Article  Google Scholar 

  29. Zhang, C., et al., Multi-Imbalance: An open-source software for multi-class imbalance learning. Knowledge-Based Systems, 2019

    Google Scholar 

  30. Wang H et al (2019) A study of graph-based system for multi-view clustering. Knowl-Based Syst 163:1009–1019

    Article  Google Scholar 

  31. Zhang Y et al (2019) A multitask multiview clustering algorithm in heterogeneous situations based on LLE and LE. Knowl-Based Syst 163:776–786

    Article  Google Scholar 

  32. Qiu X et al (2014) Ensemble deep learning for regression and time series forecasting. in Computational Intelligence in Ensemble Learning (CIEL), 2014 IEEE Symposium on. IEEE

  33. Xiao Y et al (2018) A deep learning-based multi-model ensemble method for cancer prediction. Comput Methods Prog Biomed 153:1–9

    Article  Google Scholar 

  34. Suk H-I, Lee S-W, Shen D (2017) Deep ensemble learning of sparse regression models for brain disease diagnosis. Med Image Anal 37:101–113

    Article  Google Scholar 

  35. Yin Z et al (2017) Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Comput Methods Prog Biomed 140:93–110

    Article  Google Scholar 

  36. Zhang X-L, Wang D (2016) A deep ensemble learning method for monaural speech separation. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP) 24(5):967–977

    Article  MathSciNet  Google Scholar 

  37. Qi Z et al (2016) When ensemble learning meets deep learning: a new deep support vector machine for classification. Knowl-Based Syst 107:54–60

    Article  Google Scholar 

  38. Zhu Z et al (2016) Deep learning representation using autoencoder for 3D shape retrieval. Neurocomputing 204:41–50

    Article  Google Scholar 

  39. Liu M-Y, Breuel T, Kautz J (2017) Unsupervised image-to-image translation networks. Adv Neural Inf Proces Syst

  40. Du B et al (2017) Stacked convolutional denoising auto-encoders for feature representation. IEEE Transactions on Cybernetics 47(4):1017–1027

    Article  Google Scholar 

  41. Shen F et al (2018) Unsupervised deep hashing with similarity-adaptive and discrete optimization. IEEE Trans Pattern Anal Mach Intell

  42. Romero A, Gatta C, Camps-Valls G (2016) Unsupervised deep feature extraction for remote sensing image classification. IEEE Trans Geosci Remote Sens 54(3):1349–1362

    Article  Google Scholar 

  43. Yu Y et al (2018) An Unsupervised Convolutional Feature Fusion Network for Deep Representation of Remote Sensing Images. IEEE Geosci Remote Sens Lett 15(1):23–27

    Article  MathSciNet  Google Scholar 

  44. Zhang J, Yu J, Tao D (2018) Local deep-feature alignment for unsupervised dimension reduction. IEEE Trans Image Process 27(5):2420–2432

    Article  MathSciNet  Google Scholar 

  45. Orlando JI et al (2018) An ensemble deep learning based approach for red lesion detection in fundus images. Comput Methods Prog Biomed 153:115–127

    Article  Google Scholar 

  46. Suk H-I et al (2017) Deep ensemble learning of sparse regression models for brain disease diagnosis. Med Image Anal 37:101–113

    Article  Google Scholar 

  47. Du Y, Wang W, Wang L (2015) Hierarchical recurrent neural network for skeleton based action recognition. Proc IEEE Conf Comput Vis Pattern Recognit

  48. Araque O et al (2017) Enhancing deep learning sentiment analysis with ensemble techniques in social applications. Expert Syst Appl 77:236–246

    Article  Google Scholar 

  49. Bengio Y et al (2007) Greedy layer-wise training of deep networks. Adv Neural Inf Proces Syst 19:153

    Google Scholar 

  50. Schmidhuber J (2015) Deep learning in neural networks: An overview. Neural Netw 61:85–117

    Article  Google Scholar 

  51. Yang M, Kpalma K, Ronsin J (2008) A survey of shape feature extraction techniques. In-Tech

  52. Deb K, Jain H (2014) An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part I: Solving problems with box constraints. IEEE Trans Evolutionary Computation 18(4):577–601

    Article  Google Scholar 

  53. Sastry K, Goldberg DE, Kendall G (2014) Genetic algorithms, in Search methodologies, Springer. p. 93-117

  54. Michalewicz Z (2013) Genetic algorithms+ data structures= evolution programs. Springer Science & Business Media, Berlin

    MATH  Google Scholar 

  55. Abdi H, Williams LJ (2010) Principal component analysis. Wiley Interdisciplinary Reviews: Computational Statistics 2(4):433–459

    Article  Google Scholar 

  56. Rand WM (1971) Objective criteria for the evaluation of clustering methods. J Am Stat Assoc 66(336):846–850

    Article  Google Scholar 

  57. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  Google Scholar 

  58. Turchenko V, Chalmers E, Luczak A (2017) A deep convolutional auto-encoder with pooling-unpooling layers in caffe. arXiv preprint arXiv:1701.04949

  59. Oyallon E, Mallat S (2015) Deep roto-translation scattering for object classification. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  60. Lee H et al (2009) Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: Proceedings of the 26th annual international conference on machine learning, ACM

  61. Pu Y et al (2016) A deep generative deconvolutional image model. Artificial Intelligence and Statistics

  62. Pu Y et al (2016) Variational autoencoder for deep learning of images, labels and captions. Adv Neural Inf Proces Syst

  63. Liu Q, Mukhopadhyay S (2018) Unsupervised Learning using Pretrained CNN and Associative Memory Bank. In: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE

  64. He K et al (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37(9):1904–1916

    Article  Google Scholar 

  65. Zhao Q, Li Z (2018) Unsupervised Representation Learning with Laplacian Pyramid Auto-encoders. arXiv preprint arXiv:1801.05278

  66. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  67. Coates A, Ng AY (2011) Selecting receptive fields in deep networks. Adv Neural Inf Proces Syst

  68. Lin T-H, Kung H (2014) Stable and efficient representation learning with nonnegativity constraints. International Conference on Machine Learning

  69. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434

  70. Gastaldi X (2017) Shake-shake regularization. arXiv preprint arXiv:1705.07485

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran

    Maryam Koohzadi, Nasrollah Moghadam Charkari & Foad Ghaderi

Authors
  1. Maryam Koohzadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nasrollah Moghadam Charkari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Foad Ghaderi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nasrollah Moghadam Charkari.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Koohzadi, M., Charkari, N.M. & Ghaderi, F. Unsupervised representation learning based on the deep multi-view ensemble learning. Appl Intell 50, 562–581 (2020). https://doi.org/10.1007/s10489-019-01526-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-019-01526-0

Keywords

Search

Navigation