Skip to main content

Advertisement

Springer Nature Link
Log in

A review of various semi-supervised learning models with a deep learning and memory approach

  • Review Article
  • Published:
Iran Journal of Computer Science Aims and scope Submit manuscript

Abstract

Based on data types, four learning methods have been presented to extract patterns from data: supervised, semi-supervised, unsupervised, and reinforcement. Regarding machine learning, labeled data are very hard to access, although unlabeled data are usually collected and accessed easily. On the other hand, in most projects, most of the data are unlabeled but some data are labeled. Therefore, semi-supervised learning is more practical and useful for solving most of the problems. Different semi-supervised learning models have been introduced such as iterative learning (self-training), generative models, graph-based methods, and vector-based techniques. In addition, deep neural networks are used to extract data features using a multilayer model. Various models of this method have been presented to deal with semi-supervised data such as deep generative, virtual adversarial, and Ladder models. In semi-supervised learning, labeled data can contribute significantly to accurate pattern extraction. Thus, they can result in better convergence by having greater effects on models. The aim of this paper was to analyze the available models of semi-supervised learning with an approach to deep learning. A research solution for future studies is to benefit from memory to increase such an effect. Memory-based neural networks are new models of neural networks which can be used in this area.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Zhu, X.: Semi-Supervised Learning Literature Survey. Department of Computer Sciences University of Wisconsin, ICML, Madison (2008)

    Google Scholar 

  2. Tanha, J.: Ensemble Approaches to Semi-Supervised Learning. Library of the University of Amsterdam, Amsterdam (2013)

    Google Scholar 

  3. Wang, Y., Xu, X., Zhoa, H., Hua, Z.: Semi-supervised learning based on nearest neighbor rule and cut edges. Knowl. Based Syst. 23(6), 547–554 (2010)

    Article  Google Scholar 

  4. Sathya, R., Abraham, A.: Comparison of supervised and unsupervised learning algorithms for pattern classification. Int. J. Adv. Res. Artif. Intell. 2(2), 34–38 (2013)

    Article  Google Scholar 

  5. Alpaydin, E.: Introduction to Machine Learning. MIT Press, Cambridge (2004)

    MATH  Google Scholar 

  6. Seeger, M.: Learning with labeled and unlabeled data. Technical report, University of Edinburgh (2001)

  7. Bauer, E., Kohavi, R.: An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach. Learn. 36(1), 105–139 (1999)

    Article  Google Scholar 

  8. Rezende, D.J., Mohamed, S., Danihelka, I., Gregor, K., Wierstra, D.: One-shot generalization in deep generative models. In: Proceedings of 33rd International Conference on Machine Learning (2016)

  9. Nigam, K., Mccallum, A.K.: Text Classification from Labeled and Unlabeled Documents Using EM. Kluwer Academic Publishers, Boston (1998). (manufactured in The Netherlands)

    Book  MATH  Google Scholar 

  10. Adiwardana, D.D., Matsukawa, A., Whang, J.: Using generative models for semi-supervised learning. Stanford reports (2017)

  11. Basu, A., Watters, C., Shepherd, M.: Support vector machines for text categorization. In: Proceedings of the 36th Hawaii International Conference on System Sciences (2002)

  12. Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. Pattern Anal. Mach. Intell. IEEE Trans. 35(8), 1798–1828 (2013)

    Article  Google Scholar 

  13. Latouche, P., Rossi, F.: Graphs in machine learning: an introduction. In: European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium, 22–24 April 2015. arXiv:1506.06962v1

  14. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors

  15. Triguero, I., García, S., Herrera, F.: Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowl. Inf. Syst. 42(2), 245–284 (2015)

    Article  Google Scholar 

  16. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR. arXiv:abs/1512.03385 (2015)

  18. Zeiler, M.D.: Hierarchical Convolutional Deep Learning in Computer Vision. New York University, New York (2013)

    Google Scholar 

  19. Long, J., Shelhamer, E., Darrell, T. (eds.): Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)

  20. Hinton, G., Sejnovski, T.: Learning and Re-learning in Boltzmann Machines, PDP. MTI Press, Cambridge (1986)

    Google Scholar 

  21. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A. (eds.) Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning. ACM (2008)

  22. Simoncelli, E.P.: 4.7 statistical modeling of photographic images. In: Handbook of Video and Image Processing (2005)

  23. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXivpreprint arXiv:1207.0580 (2012)

  24. van den Oord, A., Kalchbrenner, N.: Pixel recurrent neural networks. In: International Conference on Machine Learning, New York, NY, USA, 2016. arXiv:1601.06759v3 (2016)

  25. Odena, A.: Semi-supervised learning with generative adversarial networks. In: Data Efficient Machine Learning Workshop, ICML 2016. arXiv:1606.01583v2 (2016)

  26. Valpola, H.: From neural PCA to deep unsupervised learning. In: Advances in Independent Component Analysis and Learning Machines, pp. 143–171. Elsevier. arXiv:1411.7783 (2015)

  27. Sadarangani, A., Jivani, A.: A survey of semi-supervised learning. Int. J. Eng. Sci. Res. Technol. (2016). https://doi.org/10.5281/zenodo.159333

    Google Scholar 

  28. Rasmus, A., Valpola, H., Honkala, M., Berglund, M., Raiko, T.: Semi-supervised learning with ladder networks, neural and evolutionary computing (2015). arXiv:1507.02672v2 [cs.NE]

  29. Maaløe, L., Sønderby, C.K., Sønderby, S.K., Winther, O.: Auxiliary deep generative models. In: Proceedings of the 33rd International Conference on Machine Learning (2016)

  30. Jensen, R., Shen, Q.: New approaches to fuzzy-rough feature selection. IEEE Trans. Fuzzy Syst. 17(4), 824–838 (2009)

    Article  Google Scholar 

  31. Arefiyan, F., Eftekhari, M., Shen, Q.: The 8th symposium on advances in science and technology (8thSASTech), 2013, Mashhad, Iran (2013)

  32. Gallistel, C.R., King, A.P.: Memory and the Computational Brain: Why Cognitive Science will Transform Neuroscience, vol. 3. Wiley, New York (2009)

    Book  Google Scholar 

  33. Graves, A., Mohamed, A., Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6645–6649. IEEE (2013)

  34. Das, S., Giles, C.L., Sun, G.-Z.: learning context-free grammars: capabilities and limitations of a recurrent neural network with an external stack memory. In: Proceedings of the Fourteenth Annual Conference of Cognitive Science Society. Indiana University (1992)

  35. Gers, F., Schraudolph, N., Schmidhuber, J.: Learning precise timing with LSTM recurrent networks. J. Mach. Learn. Res. 3, 115–143 (2002)

    MathSciNet  MATH  Google Scholar 

  36. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997

    Article  Google Scholar 

  37. Graves, A., Wayne, G., Danihelka, I.: Neural Turing machines. arXiv:1410.5401v2 [cs.NE] (2014)

  38. Santoro, A., Bartunov, S., Botvinick, M.: Meta-learning with memory-augmented neural networks. In: Proceedings of the 33rd international conference on machine learning, New York, NY, USA (2016)

  39. Jankowski, N., Duch, W., Grabczewski, K.: Meta-Learning in Computational Intelligence, vol. 358. Springer Science & Business Media, Berlin (2011)

    MATH  Google Scholar 

  40. Graves, A., Wayne, G., Reynolds, M.: Hybrid Computing Using a Neural Network with Dynamic External Memory. Publishers Limited, part of Springer Nature, Berlin (2016). https://doi.org/10.1038/nature2010

    Book  Google Scholar 

  41. Jensen, R., Shen, Q.: Computational Intelligence and Feature Selection: Rough and Fuzzy Approaches. IEEE Press and Wiley, New York (2008)

    Book  Google Scholar 

  42. Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)

    MATH  Google Scholar 

  43. Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51(2), 181–207 (2003)

    Article  MATH  Google Scholar 

  44. Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to Boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  45. Zhu, X., Ghahramani, Z., Lafferty, J.: Semi-supervised learning using Gaussian fields and harmonic functions. In: The 20th International Conference on Machine Learning (ICML) (2003)

  46. Settles, B.: Closing the loop: fast, interactive semi-supervised annotation with queries on features and instances. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1467–1478. ACL (2011)

  47. Zhu, X., Lafferty, J., Ghahramani, Z.: Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions. In: ICML 2003 Workshop on The Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining (2003)

  48. Zhu, X., Ghahramani, Z.: Learning from labeled and unlabeled data with label propagation. Technical report CMU-CALD-02-107, Carnegie Mellon University (2002)

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Faculty of Electrical and Computer Engineering, Urmia University, Urmia, Iran

    Jamshid Bagherzadeh & Hasan Asil

Authors
  1. Jamshid Bagherzadeh
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Hasan Asil
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Hasan Asil.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bagherzadeh, J., Asil, H. A review of various semi-supervised learning models with a deep learning and memory approach. Iran J Comput Sci 2, 65–80 (2019). https://doi.org/10.1007/s42044-018-00027-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42044-018-00027-6

Keywords