Skip to main content
SpringerLink
Log in

Analysis of Deep Learning Methods in Adaptation to the Small Data Problem Solving

  • Conference paper
  • First Online:
Lecture Notes in Data Engineering, Computational Intelligence, and Decision Making (ISDMCI 2022)

Part of the book series: Lecture Notes on Data Engineering and Communications Technologies ((LNDECT,volume 149))

Abstract

This paper discusses a specific problem in the study of deep neural networks - learning on small data. Such issue happens in situation of transfer learning or applying known solutions on new tasks that involves usage of particular small portions of data. Based on previous research, some specific solutions can be applied to various tasks related to machine learning, computer vision, natural language processing, medical data study and many others. These solutions include various methods of general purpose machine and deep learning, being successfully used for these tasks. In order to do so, the paper carefully studies the problems arise in the preparation of data. For benchmark purposes, we also compared “in wild” the methods of machine learning and identified some issues in their practical application, in particular usage of specific hardware. The paper touches some other aspects of machine learning by comparing the similarities and differences of singular value decomposition and deep constrained auto-encoders. In order to test our hypotheses, we carefully studied various deep and machine learning methods on small data. As a result of the study, our paper proposes a set of solutions, which include the selection of appropriate algorithms, data preparation methods, hardware optimized for machine learning, discussion of their practical effectiveness and further improvement of approaches and methods described in the paper. Also, some problems were discussed, which have to be addressed in the following papers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Albtoush, A., Fernández-Delgado, M., Cernadas, E., Barro, S.: Quick extreme learning machine for large-scale classification. Neural Comput. Appl. 34(8), 5923–5938 (2021). https://doi.org/10.1007/s00521-021-06727-8

    Article  Google Scholar 

  2. Aloysius, N., Geetha, M.: A review on deep convolutional neural networks. In: 2017 International Conference on Communication and Signal Processing (ICCSP), pp. 588–592. IEEE (2017). https://doi.org/10.1109/iccsp.2017.8286426

  3. Alzubaidi, L., et al.: Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. J. Big Data 8(1), 1–74 (2021). https://doi.org/10.1186/s40537-021-00444-8

  4. Babichev, S., Durnyak, B., Zhydetskyy, V., Pikh, I., Senkivskyy, V.: Application of optics density-based clustering algorithm using inductive methods of complex system analysis. In: 2019 IEEE 14th International Conference on Computer Sciences and Information Technologies (CSIT), vol. 1, pp. 169–172 (2019). https://doi.org/10.1109/STC-CSIT.2019.8929869

  5. Chan, D., Rao, R., Huang, F., Canny, J.: T-SNE-CUDA: GPU-accelerated T-SNE and its applications to modern data. In: 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pp. 330–338. IEEE (2018). https://doi.org/10.1109/cahpc.2018.8645912

  6. Chawla, N., Bowyer, K., Hall, L., Kegelmeyer, W.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. (JAIR) 16, 321–357 (2002). https://doi.org/10.1613/jair.953

    Article  MATH  Google Scholar 

  7. Dongarra, J., Gates, M., Haidar, A., et al.: The singular value decomposition: anatomy of optimizing an algorithm for extreme scale. SIAM Rev. 60(4), 808–865 (2018). https://doi.org/10.1137/17m1117732

    Article  MathSciNet  MATH  Google Scholar 

  8. Hast, A., Vast, E.: Word recognition using embedded prototype subspace classifiers on a new imbalanced dataset. J. WSCG 29(1–2), 39–47 (2021). https://doi.org/10.24132/jwscg.2021.29.5

  9. He, H., Bai, Y., Garcia, E., Li, S.: ADASYN: adaptive synthetic sampling approach for imbalanced learning. In: 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pp. 1322–1328. IEEE (2008). https://doi.org/10.1109/ijcnn.2008.4633969

  10. Huang, G., Chen, D., Li, T., Wu, F., van der Maaten, L., Weinberger, K.Q.: Multi-Scale Dense Networks for Resource Efficient Image Classification (2017). https://doi.org/10.48550/arXiv.1703.09844

  11. Izonin, I., Tkachenko, R., Gregus, M., Duriagina, Z., Shakhovska, N.: PNN-SVM approach of Ti-based powder’s properties evaluation for biomedical implants production. Comput. Mater. Continua 71(3), 5933–5947 (2022). https://doi.org/10.32604/cmc.2022.022582

  12. Izonin, I., Tkachenko, R., Shakhovska, N., Lotoshynska, N.: The additive input-doubling method based on the SVR with nonlinear Kernels: small data approach. Symmetry 13(4), 1–18 (2021). https://doi.org/10.3390/sym13040612

    Article  Google Scholar 

  13. Jiang, M., et al.: Text classification based on deep belief network and softmax regression. Neural Comput. Appl. 29(1), 61–70 (2016). https://doi.org/10.1007/s00521-016-2401-x

  14. Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artif. Intell. Rev. 53(8), 5455–5516 (2020). https://doi.org/10.1007/s10462-020-09825-6

    Article  Google Scholar 

  15. Krak, I., Barmak, O., Manziuk, E.: Using visual analytics to develop human and machine-centric models: a review of approaches and proposed information technology. Comput. Intell. 1–26 (2020). https://doi.org/10.1111/coin.12289

  16. Krak, Y., Barmak, A., Baraban, E.: Usage of NURBS-approximation for construction of spatial model of human face. J. Autom. Inf. Sci. 43(2), 71–81 (2011). https://doi.org/10.1615/jautomatinfscien.v43.i2.70

    Article  Google Scholar 

  17. Krivonos, Y.G., Krak, Y., Barchukova, Y., Trotsenko, B.: Human hand motion parametrization for Dactilemes modeling. J. Autom. Inf. Sci. 43(12), 1–11 (2011). https://doi.org/10.1615/JAutomatInfScien.v43.i12.10

    Article  Google Scholar 

  18. Kryvonos, I., Krak, I.: Modeling human hand movements, facial expressions, and articulation to synthesize and visualize gesture information. Cybern. Syst. Anal. 47(4), 501–505 (2011). https://doi.org/10.1007/s10559-011-9332-4

    Article  Google Scholar 

  19. Kryvonos, I.G., Krak, I.V., Barmak, O.V., Ternov, A.S., Kuznetsov, V.O.: Information technology for the analysis of mimic expressions of human emotional states. Cybern. Syst. Anal. 51(1), 25–33 (2015). https://doi.org/10.1007/s10559-015-9693-1

    Article  Google Scholar 

  20. Lytvynenko, V., Lurie, I., Krejcí, J., Voronenko, M., Savina, N., Ali Taif, M.: Two step density-based object-inductive clustering algorithm. In: Workshop Proceedings of the 8th International Conference on “Mathematics. Information Technologies. Education” (MoMLeT and DS-2019), vol. 2386, pp. 1–19. CEUR-WS, Shatsk, Ukraine (2019). http://ceur-ws.org/Vol-2386/paper10.pdf

  21. Lytvynenko, V., Savina, N., Krejcí, J., Voronenko, M., Yakobchuk, M., Kryvoruchko, O.: Bayesian networks’ development based on noisy-MAX nodes for modeling investment processes in transport. In: Workshop Proceedings of the 8th International Conference on “Mathematics. Information Technologies. Education" (MoMLeT and DS-2019), vol. 2386, pp. 1–10. CEUR-WS, Shatsk, Ukraine (2019). http://ceur-ws.org/Vol-2386/paper1.pdf

  22. Menardi, G., Torelli, N.: Training and assessing classification rules with imbalanced data. Data Min. Knowl. Disc. 28(1), 92–122 (2012). https://doi.org/10.1007/s10618-012-0295-5

    Article  MathSciNet  MATH  Google Scholar 

  23. Python: An open-source programming language, environment and interpreter (2022). https://www.python.org/about/

  24. Romanuke, V.: An attempt of finding an appropriate number of convolutional layers in CNNs based on benchmarks of heterogeneous datasets. Electr. Control. Commun. Eng. 14(1), 51–57 (2018). https://doi.org/10.2478/ecce-2018-0006

    Article  Google Scholar 

  25. Sultana, F., Sufian, A., Dutta, P.: Advancements in image classification using convolutional neural network. In: 2018 Fourth International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN), pp. 122–129. IEEE (2018). https://doi.org/10.1109/icrcicn.2018.8718718

  26. TensorFlow: A system for large-scale machine learning (2022). https://www.tensorflow.org/about/

  27. TensorFlow-DirectML: Github repository for tensorflow fork accelerated by directml (2022). https://github.com/microsoft/tensorflow-directml

  28. Vahdat, A., Kautz, J.: Nvae: A deep hierarchical variational autoencoder (2020). 1048550/arXiv. 2007.03898

    Google Scholar 

  29. Wiatowski, T., Bolcskei, H.: A mathematical theory of deep convolutional neural networks for feature extraction. IEEE Trans. Inf. Theory 64(3), 1845–1866 (2018). https://doi.org/10.1109/tit.2017.2776228

    Article  MathSciNet  MATH  Google Scholar 

  30. Yona, G., Moran, S., Elidan, G., Globerson, A.: Active Learning with Label Comparisons (2022). https://doi.org/10.48550/ARXIV.2204.04670

  31. Zebari, R., Abdulazeez, A., Zeebaree, D., et al.: A comprehensive review of dimensionality reduction techniques for feature selection and feature extraction. J. Appl. Sci. Technol. Trends 1(2), 56–70 (2020). https://doi.org/10.38094/jastt1224

  32. Zhang, G., Chen, Y.: More informed random sample consensus. arXiv (2020). https://doi.org/10.48550/ARXIV.2011.09116

Download references

Author information

Authors and Affiliations

  1. Taras Shevchenko National University of Kyiv, Kyiv, Ukraine

    Iurii Krak, Vladyslav Kuznetsov & Serhii Kondratiuk

  2. Glushkov Institute of Cybernetics of NAS of Ukraine, Kyiv, Ukraine

    Iurii Krak & Serhii Kondratiuk

  3. Vinnytsia National Technical University, Vinnytsia, Ukraine

    Larisa Azarova

  4. Khmelnytskyi National University, Khmelnytskyi, Ukraine

    Olexander Barmak & Pavlo Padiuk

Authors
  1. Iurii Krak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vladyslav Kuznetsov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Serhii Kondratiuk
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Larisa Azarova
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Olexander Barmak
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Pavlo Padiuk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Iurii Krak .

Editor information

Editors and Affiliations

  1. Jan Evangelista Purkyně University in Ústi nad Labem, Ústi nad Labem, Czech Republic

    Sergii Babichev

  2. Kherson National Technical University, Kherson, Ukraine

    Volodymyr Lytvynenko

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Krak, I., Kuznetsov, V., Kondratiuk, S., Azarova, L., Barmak, O., Padiuk, P. (2023). Analysis of Deep Learning Methods in Adaptation to the Small Data Problem Solving. In: Babichev, S., Lytvynenko, V. (eds) Lecture Notes in Data Engineering, Computational Intelligence, and Decision Making. ISDMCI 2022. Lecture Notes on Data Engineering and Communications Technologies, vol 149. Springer, Cham. https://doi.org/10.1007/978-3-031-16203-9_20

Download citation

Publish with us

Policies and ethics

Search

Navigation