Skip to main content
SpringerLink
Log in

Synergy between traditional classification and classification based on negative features in deep convolutional neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In recent times, convolutional neural networks became an irreplaceable tool in many different machine learning applications, especially in image classification. On the other hand, new research about robustness and susceptibility of these models to different adversarial attacks has emerged. With the rise in usage and widespread adoption of these models, it is very important to make them suitable for critical applications. In our previous work, we experimented with a new type of learning applicable to all convolutional neural networks: classification based on missing (low-impact) features. In the case of partial inputs/image occlusion, we have shown that our new method creates models that are more robust and perform better when compared to traditional models of the same architecture. In this paper, we explore an interesting characteristic of our newly developed models in that while we see a general increase in validation accuracy, we also lose some important knowledge. We propose one solution to overcome this problem and validate our assumptions against CIFAR-10 image classification dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Aquino G, Rubio JDJ, Pacheco J, Gutierrez GJ, Ochoa G, Balcazar R, Cruz DR, Garcia E, Novoa JF, Zacarias A (2020) Novel nonlinear hypothesis for the delta parallel robot modeling. IEEE Access 8:46324–46334

    Article  Google Scholar 

  2. Ashfahani A, Pratama M, Lughofer E, Ong YS (2020) Devdan: deep evolving denoising autoencoder. Neurocomputing 390:297–314

    Article  Google Scholar 

  3. Assunção F, Lourenço N, Machado P, Ribeiro B (2019) Fast denser: efficient deep neuroevolution. In: European Conference on Genetic Programming, Springer, pp 197–212

  4. Bastani O, Ioannou Y, Lampropoulos L, Vytiniotis D, Nori A, Criminisi A (2016) Measuring neural net robustness with constraints. In: Advances in Neural Information Processing Systems, pp 2613–2621

  5. Becherer N, Pecarina J, Nykl S, Hopkinson K (2019) Improving optimization of convolutional neural networks through parameter fine-tuning. Neural Comput Appl 31(8):3469–3479

    Article  Google Scholar 

  6. Bojarski M, Del Testa D, Dworakowski D, Firner B, Flepp B, Goyal P, Jackel LD, Monfort M, Muller U, Zhang J et al (2016) End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316. https://doi.org/10.1109/ivs.2017.7995975

  7. Carlini N, Wagner D (2017) Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy (SP), IEEE, pp 39–57. https://doi.org/10.1109/sp.2017.49

  8. Chiang HS, Chen MY, Huang YJ (2019) Wavelet-based EEG processing for epilepsy detection using fuzzy entropy and associative petri net. IEEE Access 7:103255–103262

    Article  Google Scholar 

  9. Cohen G, Afshar S, Tapson J, van Schaik A (2017) Emnist: an extension of mnist to handwritten letters. arXiv preprint arXiv:1702.05373

  10. de Jesús Rubio J (2009) Sofmls: online self-organizing fuzzy modified least-squares network. IEEE Trans Fuzzy Syst 17(6):1296–1309

    Article  Google Scholar 

  11. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, IEEE, pp 248–255. https://doi.org/10.1109/cvprw.2009.5206848

  12. Elias I, Rubio JdJ, Cruz DR, Ochoa G, Novoa JF, Martinez DI, Muñiz S, Balcazar R, Garcia E, Juarez CF (2020) Hessian with mini-batches for electrical demand prediction. Appl Sci 10(6):2036

    Article  Google Scholar 

  13. Elsken T, Metzen JH, Hutter F (2018) Neural architecture search: a survey. arXiv preprint arXiv:1808.05377

  14. Enzweiler M, Eigenstetter A, Schiele B, Gavrila DM (2010) Multi-cue pedestrian classification with partial occlusion handling. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, pp 990–997

  15. FacebookAI: Torchvision. Computer software. Vers. 0.6.0 https://pytorch.org 2020

  16. Globerson A, Roweis S (2006) Nightmare at test time: robust learning by feature deletion. In: Proceedings of the 23rd International Conference on Machine learning, ACM, pp 353–360. https://doi.org/10.1145/1143844.1143889

  17. Goodfellow I, Shlens J, Szegedy C (2015) Explaining and harnessing adversarial examples. In: International conference on learning representations. arXiv:1412.6572

  18. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778

  19. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4700–4708

  20. Koch G, Zemel R, Salakhutdinov R (2015) Siamese neural networks for one-shot image recognition. In: ICML deep learning workshop, vol 2. Lille

  21. Krizhevsky A, Nair V, Hinton G (2014) The cifar-10 dataset. 55, online:http://www.cs.toronto.edu/kriz/cifar.html

  22. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp 1097–1105. https://doi.org/10.1145/3065386

  23. LeCun Y (1998) The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/

  24. Meng F, Qi Z, Tian Y, Niu L (2018) Pedestrian detection based on the privileged information. Neural Comput Appl 29(12):1485–1494

    Article  Google Scholar 

  25. Milošević N, Racković M (2019) Classification based on missing features in deep convolutional neural networks. Neural Netw World 221:234

    Google Scholar 

  26. Moosavi-Dezfooli SM, Fawzi A, Frossard P (2016) Deepfool: a simple and accurate method to fool deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2574–2582. https://doi.org/10.1109/cvpr.2016.282

  27. Papernot N, McDaniel P, Goodfellow I, Jha S, Celik ZB, Swami A (2017) Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp 506–519

  28. Paszke A, Gross S, Chintala S, Chanan G (2020) Pytorch. Computer software. Vers. 1.5.0 https://pytorch.org

  29. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in pytorch. NIPS 2017 Workshop Autodiff Submission. https://openreview.net/forum?id=BJJsrmfCZ

  30. Pecev P, Rackovic M (2017) LTR-MDTS structure: a structure for multiple dependent time series prediction. Comput Sci Inf Syst 14(2):467–490. https://doi.org/10.2298/CSIS150815004P

    Article  Google Scholar 

  31. Pecev P, Racković M, Ivković M (2016) A system for deductive prediction and analysis of movement of basketball referees. Multimed Tools Appl 75(23):16389–16416. https://doi.org/10.1007/s11042-015-2938-1

    Article  Google Scholar 

  32. Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R (2013)Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199. https://arxiv.org/abs/1312.6199

  33. Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J (2016) Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans Med Imaging 35(5):1299–1312

    Article  Google Scholar 

  34. Weinberger KQ, Saul LK (2009) Distance metric learning for large margin nearest neighbor classification. J Mach Learn Res 10(Feb):207–244

    MATH  Google Scholar 

  35. Weiss K, Khoshgoftaar TM, Wang D (2016) A survey of transfer learning. J Big Data 3(1):9

    Article  Google Scholar 

  36. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1492–1500

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics and Informatics, Faculty of Sciences, University of Novi Sad, Novi Sad, Serbia

    Nemanja Milošević & Miloš Racković

Authors
  1. Nemanja Milošević
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miloš Racković
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nemanja Milošević.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Milošević, N., Racković, M. Synergy between traditional classification and classification based on negative features in deep convolutional neural networks. Neural Comput & Applic 33, 7593–7602 (2021). https://doi.org/10.1007/s00521-020-05503-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05503-4

Keywords

Search

Navigation