Skip to main content

Advertisement

SpringerLink
Log in

Supervision dropout: guidance learning in deep neural network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In deep neural networks, the generalization is a vital evaluation metric. As it contributes to avoid over-fitting, Dropout plays an important role in improving the generalization of deep neural networks. Without fully utilizing the training data and the real-time performance of the networks, traditional Dropout and its variants lack of specificity in the selection of inactivated neurons and the planning of dropout rates, resulting in a weaker performance in enhancing the generalization. Therefore, this paper offers an improved Dropout method. As both the training data and the real-time performance of networks can be quantified by the loss, the method uses the loss of the network prediction to guide the selection of inactivated neurons and the determination of dropout rates. The selection is performed by the genetic algorithm, while the results of the selection are used to plan the dropout rate. In essence, this approach encourages the subset of neurons with the higher loss to be trained so as to increase the robustness of neurons and thus improves the generalization of networks. The experimental results demonstrate that the proposed method achieves better generalization on MiniImageNet and Caltech-256 datasets. Compared with the backbone network, the accuracy improves from 66.56% to 72.95%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

Data and materials are available.

Code Availability

Code is available.

References

  1. Achille A, Soatto S (2018) Information dropout: learning optimal representations through noisy computation. IEEE Trans Pattern Anal Mach Intell 40 (12):2897–2905

    Article  Google Scholar 

  2. Ambati LS, El-Gayar O (2021) Human activity recognition: a comparison of machine learning approaches. J Midwest Assoc Inf Syst (JMWAIS) 2021 (1):49

    Google Scholar 

  3. Ba J, Frey B (2013) Adaptive dropout for training deep neural networks. Advances in neural information processing systems 26

  4. Baldi P, Sadowski PJ (2013) Understanding dropout. Advances in neural information processing systems 26

  5. Chattopadhay A, Sarkar A, Howlader P, Balasubramanian VN (2018) Grad-cam++: generalized gradient-based visual explanations for deep convolutional networks. In: 2018 IEEE winter conference on applications of computer vision (WACV), IEEE, pp 839–847

  6. Chen Y, Yi Z (2021) Adaptive sparse dropout: learning the certainty and uncertainty in deep neural networks. Neurocomputing 450:354–361

    Article  Google Scholar 

  7. El-Gayar OF, Ambati LS, Nawar N (2020) Wearables, artificial intelligence, and the future of healthcare, 104–129

  8. Fan X, Zhang S, Tanwisuth K, Qian X, Zhou M (2021) Contextual dropout: an efficient sample-dependent dropout module. arXiv:2103.04181

  9. Feng X, Gao X, Luo L (2021) X-sdd: a new benchmark for hot rolled steel strip surface defects detection. Symmetry 13(4):706

    Article  Google Scholar 

  10. Gal Y, Ghahramani Z (2016) Dropout as a bayesian approximation: representing model uncertainty in deep learning. In: International conference on machine learning, PMLR, pp 1050–1059

  11. Gao W, Zhou Z-H (2016) Dropout rademacher complexity of deep neural networks. Sci China Inf Sci 59(7):1–12

    Article  Google Scholar 

  12. Griffin G, Holub A, Perona P (2007) Caltech-256 object category dataset

  13. He Y, Song K, Meng Q, Yan Y (2019) An end-to-end steel surface defect detection approach via fusing multiple hierarchical features. IEEE Trans Instrum Meas 69(4):1493–1504

    Article  Google Scholar 

  14. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  15. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012) Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580

  16. Inoue H (2019) Multi-sample dropout for accelerated training and better generalization. arXiv:1905.09788

  17. Kamili A, Fatima I, Hassan M, Parah SA, Vijaya Kumar V, Ambati L (2020) Embedding information reversibly in medical images for e-health. J Intell Fuzzy Syst 39(6):8389–8398

    Article  Google Scholar 

  18. Khan N, Stavness I (2019) Sparseout: Controlling sparsity in deep networks. In: Canadian conference on artificial intelligence, Springer, pp 296–307

  19. Konovalenko I, Maruschak P, Brezinová J, Viňáš J, Brezina J (2020) Steel surface defect classification using deep residual neural network. Metals 10(6):846

    Article  Google Scholar 

  20. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25

  21. Lambert J, Sener O, Savarese S (2018) Deep learning under privileged information using heteroscedastic dropout. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8886–8895

  22. Li M, Xu G, Lai Q, Chen J (2022) A chaotic strategy-based quadratic opposition-based learning adaptive variable-speed whale optimization algorithm. Math Comput Simul 193:71–99

    Article  MathSciNet  MATH  Google Scholar 

  23. Lian Z, Jing X, Wang X, Huang H, Tan Y, Cui Y (2016) Dropconnect regularization method with sparsity constraint for neural networks. Chin J Electron 25(1):152–158

    Article  Google Scholar 

  24. Molchanov D, Ashukha A, Vetrov D (2017) Variational dropout sparsifies deep neural networks. In: International conference on machine learning, PMLR, pp 2498–2507

  25. Morerio P, Cavazza J, Volpi R, Vidal R, Murino V (2017) Curriculum dropout. In: Proceedings of the IEEE international conference on computer vision, pp 3544–3552

  26. Nagaraj B, Arunkumar R, Nisi K, Vijayakumar P (2020) Enhancement of fraternal k-median algorithm with cnn for high dropout probabilities to evolve optimal time-complexity. Clust Comput 23(3):2001–2008

    Article  Google Scholar 

  27. Ng ST, Skitmore M, Wong KF (2008) Using genetic algorithms and linear regression analysis for private housing demand forecast. Build Environ 43 (6):1171–1184

    Article  Google Scholar 

  28. Nguyen S, Nguyen D, Nguyen K, Ho N, Than K, Bui H (2021) Improving bayesian inference in deep neural networks with variational structured dropout. arXiv e-prints 2102

  29. Ou Y, Chen J, Chen W, Cheng C, Zhu Y, Xiao W, Lv H (2022) A quasi-distributed fiber magnetic field sensor based on frequency-shifted interferometry fiber cavity ringdown technique. Opt Laser Technol, 146. https://doi.org/10.1016/j.optlastec.2021.10760

  30. Rennie SJ, Goel V, Thomas S (2014) Annealed dropout training of deep networks. In: 2014 IEEE spoken language technology workshop (SLT), IEEE, pp 159–164

  31. Roccetti M, Delnevo G, Casini L, Cappiello G (2019) Is bigger always better? a controversial journey to the center of machine learning design, with uses and misuses of big data for predicting water meter failures. J Big Data 6(1):1–23

    Article  Google Scholar 

  32. Sai Ambati L, El-Gayar OF, Nawar N (2020) Influence of the digital divide and socio-economic factors on prevalence of diabetes

  33. Santra B, Paul A, Mukherjee DP (2020) Deterministic dropout for deep neural networks using composite random forest. Pattern Recogn Lett 131:205–212

    Article  Google Scholar 

  34. Shen X, Tian X, Liu T, Xu F, Tao D (2017) Continuous dropout. IEEE Trans Neural Netw Learn Syst 29(9):3926–3937

    Article  Google Scholar 

  35. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  36. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  37. Sun Y, Wang X, Tang X (2016) Sparsifying neural network connections for face recognition

  38. Tang Y, Wang Y, Xu Y, Shi B, Xu C, Xu C, Xu C (2020) Beyond dropout: feature map distortion to regularize deep neural networks. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 5964–5971

  39. Viloria A, Lezama OBP, Mercado-Caruzo N (2020) Unbalanced data processing using oversampling: machine learning. Procedia Comput Sci 175:108–113

    Article  Google Scholar 

  40. Vinyals O, Blundell C, Lillicrap T, Wierstra D, et al. (2016) Matching networks for one shot learning. Advances in neural information processing systems 29

  41. Wan L, Zeiler M, Zhang S, Le Cun Y, Fergus R (2013) Regularization of neural networks using dropconnect. In: International conference on machine learning, PMLR, pp 1058–1066

  42. Wang G, Zhai Q (2021) Feature fusion network based on strip pooling. Sci Rep 11(1):1–8

    Google Scholar 

  43. Xie J, Ma Z, Lei J, Zhang G, Xue J-H, Tan Z-H, Guo J (2021) Advanced dropout: a model-free methodology for bayesian dropout optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence

  44. Yu F, Xu X (2014) A short-term load forecasting model of natural gas based on optimized genetic algorithm and improved bp neural network. Appl Energy 134:102–113

    Article  Google Scholar 

  45. Zeng L, Shu W, Liu Z, Zou X, Wang S, Xia J, Xu C, Xiong D, Yang Z (2022) Vision-based high-precision intelligent monitoring for shield tail clearance. Autom Constr 134:104088

    Article  Google Scholar 

  46. Zhou R, Guo F, Azarpazhooh MR, Spence JD, Ukwatta E, Ding M, Fenster A (2020) A voxel-based fully convolution network and continuous max-flow for carotid vessel-wall-volume segmentation from 3d ultrasound images. IEEE Trans Med Imaging 39(9):2844–2855

    Article  Google Scholar 

  47. Zunino A, Bargal SA, Morerio P, Zhang J, Sclaroff S, Murino V (2021) Excitation dropout: encouraging plasticity in deep neural networks. Int J Comput Vis 129(4):1139–1152

    Article  Google Scholar 

Download references

Funding

This work was in part supported by the Key Research and Development Project of Hubei Province (No. 2020BAB114), the Key Project of Science and Technology Research Program of Hubei Educational Committee (No. D20211402), the Project of Xiangyang Industrial Institute of Hubei University of Technology (No. XYYJ2022C04), and the Open Foundation of Hubei Key Laboratory for High-efficiency Utilization of Solar Energy and Operation Control of Energy Storage System (No. HBSEES201903 & HBSEES202106).

Author information

Author notes

    Authors and Affiliations

    1. School of Electrical and Electronic Engineering, Hubei University of Technology, Wuhan, 430068, Hubei Province, China

      Liang Zeng, Hao Zhang, Yanyan Li, Maodong Li & Shanshan Wang

    2. Hubei Key Laboratory for High-efficiency Utilization of Solar Energy and Operation Control of Energy Storage System, Hubei University of Technology, Wuhan, 430068, Hubei Province, China

      Liang Zeng & Shanshan Wang

    3. Xiangyang Industrial Institute of Hubei University of Technology, Xiangyang, 441100, Hubei Province, China

      Liang Zeng & Shanshan Wang

    Authors
    1. Liang Zeng
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Hao Zhang
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Yanyan Li
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Maodong Li
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Shanshan Wang
      View author publications

      You can also search for this author in PubMed Google Scholar

    Contributions

    Liang Zeng, Hao Zhang, Yanyan Li, Maodong Li and Shanshan Wang conceived the experiments, Hao Zhang conducted the experiments. All authors reviewed the manuscript.

    Corresponding authors

    Correspondence to Liang Zeng or Shanshan Wang.

    Ethics declarations

    Ethics approval

    This article does not contain any studies with human participants or animals performed by any of the authors.

    Consent for Publication

    Completed at Hubei University of Technology on February 21, 2022.

    Conflict of Interests

    The authors declare no competing interests.

    Additional information

    Consent to participate

    Welcome readers to communicate.

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Hao Zhang, Yanyan Li, Maodong Li and Shanshan Wang are contributed equally to this work.

    Rights and permissions

    Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Zeng, L., Zhang, H., Li, Y. et al. Supervision dropout: guidance learning in deep neural network. Multimed Tools Appl 82, 18831–18850 (2023). https://doi.org/10.1007/s11042-022-14274-0

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11042-022-14274-0

    Keywords

    Search

    Navigation