Skip to main content
SpringerLink
Log in

Wide deep residual networks in networks

  • Track 2: Medical Applications of Multimedia
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The Deep Residual Network in Network (DrNIN) model [18] is an important extension of the convolutional neural network (CNN). They have proven capable of scaling up to dozens of layers. This model exploits a nonlinear function, to replace linear filter, for the convolution represented in the layers of multilayer perceptron (MLP) [23]. Increasing the depth of DrNIN can contribute to improved classification and detection accuracy. However, training the deep model becomes more difficult, the training time slows down, and a problem of decreasing feature reuse arises. To address these issues, in this paper, we conduct a detailed experimental study on the architecture of DrMLPconv blocks, based on which we present a new model that represents a wider model of DrNIN. In this model, we increase the width of the DrNINs and decrease the depth. We call the result module (WDrNIN). On the CIFAR-10 dataset, we will provide an experimental study showing that WDrNIN models can gain accuracy through increased width. Moreover, we demonstrate that even a single WDrNIN outperforms all network-based models in MLPconv network models in accuracy and efficiency with an accuracy equivalent to 93.553% for WDrNIN-4-2.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

The data used to support the findings of this study are included within the article.

Abbreviations

CNN:

Convolutional Neural Network

CPU:

Central Processing Unit

GPU:

Graphics Processing Unit

RAM:

Random Access Memory

DDR:

Double Data Rate

NIN:

Network in Network

DNIN:

Deep Network in Network

DrNIN:

Deep Residual Network in Network

MLP:

Multilayer perceptron

DMLPconv:

Deep MLPconv

DrMLPconv:

Deep Residual MLPconv

ReLU:

Rectified Linear Unit

eLU:

Exponential Linear Unit

References

  1. Alaeddine H, Jihene M (2020) Deep network in network. Neural Comput Applic 134

  2. Alom MdZ, Hasan M, Yakopcic C, Taha T, Asari V (2018) Recurrent residual convolutional neural network based on U-Net (R2U-Net) for medical image segmentation

  3. Bengio Y, Glorot X (2010) Understanding the difficulty of training deep feedforward neural networks. Proceed AISTATS 9:249–256

    Google Scholar 

  4. Chan T, Jia K, Gao S, Lu J, Zeng Z and Ma Y (2014) “PCANet: a simple deep learning baseline for image classification?” , http://arxiv.org/abs/1404.3606.

  5. Chang J-R, Chen Y-S (2015) Batch-normalized maxout network in network. http://arxiv.org/abs/1511.02583

  6. Chen L-C, Papandreou G, Kokkinos I et al (2018) DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Patt Anal Machine Intell 40(4):834

    Article  Google Scholar 

  7. Chen T, Goodfellow I, Shlens J (2016) “Net2net: accelerating learning via knowledge transfer,” http://arxiv.org/abs/1511.05641.

  8. Ciresan D, Meier U, Schmidhuber J (2012) “Multi-column deep neural networks for image classification,” http://arxiv.org/abs/1202.2745.

  9. Clevert D-A, Unterthiner T, Hochreiter S (2016) Fast and accurate deep network learning by exponential linear units (ELUs) comments: published as a conference paper at ICLR 2016 subjects—learning (cs.LG), 2016

  10. Gao S, Miao Z, Zhang Q, Li Q (2019) DCRN: densely connected refinement network for object detection. J Phys: Conf Series., 1229, Article ID 012034

  11. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. J Mach Learn Res 9

  12. Gong Y, Wang L, Guo R, Lazebnik S (2014) Multi-scale orderless pooling of deep convolutional activation features. http://arxiv.org/abs/1403.1840

  13. Goodfellow IJ, Warde-Farley D, Mirza M, Courville AC, Bengio Y (2013) Maxout networks. https://arxiv.org/abs/1302.4389

  14. Graham B (2014) Fractional max-pooling. https://arxiv.org/abs/1412.6071

  15. He K, Zhang X, Ren S, Sun J (2014) Spatial pyramid pooling in deep convolutional networks for visual recognition. http://arxiv.org/abs/1406.4729

  16. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on ImageNet classification

  17. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition

    Google Scholar 

  18. Hmidi A, Malek J (2021) Deep Residual Network in Network, Computational Intelligence and Neuroscience. Hindawi 6659083:1687–5265. https://doi.org/10.1155/2021/6659083

    Article  Google Scholar 

  19. Huang G, Sun Y, Liu Z, Sedra D, Weinberger KQ (2016) Deep networks with stochastic depth. Comp Vision-ECCV 2016 9908:646–661

    Article  Google Scholar 

  20. Huang G, Liu Z, Weinberger KQ, Van Der Maaten L (2017) Densely connected convolutional networks

  21. Iandola F, Han S, Moskewicz M, Ashraf K, Dally W, Keutzer K (2017) SqueezeNet: alexnet-level accuracy with 50x fewer parameters and connected convolutional networks

  22. Ioffe S, Szegedy C (2015) “Batch normalization: accelerating deep network training by reducing internal covariate shift”, http://arxiv.org/abs/1502.03167.

  23. Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning with class imbalance. J Big Data 6:27. https://doi.org/10.1186/s40537-019-0192-5

    Article  Google Scholar 

  24. LeCun Y, Bottou L, Orr GB, Muller K-R (1998) “Efficient backprop,” in Neural Networks: Tricks of the Trade, Springer, Berlin, Germany,

  25. Lee C, Gallagher P and Tu Z (2015) Generalizing pooling functions in convolutional neural networks: mixed gated and tree,”https://arxiv.org/abs/1509.08985.

  26. Lee C-Y, Xie S, Gallagher P, Zhang Z, Tu Z(2014) “Deeply-supervised nets,” , http://arxiv.org/abs/1409.5185.

  27. Liao Z, Carneiro G (2016) On the importance of normalisation layers in deep learning with piecewise linear activation units

  28. Lin M, Chen Q, Yan S (2013) Network in network. http://arxiv.org/abs/1312.4400

  29. Murray N, Perronnin F (2015) “Generalized max pooling,” in Proceedings of the 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 2473–2480, Boston, MA USA

  30. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: proceedings of the 27th international conference on machine learning (ICML 2010), pp 807–814

  31. Raiko T, Valpola H, Lecun Y (2012) “Deep learning made easier by linear transformations in perceptrons,” in Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS-12), N. D. Lawrence and M. A. Girolami, Eds., vol. 22, pp. 924–932, La Palma, Canary Islands, Spain

  32. Romero A, Ballas N, Kahou SE, Antoine C, Gatta C, Bengio Y (2014) FitNets: hints for thin deep nets

  33. Schmidhuber J (1992) Learning complex, extended sequences using the principle of history compression. Neural Comput 4(2):234–242

    Article  Google Scholar 

  34. Simonyan K, Zisserman A (2014) Very deep convolutional networks for largescale image recognition

  35. Springenberg J, Dosovitskiy A, Brox TT, Riedmiller M (2014) Striving for simplicity: the all convolutional net. http://arxiv.org/abs/1412.6806

  36. Srivastava N, Geoffrey H, Krizhevsky A, Ilya S, Ruslan S, Dropout (2014) A simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    MATH  Google Scholar 

  37. Sutskever I, Martens J, Dahl GE, and Hinton GE(2013) “On the importance of initialization and momentum in deep learning,” in Proceedings of the 30th International Conference on Machine Learning (ICML-13), S. Dasgupta and D. Mcallester, Eds., vol. 28, pp. 1139–1147, JMLR Workshop and Conference Proceedings, New Brunswick, NJ, USA

  38. Szegedy C (2015) “Going deeper with convolutions,” in Proceedingsof the 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9, Boston, MA USA

  39. Yoo D, Park S, Lee J, Kweon I (2015) “Multi-scale pyramid pooling for deep convolutional representation,” in Proceedings of the 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 1–5, Boston, MA USA

  40. Zagoruyko S, Komodakis N (2017) Wide Residual Networks, 1605.07146, arXiv, pp 87.1–87.12 https://doi.org/10.5244/C.30.87.

  41. Zeiler MD, Fergus R (2013) Visualizing and understanding convolutional networks

Download references

Acknowledgments

This work was supported by the Electronics and Microelectronics Laboratory.

Author information

Authors and Affiliations

  1. Faculty of Sciences of Monastir, Laboratory of Electronics and Microelectronics, LR99ES30, Monastir University, 5000, Monastir, Tunisia

    Hmidi Alaeddine & Malek Jihene

  2. Higher Institute of Applied Sciences and Technology of Sousse, Sousse University, 4000, Sousse, Tunisia

    Malek Jihene

Authors
  1. Hmidi Alaeddine
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Malek Jihene
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hmidi Alaeddine.

Ethics declarations

Conflict of interest

The authors declare that there are no conflicts of interest regarding the publication of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alaeddine, H., Jihene, M. Wide deep residual networks in networks. Multimed Tools Appl 82, 7889–7899 (2023). https://doi.org/10.1007/s11042-022-13696-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13696-0

Keywords

Search

Navigation