Abstract
The Deep Residual Network in Network (DrNIN) model [18] is an important extension of the convolutional neural network (CNN). They have proven capable of scaling up to dozens of layers. This model exploits a nonlinear function, to replace linear filter, for the convolution represented in the layers of multilayer perceptron (MLP) [23]. Increasing the depth of DrNIN can contribute to improved classification and detection accuracy. However, training the deep model becomes more difficult, the training time slows down, and a problem of decreasing feature reuse arises. To address these issues, in this paper, we conduct a detailed experimental study on the architecture of DrMLPconv blocks, based on which we present a new model that represents a wider model of DrNIN. In this model, we increase the width of the DrNINs and decrease the depth. We call the result module (WDrNIN). On the CIFAR-10 dataset, we will provide an experimental study showing that WDrNIN models can gain accuracy through increased width. Moreover, we demonstrate that even a single WDrNIN outperforms all network-based models in MLPconv network models in accuracy and efficiency with an accuracy equivalent to 93.553% for WDrNIN-4-2.
Similar content being viewed by others
Data availability
The data used to support the findings of this study are included within the article.
Abbreviations
- CNN:
-
Convolutional Neural Network
- CPU:
-
Central Processing Unit
- GPU:
-
Graphics Processing Unit
- RAM:
-
Random Access Memory
- DDR:
-
Double Data Rate
- NIN:
-
Network in Network
- DNIN:
-
Deep Network in Network
- DrNIN:
-
Deep Residual Network in Network
- MLP:
-
Multilayer perceptron
- DMLPconv:
-
Deep MLPconv
- DrMLPconv:
-
Deep Residual MLPconv
- ReLU:
-
Rectified Linear Unit
- eLU:
-
Exponential Linear Unit
References
Alaeddine H, Jihene M (2020) Deep network in network. Neural Comput Applic 134
Alom MdZ, Hasan M, Yakopcic C, Taha T, Asari V (2018) Recurrent residual convolutional neural network based on U-Net (R2U-Net) for medical image segmentation
Bengio Y, Glorot X (2010) Understanding the difficulty of training deep feedforward neural networks. Proceed AISTATS 9:249–256
Chan T, Jia K, Gao S, Lu J, Zeng Z and Ma Y (2014) “PCANet: a simple deep learning baseline for image classification?” , http://arxiv.org/abs/1404.3606.
Chang J-R, Chen Y-S (2015) Batch-normalized maxout network in network. http://arxiv.org/abs/1511.02583
Chen L-C, Papandreou G, Kokkinos I et al (2018) DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Patt Anal Machine Intell 40(4):834
Chen T, Goodfellow I, Shlens J (2016) “Net2net: accelerating learning via knowledge transfer,” http://arxiv.org/abs/1511.05641.
Ciresan D, Meier U, Schmidhuber J (2012) “Multi-column deep neural networks for image classification,” http://arxiv.org/abs/1202.2745.
Clevert D-A, Unterthiner T, Hochreiter S (2016) Fast and accurate deep network learning by exponential linear units (ELUs) comments: published as a conference paper at ICLR 2016 subjects—learning (cs.LG), 2016
Gao S, Miao Z, Zhang Q, Li Q (2019) DCRN: densely connected refinement network for object detection. J Phys: Conf Series., 1229, Article ID 012034
Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. J Mach Learn Res 9
Gong Y, Wang L, Guo R, Lazebnik S (2014) Multi-scale orderless pooling of deep convolutional activation features. http://arxiv.org/abs/1403.1840
Goodfellow IJ, Warde-Farley D, Mirza M, Courville AC, Bengio Y (2013) Maxout networks. https://arxiv.org/abs/1302.4389
Graham B (2014) Fractional max-pooling. https://arxiv.org/abs/1412.6071
He K, Zhang X, Ren S, Sun J (2014) Spatial pyramid pooling in deep convolutional networks for visual recognition. http://arxiv.org/abs/1406.4729
He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on ImageNet classification
He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition
Hmidi A, Malek J (2021) Deep Residual Network in Network, Computational Intelligence and Neuroscience. Hindawi 6659083:1687–5265. https://doi.org/10.1155/2021/6659083
Huang G, Sun Y, Liu Z, Sedra D, Weinberger KQ (2016) Deep networks with stochastic depth. Comp Vision-ECCV 2016 9908:646–661
Huang G, Liu Z, Weinberger KQ, Van Der Maaten L (2017) Densely connected convolutional networks
Iandola F, Han S, Moskewicz M, Ashraf K, Dally W, Keutzer K (2017) SqueezeNet: alexnet-level accuracy with 50x fewer parameters and connected convolutional networks
Ioffe S, Szegedy C (2015) “Batch normalization: accelerating deep network training by reducing internal covariate shift”, http://arxiv.org/abs/1502.03167.
Johnson JM, Khoshgoftaar TM (2019) Survey on deep learning with class imbalance. J Big Data 6:27. https://doi.org/10.1186/s40537-019-0192-5
LeCun Y, Bottou L, Orr GB, Muller K-R (1998) “Efficient backprop,” in Neural Networks: Tricks of the Trade, Springer, Berlin, Germany,
Lee C, Gallagher P and Tu Z (2015) Generalizing pooling functions in convolutional neural networks: mixed gated and tree,”https://arxiv.org/abs/1509.08985.
Lee C-Y, Xie S, Gallagher P, Zhang Z, Tu Z(2014) “Deeply-supervised nets,” , http://arxiv.org/abs/1409.5185.
Liao Z, Carneiro G (2016) On the importance of normalisation layers in deep learning with piecewise linear activation units
Lin M, Chen Q, Yan S (2013) Network in network. http://arxiv.org/abs/1312.4400
Murray N, Perronnin F (2015) “Generalized max pooling,” in Proceedings of the 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 2473–2480, Boston, MA USA
Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: proceedings of the 27th international conference on machine learning (ICML 2010), pp 807–814
Raiko T, Valpola H, Lecun Y (2012) “Deep learning made easier by linear transformations in perceptrons,” in Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS-12), N. D. Lawrence and M. A. Girolami, Eds., vol. 22, pp. 924–932, La Palma, Canary Islands, Spain
Romero A, Ballas N, Kahou SE, Antoine C, Gatta C, Bengio Y (2014) FitNets: hints for thin deep nets
Schmidhuber J (1992) Learning complex, extended sequences using the principle of history compression. Neural Comput 4(2):234–242
Simonyan K, Zisserman A (2014) Very deep convolutional networks for largescale image recognition
Springenberg J, Dosovitskiy A, Brox TT, Riedmiller M (2014) Striving for simplicity: the all convolutional net. http://arxiv.org/abs/1412.6806
Srivastava N, Geoffrey H, Krizhevsky A, Ilya S, Ruslan S, Dropout (2014) A simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958
Sutskever I, Martens J, Dahl GE, and Hinton GE(2013) “On the importance of initialization and momentum in deep learning,” in Proceedings of the 30th International Conference on Machine Learning (ICML-13), S. Dasgupta and D. Mcallester, Eds., vol. 28, pp. 1139–1147, JMLR Workshop and Conference Proceedings, New Brunswick, NJ, USA
Szegedy C (2015) “Going deeper with convolutions,” in Proceedingsof the 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9, Boston, MA USA
Yoo D, Park S, Lee J, Kweon I (2015) “Multi-scale pyramid pooling for deep convolutional representation,” in Proceedings of the 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 1–5, Boston, MA USA
Zagoruyko S, Komodakis N (2017) Wide Residual Networks, 1605.07146, arXiv, pp 87.1–87.12 https://doi.org/10.5244/C.30.87.
Zeiler MD, Fergus R (2013) Visualizing and understanding convolutional networks
Acknowledgments
This work was supported by the Electronics and Microelectronics Laboratory.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there are no conflicts of interest regarding the publication of this article.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Alaeddine, H., Jihene, M. Wide deep residual networks in networks. Multimed Tools Appl 82, 7889–7899 (2023). https://doi.org/10.1007/s11042-022-13696-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-13696-0