Skip to main content
SpringerLink
Log in

Deep network in network

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The different CNN models use many layers that typically include a stack of linear convolution layers combined with pooling and normalization layers to extract the characteristics of the images. Unlike these models, and instead of using a linear filter for convolution, the network in network (NiN) model uses a multilayer perception (MLP), a nonlinear function, to replace the linear filter. This article presents a new deep network in network (DNIN) model based on the NiN structure, NiN drag a universal approximator, (MLP) with rectified linear unit (ReLU) to improve classification performance. The use of MLP leads to an increase in the density of the connection. This makes learning more difficult and time learning slower. In this article, instead of ReLU, we use the linear exponential unit (eLU) to solve the vanishing gradient problem that can occur when using ReLU and to speed up the learning process. In addition, a reduction in the convolution filters size by increasing the depth is used in order to reduce the number of parameters. Finally, a batch normalization layer is applied to reduce the saturation of the eLUs and the dropout layer is applied to avoid overfitting. The experimental results on the CIFAR-10 database show that the DNIN can reduce the complexity of implementation due to the reduction in the adjustable parameters. Also the reduction in the filters size shows an improvement in the recognition accuracy of the model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  2. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Computer vision–ECCV 2014. Springer, pp 818–833

  3. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. In: CoRR. http://arxiv.org/abs/1409.1556

  4. Szegedy C et al (2015) Going deeper with convolutions. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), Boston, MA USA, pp 1–9. https://doi.org/10.1109/cvpr.2015.7298594

  5. Lin M, Chen Q, Yan S (2014) Network in network. In: International conference on learning representations. http://arxiv.org/abs/1312.4400

  6. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. In: CoRR. http://arxiv.org/abs/1512.03385

  7. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer vision – ECCV 2016. ECCV 2016. Lecture notes in computer science, vol 9908. Springer, Cham, pp 630–645. https://doi.org/10.1007/978-3-319-46493-0_38

  8. Zagoruyko S, Komodakis N (2016) Wide residual networks. Sergey Zagoruyko Nikos Komodakis: Wide Residual Networks, pp 87.1–87.12 https://doi.org/10.5244/C.30.87

  9. Xie S, Girshick R, Dollár P, Tu Z, He K (2016) Aggregated residual transformations for deep neural networks. arXiv:1611.05431

  10. Huang G, Sun Y, Liu Z, Sedra D, Weinberger KQ (2016) Deep networks with stochastic depth, vol $9908, pp 646–661. https://doi.org/10.1007/978-3-319-46493-0_39

  11. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks

  12. Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, LeCun Y (2014) OverFeat: integrated recognition, localization and detection using convolutional networks. In: ICLR, 2014

  13. Cao J, Pang Y, Li X (2016) Pedestrian detection inspired by appearance constancy and shape symmetry. In: Proceedings of IEEE international conference on computer vision and pattern recognition, pp 1316–1324

  14. Gong M, Zhao J, Liu J, Miao Q, Jiao L (2016) Change detection in synthetic aperture radar images based on deep neural networks. IEEE Trans Neural Netw Learn Syst 27(1):125–138

    Article  MathSciNet  Google Scholar 

  15. Liu J, Gong M, Qin K, Zhang P (2018) A deep convolutional coupling network for change detection based on heterogeneous optical and radar images. IEEE Trans Neural Netw Learn Syst 29(3):545–559

    Article  MathSciNet  Google Scholar 

  16. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):127–131

    Article  MathSciNet  Google Scholar 

  17. Chang C-H (2015) Deep and shallow architecture of multilayer neural networks. IEEE Trans Neural Netw Learn Syst 26(10):2477–2486

    Article  MathSciNet  Google Scholar 

  18. Gong M, Liu J, Li H, Cai Q, Su L (2015) A multi objective sparse feature learning model for deep neural networks. IEEE Trans Neural Netw Learn Syst 26(12):3263–3277

    Article  MathSciNet  Google Scholar 

  19. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  20. Liu J, Gong M, Zhao J, Li H, Jiao L (2016) Difference representation learning using stacked restricted Boltzmann machines for change detection in SAR images. Soft Comput 20(12):4645–4657

    Article  Google Scholar 

  21. Zhang P, Gong M, Su L, Liu J, Li Z (2016) Change detection based on deep feature representation and mapping transformation for multi-spatial-resolution remote sensing images. Photogram Remote Sens 116:24–41

    Article  Google Scholar 

  22. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  23. Schmidhuber J (2014) Deep learning in neural networks: an overview. Neural Networks 61:85–117

  24. Rumelhart D, Hinton G, Williams R (1986) Learning representations by back propagating error. Nature 323:533–536

    Article  Google Scholar 

  25. Agostinelli F, Hoffman M, Sadowski P, Baldi P (2014) Learning activation functions to improve deep neural networks. In: CoRR. http://arxiv.org/abs/1412.6830

  26. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML 2010), pp 807–814

  27. Goodfellow IJ, Warde-Farley D, Mirza M, Courville AC, Bengio Y (2013) Maxout networks. In: Proceedings of the 30th international conference on machine learning (ICML 2013), volume 28 of JMLR proceedings, pp 1319–1327. http://jmlr.org/

  28. Liao Z, Carneiro G (2016) On the importance of normalisation layers in deep learning with piecewise linear activation units. arXiv:1508.00330

  29. Clevert D-A, Unterthiner T, Hochreiter S (2016) Fast and accurate deep network learning by exponential linear units (ELUs) comments: published as a conference paper at ICLR 2016 subjects—learning (cs.LG), 2016

  30. Lee C-Y, Xie S, Gallagher P, Zhang Z, Tu Z (2015) Deeply supervised nets. In: Proceedings of AISTATS 2015

  31. Srivastava RK, Greff K, Schmidhuber J (2015) Highway networks. In: CoRR. http://arxiv.org/abs/1505.00387

  32. Jarrett K, Kavukcuoglu K, Ranzato MA, LeCun Y (2009) What is the best multi-stage architecture for object recognition?. In: Proceedings of the IEEE international conference on computer vision, pp 2146–2153

  33. LeCun Y, Kavukcuoglu K, Farabet C (2010) Convolutional networks and applications in vision. In: Proceedings of the IEEE international symposium on circuits and systems, pp 253–256

  34. Chan T, Jia K, Gao S, Lu J, Zeng Z, Ma Y (2014) PCANet: a simple deep learning baseline for image classification? In: CoRR. http://arxiv.org/abs/1404.3606

  35. Gong Y, Wang L, Guo R, Lazebnik S (2014) Multi-scale orderless pooling of deep

  36. Graham B (2014) Fractional max-pooling. In: CoRR. https://arxiv.org/abs/1412.6071

  37. He K, Zhang X, Ren S, Sun J (2014) Spatial pyramid pooling in deep convolutional networks for visual recognition. In: Proceedings of European conference on computer vision, pp 346–361

  38. Lee C, Gallagher P, Tu Z (2015) Generalizing pooling functions in convolutional neural networks: mixed gated and tree. In: CoRR. http://arxiv.org/abs/1509.08985

  39. Murray N, Perronnin F (2014) Generalized max pooling. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 2473–2480

  40. Springenberg J, Dosovitskiy A, Brox TT, Riedmiller M (2014) Striving for simplicity: the all convolutional net. In: CoRR. http://arxiv.org/abs/1412.6806

  41. Yoo D, Park S, Lee J, Kweon I (2015) Multi-scale pyramid pooling for deep convolutional representation. In: Proceedings of IEEE workshop computer vision and pattern recognition, pp 1–5

  42. Zeiler MD, Fergus R (2013) Stochastic pooling for regularization of deep convolutional neural networks. In: CoRR. http://arxiv.org/abs/1301.3557

  43. Chang J, Chen Y (2015) Batch-normalized maxout network in network. arXiv:1511.02583

  44. Castaneda G, Morris P, Khoshgoftaar TM (2019) Evaluation of maxout activations in deep learning across several big data domains. J Big Data 6:72. https://doi.org/10.1186/s40537-019-0233-0

    Article  Google Scholar 

  45. Shao L, Wu D, Li X (2014) Learning deep and wide: a spectral method for learning deep networks. IEEE Trans Neural Netw Learn Syst 25(12):2303–2308

    Article  Google Scholar 

  46. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: CoRR, vol. abs/1502.03167

  47. Srivastava Nitish, Hinton Geoffrey, Krizhevsky Alex, Sutskever Ilya, Salakhutdinov Ruslan (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    MathSciNet  MATH  Google Scholar 

  48. Russakovsky O, Deng J, Su H et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis (IJCV) 115(3):211–252

    Article  MathSciNet  Google Scholar 

  49. Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data 6:60. https://doi.org/10.1186/s40537-019-0197-0

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Sciences of Monastir, Laboratory of Electronics and Microelectronics, LR99ES30, Monastir University, 5000, Monastir, Tunisia

    Hmidi Alaeddine & Malek Jihene

  2. Higher Institute of Applied Sciences and Technology of Sousse, Sousse University, 4000, Sousse, Tunisia

    Malek Jihene

Authors
  1. Hmidi Alaeddine
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Malek Jihene
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hmidi Alaeddine.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alaeddine, H., Jihene, M. Deep network in network. Neural Comput & Applic 33, 1453–1465 (2021). https://doi.org/10.1007/s00521-020-05008-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05008-0

Keywords

Search

Navigation