Skip to main content
SpringerLink
Log in

Restricted Convolutional Neural Networks

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

In this paper, a new type of convolutional neural network is proposed which is inspired by cellular automata research. This model is referred to as “restricted convolutional neural network” and its characteristic is that the feature maps are not fully connected, i.e. each feature map is only connected to a small neighborhood of previous feature maps. First this model is formally defined. Then it is used for image classification. Two layerwise pretraining methods have been proposed, and some structural variations have been analyzed. The model is tested on both MNIST and CIFAR-10 datasets. Results suggest that this model in some cases can outperform a convolutional neural network with similar architecture.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Notes

  1. We refer to both cellular automaton and cellular automata (plural) as CA. The distinction would be clear from the context.

References

  1. Baldi P (2012) Autoencoders, unsupervised learning, and deep architectures. In: Proceedings of ICML workshop on unsupervised and transfer learning, pp 37–49

  2. Beigy H, Meybodi MR (2004) A mathematical framework for cellular learning automata. Adv Complex Syst 7(03n04):295–319

    Article  MathSciNet  Google Scholar 

  3. Chua LO, Yang L (1988) Cellular neural networks: applications. IEEE Trans Circuits Syst 35(10):1273–1290

    Article  MathSciNet  Google Scholar 

  4. de Korte A, Brouwers H (2013) A cellular automata approach to chemical reactions: 1 reaction controlled systems. Chem Eng J 228:172–178. https://doi.org/10.1016/j.cej.2013.04.084

    Article  Google Scholar 

  5. Gardner M (1970) Mathematical games: the fantastic combinations of john conways new solitaire game life. Sci Am 223(4):120–123

    Article  Google Scholar 

  6. Gehring WJ (2005) New perspectives on eye development and the evolution of eyes and photoreceptors. J Hered 96(3):171–184

    Article  Google Scholar 

  7. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    MATH  Google Scholar 

  8. Guo Y, Yao A, Chen Y (2016) Dynamic network surgery for efficient DNNs. In: Lee DD, Sugiyama M, Luxburg UV, Guyon I, Garnett R (eds) Advances in neural information processing systems 29. Curran Associates, Inc., pp 1379–1387

  9. Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems, pp 1135–1143

  10. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR. arXiv:1512.03385

  11. Hinton G (2010) A practical guide to training restricted boltzmann machines. Momentum 9(1):926

    Google Scholar 

  12. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  Google Scholar 

  13. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  Google Scholar 

  14. Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366. https://doi.org/10.1016/0893-6080(89)90020-8

    Article  MATH  Google Scholar 

  15. Huberman B (1985) Probabilistic cellular automata. In: Nonlinear phenomena in physics. Springer, pp 129–137

  16. Jump JR, Kirtane JS (1974) On the interconnection structure of cellular networks. Inf Control 24(1):74–91

    Article  MathSciNet  Google Scholar 

  17. Kohonen T (1982) Self-organized formation of topologically correct feature maps. Biol Cybern 43(1):59–69

    Article  MathSciNet  Google Scholar 

  18. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf; https://www.cs.toronto.edu/~kriz/cifar.html

  19. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  20. LeCun Y, Cortes C, Burges CJ (2010) Mnist handwritten digit database. AT&T Labs (Online). http://yann.lecun.com/exdb/mnist

  21. Margenstern M (2007) Cellular automata in hyperbolic spaces: theory, vol 1. Archives contemporaines

  22. McCulloch WS, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 5(4):115–133

    Article  MathSciNet  Google Scholar 

  23. Molchanov P, Tyree S, Karras T, Aila T, Kautz J (2016) Pruning convolutional neural networks for resource efficient transfer learning. CoRR. arXiv:1611.06440

  24. Osindero S, Hinton GE (2008) Modeling image patches with a directed hierarchy of Markov random fields. In: Platt JC, Koller D, Singer Y, Roweis ST (eds) Advances in neural information processing systems 20. Curran Associates, Inc., pp 1121–1128

  25. O’Sullivan D (2001) Exploring spatial process dynamics using irregular cellular automaton models. Geogr Anal 33(1):1–18

    Article  Google Scholar 

  26. Rosenblatt F (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev 65(6):386

    Article  Google Scholar 

  27. Sarkar P (2000) A brief history of cellular automata. ACM Comput Surv (CSUR) 32(1):80–107

    Article  Google Scholar 

  28. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  29. Toffoli T, Margolus N (1987) Cellular automata machines: a new environment for modeling. MIT Press, Cambridge

    Book  Google Scholar 

  30. Von Neumann J, Burks AW et al (1966) Theory of self-reproducing automata. IEEE Trans Neural Netw 5(1):3–14

    Google Scholar 

  31. Wang L, Zhang J, Shao H (2014) Existence and global stability of a periodic solution for a cellular neural network. Commun Nonlinear Sci Numer Simul 19(9):2983–2992

    Article  MathSciNet  Google Scholar 

  32. Werbos PJ (1974) Beyond regression: new tools for prediction and analysis in the behavioral sciences. Doctoral dissertation, Applied Mathematics, Harvard University, MA

  33. Widrow B, Hoff ME (1960) Adaptive switching circuits. Technical report, Stanford University, Stanford Electronics Labs

  34. Wolfram S (1994) Cellular automata and complexity: collected papers, vol 1. Addison-Wesley, Reading

    MATH  Google Scholar 

  35. Wolfram S (2002) A new kind of science, vol 5. Wolfram Media, Champaign

    MATH  Google Scholar 

  36. Yang T, Yang LB, Wu CW, Chua LO (1996) Fuzzy cellular neural networks: theory. In: 1996 4th IEEE international workshop on cellular neural networks and their applications, 1996. CNNA-96. Proceedings. IEEE, pp 181–186

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering and Information Technology, Amirkabir University of Technology, Tehran, Iran

    Mehran Mirkhan & Mohammad Reza Meybodi

Authors
  1. Mehran Mirkhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Reza Meybodi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehran Mirkhan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mirkhan, M., Meybodi, M.R. Restricted Convolutional Neural Networks. Neural Process Lett 50, 1705–1733 (2019). https://doi.org/10.1007/s11063-018-9954-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-018-9954-x

Keywords

Search

Navigation