Skip to main content
SpringerLink
Log in

Feed-forward versus recurrent architecture and local versus cellular automata distributed representation in reservoir computing for sequence memory learning

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Reservoir computing based on cellular automata (ReCA) constructs a novel bridge between automata computational theory and recurrent neural networks. ReCA has been trained to solve 5-bit memory tasks. Several methods are proposed to implement the reservoir where the distributed representation of cellular automata (CA) in recurrent architecture could solve the 5-bit tasks with minimum complexity and minimum number of training examples. CA distributed representation in recurrent architecture outperforms the local representation in recurrent architecture (stack reservoir), then echo state networks and feed-forward architecture using local or distributed representation. Extracted features from the reservoir, using the natural diffusion of CA states in the reservoir offers the state-of-the-art results in terms of feature vector length and the required training examples. Another extension is obtained by combining the reservoir CA states using XOR, Binary or Gray operator to produce a single feature vector to reduce the feature space. This method gives promising results, however using the natural diffusion of CA states still outperform. ReCA can be considered to operate around the lower bound of complexity; due to using the elementary CA in the reservoir.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. The network should gradually lose information that has been received from previous states and inputs.

  2. In normalized addition; \(0 + 0 = 0, 1 + 1 = 1\) and for \(1 + 0\) or \(0 + 1\) the result is decided randomly 0 or 1.

  3. Insertion function is to insert a new input time step into the reservoir.

  4. The LSB can also be at the last row.

  5. Because, the evolution of class I rules vanishes after the first iteration in 5-bit task, due to the single nonzero in its input at each time step as shown in Fig. 6.

  6. Since, \(L_{in}\) and T are constants in all experiments, hence the value of \(L_{CA}\) depends only on I, k,  and f as illustrated in Table 6. Then, the feature vector with dimension \(L_{CA}\) will be used in the regressor (read-out stage) to find the pseudo-inverse that implies the most expensive computational part in the model.

  7. One side of propagation as shown in Fig. 9.

  8. The experiments have been repeated 100 times, i.e., \(N_{trials} = 100\) runs (trials).

  9. The shift property is especially for a single non-zero initial state as in our case for 5-bit task.

  10. For Normal and Overwrite, \(L_{CA}\) depends on I, k, and f, but it depends only on I and f for XOR, Binary and Gray.

References

  • Adamatzky A (2001) Computing in nonlinear media and automata collectives. CRC Press, Boca Raton

    MATH  Google Scholar 

  • Alonso-Sanz R, Martin M (2006) Elementary cellular automata with elementary memory rules in cells: the case of linear rules. J Cell Autom 1(1):71–87

    MathSciNet  MATH  Google Scholar 

  • Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157–166

    Google Scholar 

  • Bertschinger N, Natschläger T (2004) Real-time computation at the edge of chaos in recurrent neural networks. Neural Comput 16(7):1413–1436

    MATH  Google Scholar 

  • Cook M (2004) Universality in elementary cellular automata. Complex Syst 15(1):1–40

    MathSciNet  MATH  Google Scholar 

  • Dai X (2004) Genetic regulatory systems modeled by recurrent neural network. In: International symposium on neural networks. Springer, pp 519–524

  • Dale M, Miller JF, Stepney S, Trefzer MA (2016) Evolving carbon nanotube reservoir computers. In: International conference on unconventional computation and natural computation. Springer, pp 49–61

  • Dale M, Miller JF, Stepney S (2017) Reservoir computing as a model for in-materio computing. In: Adamatzky A (ed) Advances in unconventional computing. Springer, pp 533–571

  • Deypir M, Sadreddini MH, Hashemi S (2012) Towards a variable size sliding window model for frequent itemset mining over data streams. Comput Ind Eng 63(1):161–172

    Google Scholar 

  • Dietterich TG (2002) Machine learning for sequential data: a review. In: Joint IAPR international workshops on statistical techniques in pattern recognition (SPR) and structural and syntactic pattern recognition (SSPR), vol 88. Springer, pp 15–30

  • Doya K (1992) Bifurcations in the learning of recurrent neural networks. In: IEEE international symposium on circuits and systems, vol 6. IEEE, pp 2777–2780

  • Fernando C, Sojakka S (2003) Pattern recognition in a bucket. In: European conference on artificial life. Springer, pp 588–597

  • Funahashi K, Nakamura Y (1993) Approximation of dynamical systems by continuous time recurrent neural networks. Neural Netw 6(6):801–806

    Google Scholar 

  • Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    MATH  Google Scholar 

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Google Scholar 

  • Hochreiter S, Bengio Y, Frasconi P, Schmidhuber J (2001) Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In: A field guide to dynamical recurrent neural networks, chap 14. Wiley-IEEE Press, pp 237–244

  • Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: IEEE international joint conference on neural networks, IJCNN. 2004, vol 2, pp 985–990

  • Huang GB, Wang DH, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122

    Google Scholar 

  • Jaeger H (2001) The echo state approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report 148(34):13

  • Jaeger H (2012) Long short-term memory in echo state networks: details of a simulation study. Technical Report 27, Jacobs University Bremen

  • Jones B, Stekel D, Rowe J, Fernando C (2007) Is there a liquid state machine in the bacterium Escherichia coli? In: IEEE symposium on artificial life. IEEE, pp 187–191

  • Kang G, Guo S (2009) Variable sliding window DTW speech identification algorithm. In: Ninth international conference on hybrid intelligent systems, vol 1. IEEE, pp 304–307

  • Kuhn M, Johnson K (2013) Applied predictive modeling, vol 26. Springer, New York

    MATH  Google Scholar 

  • Langton CG (1986) Studying artificial life with cellular automata. Physica D 22(1–3):120–149

    MathSciNet  Google Scholar 

  • Larger L, Soriano MC, Brunner D, Appeltant L, Gutiérrez JM, Pesquera L, Mirasso CR, Fischer I (2012) Photonic information processing beyond turing: an optoelectronic implementation of reservoir computing. Opt Express 20(3):3241–3249

    Google Scholar 

  • Legenstein R, Maass W (2007) Edge of chaos and prediction of computational performance for neural circuit models. Neural Netw 20(3):323–334

    MATH  Google Scholar 

  • Li W, Packard N (1990) The structure of the elementary cellular automata rule space. Complex Syst 4(3):281–297

    MathSciNet  MATH  Google Scholar 

  • Lukoševičius M (2012) A practical guide to applying echo state networks. In: Montavon G, Orr G, Müller KR (eds) Neural networks: tricks of the trade. Springer, pp 659–686

  • Lukoševičius M, Jaeger H (2009) Reservoir computing approaches to recurrent neural network training. Comput Sci Rev 3(3):127–149

    MATH  Google Scholar 

  • Maass W, Natschläger T, Markram H (2002) Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput 14(11):2531–2560

    MATH  Google Scholar 

  • Margem M, Gedik OS (2019) Reservoir computing based on cellular automata (ReCA) in sequence learning. J Cell Autom 14(1–2):153–170

    MathSciNet  Google Scholar 

  • Margem M, Yilmaz O (2016) How much computation and distributedness is needed in sequence learning tasks? In: International conference on artificial general intelligence, AGI-16. Springer, pp 274–283

  • Martens J, Sutskever I (2011) Learning recurrent neural networks with hessian-free optimization. In: the 28th International conference on machine learning (ICML-11), pp 1033–1040

  • Martnez GJ (2013) A note on elementary cellular automata classification. J Cell Autom 8(3–4):233–259

    MathSciNet  Google Scholar 

  • Martnez GJ, Adamatzky A, Alonso-Sanz R (2013a) Designing complex dynamics in cellular automata with memory. Int J Bifurc Chaos 23(10):1330035

    MathSciNet  MATH  Google Scholar 

  • Martnez GJ, Seck-Tuoh-Mora JC, Zenil H (2013b) Computation and universality: class iv versus class iii cellular automata. J Cell Autom 7(5–6):393–430

    MathSciNet  MATH  Google Scholar 

  • Martnez GJ, Seck-Tuoh-Mora JC, Zenil H (2013c) Wolframs classification and computation in cellular automata classes iii and iv. In: Zenil H (ed) Irreducibility and computational equivalence. Springer, pp 237–259

  • McDonald N (2017) Reservoir computing & extreme learning machines using pairs of cellular automata rules. In: International joint conference on neural networks (IJCNN), USA, vol 88. pp 2429–2436

  • Morn A, Frasser CF, Rossell JL (2018) Reservoir computing hardware with cellular automata. arXiv:1806.04932

  • Nichele S, Gundersen MS (2017) Reservoir computing using non-uniform binary cellular automata. Complex Syst 26(3):225–245

    Google Scholar 

  • Nichele S, Molund A (2017) Deep learning with cellular automaton-based reservoir computing. Complex Syst 26(4):319–339

    MathSciNet  Google Scholar 

  • Ortín S, Soriano MC, Pesquera L, Brunner D, San-Martín D, Fischer I, Mirasso C, Gutiérrez J (2015) A unified framework for reservoir computing and extreme learning machines based on a single time-delayed neuron. Sci Rep 5:14945

    Google Scholar 

  • Paquot Y, Duport F, Smerieri A, Dambre J, Schrauwen B, Haelterman M, Massar S (2012) Optoelectronic reservoir computing. Sci Rep 2:287

    Google Scholar 

  • Pascanu R, Mikolov T, Bengio Y (2013) On the difficulty of training recurrent neural networks. In: the 30th International conference on machine learning, Atlanta, Georgia, USA

  • Siegelmann HT, Sontag ED (1995) On the computational power of neural nets. J Comput Syst Sci 50(1):132–150

    MathSciNet  MATH  Google Scholar 

  • Snyder D, Goudarzi A, Teuscher C (2013) Computational capabilities of random automata networks for reservoir computing. Phys Rev E 87(4):042808

    Google Scholar 

  • Toffoli T, Margolus N (1987) Cellular automata machines: a new environment for modeling. MIT Press, Cambridge

    MATH  Google Scholar 

  • Von Neumann J (1951) The general and logical theory of automata. Cereb Mech Behav Hixon Symp 1:1–41

    MathSciNet  Google Scholar 

  • Wolfram S (1994) Tables of cellular automaton properties. In: Wolfram S (ed) Cellular automata and complexity: collected papers. Westview Press, pp 513–584

  • Wolfram S (2002) A new kind of science, vol 5. Wolfram Media, Champaign

    MATH  Google Scholar 

  • Yilmaz O (2014) Reservoir computing using cellular automata. arXiv:1410.0162

  • Yilmaz O (2015a) Analogy making and logical inference on images using cellular automata based hyperdimensional computing. In: NIPS, workshop on cognitive computation

  • Yilmaz O (2015b) Machine learning using cellular automata based feature expansion and reservoir computing. J Cell Autom 10(5–6):435–472

    MathSciNet  MATH  Google Scholar 

  • Yilmaz O (2015c) Symbolic computation using cellular automata-based hyperdimensional computing. Neural Comput 27(12):2661–2692

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Control Engineering, College of Electronic Technology-Tripoli, Al-Jaraba Street, 21821, Tripoli, Libya

    Mrwan Margem

  2. Department of Computer Engineering, Ankara Yildirim Beyazit University, 06220, Keçiören, Ankara, Turkey

    Osman S. Gedik

Authors
  1. Mrwan Margem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Osman S. Gedik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mrwan Margem.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1: Calculation details of the result comparison

See Table 6.

Table 6 The calculation details of minimum complexity results in Table 4 for 5-bit task using several approaches
Full size table

Appendix 2: Pseudo-code

Algorithm 2.1 is the Pseudo-code that has been used to create the matrix of training features \(A_{train}\) which is used in read-out stage to find \(\mathbf{W }_{out}\). The output weight matrix \(\mathbf{W }_{out}\) will be used to find the predicted output \(\hat{\mathbf{y }}_{train}\) in 5-bit task and \(\hat{\mathbf{y }}_{test}\) in generalized 5-bit task as explained in Sect. 4.2.

figure a

The algorithm 2.1 should be repeated for \(N_{train}\) training examples and placed the obtained CA_Train matrices consecutively to produce the matrix \(\mathbf{A }_{train}\) with size of (\(k\times L, N_{train}\times T\)). Then, we use Eq. (7) to find the output weight matrix \(\mathbf{W }_{out}\).

The line 12 in algorithm 2.1 should be replaced by the following line: InitialState = [CA_output(\(I, 1:R, i-1\)) CA_Input(\(i, R+1:R+L_{in}\)) CA_output(\(I, R+L_{in}+1:L, i-1\))] for the Overwrite insertion function.

The function (Concatenate) in line 15 of the algorithm 2.1 will be replaced by (XOR), (Binary), or (Gray) according to the used option.

To find \(\mathbf{A }_{test}\) in generalized 5-bit task, the algorithm 2.1 is used; where the Input_Train set is replaced by Input_Test set. Then, \(N_{test}\) examples will be used from the testing set.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Margem, M., Gedik, O.S. Feed-forward versus recurrent architecture and local versus cellular automata distributed representation in reservoir computing for sequence memory learning. Artif Intell Rev 53, 5083–5112 (2020). https://doi.org/10.1007/s10462-020-09815-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-020-09815-8

Keywords

Search

Navigation