Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Teaching recurrent neural networks to infer global temporal structure from local examples

Nature Machine Intelligence volume 3, pages 316–323 (2021)Cite this article

Subjects

Abstract

The ability to store and manipulate information is a hallmark of computational systems. Whereas computers are carefully engineered to represent and perform mathematical operations on structured data, neurobiological systems adapt to perform analogous functions without needing to be explicitly engineered. Recent efforts have made progress in modelling the representation and recall of information in neural systems. However, precisely how neural systems learn to modify these representations remains far from understood. Here, we demonstrate that a recurrent neural network (RNN) can learn to modify its representation of complex information using only examples, and we explain the associated learning mechanism with new theory. Specifically, we drive an RNN with examples of translated, linearly transformed or pre-bifurcated time series from a chaotic Lorenz system, alongside an additional control signal that changes value for each example. By training the network to replicate the Lorenz inputs, it learns to autonomously evolve about a Lorenz-shaped manifold. Additionally, it learns to continuously interpolate and extrapolate the translation, transformation and bifurcation of this representation far beyond the training data by changing the control signal. Furthermore, we demonstrate that RNNs can infer the bifurcation structure of normal forms and period doubling routes to chaos, and extrapolate non-dynamical, kinematic trajectories. Finally, we provide a mechanism for how these computations are learned, and replicate our main results using a Wilson–Cowan reservoir. Together, our results provide a simple but powerful mechanism by which an RNN can learn to manipulate internal representations of complex information, enabling the principled study and precise design of RNNs.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Representing chaotic attractors with reservoirs.
Fig. 2: Learning and extrapolating translations and transformations by example.
Fig. 3: Inferring and extrapolating the bifurcation of the Lorenz.
Fig. 4: Changing the control parameter changes the reservoir dynamics to manipulate representations.
Fig. 5: Inferring bifurcation normal forms and extrapolating kinematic trajectories.
Fig. 6: Flight of the Lorenz.

Similar content being viewed by others

Data availability

All data generated or analysed during this study can be found in the associated code.

Code availability

The code used to generate and analyse the data, and to produce the figures in the main text and the Supplementary Information, is available on CodeOcean at https://codeocean.com/capsule/2107188/tree/v1 (ref. 53).

References

  1. von Neumann, J. First draft of a report on the EDVAC. IEEE Ann. History Comput. 15, 27–75 (1993).

    Article  MathSciNet  MATH  Google Scholar 

  2. Alglave, J. et al. The semantics of power and ARM multiprocessor machine code. In Proc. 4th Workshop on Declarative Aspects of Multicore Programming—DAMP ’09, 13 (ACM Press, 2008).

  3. Zhang, Z., Jiao, Y.-Y. & Sun, Q.-Q. Developmental maturation of excitation and inhibition balance in principal neurons across four layers of somatosensory cortex. Neuroscience 174, 10–25 (2011).

    Article  Google Scholar 

  4. Faulkner, R. L. et al. Development of hippocampal mossy fiber synaptic outputs by new neurons in the adult brain. Proc. Natl Acad. Sci. USA 105, 14157–14162 (2008).

    Article  Google Scholar 

  5. Dunn, F. A. & Wong, R. O. L. Diverse strategies engaged in establishing stereotypic wiring patterns among neurons sharing a common input at the visual system’s first synapse. J. Neurosci. 32, 10306–10317 (2012).

    Article  Google Scholar 

  6. Craik, F. I. & Bialystok, E. Cognition through the lifespan: mechanisms of change. Trends Cogn. Sci. 10, 131–138 (2006).

    Article  Google Scholar 

  7. Tacchetti, A., Isik, L. & Poggio, T. A. Invariant recognition shapes neural representations of visual input. Annu. Rev. Vis. Sci. 4, 403–422 (2018).

    Article  Google Scholar 

  8. Moser, E. I., Kropff, E. & Moser, M.-B. Place cells, grid cells and the brain’s spatial representation system. Annu. Rev. Neurosci. 31, 69–89 (2008).

    Article  Google Scholar 

  9. Ifft, P. J., Shokur, S., Li, Z., Lebedev, M. A. & Nicolelis, M. A. L. A brain–machine interface enables bimanual arm movements in monkeys. Sci. Transl. Med. 5, 210ra154 (2013).

    Article  Google Scholar 

  10. Sainath, T. N. et al. Deep convolutional neural networks for large-scale speech tasks. Neural Netw. 64, 39–48 (2015).

    Article  Google Scholar 

  11. Jarrell, T. A. et al. The connectome of a decision-making neural network. Science 337, 437–444 (2012).

    Article  Google Scholar 

  12. Lee, J. & Tashev, I. High-level feature representation using recurrent neural network for speech emotion recognition. In Proc. Annual Conference of the International Speech Communication Association, INTERSPEECH, Vol. 2015, January 1537–1540 (ISCA, 2015).

  13. Wang, J., Narain, D., Hosseini, E. A. & Jazayeri, M. Flexible timing by temporal scaling of cortical responses. Nat. Neurosci. 21, 102–110 (2018).

    Article  Google Scholar 

  14. Weber, M., Maia, P. D. & Kutz, J. N. Estimating memory deterioration rates following neurodegeneration and traumatic brain injuries in a Hopfield network model. Front. Neurosci 11, 623 (2017).

    Article  Google Scholar 

  15. Burak, Y. & Fiete, I. R. Accurate path integration in continuous attractor network models of grid cells. PLoS Comput. Biol. 5, e1000291 (2009).

    Article  MathSciNet  Google Scholar 

  16. Yoon, K. et al. Specific evidence of low-dimensional continuous attractor dynamics in grid cells. Nat. Neurosci. 16, 1077–1084 (2013).

    Article  Google Scholar 

  17. Kumar, S., Dasgupta, I., Cohen, J. D., Daw, N. D. & Griffiths, T. L. Meta-learning of compositional task distributions in humans and machines. Preprint at https://arxiv.org/pdf/2010.02317.pdf (2020).

  18. Schweighofer, N. & Doya, K. Meta-learning in reinforcement learning. Neural Netw. 16, 5–9 (2003).

    Article  Google Scholar 

  19. Santiago, R. A. Context discerning multifunction networks: reformulating fixed weight neural networks. In 2004 IEEE International Joint Conference on Neural Networks Vol. 1, 189–194 (IEEE, 2004).

  20. Feldkamp, L. A., Puskorius, G. & Moore, P. Adaptive behavior from fixed weight networks. Inform. Sci. 98, 217–235 (1997).

    Article  Google Scholar 

  21. Hegarty, M. Mechanical reasoning by mental simulation. Trends Cogn. Sci. 8, 280–285 (2004).

    Article  Google Scholar 

  22. Kubricht, J. R., Holyoak, K. J. & Lu, H. Intuitive physics: current research and controversies. Trends Cogn. Sci. 21, 749–759 (2017).

    Article  Google Scholar 

  23. Pfeiffer, B. E. & Foster, D. J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74–79 (2013).

    Article  Google Scholar 

  24. Gold, J. I. & Shadlen, M. N. The neural basis of decision making. Annu. Rev. Neurosci. 30, 535–574 (2007).

    Article  Google Scholar 

  25. Strogatz, S. H. Nonlinear Dynamics and Chaos 1st edn (Perseus Books, 1994).

  26. Yang, J., Wang, L., Wang, Y. & Guo, T. A novel memristive Hopfield neural network with application in associative memory. Neurocomputing 227, 142–148 (2017).

    Article  Google Scholar 

  27. Wu, S., Wong, K. Y. M., Fung, C. C. A., Mi, Y. & Zhang, W. Continuous attractor neural networks: candidate of a canonical model for neural information representation. F1000Res. 5, 156 (2016).

    Article  Google Scholar 

  28. Eliasmith, C. & Anderson, C. H. Neural Engineering: Computation, Representation and Dynamics in Neurobiological Systems (MIT Press, 2003).

  29. Bekolay, T. et al. Nengo: a Python tool for building large-scale functional brain models. Front. Neuroinf. 7, 48 (2014).

    Article  Google Scholar 

  30. Graves, A. et al. Hybrid computing using a neural network with dynamic external memory. Nature 538, 471–476 (2016).

    Article  Google Scholar 

  31. Carroll, J. M. Letter knowledge precipitates phoneme segmentation, but not phoneme invariance. J. Res. Reading 27, 212–225 (2004).

    Article  Google Scholar 

  32. Fee, M. S. & Scharff, C. The songbird as a model for the generation and learning of complex sequential behaviors. ILAR J. 51, 362–377 (2010).

    Article  Google Scholar 

  33. Donnay, G. F., Rankin, S. K., Lopez-Gonzalez, M., Jiradejvong, P. & Limb, C. J. Neural substrates of interactive musical improvisation: an FMRI study of trading fours in jazz. PLoS ONE 9, e88665 (2014).

    Article  Google Scholar 

  34. Qiao, J., Li, F., Han, H. & Li, W. Growing echo-state network with multiple subreservoirs. IEEE Trans. Neural Netw. Learn. Syst. 28, 391–404 (2017).

    Article  Google Scholar 

  35. Lorenz, E. N. Deterministic nonperiodic flow. J. Atmos. Sci. 20, 130–141 (1963).

    Article  MathSciNet  MATH  Google Scholar 

  36. Jaeger, H. The echo state approach to analysing and training recurrent neural networks—with an erratum note. GMD Rep. 1, 1–47 (2010).

    Google Scholar 

  37. Sussillo, D. & Abbott, L. Generating coherent patterns of activity from chaotic neural networks. Neuron 63, 544–557 (2009).

    Article  Google Scholar 

  38. Tyukin, I. Y., Prokhorov, D. & Van Leeuwen, C. Adaptive classification of temporal signals in fixed-weight recurrent neural networks: an existence proof. Neural Comput. 20, 2564–2596 (2008).

    Article  MathSciNet  MATH  Google Scholar 

  39. Klos, C., Kossio, Y. F. K., Goedeke, S., Gilra, A. & Memmesheimer, R.-M. Dynamical learning of dynamics. Phys. Rev. Lett. 125, 088103 (2020).

    Article  Google Scholar 

  40. Słowik, R. Inverses and determinants of Toeplitz–Hessenberg matrices. Taiwan. J. Math. 22, 901–908 (2018).

    Article  MathSciNet  MATH  Google Scholar 

  41. Nansai, S., Elara, M. R. & Iwase, M. Dynamic analysis and modeling of Jansen mechanism. Proc. Eng. 64, 1562–1571 (2013).

    Article  Google Scholar 

  42. Seung, H. S. Learning continuous attractors in recurrent networks. In Advances in Neural Information Processing Systems 654–660 (MIT Press, 1998).

  43. Rulkov, N. F., Sushchik, M. M., Tsimring, L. S. & Abarbanel, H. D. I. Generalized synchronization of chaos in directionally coupled chaotic systems. Phys. Rev. E 51, 980–994 (1995).

    Article  Google Scholar 

  44. Ott, E., Grebogi, C. & Yorke, J. A. Controlling chaos. Phys. Rev. Lett. 64, 1196–1199 (1990).

    Article  MathSciNet  MATH  Google Scholar 

  45. Wilson, H. R. & Cowan, J. D. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys. J. 12, 1–24 (1972).

    Article  Google Scholar 

  46. Zurn, P., Bassett, D. S. & Rust, N. C. The citation diversity statement: a practice of transparency, a way of life. Trends Cogn. Sci. 24, 669–672 (2020).

    Article  Google Scholar 

  47. Dworkin, J. D. et al. The extent and drivers of gender imbalance in neuroscience reference lists. Nat. Neurosci. 23, 918–926 (2020).

    Article  Google Scholar 

  48. Maliniak, D., Powers, R. & Walter, B. F. The gender citation gap in international relations. Int. Organ. 67, 889–922 (2013).

    Article  Google Scholar 

  49. Caplar, N., Tacchella, S. & Birrer, S. Quantitative evaluation of gender bias in astronomical publications from citation counts. Nat. Astron. 1, 0141 (2017).

    Article  Google Scholar 

  50. Mitchell, S. M., Lange, S. & Brus, H. Gendered citation patterns in international relations journals. Int. Studies Perspect. 14, 485–492 (2013).

    Article  Google Scholar 

  51. Dion, M. L., Sumner, J. L. & Mitchell, S. M. Gendered citation patterns across political science and social science methodology fields. Political Anal. 26, 312–327 (2018).

    Article  Google Scholar 

  52. Zhou, D. et al. Gender diversity statement and code notebook v1.0 (2020); https://doi.org/10.5281/zenodo.3672110

  53. Kim, J. Z., Lu, Z., Nozari, E., Pappas, G. J. & Bassett, D. S. Teaching recurrent neural networks to infer global temporal structure from local examples (2020); https://codeocean.com/capsule/2107188/tree/v1

  54. Langer, G. & Parlitz, U. Modeling parameter dependence from time series. Phys. Rev. E 70, 056217 (2004).

    Article  Google Scholar 

Download references

Acknowledgements

We are thankful for the insightful feedback and comments from H. Ju and K. Wiley. We acknowledge support from the John D. and Catherine T. MacArthur Foundation, the Alfred P. Sloan Foundation, the ISI Foundation, the Paul Allen Foundation, the Army Research Laboratory (grant no. W911NF-10-2-0022), the Army Research Office (grants nos. Bassett-W911NF-14-1-0679, Grafton-W911NF-16-1-0474 and DCIST-W911NF-17-2-0181), the Office of Naval Research (ONR), the National Institute of Mental Health (grants nos. 2-R01-DC-009209-11, R01-MH112847, R01-MH107235 and R21-M MH-106799), the National Institute of Child Health and Human Development (grant no. 1R01HD086888-01), the National Institute of Neurological Disorders and Stroke (grant no. R01 NS099348) and the National Science Foundation (NSF; grants nos. DGE-1321851, NSF PHY-1554488 and BCS-1631550). The content is solely the responsibility of the authors and does not necessarily represent the official views of any of the funding agencies.

Author information

Authors and Affiliations

  1. Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA

    Jason Z. Kim & Zhixin Lu

  2. Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, USA

    Erfan Nozari & George J. Pappas

  3. Department of Mechanical Engineering, University of California, Riverside, CA, USA

    Erfan Nozari

  4. Departments of Bioengineering, Physics & Astronomy, Electrical & Systems Engineering, Neurology, and Psychiatry, University of Pennsylvania, Philadelphia, PA, USA

    Danielle S. Bassett

  5. Santa Fe Institute, Santa Fe, NM, USA

    Danielle S. Bassett

Authors
  1. Jason Z. Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhixin Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Erfan Nozari
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. George J. Pappas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Danielle S. Bassett
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J.Z.K. conceived the initial idea and developed the analyses in conversation with Z.L., E.N., G.J.P. and D.S.B. J.Z.K. and D.S.B. prepared the manuscript with feedback from Z.L., E.N. and G.J.P. All authors contributed to discussions and approved the manuscript.

Corresponding author

Correspondence to Danielle S. Bassett.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Machine Intelligence thanks Wilten Nicola and the other, anonymous, reviewer(s), for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplemental Figs. 1–7, Table 1, derivations, discussion and Methods.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, J.Z., Lu, Z., Nozari, E. et al. Teaching recurrent neural networks to infer global temporal structure from local examples. Nat Mach Intell 3, 316–323 (2021). https://doi.org/10.1038/s42256-021-00321-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-021-00321-2

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing