Abstract
Tensor contexts enlarge the performances and computational powers of many neural models of language by generating a double filtering of incoming data. Applied to the linguistic domain, its implementation enables a very efficient disambiguation of polysemous and homonymous words. For the neurocomputational modeling of language, the simultaneous tensor contextualization of inputs and outputs inserts into the models strategic passwords that rout words towards key natural targets, thus allowing for the creation of meaningful phrases. In this work, we present the formal properties of these models and describe possible ways to use contexts to represent plausible neural organizations of sequences of words. We include an illustration of how these contexts generate topographic or thematic organization of data. Finally, we show that double contextualization opens promising ways to explore the neural coding of episodes, one of the most challenging problems of neural computation.
Gradually, it saw itself (like us) imprisoned in this sonorous web of Before, After, Yesterday, While, Now, Right, Left, Me, You, Those, Others. From “The Golem” by J.L. Borges
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Luria, A.R.: The Working Brain. Basic Books, New York City (1973)
Kimura, D.: Neuromotor mechanisms in the evolution of human communication. In: Steklis, H.D., Raleigh, M.J. (eds.) Neurobiology of Social Communication in Primates, pp. 197–219. Academic Press, New York (1979)
Calvin, W.H.: A stone’s throw and its launch window: timing precision and its implications for language and hominid brains. J. Theor. Biol. 104, 121–135 (1983)
Calvin, W.H.: The unitary hypothesis: a common neural circuitry for novel manipulations, language, plan-ahead, and throwing? In: Gibson, K.R., Ingold, T. (eds.) Tools, Language, and Cognition in Human Evolution, pp. 230–250. Cambridge University Press, Cambridge (1993)
Ojemann, G.A.: Brain organization for language from the perspective of electrical stimulation mapping. Behav. Brain Sci. 6, 189–206 (1983)
Anderson, J.A.: A simple neural network generating an interactive memory. Math. Biosci. 14, 197–220 (1972)
Anderson, J.A.: An introduction to neural networks. MIT Press, Cambridge (1995)
Cooper, L.N.: A possible organization of animal memory and learning. In: Lundquist, B., Lundquist, S. (eds.) Proceedings of the Nobel Symposium on Collective Properties of Physical Systems, pp. 252–264. Academic Press, New York (1973)
Kohonen, T.: Correlation matrix memories. IEEE Trans. Comput. C-21, 353–359 (1972)
Kohonen, T.: Associative Memory: A System Theoretical Approach. Springer, Heidelberg (1977). https://doi.org/10.1007/978-3-642-96384-1. Chap. 3
Beim Graben, P., Potthast, R.: Inverse problems in dynamic cognitive modeling. Chaos Interdiscip. J. Nonlinear Sci. 19, 015103 (2009)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986)
Carmantini, G.S., Beim Graben, P., Desroches, M., Rodrigues, S.: A modular architecture for transparent computation in Recurrent Neural Networks. Neural Netw. 85, 85–107 (2017)
Mizraji, E.: Context-dependent associations in linear distributed memories. Bull. Math. Biol. 51, 195–205 (1989)
Smolensky, P.: Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artif. Intell. 46, 159–216 (1990)
Graham, A.: Kronecker Products and Matrix Calculus With Applications. Ellis Horwood, Chichester (1981)
Pomi, A., Mizraji, E.: Semantic graphs and associative memories. Phys. Rev. E 70, 066136 (2004)
Pomi, A.: A possible neural representation of mathematical group structures. Bull. Math. Biol. 78, 1847–1865 (2016)
Mizraji, E., Pomi, A., Valle-Lisboa, J.C.: Dynamic searching in the brain. Cogn. Neurodyn. 3, 401–414 (2009)
Mizraji, E., Lin, J.: Modeling spatial-temporal operations with context-dependent associative memories. Cognit. Neurodyn. 9, 523–534 (2015)
James, W.: Principles of Psychology. The Great Books of the Western World, vol. 53. The University of Chicago (1890)
Nishitani, N., Schürmann, M., Amunts, K., Har, R.: Broca’s region: from action to language. Physiology 20, 60–69 (2005)
Jurafsky, D., Bell, A., Gregory, M., Raymond, W.D.: Probabilistic relations between words: evidence from reduction in lexical production. Typol. Stud. Lang. 45, 229–254 (2001)
Jurafsky, D.: Probabilistic modeling in psycholinguistics: linguistic comprehension and production. In: Bod, R., Hay, J., Jannedy, S. (eds.) Probabilistic Linguistics, p. 21. MIT Press, Cambridge (2003). Chap. 3
Nowak, M.A., Komarova, N.L., Niyogi, P.: Computational and evolutionary aspects of language. Nature 417, 611–617 (2002)
Chater, N., Manning, C.D.: Probabilistic models of language processing and acquisition. Trends Cognit. Sci. 10, 335–344 (2006)
Kohonen, T.: Self-Organizing Maps. Springer, Heidelberg (1997). https://doi.org/10.1007/978-3-642-97966-8
Huth, A.G., Nishimoto, S., Vu, A.T., Gallant, J.L.: A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 1210–1224 (2012)
Tulving, E.: Episodic memory. Annu. Rev. Psychol. 53, 1–25 (2002)
Baddeley, A.: Working memory: looking back and looking forward. Nat. Rev. Neurosci. 4, 829–839 (2003)
Jonides, J.R., et al.: The mind and brain of short-term memory. Ann. Rev. Psychol. 59, 193–224 (2008)
Repovs, G., Baddeley, A.: The multi-component model of working memory: explorations in experimental cognitive psychology. Neuroscience 139, 5–21 (2006)
Eichenbaum, H.: Prefrontal–hippocampal interactions in episodic memory. Nature Rev. Neurosci. 18, 547–558 (2017)
Schapiro, A.C., Turk-Browne, N.B., Botvinick, M.M., Norman, K.A.: Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning. Philos. Trans. R. Soc. Lond B 372, 20160049 (2017)
Schacter, D.L., et al.: The future of memory: remembering, imagining, and the brain. Neuron 76, 677–694 (2012)
Schacter, D.L., Benoit, R.G., Szpunar, K.K.: Episodic future thinking: mechanisms and functions. Curr. Opin. Behav. Sci. 17, 41–50 (2017)
Acknowledgments
AP and EM acknowledge partial financial support by PEDECIBA and CSIC-UdelaR.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Mizraji, E., Pomi, A., Lin, J. (2018). Improving Neural Models of Language with Input-Output Tensor Contexts. In: Karpov, A., Jokisch, O., Potapova, R. (eds) Speech and Computer. SPECOM 2018. Lecture Notes in Computer Science(), vol 11096. Springer, Cham. https://doi.org/10.1007/978-3-319-99579-3_45
Download citation
DOI: https://doi.org/10.1007/978-3-319-99579-3_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99578-6
Online ISBN: 978-3-319-99579-3
eBook Packages: Computer ScienceComputer Science (R0)