Abstract
Human intelligence consists largely of the ability to recognize and exploit structural systematicity in the world, relating our senses simultaneously to each other and to our cognitive state. Language abilities, in particular, require a learned mapping between the linguistic input and one’s internal model of the real world. In order to demonstrate that connectionist methods excel at this task, we teach a deep, recurrent neural network—a variant of the long short-term memory (LSTM)—to ground language in a micro-world. The network integrates two inputs—a visual scene and an auditory sentence—to produce the meaning of the sentence in the context of the scene. Crucially, the network exhibits strong systematicity, recovering appropriate meanings even for novel objects and descriptions. With its ability to exploit systematic structure across modalities, this network fulfills an important prerequisite of general machine intelligence.
Keywords
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Goldberg, A.E.: Learning Linguistic Patterns. The Psychology of Learning and Motivation 47, 33–63 (2006)
Turing, A.M.: Computing Machinery and Intelligence. Mind 59(236), 433–460 (1950)
Searle, J.R.: Minds, Brains, and Programs. Behavioral and Brain Sciences 3(03), 417–457 (1980)
Harnad, S.: The Symbol Grounding Problem. Physica D: Nonlinear Phenomena 42(1-3), 335–346 (1990)
Hochreiter, S., Schmidhuber, J.: Long Short-Term Memory. Neural Computation 9(8), 1735–1780 (1997)
Gers, F.A., Cummins, F.: Learning to Forget: Continual Prediction with LSTM. Neural Computation 12(10), 2451–2471 (2000)
Hadley, R.: Systematicity in Connectionist Language Learning. Mind & Language (1994)
Feldman, J.A., Lakoff, G., Stolcke, A., Weber, S.H.: Miniature Language Acquisition: A Touchstone for Cognitive Science. In: Proceedings of the Annual Conference of the Cognitive Science Society, pp. 686–693 (1990)
Hadley, R.F., Cardei, V.C.: Language Acquisition from Sparse Input Without Error Feedback. Neural Networks 12(2), 217–235 (1999)
Riga, T., Cangelosi, A., Greco, A.: Symbol Grounding Transfer with Hybrid Self-Organizing/Supervised Neural Networks. In: Proceedings of the International Joint Conference on Neural Networks, pp. 2865–2869 (2004)
Williams, P., Miikkulainen, R.: Grounding Language in Descriptions of Scenes. In: Proceedings of the 8th Annual Conference of the Cognitive Science Society, pp. 2381–2386 (2006)
Kohonen, T.: The Self-Organizing Map. Proceedings of the IEEE 78, 1464–1480 (1990)
Frank, S.L., Haselager, W.F.G., van Rooij, I.: Connectionist Semantic Systematicity. Cognition 110(3), 358–379 (2009)
Elman, J.L.: Finding Structure in Time. Cognitive Science 14, 179–211 (1990)
Weems, S.A., Reggia, J.A.: Simulating Single Word Processing in the Classic Aphasia Syndromes Based on the Wernicke-Lichtheim-Geschwind Theory. Brain and Language 98(3), 291–309 (2006)
Tomasello, M.: Constructing a Language: A Usage-Based Theory of Language Acquisition. Harvard University Press, Cambridge (2003)
Monner, D.D., Reggia, J.A.: A Generalized LSTM-Like Training Algorithm for Second-Order Recurrent Neural Networks (2011) (submitted)
Xie, X., Seung, H.S.: Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network. Neural Computation 15(2), 441–454 (2003)
O’Reilly, R.C.: Generalization in Interactive Networks: The Benefits of Inhibitory Competition and Hebbian Learning. Neural Computation 13(6), 1199–1241 (2001)
O’Reilly, R.C., Frank, M.J.: Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia. Neural Computation 18(2), 283–328 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Monner, D.D., Reggia, J.A. (2011). Systematically Grounding Language through Vision in a Deep, Recurrent Neural Network. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds) Artificial General Intelligence. AGI 2011. Lecture Notes in Computer Science(), vol 6830. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22887-2_12
Download citation
DOI: https://doi.org/10.1007/978-3-642-22887-2_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22886-5
Online ISBN: 978-3-642-22887-2
eBook Packages: Computer ScienceComputer Science (R0)