Abstract
Recent “visual worlds” studies, wherein researchers study language in context by monitoring eye-movements in a visual scene during sentence processing, have revealed much about the interaction of diverse information sources and the time course of their influence on comprehension. In this study, five experiments that trade off scene context with a variety of linguistic factors are modelled with a Simple Recurrent Network modified to integrate a scene representation with the standard incremental input of a sentence. The results show that the model captures the qualitative behavior observed during the experiments, while retaining the ability to develop the correct interpretation in the absence of visual input.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Pinker, S.: How could a child use verb syntax to learn verb semantics? In: Gleitman, L., Landau, B. (eds.) The acquisition of the lexicon, pp. 377–410. MIT Press, Cambridge (1994)
Fisher, C., Hall, D.G., Rakowitz, S., Gleitman, L.: When it is better to receive than to give: Syntactic and conceptual constraints on vocabulary growth. In: Gleitman, L., Landau, B. (eds.) The acquisition of the lexicon, pp. 333–375. MIT Press, Cambridge (1994)
Elman, J.L.: Finding structure in time. Cognitive Science 14, 179–211 (1990)
Redington, M., Chater, N., Finch, S.: Distributional information: A powerful cue for acquiring syntactic categories. Cognitive Science 22, 425–469 (1998)
Roy, D., Pentland, A.: Learning words from sights and sounds: A computational model. Cognitive Science 26(1), 113–146 (2002)
Knoeferle, P., Crocker, M.W.: Stored knowledge versus depicted events: what guides auditory sentence comprehension. In: Proceedings of the 26th Annual Conference of the Cognitive Science Society, Mahawah, NJ, Erlbaum, pp. 714–719 (2004)
Knoeferle, P., Crocker, M.W.: The coordinated interplay of scene, utterance, and world knowledge: evidence from eye-tracking (submitted)
Kamide, Y., Scheepers, C., Altmann, G.T.M.: Integration of syntactic and semantic information in predictive processing: Cross-linguistic evidence from German and English. Journal of Psycholinguistic Research 32(1), 37–55 (2003)
Knoeferle, P., Crocker, M.W., Scheepers, C., Pickering, M.J.: The influence of the immediate visual context on incremental thematic role-assignment: evidence from eye-movements in depicted events. Cognition 95, 95–127 (2005)
Miikkulainen, R.: Natural language processing with subsymbolic neural networks. In: Browne, A. (ed.) Neural Network Perspectives on Cognition and Adaptive Robotics, pp. 120–139. Institute of Physics Publishing, Bristol (1997)
Knoeferle, P., Crocker, M.W.: The coordinated processing of scene and utterance: evidence from eye-tracking in depicted events. In: Proceedings of International Conference on Cognitive Science, Allahabad, India (2004)
Roy, D., Mukherjee, N.: Towards situated speech understanding: Visual context priming of language models. Computer Speech and Language 19(2), 227–248 (2005)
Elman, J.L., Bates, E.A., Johnson, M.H., Karmiloff-Smith, A., Parisi, D., Plunkett, K.: Rethinking Innateness: A Connectionist Perspective on Development. MIT Press, Cambridge (1996)
Mayberry, M.R., Crocker, M.W.: Generating semantic graphs through self-organization. In: Proceedings of the AAAI Symposium on Compositional Connectionism in Cognitive Science, Washington, D.C., pp. 40–49 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mayberry, M.R., Crocker, M.W., Knoeferle, P. (2005). A Connectionist Model of Anticipation in Visual Worlds. In: Dale, R., Wong, KF., Su, J., Kwong, O.Y. (eds) Natural Language Processing – IJCNLP 2005. IJCNLP 2005. Lecture Notes in Computer Science(), vol 3651. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11562214_74
Download citation
DOI: https://doi.org/10.1007/11562214_74
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29172-5
Online ISBN: 978-3-540-31724-1
eBook Packages: Computer ScienceComputer Science (R0)