Skip to main content
SpringerLink
Log in

Template Construction Grammar: From Visual Scene Description to Language Comprehension and Agrammatism

  • Original Article
  • Published:
Neuroinformatics Aims and scope Submit manuscript

Abstract

How does the language system coordinate with our visual system to yield flexible integration of linguistic, perceptual, and world-knowledge information when we communicate about the world we perceive? Schema theory is a computational framework that allows the simulation of perceptuo-motor coordination programs on the basis of known brain operating principles such as cooperative computation and distributed processing. We present first its application to a model of language production, SemRep/TCG, which combines a semantic representation of visual scenes (SemRep) with Template Construction Grammar (TCG) as a means to generate verbal descriptions of a scene from its associated SemRep graph. SemRep/TCG combines the neurocomputational framework of schema theory with the representational format of construction grammar in a model linking eye-tracking data to visual scene descriptions. We then offer a conceptual extension of TCG to include language comprehension and address data on the role of both world knowledge and grammatical semantics in the comprehension performances of agrammatic aphasic patients. This extension introduces a distinction between heavy and light semantics. The TCG model of language comprehension offers a computational framework to quantitatively analyze the distributed dynamics of language processes, focusing on the interactions between grammatical, world knowledge, and visual information. In particular, it reveals interesting implications for the understanding of the various patterns of comprehension performances of agrammatic aphasics measured using sentence-picture matching tasks. This new step in the life cycle of the model serves as a basis for exploring the specific challenges that neurolinguistic computational modeling poses to the neuroinformatics community.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Aine, C. J., Sanfratello, L., Ranken, D., Best, E., MacArthur, J. A., et al. (2012). MEG-SIM: a web portal for testing MEG analysis methods using realistic simulated and empirical data. Neuroinformatics, 10, 141–158.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs: restricting the domain of subsequent reference. Cognition, 73, 247–264.

    Article  CAS  PubMed  Google Scholar 

  • Ansell, B. J., & Flowers, C. R. (1982). Aphasic adults’ use of heuristic and structural linguistic cues for sentence analysis. Brain and Language, 16, 61–72.

    Article  CAS  PubMed  Google Scholar 

  • Arbib, M. A. (1981). Perceptual structures and distributed motor control. In V. B. Brooks (Ed.), Handbook of physiology — The nervous system II. Motor control (pp. 1449–1480). Bethesda: American Physiological Society.

    Google Scholar 

  • Arbib, M. A. (2012). How the brain got language: the mirror system hypothesis. New York & Oxford: Oxford University Press.

    Google Scholar 

  • Arbib, M. A., & Bota, M. (2003). Language evolution: neural homologies and neuroinformatics. Neural Networks: The Official Journal of the International Neural Network Society, 16, 1237–1260.

    Article  Google Scholar 

  • Arbib, M. A., & Caplan, D. (1979). Neurolinguistics must be computational. The Behavioral and Brain Sciences, 2, 449–483.

    Article  Google Scholar 

  • Arbib, M. A., & Lee, J. Y. (2008). Describing visual scenes: towards a neurolinguistics based on construction grammar. Brain Research, 1225, 146–162.

    Article  CAS  PubMed  Google Scholar 

  • Arbib, M. A., Conklin, E. J., & Hill, J. C. (1987). From schema theory to language. New York: Oxford University Press. x + 253 pp.

    Google Scholar 

  • Arbib, M. A., Billard, A., Iacoboni, M., & Oztop, E. (2000). Synthetic brain imaging: grasping, mirror neurons and imitation. Neural Networks: The Official Journal of the International Neural Network Society, 13, 975–997.

    Article  CAS  Google Scholar 

  • Arbib, M. A., Plangprasopchok, A., Bonaiuto, J. J., Schuler, R. E. (2013). A neuroinformatics of brain modeling and its implementation in the Brain Operation Database BODB. Neuroinformatics, in press.

  • Baker, C., Fillmore, C. J., Lowe, J. B. (1998). The {B}erkeley {F}rame{N}et project.

  • Bakker, R., Wachtler, T., Diesmann, M. (2012). CoCoMac 2.0 and the future of tract-tracing databases. Frontiers in Neuroinformatics, 6.

  • Barrès, V., Simons, A., & Arbib, M. A. (2013). Synthetic event-related potentials: a computational bridge between neurolinguistic models and experiments. Neural Networks, 37, 66–92.

    Article  PubMed  Google Scholar 

  • Bergen B. K, Chang N. (2005a). Embodied construction grammar in simulation-based language understanding. In Construction Grammars: cognitive grounding and theoretical extensions.

  • Bergen, B. K., & Chang, N. (2005b). Embodied construction grammar in simulation-based language understanding. In J.-O. OÖstman & M. Fried (Eds.), Construction grammar(s): cognitive and cross-language dimensions (pp. 147–190). Amsterdam: John Benjamins.

    Google Scholar 

  • Berndt, R. S., & Caramazza, A. (1999). How “regular” is sentence comprehension in Broca's aphasia? It depends on how you select the patients. Brain and Language, 67, 242–247.

    Article  CAS  PubMed  Google Scholar 

  • Berndt, R. S., Mitchum, C. C., & Haendiges, A. N. (1996). Comprehension of reversible sentences in “agrammatism”: a meta-analysis. Cognition, 58, 289–308.

    Article  CAS  PubMed  Google Scholar 

  • Bornkessel, I., & Schlesewsky, M. (2006). The extended argument dependency model: a neurocognitive approach to sentence comprehension across languages. Psychological Review, 113, 787–821.

    Article  PubMed  Google Scholar 

  • Caramazza, A., & Zurif, E. B. (1976). Dissociation of algorithmic and heuristic processes in language comprehension: evidence from aphasia. Brain and Language, 3, 572–582.

    Article  CAS  PubMed  Google Scholar 

  • Caramazza, A., Capasso, R., Capitani, E., & Miceli, G. (2005). Patterns of comprehension performance in agrammatic Broca’s aphasia: a test of the trace deletion hypothesis. Brain and Language, 94, 43–53.

    Article  PubMed  Google Scholar 

  • Chambers, C. G., Tanenhaus, M. K., & Magnuson, J. S. (2004). Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology Learning, Memory, and Cognition, 30, 687–696.

    Article  PubMed  Google Scholar 

  • Chen, R., Hillis, A. E., Pawlak, M., & Herskovits, E. H. (2008). Voxelwise Bayesian lesion-deficit analysis. NeuroImage, 40, 1633–1642.

    Article  PubMed Central  PubMed  Google Scholar 

  • Christianson, K., & Luke, S. G. (2011). Context strengthens initial misinterpretations of text. Scientific Studies of Reading, 15, 136–166.

    Article  Google Scholar 

  • Christianson, K., Hollingworth, A., Halliwell, J. F., & Ferreira, F. (2001). Thematic roles assigned along the garden path linger. Cognitive Psychology, 42, 368–407.

    Article  CAS  PubMed  Google Scholar 

  • Croft, W. (2001). Radical construction grammar: syntactic theory in typological perspective. Oxford: Oxford University Press.

    Book  Google Scholar 

  • Croft, W., & Cruse, D. A. (2005). Cognitive linguistics. Cambridge: Cambridge University Press.

    Google Scholar 

  • David, O., Kiebel, S. J., Harrison, L. M., Mattout, J., Kilner, J. M., & Friston, K. J. (2006). Dynamic causal modeling of evoked responses in EEG and MEG. NeuroImage, 30, 1255–1272.

    Article  PubMed  Google Scholar 

  • De Beule, J., & Steels, L. (2005). Hierarchy in fluid construction grammar. In U. Furbach (Ed.), Proceedings of the 28th annual German conference on AI, KI 2005, lecture notes in artificial intelligence (vol. 3698) (pp. 1–15). Berlin, Heidelberg: Springer.

    Google Scholar 

  • Dominey, P. F., & Boucher, J.-D. (2005). Learning to talk about events from narrated video in a construction grammar framework. Artificial Intelligence, 167, 31–61.

    Article  Google Scholar 

  • Dominey, P. F., Hoen, M., & Inui, T. (2006a). A neurolinguistic model of grammatical construction processing. Journal of Cognitive Neuroscience, 18, 2088–2107.

    Article  PubMed  Google Scholar 

  • Dominey, P. F., Hoen, M., & Inui, T. (2006b). A neurolinguistic model of grammatical construction processing. Journal of Cognitive Neuroscience, 18, 2088–2107.

    Article  PubMed  Google Scholar 

  • Dou, D., Frishkoff, G., Rong, J., Frank, R., Malony, A., & Tucker, D. (2007). Development of NeuroElectroMagnetic Ontologies(NEMO): a framework for mining brainwave ontologies. New York: Assoc Computing Machinery.

    Google Scholar 

  • Draper, B. A., Collins, R. T., Brolio, J., Hanson, A. R., & Riseman, E. M. (1989). The schema system. International Journal of Computer Vision, 2, 209–250.

    Article  Google Scholar 

  • Evans. A. C., Collins, D. L., Mills, S. R., Brown, E. D., Kelly, R. L., Peters, T. M. (1993). 1813–17 vol.3-13 -17 vol.3.

  • Fellbaum, C. (2010). WordNet. In R. Poli, M. Healy, & A. Kameas (Eds.), Theory and applications of ontology: computer applications (pp. 231–43). Springer: Netherlands.

    Chapter  Google Scholar 

  • Ferreira, F. (2003). The misinterpretation of noncanonical sentences. Cognitive Psychology, 47, 164–203.

    Article  PubMed  Google Scholar 

  • Ferreira, F., & Patson, N. D. (2007). The ‘good enough’ approach to language comprehension. Language and Linguistics Compass, 1, 71–83.

    Article  Google Scholar 

  • Fox, P. T., & Lancaster, J. L. (2002). Mapping context and content: the BrainMap model. Nature Reviews Neuroscience, 3, 319–321.

    Article  CAS  PubMed  Google Scholar 

  • Fox, P. T., Laird, A. R., Fox, S. P., Fox, P. M., Uecker, A. M., et al. (2005). Brainmap taxonomy of experimental design: description and evaluation. Human Brain Mapping, 25, 185–198.

    Article  PubMed  Google Scholar 

  • Frazier, L., & Fodor, J. D. (1978). The sausage machine: a new two-stage parsing model. Cognition, 6, 291–325.

    Article  Google Scholar 

  • Friederici, A. D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6, 78–84.

    Article  PubMed  Google Scholar 

  • Friederici, A. D. (2009). Pathways to language: fiber tracts in the human brain. Trends in Cognitive Sciences, 13, 175–181.

    Article  PubMed  Google Scholar 

  • Gleitman, L. R., January, D., Nappa, R., & Trueswell, J. C. (2007). On the give and take between even apprehension and utterance formulation. Journal of Memory and Language, 57, 544–569.

    Article  PubMed Central  PubMed  Google Scholar 

  • Goldberg, A. E. (1995). Constructions: a construction grammar approach to argument structure. Chicago: The University of Chicago Press.

    Google Scholar 

  • Goodglass, H. (1976). Agrammatism. Studies in Neurolinguistics, 1, 237–260.

    Google Scholar 

  • Grodzinsky, Y. (2000). The neurology of syntax: language use without Broca’s area. The Behavioral and Brain Sciences, 23, 1–21.

    Article  CAS  PubMed  Google Scholar 

  • Grodzinsky, Y., Piñango, M. M., Zurif, E., & Drai, D. (1999). The critical role of group studies in neuropsychology: comprehension regularities in Broca’s Aphasia. Brain and Language, 67, 134–147.

    Article  CAS  PubMed  Google Scholar 

  • Hagoort, P. (2005). On Broca, brain, and binding: a new framework. Trends in Cognitive Sciences, 9, 416–423.

    Article  PubMed  Google Scholar 

  • Hanson, A. R., & Riseman, E. M. (1978). VISIONS: a computer system for interpreting scenes. In A. R. Hanson & E. M. Riseman (Eds.), Computer vision systems (pp. 129–163). New York: Academic.

    Google Scholar 

  • Hawkins, J. A. (1999). Processing complexity and Filler-Gap dependencies across grammars. Language & Cognitive Processes, 75, 244–285.

    Google Scholar 

  • Hickok, G., & Poeppel, D. (2004). Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition, 92, 67–99.

    Article  PubMed  Google Scholar 

  • Hurford, J. R. (2011). The origins of grammar II: language in the light of evolution. Oxford, New York: Oxford University Press.

    Google Scholar 

  • Kay, P. (2002). An informal sketch of the formal architecture of construction grammar. Grammars, 5, 1–19.

    Article  Google Scholar 

  • Kay, P., Fillmore, C. J. (1999). Grammatical constructions and linguistic generalizations: the what’s X doing Y? Construction.

  • Kemmerer, D. (2000a). Grammatically relevant and grammatically irrelevant features of verb meaning can be independently impaired. Aphasiology, 14, 997–1020.

    Article  Google Scholar 

  • Kemmerer, D. (2000b). Selective impairment of knowledge underlying prenominal adjective order: evidence for the autonomy of grammatical semantics. Journal of Neurolinguistics, 13, 57–82.

    Article  Google Scholar 

  • Kemmerer, D. (2003). Why can you hit someone on the arm but not break someone on the arm?—a neuropsychological investigation of the English body-part possessor ascension construction. Journal of Neurolinguistics, 16, 13–36.

    Article  Google Scholar 

  • Kemmerer, D., & Wright, S. K. (2002). Selective impairment of knowledge underlying un- prefixation: further evidence for the autonomy of grammatical semantics. Journal of Neurolinguistics, 15, 403–432.

    Article  Google Scholar 

  • Kemmerer, D., Tranel, D., & Zdanczyk, C. (2009). Knowledge of the semantic constraints on adjective order can be selectively impaired. Journal of Neurolinguistics, 22, 91–108.

    Article  PubMed Central  PubMed  Google Scholar 

  • Kempen, G., Olsthoorn, N., & Sprenger, S. (2012). Grammatical workspace sharing during language production and language comprehension: evidence from grammatical multitasking. Language & Cognitive Processes, 27, 345–380.

    Article  Google Scholar 

  • Kim, & Osterhout. (2005). The independence of combinatory semantic processing: evidence from event-related potentials. Journal of Memory and Language, 52, 205–225.

    Article  Google Scholar 

  • Kos, M., Vosse, T., Dvd, B., & Hagoort, P. (2010). About edible restaurants: conflicts between syntax and semantics as revealed by ERPs. Frontiers in Language Sciences, 1, 222–22.

    Google Scholar 

  • Kudo, T. (1984). The effect of semantic plausibility on sentence comprehension in aphasia. Brain and Language, 21, 208–218.

    Article  CAS  PubMed  Google Scholar 

  • Kuperberg, G. R. (2007). Neural mechanisms of language comprehension: challenges to syntax. Brain Research, 1146, 23–49.

    Article  CAS  PubMed  Google Scholar 

  • Laird, A. R., Eickhoff, S. B., Kurth, F., Fox, P. M., Uecker, A. M., et al. (2009). ALE meta-analysis workflows via the brainmap database: progress towards a probabilistic functional brain atlas. Frontiers in Neuroinformatics, 3, 23–23.

    Article  PubMed Central  PubMed  Google Scholar 

  • Lee J. (2012). Linking eyes to mouth: a schema-based computational model for describing visual scenes. Ph.D. Thesis, Computer Science, University of Southern California, Los Angeles, CA.

  • Lee J. (In preparation-a). Implementing Template Construction Grammar (TCG) for visual scene description.

  • Lee J. (In preparation-b). The temporal unfolding of eye movements and utterance formulation.

  • Lesser, V. R., Fennel, R. D., Erman, L. D., & Reddy, D. R. (1975). Organization of the HEARSAY-II speech understanding system. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23, 11–23.

    Article  Google Scholar 

  • Letovsky, S. I., Whitehead, S. H., Paik, C. H., Miller, G. A., Gerber, J., et al. (1998). A brain image database for structure/function analysis. AJNR. American Journal of Neuroradiology, 19, 1869–1877.

    CAS  PubMed  Google Scholar 

  • Levin B. (1993). English verb classes and alternations: a preliminary investigation. University of Chicago Press.

  • Luria, A. R. (1973). The working brain. Harmondsworth: Penguin.

    Google Scholar 

  • MacWhinney, B. (2007). The TalkBank Project.

  • MacWhinney, B., Fromm, D., Forbes, M., & Holland, A. (2011). AphasiaBank: methods for studying discourse. Aphasiology, 25, 1286–1307.

    Article  PubMed Central  PubMed  Google Scholar 

  • Makkai, A. (1972). Idiom structure in English

  • Marcus, D. S., Harwell, J., Olsen, T., Hodge, M., Glasser, M. F., et al. (2011). Informatics and data mining tools and strategies for the human connectome project. Frontiers in Neuroinformatics, 5.

  • Mayberry, M., Crocker, M. W., Knoeferle, P. (2006). A connectionist model of the coordinated interplay of scene, utterance, and world knowledge.

  • Menenti, L., Gierhan, S. M. E., Segaert, K., & Hagoort, P. (2011). Shared language overlap and segregation of the neuronal infrastructure for speaking and listening revealed by functional MRI. Psychological Science, 22, 1173–1182.

    Article  PubMed  Google Scholar 

  • Miyake, A., Carpenter, P. A., & Just, M. A. (1994). A capacity approach to syntactic comprehension disorders: making normal adults perform like aphasic patients. Cognitive Neuropsychology, 11, 671–717.

    Article  Google Scholar 

  • Miyake, A., Carpenter, P. A., & Just, M. A. (1995). Reduced resources and specific impairments in normal and aphasic sentence comprehension. Cognitive Neuropsychology, 12, 651–679.

    Article  Google Scholar 

  • Mohanan, T., Wee, L. (1999). Grammatical semantics: evidence for structure in meaning. CSLI.

  • Osterhout, L., Albert, K., Kuperberg, G. (2007). The neurobiology of sentence comprehension, CiteSeerX.

  • Pinker, S. (1989). Learnability and cognition: the acquisition of argument structure. Cambridge: The MIT Press. 411 pp.

    Google Scholar 

  • Poeppel, D., Emmorey, K., Hickok, G., & Pylkkänen, L. (2012). Towards a new neurobiology of language. The Journal of Neuroscience, 32, 14125–14131.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  • Saffran, E. M., Schwartz, M. F., & Linebarger, M. C. (1998). Semantic influences on thematic role assignment: evidence from normals and aphasics. Brain and Language, 62, 255–297.

    Article  CAS  PubMed  Google Scholar 

  • Schuler, K. K. (2005). VerbNet: a broad-coverage, comprehensive verb lexicon. University of Pennsylvania

  • Schwartz, M. F., Linebarger, M. C., Saffran, E. M., & Pate, D. S. (1987). Syntactic transparency and sentence interpretation in aphasia. Language & Cognitive Processes, 2, 85–113.

    Article  Google Scholar 

  • Segaert, K., Menenti, L., Weber, K., Petersson, K. M., & Hagoort, P. (2012). Shared syntax in language production and language comprehension—an fMRI study. Cerebral Cortex, 22, 1662–1670.

    Article  PubMed  Google Scholar 

  • Sherman, J. C., & Schweickert, J. (1989). Syntactic and semantic contributions to sentence comprehension in agrammatism. Brain and Language, 37, 419–439.

    Article  CAS  PubMed  Google Scholar 

  • Spivey, M. J., Richardson, D. C., & Fitneva, S. A. (2005). Thinking outside the brain: spatial indices to visual and linguistic information. In J. M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: eye movements and the visual world (pp. 161–190). New York, Hove: Psychology Press.

    Google Scholar 

  • Sporns, O., Tononi, G., & Kötter, R. (2005). The human connectome: a structural description of the human brain. PLoS Computational Biology, 1, e42–e42.

    Article  PubMed Central  PubMed  Google Scholar 

  • Steels, L. (1999). The talking heads experiment.

  • Steels, L., & De Beule, J. (2006). Unify and merge in fluid construction grammar. In P. Vogt, Y. Sugita, E. Tuci, & C. Nehaniv (Eds.), Symbol grounding and beyond, proceedings (pp. 197–223). Berlin: Springer-Verlag Berlin.

    Chapter  Google Scholar 

  • Stefanowitsch, A., & Gries, S. T. (2003). Collostructions: investigating the interaction of words and constructions. International Journal of Corpus Linguistics, 8, 209–243.

    Article  Google Scholar 

  • Stephan, K. E., Kamper, L., Bozkurt, A., Burns, G. A. P. C., Young, M. P., & Kotter, R. (2001). Advanced database methodology for the collation of connectivity data on the Macaque brain (CoCoMac). Philosophical Transactions of the Royal Society B: Biological Sciences, 356, 1159–1186.

    Article  CAS  Google Scholar 

  • Talairach, Tournoux P. (1988). Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system: an approach to cerebral imaging. Thieme.

  • Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632–1634.

    Article  CAS  PubMed  Google Scholar 

  • Van Essen, D. C. (2009). Lost in localization–but found with foci?! NeuroImage, 48, 14–17.

    Article  PubMed Central  PubMed  Google Scholar 

  • Vigneau, M., Beaucousin, V., Hervé, P. Y., Duffau, H., Crivello, F., et al. (2006). Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. NeuroImage, 30, 1414–1432.

    Article  CAS  PubMed  Google Scholar 

  • Vosse, T., & Kempen, G. (2009). In defense of competition during syntactic ambiguity resolution. Journal of Psycholinguistic Research, 38, 1–9.

    Article  PubMed Central  PubMed  Google Scholar 

  • Wendel, K., Väisänen, O., Malmivuo. J., Gencer, N. G., Vanrumste. B., et al. (2009). EEG/MEG source imaging: methods, challenges, and open issues. Intelligence and Neuroscience, 2009: 13:1–13:12–13:1–13:12.

    Google Scholar 

  • Wilbur, R., Kak, A. (2006). Purdue RVL-SLLL American Sign Language Database. ECE Technical Reports.

  • Zurif, E. B., & Piñango, M. M. (1999). The existence of comprehension patterns in Broca’s Aphasia. Brain and Language, 70, 133–138.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

This research was supported by the National Science Foundation under Grant No. 0924674 (M.A. Arbib, Principal Investigator). We thank Brad Gasser and Michael Arbib for their fruitful comments on the model.

Author information

Authors and Affiliations

  1. Neuroscience Graduate Program and USC Brain Project, University of Southern California, Los Angeles, CA, USA

    Victor Barrès

  2. Computer Science Department and USC Brain Project, University of Southern California, Los Angeles, CA, USA

    Jinyong Lee

Authors
  1. Victor Barrès
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinyong Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Victor Barrès.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Barrès, V., Lee, J. Template Construction Grammar: From Visual Scene Description to Language Comprehension and Agrammatism. Neuroinform 12, 181–208 (2014). https://doi.org/10.1007/s12021-013-9197-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12021-013-9197-y

Keywords

Search

Navigation