Elsevier

NeuroImage

Volume 22, Issue 1, May 2004, Pages 11-21
NeuroImage

Sentence complexity and input modality effects in sentence comprehension: an fMRI study

https://doi.org/10.1016/j.neuroimage.2004.01.001Get rights and content

Abstract

Cortical regions engaged by sentence processing were mapped using functional MRI. The influence of input modality (spoken word vs. print input) and parsing difficulty (sentences containing subject-relative vs. object-relative clauses) was assessed. Auditory presentation was associated with pronounced activity at primary auditory cortex and across the superior temporal gyrus bilaterally. Printed sentences by contrast evoked major activity at several posterior sites in the left hemisphere, including the angular gyrus, supramarginal gyrus, and the fusiform gyrus in the occipitotemporal region. In addition, modality-independent regions were isolated, with greatest overlap seen in the inferior frontal gyrus (IFG). With respect to sentence complexity, object-relative sentences evoked heightened responses in comparison to subject-relative sentences at several left hemisphere sites, including IFG, the middle/superior temporal gyrus, and the angular gyrus. These sites showing modulation of activity as a function of sentence type, independent of input mode, arguably form the core of a cortical system essential to sentence parsing.

Introduction

With continuing advances in functional neuroimaging tools such as fMRI, which can detect cortical regions implicated in a wide range of cognitive behaviors, the goal of mapping the language brain is within our grasp. Examination of patterns of co-varying regional brain activity across manipulations of sentence structure and input modality can help to narrow in on the brain circuitry that instantiates abilities essential to comprehension of spoken discourse and printed text. Identifying these critical regions and their connections can also advance our understanding of the impact on language systems of developmental and pathologic conditions that affect brain function Carpentier et al., 2001, Eden and Zeffiro, 1998, Pugh et al., 1996, Pugh et al., 2001. An important potential application of mapping studies is to aid in planning for neurosurgical procedures in the treatment of focal epilepsy and other conditions Binder et al., 1996, Bookheimer et al., 1997, Carpentier et al., 2001, Desmond et al., 1995, Simos et al., 1999, Springer et al., 1999. Before functional imaging paradigms can be used routinely in clinical practice, a fuller understanding of the functional anatomy of language must be obtained. Indeed, a clearer view is needed of how distributed neural systems function cooperatively in the performance of essential language abilities.

Until relatively recently, most knowledge of the neurological foundations of language had been gained from studies of patients with localized cortical damage. Specific deficits in language processing have been associated with damage to local cortical regions. Damage to Broca's area (BA 44, 45) in left inferior frontal gyrus (IFG) is associated with a range of deficits in spoken language production, including phonetic production of words and grammatical organization of sentences. Clinical and psycholinguistic studies also implicate this region in sentence comprehension, both in speech and reading. Recent neuroimaging studies have confirmed a strong role for pars triangularis (Brodmann area 45) and pars opercularis (BA 44) within IFG for core reading operations, including print-to-phonology decoding, working memory, and aspects of sentence parsing Fiez and Peterson, 1998, Fiez et al., 1999, Hagoort et al., 1999, Herbster et al., 1997, Ni et al., 2000, Pugh et al., 1997.

Wernicke's area in the posterior portion of the left STG is the other classical language zone where lesions are known to produce major deficits in both spoken and written language comprehension. Imaging studies suggest that this region serves a multiplicity of functions, playing a role in ordered recall of words during verbal fluency tasks (Wise et al., 2001), phonological memory storage (Paulesu et al., 1993), phonological analysis, lexical semantic processing Just et al., 1996, Keller et al., 2001, Michael et al., 2001, Pugh et al., 1996, and sentence interpretation Helenius et al., 1998, Ni et al., 2000. Deficits in accessing visual word forms for reading have been associated with damage to the left angular gyrus while impairments in name retrieval and semantic processing of words have been associated with damage to the left basal temporal area Hodges et al., 1992, Krauss et al., 1996, Roeltgen and Heilman, 1984.

At present, there is a good deal of evidence, stemming both from lesion studies and neuroimaging studies on neurologically intact persons, pointing to a role for multiple regions, beyond the classical Broca's and Wernicke's areas, in various aspects of language processing of spoken and printed materials Caplan et al., 1996, Howard et al., 1992, Peterson et al., 1989, Pugh et al., 1996, Warburton et al., 1996. Thus, findings from both these sources converge on a complex, richly interconnected network of cortical and subcortical regions distributed in both hemispheres, but predominantly in the left, which are relevant to one or another aspect of language behavior.

A majority of neuroimaging studies of language processing to date have been carried out with simple materials, usually lists of isolated words. Examples of such studies include experiments aimed at identifying the brain systems active in the several component processes of word recognition, involving orthographic, phonological, and lexical semantic information, as well as those systems that support fluent word generation (Beauregard et al., 1997, Pugh et al., 1996, Shaywitz et al., 1998). From the many studies of this type, there is an emerging consensus on the functional architecture of word recognition systems for print and speech (Hickok and Poeppel, 2000, Pugh et al., 2000, Pugh et al., 2001, Tagamets et al., 2000. Neuroimaging studies have revealed a set of left hemisphere brain areas critical for word identification in reading. The left hemisphere reading circuit contains ventral (occipitotemporal), dorsal (temporoparietal), and anterior (inferior frontal gyrus) components.

A factor, which is sometimes neglected in analyses of language processing, is the influence of input modality. Both in the history of the human species and the development of the individual child, spoken language capacity is the secondary derived language abilities of reading and writing. Although parsimony would dictate that these secondary language functions would exploit much of the preexisting spoken language machinery rather than recreating it, nature does not always take the parsimonious route. Hence, the degree of overlap in regional brain activity underlying speech perception and reading is an empirical question. Studies that directly compare spoken with printed word identification are relatively few. These have generally found largely overlapping neural networks across left hemisphere cortex when activations associated with basic auditory and visual sensory processing are removed statistically Chee et al., 1999, Howard et al., 1992, Michael et al., 2001, Shaywitz et al., 2001, Simos et al., 1999. While the center of activation induced by speech and printed material can vary somewhat within major left hemisphere regions (Howard et al., 1992), the overlap in activation is a more prominent feature of such comparisons than are the differences.

To illustrate, Shaywitz et al. (2001) examined auditory and visual word recognition across conditions that also modulated demands on selective and divided attention. This study revealed common activation for print and speech in left IFG, supramarginal gyrus, and occipitotemporal sites, with unique foci for spoken words in a localized region within the superior temporal gyrus (see Chee et al., 1999, Howard et al., 1992 for similar findings). Thus, it appears that lexical access and lexical semantic processing of single printed and spoken words share many neural networks within left perisylvian language areas. There is evidence for a language modality that transcends sensory modality.

Although much valuable information has been obtained from controlled studies of processing isolated words, such studies necessarily present an incomplete and possibly a distorted view of brain operations occurring during the processing of language in sentence contexts. Those circuits that are responsible for linking words in text or discourse and that underlie multiword syntactic and semantic operations are relatively less well charted (Hagoort and Brown, 2000). Accordingly, this study focused on sentence processing. Given the high degree of overlap of structures engaged by speech and print at the word level, it is reasonable to expect that the processing of connected text or discourse will conform to this picture of largely overlapping foci for the two input modalities. Nevertheless, lesion studies have occasionally shown selective loss of comprehension for spoken discourse or text, so there is reason to believe that the neural systems supporting abstract components of language processing in the two modalities are partially non-overlapping (Shallice, 1987).

Previous investigations of sentence processing with PET or fMRI have consistently shown broad regions of activity across left perisylvian areas irrespective of mode of presentation, though few studies have directly contrasted speech and print input within the same study under controlled conditions Bookheimer et al., 1997, Carpentier et al., 2001, Michael et al., 2001. In two recent studies by our group, subjects heard or read sentences that were either well-formed or syntactically or semantically anomalous Carpentier et al., 2001, Ni et al., 2000. In Carpentier et al., there was an explicit speech–print comparison, which showed that many left hemisphere regions were activated in common across both modalities. Greater levels of activity for spoken sentences were found primarily in bilateral auditory cortex and at several sites in the superior temporal gyrus, while printed sentences showed heightened responses at ventral occipitotemporal sites and some temporoparietal sites. Within IFG, common activation was seen anteriorally in BA 45, but posteriorly in BA 44, greater activation for printed than spoken sentences was observed. This led us to speculate that BA 44 is associated with orthographic-to-phonological recoding for print, in keeping with some previous studies of processing isolated individual words Fiez and Peterson, 1998, Herbster et al., 1997, Pugh et al., 1997. It would also suggest that BA 45, an area that was insensitive to modality of presentation, might be a good candidate for higher order processing related to sentence parsing operations and/or language-related working memory (a speculation which is also consistent with the findings of Ni et al., 2000).

To test this speculation about division of labor within IFG and other regions, it is critical to experimentally cross the factors of speech/print modality with sentence complexity to isolate modality-independent areas that modulate activity in response to differences in sentence complexity. This approach was adopted in the present investigation. The underlying premise can be stated simply: Sentence-level processing of linguistic information should be largely independent of input modality because it is carried out on linguistic representations that are neither specifically auditory nor visual. Accordingly, we could expect that the neural substrate of parsing operations comprises those regions that are activated by both spoken and written sentences and that also respond differentially to sentences that are easier or harder to parse.

Sentence complexity effects involving subject- and object-relative clause structures have been demonstrated both in behavioral and imaging studies. The restrictive relative clause is one structure that has received much study. Relative clauses can be distinguished by the syntactic role of the noun phrase within the matrix sentence that bears the relative clause (subject or object) and the role of the missing noun phrase within the relative clause (again, subject or object). With respect to the latter distinction, object relatives have uniformly been found more difficult and slower to process than subject relatives Hamburger and Crain, 1984, Just et al., 1996, Ni et al., 1996. There are a variety of theories to explain the greater difficulty of processing object relatives (Gibson and Pearlmutter, 2000). For the most part, these focus on the problems of assigning thematic role to the head noun of the relative clause. Apparently, object relatives present greater processing difficulties than subject relatives because the assignment of thematic role is more difficult in these cases.

Recently, in a design similar to the one employed in the current report, Michael et al. (2001) varied sentence complexity for spoken and written sentences. Subjects either heard or read two kinds of sentences: Compound simple sentences containing two clauses joined by the word and (“The coach saw the actress and ran rapidly up the steep hill”) and complex sentences with an embedded clause (“The monk that the astronaut watched entered the room quietly at noon”). Each condition was examined relative to a rest/fixation baseline condition. Using anatomically defined regions of interest (ROI), Michael et al. found sentence complexity effects that were modality independent in the IFG ROI and in the temporal lobe ROI. Increased activation for spoken sentences was seen at anterior temporal lobe sites and in anterior aspects of IFG (probably BA 45), while print showed reliable increases in a broadly defined extrastriate ROI. Moreover, as in the study by Carpentier et al. (2001), spoken sentences were associated with greater right hemispheric activity (especially in STG and IFG) than printed sentences. The use by Michael et al. of anatomically defined ROIs does, however, limit the grain-size of spatial resolution. Moreover, the use of rest/fixation as a baseline task does not permit subtraction of primary visual and auditory processes. The present study employs modality-specific control tasks and voxel-based analyses.

Several other studies have examined whole sentence processing in one or the other modality to map activity associated with particular demands placed on the brain in comprehending sentences Caplan, 2001, Caplan et al., 2000, Caplan et al., 2002, Carpentier et al., 2001, Michael et al., 2001, Muller et al., 1997, Ni et al., 2000, Kang et al., 1999, Schmolck et al., 2000. The most frequent finding in studies that examine sentence complexity is that sites in IFG, particularly BA 45, consistently discriminate easy-to-parse from difficult-to-parse structures. Less consistently reported effects have been reported in posterior STG, MTG, and in one recent study, effects of complexity were observed in the angular gyrus (Caplan, 2001). The current study examines all of these regions (i.e., whole brain analyses are employed), with both auditory and visually presented sentences to distinguish possible modality-specific from modality-independent sentence complexity effects. While STG and IFG activations are anticipated, the angular gyrus finding by Caplan et al. stands as unexpected. Given that stimuli in this event-related experiment were presented visually, it remains to be seen if such effects in this putatively reading-related region are modality specific or, alternatively, whether they cut across both speech and print modes.

The present study examined the influence of sentence structure (center-embedded, object-relative clause vs. subject-relative clause) and input modality (print vs. speech) in a sentence comprehension fMRI paradigm with full brain coverage. As noted, a reason to examine sentence processing of comparable materials presented in both spoken and written form is to clearly distinguish regions that are active in one modality or the other from those that show overlapping excitation. The latter amodal areas likely constitute the critical sites for the more abstract dimensions of linguistic processing. A candidate neural system for sentence parsing is one that is both modality independent and discriminates subject-relative from object-relative sentences. This approach allows us, in principle, to examine sentential and modality effects and their interactions.

Subjects were healthy right-handed individuals aged 18–40, mean 24, 10 males and 10 females. A total of 21 subjects were studied, with 1 subject discarded who was judged to be right hemisphere dominant for language (based on a laterality score from the main subtractions in both the visual and auditory presentation modes). Final analysis was performed on 20 left hemisphere dominant subjects. All participants gave informed consent, had normal or corrected-to-normal vision, and were paid $40 for their participation. Approval for this study was obtained from the Yale Medical Human Investigation Committee.

Section snippets

Materials

The stimulus materials consisted of 120 center-embedded sentences containing relative clauses. Half of these were subject relatives (as in example 1 below) and half were object relatives (as in example 2 below). Within each pair, the sentences contained the same words; only the structure of the relative clause varied.1

Procedure

The task involved reading or listening to the sentences and making a sentence goodness judgment following each sentence, or a line orientation or a pitch judgment for the control stimuli. Subjects were instructed to press the yes key if the sentence seemed to be grammatical and to make sense, otherwise they were to press the no key (criteria employed for making this judgment were not controlled and presumably varied across subjects; our primary consideration was to ensure that subjects were

Image acquisition

Imaging was performed using a 1.5 T GE, Signa LX system (General Electric, Waukesha, WI). Subjects were placed supine in the MR scanner, and their heads were placed within the standard quadrature head coil. Care was taken to ensure that the subjects were looking straight up; foam was placed on either side of the head to minimize motions. Also to minimize motion, a piece of tape was placed across the forehead of each subject and attached to the head holder within the head coil. Once inside the

Data analysis

Data were motion corrected using SPM-99 (Friston et al., 1995) and aligned to a reference anatomic scan taken at the beginning of the study (Studholme et al., 2001). A Student's t test was applied to the task/control conditions within a run using in-house developed software. Specifically, within each run, the two subject-relative blocks were compared with the baseline blocks on either side of each activation block, in a single t test. The analogous operation was performed for the

Modality-specific and modality-independent effects

As shown in Figs. 1a and b and Table 1, Table 2, activation for printed and spoken sentences (each in contrast to its within modality control task) was broadly distributed with spoken sentences showing a stronger right hemisphere response, especially at STG, relative to printed sentences. In Fig. 2a, modality-independent sites (i.e., sites considered activated above threshold in both modalities) are largely left lateralized and include the occipitotemporal area, MTG, STG, and several foci in

Discussion

The present study was conducted to examine co-varying effects of input modality, sentence complexity, and their interactions on those cortical systems that participate in language processing. Several important findings emerged. Most prominent, perhaps, is the observation that while the neural networks that serve printed and spoken sentence processing are not identical, the sites within these systems that were sensitive to parsing difficulty were largely overlapping, especially in IFG. On the

Conclusions

The influence of input modality demonstrated that auditory sentence presentation was associated with pronounced activity at primary auditory cortex and across the superior temporal gyrus bilaterally, whereas printed sentences by contrast evoked major activity at several posterior sites in the left hemisphere, including the angular gyrus, supramarginal gyrus, and the fusiform gyrus in the occipitotemporal region. The intersection of these modality-dependent maps indicated modality-independent

Acknowledgements

The research and preparation of the manuscript was supported in part by the following grants to Yale University School of Medicine (NIH R01 NS38467, NS40497) and Haskins Laboratories (NICHD grants HD-01994, HD 40353, and HD 40411).

References (55)

  • J.R Binder et al.

    Determination of language dominance using functional MRI

    Neurology

    (1996)
  • S.Y Bookheimer et al.

    A direct comparison of PET activation and electrocortical stimulation mapping for language localization

    Neurology

    (1997)
  • D Caplan et al.

    Location of lesions in stroke patients with deficits in syntactic processing in sentence comprehension

    Brain

    (1996)
  • D Caplan et al.

    Activation of Broca's area by syntactic processing under conditions of concurrent articulation

    Hum. Brain Mapp.

    (2000)
  • D Caplan

    Functional neuroimaging studies of syntactic processing

    Psycholing. Res.

    (2001)
  • D Caplan et al.

    Vascular responses to syntactic processing: event-related fMRI study of relative clauses

    Hum. Brain Mapp.

    (2002)
  • A.C Carpentier et al.

    Functional MRI of language processing: dependence on input modality and temporal lobe epilepsy

    Epilepsia

    (2001)
  • M.W.L Chee et al.

    Auditory and visual word processing studied with fMRI

    Hum. Brain Mapp.

    (1999)
  • R.T Constable et al.

    An ROC approach for evaluating functional brain MR imaging and postprocessing protocols

    Magn. Reson. Med.

    (1995)
  • J.E Desmond et al.

    Functional MRI measurement of language lateralization in Wada-tested patients

    Brain

    (1995)
  • B Efron et al.

    An Introduction to the Bootstrap

    (1993)
  • J.A Fiez et al.

    Neuroimaging studies of word reading

    Proc. Natl. Acad. Sci. U. S. A.

    (1998)
  • A.D Friederici et al.

    Syntactic, prosodic, and semantic processes in the brain: evidence from event-related neuroimaging

    J. Psychol. Res.

    (2000)
  • K.J Friston et al.

    Spatial registration and normalization of images

    Hum. Brain Mapp.

    (1995)
  • N Geschwind

    Disconnexion syndromes in animals and man

    Brain

    (1965)
  • E Gibson et al.

    Distinguishing serial and parallel parsing

    J. Psycholing

    (2000)
  • Y Grodzinsky

    The neurology of syntax: language use without Broca's area

    Behav. Brain Sci.

    (2000)
  • Cited by (0)

    View full text