Elsevier

NeuroImage

Volume 49, Issue 4, 15 February 2010, Pages 3295-3307
NeuroImage

The amodal system for conscious word and picture identification in the absence of a semantic task

https://doi.org/10.1016/j.neuroimage.2009.12.005Get rights and content

Abstract

Previous studies using explicit semantic tasks, such as category or similarity judgments, have revealed considerable neuroanatomical overlap between processing of the meaning of words and pictures. This result may have been influenced by the semantic executive control required by such tasks. We examined the degree of overlap while minimizing semantic executive demands. In a first fMRI experiment (n = 28), we titrated word (35.3 ms, SD = 9.6) and picture presentation duration (50.7 ms, SD = 15.8) such that conscious stimulus identification became a stochastic process, with a 50% chance of success. Subjects had to indicate by key press whether or not they had been able to identify the stimulus. In a second fMRI experiment (n = 19), the identification runs were followed by a surprise forced-choice recognition task and events were sorted on the basis of subsequent memory retrieval success rather than a subjective consciousness report. For both words and pictures, when stimulus processing exceeded the conscious identification threshold, the left occipitotemporal sulcus (OTS), intraparietal sulcus, inferior frontal junction, and middle third of the inferior frontal sulcus (IFS) were more active than when subjects had been unable to identify the stimulus. For both words and pictures, activity in two of these regions, IFS and OTS, predicted subsequent memory retrieval success. A Bayesian comparison revealed that the effective connectivity between IFS and the word- or picture-specific systems was mainly mediated via its connections with OTS. The amodal nature of left OTS and IFS involvement in word and picture processing extends to tasks with minimal semantic executive demands.

Introduction

The substrate for shared versus input modality-specific processing of meaning of words versus pictures has received a great deal of attention, both in patient lesion studies (Rapp et al., 1993, Chertkow et al., 1997, Rogers and McClelland, 2004) and in functional imaging of the intact brain (Vandenberghe et al., 1996, Wagner et al., 1997, Buckner et al., 2000, Thierry and Price, 2006, Vandenbulcke et al., 2006, Vandenbulcke et al., 2007). Candidate regions involved in amodal processing, i.e., processing of meaning of concrete words as well as pictures, are the ventral occipitotemporal cortex (Buckner et al., 2000), the posterior third of the left middle temporal gyrus (Chertkow et al., 1997, Vandenbulcke et al., 2007, Kircher et al., 2009), the anterior temporal pole (Hodges et al., 1992, Vandenberghe et al., 1996, Rogers and McClelland, 2004), the left inferior frontal sulcus (Wagner et al., 1997), the anterior inferior frontal gyrus (Goldberg et al., 2007), and the left middle frontal gyrus (Demb et al., 1995, Vandenberghe et al., 1996).

Many tasks that have been used to study commonality between cognitive processing of words and pictures have required at least some degree of explicit task-related retrieval of the associations and meaning of the referent. This is true for relatively demanding tasks with a high decision weighting component such as the Pyramids and Palm Trees test (Howard and Patterson, 1992, Vandenberghe et al., 1996) but also for relatively easy tasks that require subjects to explicitly retrieve well-known and typical features of the referent such as living–nonliving judgments (Wagner et al., 1997), sound, or colour matching (Garrard and Carroll, 2006), etc. Task-related deployment of strategies for retrieving, comparing, or deciding about semantic properties and associations may partly account for the commonality in activity pattern between words and pictures in these studies, in particular in the prefrontal cortex.

Other neuroimaging studies of amodal processing have made use of automatic semantic priming (Rissman et al., 2003, Sachs et al., 2008, Kircher et al., 2009, Sass et al., 2009) and combined words and pictures cross-modally (Kircher et al., 2009, Sass et al., 2009). During speeded lexical decision, the left posterior middle and superior temporal cortex shows a semantic relatedness effect during automatic priming (Copland et al., 2003, Gold et al., 2006) for word–word pairs and for picture–word pairs (Sachs et al., 2008, Kircher et al., 2009, Sass et al., 2009). Subliminal (masked) priming removes the strategic semantic effects altogether (Forster, 1998, Holcomb and Grainger, 2006, Diaz and McCarthy, 2007, Grainger and Holcomb, 2009). Neuroimaging studies of subliminal (masked) priming have reported differential effects in left fusiform cortex for word–word pairs comparing same versus different words (Dehaene et al., 2001, Qiao et al., 2010) and in the left posterior fusiform cortex for picture–picture pairs comparing repeated versus unrepeated pictures (Eddy et al., 2007). Cross-modal effects or within-study overlap between word–word and picture–picture priming effects have not been reported in neuroimaging studies of subliminal priming until now to the best of our knowledge. Prefrontal supraliminal priming effects (suppression or enhancement) have been reported at various locations distributed over the lateral convexity (Wagner et al., 1997, Copland et al., 2003, Rissman et al., 2003, Raposo et al., 2006, Race et al., 2009). This variability may relate to differences in the balance between automatic versus strategic component processes (Copland et al., 2003, Gold et al., 2006), but it could also be a consequence of the relatively small effect sizes and the inherent inability of fMRI to capture the time dependency of priming effects.

For the first time, we applied a method to the study of amodal processing that stems from the consciousness research field (Kanwisher, 2001, Bar et al., 2001, Marois et al., 2004, Carmel et al., 2006, Wilenius-Emet et al., 2004). At brief stimulus durations, conscious stimulus identification is a probabilistic process. For each individual and each input modality, we selected a duration of word or picture presentation so that subjects reported conscious identification of the stimulus only with a probability of 50% across items (Marois et al., 2004, Carmel et al., 2006, Wilenius-Emet et al., 2004) and also within items across subjects. In this way, sensory input was matched between events that were associated with conscious processing and events in which processing remained subliminal. We tried to detect zones of amodal cognitive processing during supra- versus subliminal word and picture identification. In a previous study of speeded picture identification (Bar et al., 2001, Bar et al., 2006), activity in mid and anterior fusiform cortex correlated with the subjects' level of confidence that they had been able to identify the pictures correctly, ranging from zero (no stimulus perceived) to five (fully confident about stimulus identity), with also orbitofrontal involvement (Bar et al., 2006). In that study, confidence was manipulated by repeating the same stimuli up to 6 times. Only pictures were used (Bar et al., 2001, Bar et al., 2006). Our cross-modal design allowed us to examine whether these fusiform and other regions are shared between words and pictures during supra- versus subthreshold processing while sensory characteristics and prior exposure were strictly matched between supra- and subliminally processed stimuli.

How does conscious processing of a word or picture differ from subliminal processing? According to the Theory of Visual Attention (TVA) (Bundesen, 1990, Bundesen et al., 2005, Bundesen and Habekost, 2008), conscious processing takes place when a perceptual entity is selected among other competing entities and gains access to visual short-term memory (VSTM) according to a winner-takes-all principle. From VSTM, the consciously perceived unit is made available (“broadcasted”) to other cognitive brain systems, such as declarative memory (Baars, 1988); 2002) or phonological retrieval (Jackendoff, 2007) systems. When we consciously identify a stimulus, the surface features of that stimulus and its visual form become available for subsequent explicit retrieval, as well as the identity of the referent. The central question in this study is whether similar brain networks are involved for word and picture input when stimuli gain access to consciousness. Conscious stimulus identification may in its turn trigger explicit lexical– or associative–semantic processes but the short stimulus durations and the task instruction in our study directed attentional resources towards perceptual identification and away from downstream processes of explicit semantic elaboration.

We determined which regions became activated when perceptual processing exceeded the threshold of conscious identification on the basis of a subjective consciousness report (Frith et al., 1999, Beck et al., 2001, Bar et al., 2001, Haynes et al., 2005). We dealt with the inherent subjectivity of this measure (Frith et al., 1999) in two ways: first, we also included foil stimuli consisting of nonexisting chimeras. Subjects were instructed to respond negatively to the foil stimuli, i.e., in the same way as to stimuli that they had not been able to identify. When the false-positive response rate to foil stimuli was too high, we excluded the run. In case of pictures this task is very similar to the object decision test (Riddoch and Humphreys, 1993), a classical neuropsychological task that probes the structural description level of processing. According to the Hierarchical Interactive Theory (HIT) (Humphreys and Forde, 2001), the activation of a structural description of the object is necessary for object identification and distinct from the associative–semantic level (Humphreys and Forde, 2001). At the structural description level, visual percepts are matched with mnemonic presentations of real-life entities in a way that is invariant for viewpoint, size, orientation, etc., and generalizes across different exemplars of a same entity (Humphreys and Forde, 2001).

As a second measure to circumvent the subjectivity of a consciousness report, we only retained regions that also fulfilled a second, more objective criterion: following runs of subjective consciousness report, we conducted a surprise forced-choice yes/no recognition task and sorted events on the basis of subsequent memory retrieval success rather than subjective consciousness report. Only regions that stringently fulfilled both criteria, an association with a positive consciousness report and with successful encoding, conjointly for words and for pictures, were retained as amodal zones of supra- versus subliminal processing. To further verify the level and accuracy of stimulus identification and also to evaluate how the extent of the amodal activations obtained in the first and second experiment related to those obtained in more classical tasks, we conducted a third, overt naming/reading fMRI experiment.

Section snippets

Subjects

Twenty-eight healthy native Dutch speakers (15 women and 13 men, between 19 and 29 years of age) participated in the first fMRI experiment, which was based on the subjective consciousness report, and 8 additional subjects (3 women and 5 men, age range 19–25 years) in a control experiment. Nineteen other subjects (12 women and 7 men, between 19 and 35 years of age) participated in a third fMRI experiment, the subsequent memory retrieval experiment. Four other subjects (1 man and 3 women, between

First experiment

In the first experiment, our titration procedure yielded an average word presentation duration across runs and subjects of 35.3 ms (SD = 9.6) and an average picture presentation duration of 50.7 ms (SD = 15.8). Thirty-four out of 156 runs were excluded because subjects responded false-positively to more than 3 out of 8 catch trials in these runs, indicating that in these runs subjects produced a positive key press even when they had failed to identify the stimuli at the basic level. Subjects

First experiment

Regardless of input modality (conjunction contrast 1), words or pictures, a positive consciousness report was associated with significantly higher activity in the middle third of the left inferior frontal sulcus (IFS) (Figs. 3A and C), the left inferior frontal junction (IFJ) (Figs. 3B and C), the border between the posterior and the middle third of the left occipitotemporal sulcus (OTS) (Figs. 3C and D) and the middle third of the left intraparietal sulcus (IPS) (Figs. 3C and E) (Table 2). A

Discussion

Two regions, the middle third of the left IFS (Figs. 3A and C) and the left occipitotemporal sulcus (Figs. 3C and D), were activated in an amodal manner when a stimulus, word or picture, gained access to consciousness. In left OTS, and to a lesser degree also in IFS, this effect consisted of an enhancement of a response that was also present for subliminal stimuli. Activation in these two regions predicted subsequent memory retrieval success for both words and pictures (Fig. 5, orange outline).

Acknowledgments

This work was supported by Research Foundation Flanders (FWO) [G.0076.02 and G0668.07 to R.V.], K.U.Leuven [OT/04/41, OT/08/056, EF/05/014 to R.V.], and Federaal Wetenschapsbeleid belspo [IUAP P6/29].

References (88)

  • FristonK. et al.

    Dynamic causal modelling

    NeuroImage

    (2003)
  • FristonK. et al.

    Stochastic designs in event-related fMRI

    NeuroImage

    (1999)
  • FrithC. et al.

    The neural correlates of conscious experience: an experimental framework

    Trends Cogn. Sci.

    (1999)
  • GaillardR. et al.

    Direct intracranial, fMRI and lesion evidence for the causal role of left inferotemporal cortex in reading

    Neuron

    (2006)
  • HaynesJ. et al.

    Visibility reflects dynamic changes of effective connectivity between V1 and fusiform cortex

    Neuron

    (2005)
  • KanwisherN.

    Neural events and perceptual awareness

  • MaroisR. et al.

    The neural fate of consciously perceived and missed events in the attentional blink

    Neuron

    (2004)
  • NicholsT. et al.

    Valid conjunction inference with the minimum statistic

    Neuroimage

    (2005)
  • OldfieldR.

    The assessment and analysis of handedness: the Edinburgh Inventory

    Neuropsychologia

    (1971)
  • PennyW. et al.

    Comparing dynamic causal models

    NeuroImage

    (2004)
  • PoldrackR. et al.

    Functional specialization for semantic and phonological processing in the left inferior prefrontal cortex

    NeuroImage

    (1999)
  • PolineJ. et al.

    Combining spatial extent and peak intensity to test for activations in functional imaging

    NeuroImage

    (1997)
  • PriceC.

    The functional anatomy of word comprehension and production

    Trends Cogn. Sci.

    (1998)
  • PriceC. et al.

    The pro and cons of labelling a left occipitotemporal region ‘visual word form area’

    Neuroimage

    (2004)
  • RaposoA. et al.

    Repetition suppression and semantic enhancement: an investigation of the neural correlates of priming

    Neuropsychologia

    (2006)
  • RuffI. et al.

    Recruitment of anterior and posterior structures in lexical–semantic processing: an fMRI study comparing implicit and explicit tasks

    Brain Lang.

    (2008)
  • SachsO. et al.

    Automatic processing of semantic relations in fMRI: neural activation during semantic priming of taxonomic and thematic categories

    Brain Res.

    (2008)
  • SassK. et al.

    Lion–tiger–stripes: neural correlates of indirect semantic priming across processing modalities

    Neuroimage

    (2009)
  • VinckierF. et al.

    Hierarchical coding of letter strings in the ventral stream: dissecting the inner organization of the visual word-form system

    Neuron

    (2007)
  • Wilenius-EmetM. et al.

    An electrophysiological correlate of human visual awareness

    Neurosci. Lett.

    (2004)
  • BaarsB.

    A cognitive theory of consciousness

    Cambridge University Press

    (1988)
  • BaayenH. et al.

    The CELEX lexical database (CD-ROM)

    Linguistic Data Consortium

    (1993)
  • BarM.

    A cortical mechanism for triggering top-down facilitation in visual object recognition

    J. Cogn. Neurosci.

    (2003)
  • BarM. et al.

    Top-down facilitation of visual recognition

    Proc. Nat. Acad. Sci.

    (2006)
  • BeckD. et al.

    Neural correlates of change detection and change blindness

    Nat. Neurosci.

    (2001)
  • Ben-ShacharM. et al.

    Differential sensitivity to words and shapes in ventral occipito-temporal cortex

    Cereb. Cortex

    (2007)
  • BucknerR. et al.

    Functional MRI evidence for a role of frontal and inferior temporal cortex in amodal components of priming

    Brain

    (2000)
  • BundesenC.

    A theory of visual attention

    Psychol. Rev.

    (1990)
  • BundesenC. et al.

    Principles of visual attention

    Linking Mind and Brain

    (2008)
  • BundesenC. et al.

    A neural theory of visual attention: Bridging cognition and neurophysiology

    Psychol. Rev.

    (2005)
  • CohenL. et al.

    The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split brain patients

    Brain

    (2000)
  • CohenL. et al.

    Language-specific tuning of visual cortex? Functional properties of the visual word form area

    Brain

    (2002)
  • CohenL. et al.

    Visual word recognition in the left and right hemispheres: anatomical and functional correlates of peripheral alexias

    Cereb. Cortex

    (2003)
  • CollinsA. et al.

    A spreading–activation theory of semantic processing

    Psychol. Rev.

    (1975)
  • Cited by (0)

    View full text