The amodal system for conscious word and picture identification in the absence of a semantic task
Introduction
The substrate for shared versus input modality-specific processing of meaning of words versus pictures has received a great deal of attention, both in patient lesion studies (Rapp et al., 1993, Chertkow et al., 1997, Rogers and McClelland, 2004) and in functional imaging of the intact brain (Vandenberghe et al., 1996, Wagner et al., 1997, Buckner et al., 2000, Thierry and Price, 2006, Vandenbulcke et al., 2006, Vandenbulcke et al., 2007). Candidate regions involved in amodal processing, i.e., processing of meaning of concrete words as well as pictures, are the ventral occipitotemporal cortex (Buckner et al., 2000), the posterior third of the left middle temporal gyrus (Chertkow et al., 1997, Vandenbulcke et al., 2007, Kircher et al., 2009), the anterior temporal pole (Hodges et al., 1992, Vandenberghe et al., 1996, Rogers and McClelland, 2004), the left inferior frontal sulcus (Wagner et al., 1997), the anterior inferior frontal gyrus (Goldberg et al., 2007), and the left middle frontal gyrus (Demb et al., 1995, Vandenberghe et al., 1996).
Many tasks that have been used to study commonality between cognitive processing of words and pictures have required at least some degree of explicit task-related retrieval of the associations and meaning of the referent. This is true for relatively demanding tasks with a high decision weighting component such as the Pyramids and Palm Trees test (Howard and Patterson, 1992, Vandenberghe et al., 1996) but also for relatively easy tasks that require subjects to explicitly retrieve well-known and typical features of the referent such as living–nonliving judgments (Wagner et al., 1997), sound, or colour matching (Garrard and Carroll, 2006), etc. Task-related deployment of strategies for retrieving, comparing, or deciding about semantic properties and associations may partly account for the commonality in activity pattern between words and pictures in these studies, in particular in the prefrontal cortex.
Other neuroimaging studies of amodal processing have made use of automatic semantic priming (Rissman et al., 2003, Sachs et al., 2008, Kircher et al., 2009, Sass et al., 2009) and combined words and pictures cross-modally (Kircher et al., 2009, Sass et al., 2009). During speeded lexical decision, the left posterior middle and superior temporal cortex shows a semantic relatedness effect during automatic priming (Copland et al., 2003, Gold et al., 2006) for word–word pairs and for picture–word pairs (Sachs et al., 2008, Kircher et al., 2009, Sass et al., 2009). Subliminal (masked) priming removes the strategic semantic effects altogether (Forster, 1998, Holcomb and Grainger, 2006, Diaz and McCarthy, 2007, Grainger and Holcomb, 2009). Neuroimaging studies of subliminal (masked) priming have reported differential effects in left fusiform cortex for word–word pairs comparing same versus different words (Dehaene et al., 2001, Qiao et al., 2010) and in the left posterior fusiform cortex for picture–picture pairs comparing repeated versus unrepeated pictures (Eddy et al., 2007). Cross-modal effects or within-study overlap between word–word and picture–picture priming effects have not been reported in neuroimaging studies of subliminal priming until now to the best of our knowledge. Prefrontal supraliminal priming effects (suppression or enhancement) have been reported at various locations distributed over the lateral convexity (Wagner et al., 1997, Copland et al., 2003, Rissman et al., 2003, Raposo et al., 2006, Race et al., 2009). This variability may relate to differences in the balance between automatic versus strategic component processes (Copland et al., 2003, Gold et al., 2006), but it could also be a consequence of the relatively small effect sizes and the inherent inability of fMRI to capture the time dependency of priming effects.
For the first time, we applied a method to the study of amodal processing that stems from the consciousness research field (Kanwisher, 2001, Bar et al., 2001, Marois et al., 2004, Carmel et al., 2006, Wilenius-Emet et al., 2004). At brief stimulus durations, conscious stimulus identification is a probabilistic process. For each individual and each input modality, we selected a duration of word or picture presentation so that subjects reported conscious identification of the stimulus only with a probability of 50% across items (Marois et al., 2004, Carmel et al., 2006, Wilenius-Emet et al., 2004) and also within items across subjects. In this way, sensory input was matched between events that were associated with conscious processing and events in which processing remained subliminal. We tried to detect zones of amodal cognitive processing during supra- versus subliminal word and picture identification. In a previous study of speeded picture identification (Bar et al., 2001, Bar et al., 2006), activity in mid and anterior fusiform cortex correlated with the subjects' level of confidence that they had been able to identify the pictures correctly, ranging from zero (no stimulus perceived) to five (fully confident about stimulus identity), with also orbitofrontal involvement (Bar et al., 2006). In that study, confidence was manipulated by repeating the same stimuli up to 6 times. Only pictures were used (Bar et al., 2001, Bar et al., 2006). Our cross-modal design allowed us to examine whether these fusiform and other regions are shared between words and pictures during supra- versus subthreshold processing while sensory characteristics and prior exposure were strictly matched between supra- and subliminally processed stimuli.
How does conscious processing of a word or picture differ from subliminal processing? According to the Theory of Visual Attention (TVA) (Bundesen, 1990, Bundesen et al., 2005, Bundesen and Habekost, 2008), conscious processing takes place when a perceptual entity is selected among other competing entities and gains access to visual short-term memory (VSTM) according to a winner-takes-all principle. From VSTM, the consciously perceived unit is made available (“broadcasted”) to other cognitive brain systems, such as declarative memory (Baars, 1988); 2002) or phonological retrieval (Jackendoff, 2007) systems. When we consciously identify a stimulus, the surface features of that stimulus and its visual form become available for subsequent explicit retrieval, as well as the identity of the referent. The central question in this study is whether similar brain networks are involved for word and picture input when stimuli gain access to consciousness. Conscious stimulus identification may in its turn trigger explicit lexical– or associative–semantic processes but the short stimulus durations and the task instruction in our study directed attentional resources towards perceptual identification and away from downstream processes of explicit semantic elaboration.
We determined which regions became activated when perceptual processing exceeded the threshold of conscious identification on the basis of a subjective consciousness report (Frith et al., 1999, Beck et al., 2001, Bar et al., 2001, Haynes et al., 2005). We dealt with the inherent subjectivity of this measure (Frith et al., 1999) in two ways: first, we also included foil stimuli consisting of nonexisting chimeras. Subjects were instructed to respond negatively to the foil stimuli, i.e., in the same way as to stimuli that they had not been able to identify. When the false-positive response rate to foil stimuli was too high, we excluded the run. In case of pictures this task is very similar to the object decision test (Riddoch and Humphreys, 1993), a classical neuropsychological task that probes the structural description level of processing. According to the Hierarchical Interactive Theory (HIT) (Humphreys and Forde, 2001), the activation of a structural description of the object is necessary for object identification and distinct from the associative–semantic level (Humphreys and Forde, 2001). At the structural description level, visual percepts are matched with mnemonic presentations of real-life entities in a way that is invariant for viewpoint, size, orientation, etc., and generalizes across different exemplars of a same entity (Humphreys and Forde, 2001).
As a second measure to circumvent the subjectivity of a consciousness report, we only retained regions that also fulfilled a second, more objective criterion: following runs of subjective consciousness report, we conducted a surprise forced-choice yes/no recognition task and sorted events on the basis of subsequent memory retrieval success rather than subjective consciousness report. Only regions that stringently fulfilled both criteria, an association with a positive consciousness report and with successful encoding, conjointly for words and for pictures, were retained as amodal zones of supra- versus subliminal processing. To further verify the level and accuracy of stimulus identification and also to evaluate how the extent of the amodal activations obtained in the first and second experiment related to those obtained in more classical tasks, we conducted a third, overt naming/reading fMRI experiment.
Section snippets
Subjects
Twenty-eight healthy native Dutch speakers (15 women and 13 men, between 19 and 29 years of age) participated in the first fMRI experiment, which was based on the subjective consciousness report, and 8 additional subjects (3 women and 5 men, age range 19–25 years) in a control experiment. Nineteen other subjects (12 women and 7 men, between 19 and 35 years of age) participated in a third fMRI experiment, the subsequent memory retrieval experiment. Four other subjects (1 man and 3 women, between
First experiment
In the first experiment, our titration procedure yielded an average word presentation duration across runs and subjects of 35.3 ms (SD = 9.6) and an average picture presentation duration of 50.7 ms (SD = 15.8). Thirty-four out of 156 runs were excluded because subjects responded false-positively to more than 3 out of 8 catch trials in these runs, indicating that in these runs subjects produced a positive key press even when they had failed to identify the stimuli at the basic level. Subjects
First experiment
Regardless of input modality (conjunction contrast 1), words or pictures, a positive consciousness report was associated with significantly higher activity in the middle third of the left inferior frontal sulcus (IFS) (Figs. 3A and C), the left inferior frontal junction (IFJ) (Figs. 3B and C), the border between the posterior and the middle third of the left occipitotemporal sulcus (OTS) (Figs. 3C and D) and the middle third of the left intraparietal sulcus (IPS) (Figs. 3C and E) (Table 2). A
Discussion
Two regions, the middle third of the left IFS (Figs. 3A and C) and the left occipitotemporal sulcus (Figs. 3C and D), were activated in an amodal manner when a stimulus, word or picture, gained access to consciousness. In left OTS, and to a lesser degree also in IFS, this effect consisted of an enhancement of a response that was also present for subliminal stimuli. Activation in these two regions predicted subsequent memory retrieval success for both words and pictures (Fig. 5, orange outline).
Acknowledgments
This work was supported by Research Foundation Flanders (FWO) [G.0076.02 and G0668.07 to R.V.], K.U.Leuven [OT/04/41, OT/08/056, EF/05/014 to R.V.], and Federaal Wetenschapsbeleid belspo [IUAP P6/29].
References (88)
The conscious access hypothesis: origins and recent evidence
Trends Cogn. Sci.
(2002)- et al.
Cortical mechanisms specific to explicit visual object recognition
Neuron
(2001) - et al.
Conscious awareness of flicker in humans involves frontal and parietal cortex
Curr. Biol.
(2006) - et al.
On the status of object concepts in aphasia
Brain Lang.
(1997) - et al.
Brain activity during automatic semantic priming revealed by event-related functional magnetic resonance imaging
NeuroImage
(2003) - et al.
Susceptibility-induced loss of signal: comparing PET and fMRI on a semantic task
Neuroimage
(2000) - et al.
Common regions of the human frontal lobe recruited by diverse cognitive demands
Trends Neurosci.
(2000) - et al.
Spatial dynamics of masked picture repetition effects
Neuroimage
(2007) - et al.
Specific reading and phonological processing deficits are associated with damage to the left frontal operculum
Cortex
(2006) - et al.
Other minds in the brain: a functional imaging study of “theory of mind” in story comprehension
Cognition
(1995)