Abstract
While previous research has shown that during mental imagery participants look back to areas visited during encoding it is unclear what happens when information presented during encoding is incongruent. To investigate this research question, we presented 30 participants with incongruent audio-visual associations (e.g. the image of a car paired with the sound of a cat) and later asked them to create a congruent mental representation based on the auditory cue (e.g. to create a mental representation of a cat while hearing the sound of a cat). The results revealed that participants spent more time in the areas where they previously saw the object and that incongruent audio-visual information during encoding did not appear to interfere with the generation and maintenance of mental images. This finding suggests that eye movements can be flexibly employed during mental imagery depending on the demands of the task.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ballard DH, Hayhoe MM, Pook PK, Rao RP (1997) Deictic codes for the embodiment of cognition. Behav Brain Sci 20(4):723–742
Berger CC, Ehrsson HH (2018) Mental imagery induces cross-modal sensory plasticity and changes future auditory perception. Psychol Sci 29(6):926–935
Borst G, Kosslyn SM (2008) Visual mental imagery and visual perception: structural equivalence revealed by scanning processes. Mem Cogn 36(4):849–862
Brandt SA, Stark LW (1997) Spontaneous eye movements during visual imagery reflect the content of the visual scene. J Cognit Neurosci 9(1):27–38
Ferreira F, Apel J, Henderson JM (2008) Taking a new look at looking at nothing. Trends Cognit Sci 12(11):405–410
Hebb DO (1968) Concerning imagery. Psychol Rev 75(6):466
Johansson R, Johansson M (2014) Look here, eye movements play a functional role in memory retrieval. Psychol Sci 25(1):236–242
Johansson R, Holsanova J, Dewhurst R, Holmqvist K (2012) Eye movements during scene recollection have a functional role, but they are not reinstatements of those produced during encoding. J Exp Psychol Hum Percept Perform 38(5):1289
Kumcu A, Thompson RL (2016) Spatial interference and individual differences in looking at nothing for verbal memory. Paper presented at the CogSci
Kumcu A, Thompson RL (2018) Less imageable words lead to more looks to blank locations during memory retrieval. Psychol Res 84:667–684
Lacey S, Lawson R (2013) Multisensory imagery. Springer, Berlin
Laeng B, Teodorescu D-S (2002) Eye scanpaths during visual imagery reenact those of perception of the same visual scene. Cognit Sci 26(2):207–231
Laeng B, Bloem IM, D’Ascenzo S, Tommasi L (2014) Scrutinizing visual images: the role of gaze in mental imagery and memory. Cognition 131(2):263–283
Martarelli CS, Mast FW (2011) Preschool children’s eye-movements during pictorial recall. Br J Dev Psychol 29(3):425–436
Martarelli CS, Mast FW (2013) Eye movements during long-term pictorial recall. Psychol Res 77(3):303–309
Martarelli CS, Chiquet S, Laeng B, Mast FW (2017) Using space to represent categories: Insights from gaze position. Psychol Res 81(4):721–729
Richardson DC, Spivey MJ (2000) Representation, space and Hollywood Squares: looking at things that aren’t there anymore. Cognition 76(3):269–295
Richardson DC, Altmann GT, Spivey MJ, Hoover MA (2009) Much ado about eye movements to nothing: a response to Ferreira et al.: taking a new look at looking at nothing. Trends Cognit Sci 13(6):235–236
Ryan JD, Shen K (2020) The eyes are a window into memory. Curr Opin Behav Sci 32:1–6
Scholz A, Mehlhorn K, Krems JF (2016) Listen up, eye movements play a role in verbal memory retrieval. Psychol Res 80(1):149–158
Scholz A, Klichowicz A, Krems JF (2018) Covert shifts of attention can account for the functional role of “eye movements to nothing.” Mem Cognit 46(2):230–243
Viggiano MP, Giovannelli F, Giganti F, Rossi A, Metitieri T, Rebai M, Cincotta M (2017) Age-related differences in audiovisual interactions of semantically different stimuli. Dev Psychol 53(1):138
Funding
This work was supported by a scholarship from the Ministry of Education Malaysia to HU (KPT(BS)850610065436).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
The study was in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Reviewers: Luca Tommasi (University of Chieti) and a second reviewer who prefers to remain anonymous.
Handling editor: Bruno Laeng (University of Oslo).
Appendices
Appendix A: list of quadrant allocation for image-sound pairs used during the tasks
In the intra-categorical group (Group A) as well as in the extra-categorical group (Group B), 32 stimuli were presented. Please find in the table below a list of stimuli used during the experiment as well as the position of the images on the screen. AOI 1 corresponds to the upper-left quadrant, AOI 2 corresponds to the upper-right quadrant, AOI 3 corresponds to the lower-right quadrant, and AOI 4 corresponds to the lower-left quadrant.
Group A (intra-categorical pairs) | Group B (extra-categorical pairs) | ||||
|---|---|---|---|---|---|
Quadrant | Image | Sound | Quadrant | Image | Sound |
AOI 1 | Dog | Hen | AOI 1 | Cat | Car |
AOI 4 | Hen | Dog | AOI 4 | Car | Cat |
AOI 4 | Tiger | Horse | AOI 4 | Lion | Hand bell |
AOI 1 | Horse | Tiger | AOI 1 | Hand bell | Lion |
AOI 2 | Pig | Frog | AOI 2 | Monkey | Guitar |
AOI 3 | Frog | Pig | AOI 3 | Guitar | Monkey |
AOI 3 | Elephant | Mosquito | AOI 3 | Donkey | Police siren |
AOI 2 | Mosquito | Elephant | AOI 2 | Police car | Donkey |
AOI 1 | Camel | Dolphin | AOI 1 | Duck | Scissors |
AOI 3 | Dolphin | Camel | AOI 3 | Scissors | Duck |
AOI 3 | Goose | Crocodile | AOI 3 | Sheep | Vacuum |
AOI 1 | Crocodile | Goose | AOI 1 | Vacuum | Sheep |
AOI 2 | Canary bird | Goat | AOI 2 | Cow | Washer |
AOI 4 | Goat | Canary bird | AOI 4 | Washer | Cow |
AOI 4 | Turkey | Bee | AOI 4 | Wolf | Bagpipes |
AOI 2 | Bee | Turkey | AOI 2 | Bagpipes | Wolf |
AOI 1 | Ship | Fire truck siren | AOI 1 | Flies | Train |
AOI 4 | Fire Truck | Ship | AOI 4 | Train | Flies |
AOI 4 | Piano | Trumpet | AOI 4 | Snake | Airplane |
AOI 1 | Trumpet | Piano | AOI 1 | Airplane | Snake |
AOI 2 | Helicopter | Motorcycle | AOI 2 | Owl | Bicycle |
AOI 3 | Motorcycle | Helicopter | AOI 3 | Bicycle | Owl |
AOI 3 | Violin | Bongos | AOI 3 | Rooster | Tambourine |
AOI 2 | Bongos | Violin | AOI 2 | Tambourine | Rooster |
AOI 1 | Whistle | Hair dryer | AOI 1 | Eagle | Telephone |
AOI 3 | Hair Dryer | Whistle | AOI 3 | Telephone | Eagle |
AOI 3 | Lawn Mower | Hammer | AOI 3 | Mouse | Flute |
AOI 1 | Hammer | Lawn mower | AOI 1 | Flute | Mouse |
AOI 2 | Alarm clock | Blender | AOI 2 | Bear | Drum |
AOI 4 | Blender | Alarm clock | AOI 4 | Drum | Bear |
AOI 4 | Printer | Microwave | AOI 4 | Crickets | Electric drill |
AOI 2 | Microwave | Printer | AOI 2 | Electric drill | Crickets |
Appendix B: gender differences
Given the large overrepresentation of females (n = 24) over males (n = 6) in our sample, we run the mixed ANOVA presented in paragraph ‘Imagery phase: percentage of dwell time in the three imagery tasks’ by excluding the six men from the analyses. The analysis again revealed a significant main effect of Cue Quadrant (F(1.82, 40) = 8.00, p = 0.002, η2p = 0.27). Dwell time was significantly higher in VisualCQ (35.6%, SD 16.8) than in the other Cue Quadrants (AudioCQ: 22%, SD 5.56, p = 0.002; NoCQ1: 19.3%, SD 9.22, p < 0.001; NoCQ2: 23.1%, SD 12.2, p = 0.005). However, in this analysis the interaction of Task with Cue Quadrant (F(3.65, 80.33) = 2.75, p = 0.038, η2p = 0.11) turned out to be significant. All other effects were non-significant. Please find in Table
1 a summary of the results of the ANOVAs for percentage of dwell time in the Imagery phase.
Given the significant interaction of Task with Cue Quadrant, we conducted three additional ANOVAs for each task on the percentage of dwell time spent in the quadrants. In the Image Generation task (Task 1), we found a significant effect of Cue Quadrant (F(2.32, 53.29) = 8.06, p < 0.001, η2p = 0.26). Post hoc tests with Tukey correction showed that dwell time was significantly higher in VisualCQ (33%, SD 10.8) than in the other Cue Quadrants (AudioCQ: 22.8%, SD 4.84, p = 0.002; NoCQ1: 21.4%, SD 6.86, p < 0.001; NoCQ2: 22.8%, SD 8.53, p = 0.002). Similarly, in the Image Inspection task (Task 2), we also found a significant effect of Cue Quadrant (F(1.90, 43.81) = 7.08, p = 0.002, η2p = 0.24) and post hoc tests with Tukey correction showed that dwell time was significantly higher in VisualCQ (37.6%, SD 19.4) than in the other Cue Quadrants (AudioCQ: 21.7%, SD 6.24, p = 0.004; NoCQ1: 19%, SD 11.4, p < 0.001; NoCQ2: 21.7%, SD 13.8, p = 0.004). Finally, in the Vividness Rating task (Task 3), we also found a significant effect of Cue Quadrant (F(1.62, 37.26) = 6.89, p = 0.005, η2p = 0.23). Post hoc tests with Tukey correction showed that dwell time was significantly higher in VisualCQ (36.1%, SD 19.3) than in the AudioCQ (21.4%, SD 5.64, p = 0.006) and NoCQ1 (17.5%, SD 8.87 p > 0.001). The difference between VisualCQ and NoCQ2 (25%, SD 13.8, p = 0.056) just missed significance. The pattern of results turned out to be very similar, when excluding males from the sample. Future research should include a larger sample of males to investigate possible gender differences.
Rights and permissions
About this article
Cite this article
Umar, H., Mast, F.W., Cacchione, T. et al. The prioritization of visuo-spatial associations during mental imagery. Cogn Process 22, 227–237 (2021). https://doi.org/10.1007/s10339-020-01010-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10339-020-01010-5