Olli Rummukainen
Abstract
Most research on multisensory processing focuses on impoverished stimuli and simple tasks. In consequence, very little is known about the sensory contributions in the perception of real environments. Here, we presented 23 participants with paired comparison tasks, where natural scenes were discriminated in three perceptually meaningful attributes: movement, openness, and noisiness. The goal was to assess the auditory and visual modality contributions in scene discrimination with short (≤500ms) natural scene exposures. The scenes were reproduced in an immersive audiovisual environment with 3D sound and surrounding visuals. Movement and openness were found to be mainly visual attributes with some input from auditory information. In some scenes, the auditory system was able to derive information about movement and openness that was comparable with audiovisual condition already after 500ms stimulation. Noisiness was mainly auditory, but visual information was found to have a facilitatory role in a few scenes. The sensory weights were highly imbalanced in favor of the stronger modality, but the weaker modality was able to affect the bimodal estimate in some scenes.
- D. Alais and D. Burr. 2004. The ventriloquist effect results from near-optimal bimodal integration. Current Biology 14, 257--262.Google Scholar
Cross Ref
- Alan Armstrong and Johann Issartel. 2014. Sensorimotor synchronization with audio-visual stimuli: Limited multisensory integration. Experimental Brain Research 232, 11.Google ScholarCross Ref
- D. Bates, M. Mächler, B. Bolker, and S. Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67, 1, 1--48.Google ScholarCross Ref
- Yi Chuan Chen and Charles Spence. 2010. When hearing the bark helps to identify the dog: Semantically-congruent sounds modulate the identification of masked pictures. Cognition 114, 389--404.Google ScholarCross Ref
- Yi-Chuan Chen, Su-Ling Yeh, and Charles Spence. 2011. Crossmodal constraints on human perceptual awareness: Auditory semantic modulation of binocular rivalry. Frontiers in Psychology 2, 1--13.Google ScholarCross Ref
- Ken Cheng, Sara J. Shettleworth, Janellen Huttenlocher, and John J. Rieser. 2007. Bayesian integration of spatial information. Psychological Bulletin 133, 4, 625--637.Google ScholarCross Ref
- Francis B. Colavita. 1974. Human sensory dominance. Perception & Psychophysics 16, 2, 409--412.Google ScholarCross Ref
- Verena Conrad, Mario Kleiner, Andreas Bartels, Jessica Hartcher O’Brien, Heinrich H. Bülthoff, and Uta Noppeney. 2013. Naturalistic stimulus structure determines the integration of audiovisual looming signals in binocular rivalry. PLoS One 8, 8, e70710.Google ScholarCross Ref
- Beatrice De Gelder and Paul Bertelson. 2003. Multisensory integration, perception and ecological validity. Trends in Cognitive Sciences 7, 10, 460--467.Google ScholarCross Ref
- Oliver Doehrmann and Marcus J. Naumer. 2008. Semantics and the multisensory brain: How meaning modulates processes of audio-visual integration. Brain Research 1242, 136--150.Google ScholarCross Ref
- A. J. Ecker and L. M. Heller. 2005. Auditory-visual interactions in the perception of a ball’s path. Perception 34, 1, 59--75.Google ScholarCross Ref
- Marc O. Ernst and Heinrich H. Bülthoff. 2004. Merging the senses into a robust percept. Trends in Cognitive Sciences 8, 4, 162--169.Google ScholarCross Ref
- K. K. Evans and Anne Treisman. 2010. Natural cross-modal mappings between visual and auditory features. Journal of Vision 10, 1, 1--12.Google Scholar
- J. Gómez Bolaños and V. Pulkki. 2012. Immersive audiovisual environment with 3D audio playback. In 132nd Convention of the Audio Engineering Society. Budapest, Hungary, 1--9.Google Scholar
- David Hecht and Miriam Reiner. 2009. Sensory dominance in combinations of audio, visual and haptic stimuli. Experimental Brain Research 193, 2, 307--314.Google ScholarCross Ref
- A. Koene, Derek Arnold, and Alan Johnston. 2007. Bimodal sensory discrimination is finer than dual single modality discrimination. Journal of Vision 7, 11, 1--11.Google ScholarCross Ref
- J. R. Landis and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 159--174.Google ScholarCross Ref
- Paul J. Laurienti, Robert A. Kraft, Joseph A. Maldjian, Jonathan H. Burdette, and Mark T. Wallace. 2004. Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research 158, 4, 405--414.Google ScholarCross Ref
- J. López-Moliner and S. Soto-Faraco. 2007. Vision affects how fast we hear sounds move. Journal of Vision 7, 12, 1--7.Google ScholarCross Ref
- Catarina Mendonça, Olli Rummukainen, and Ville Pulkki. 2015. 3D sound can have a negative impact on the perception of visual content in audiovisual reproductions. In 21st International Conference on Auditory Display (ICAD’15). Graz, Austria.Google Scholar
- Catarina Mendonça, Jorge A. Santos, and Joan López-Moliner. 2011. The benefit of multisensory integration with biological motion signals. Experimental Brain Research 213, 2--3, 185--192.Google ScholarCross Ref
- Alexis Pérez-Bellido, Salvador Soto-Faraco, and Joan López-Moliner. 2013. Sound-driven enhancement of vision: Disentangling detection-level from decision-level contributions. Journal of Neurophysiology 109, 4, 1065--1077.Google ScholarCross Ref
- Robert J. Pheasant, Mark N. Fisher, Greg R. Watts, David J. Whitaker, and Kirill V. Horoshenkov. 2010. The importance of auditory-visual interaction in the construction of tranquil space. Journal of Environmental Psychology 30, 4, 501--509.Google ScholarCross Ref
- A. Politis and V. Pulkki. 2011. Broadband analysis and synthesis for DirAC using A-format. In 131st Convention of the Audio Engineering Society. New York, NY, 1--11.Google Scholar
- V. Pulkki. 2007. Spatial sound reproduction with directional audio coding. Journal of the Audio Engineering Society 55, 6, 503--516.Google Scholar
- R Core Team. 2015. R: A Language and Environment for Statistical Computing. http://www.R-project.org.Google Scholar
- O. Rummukainen, J. Radun, T. Virtanen, and V. Pulkki. 2014. Categorization of natural dynamic audiovisual scenes. PLoS One 9, 5, e95848.Google ScholarCross Ref
- Charles Spence and Sarah Squire. 2003. Multisensory integration: Maintaining the perception of synchrony. Current Biology 13, 519--521.Google ScholarCross Ref
- Jye-sheng Tan and Su-ling Yeh. 2015. Audiovisual integration facilitates unconscious visual scene processing. Journal of Experimental Psychology: Human Perception and Performance 41, 5, 1325--1335.Google ScholarCross Ref
- Michel Treisman. 1998. Combining information: Probability summation and probability averaging in detection and discrimination. Psychological Methods 3, 2, 252--265.Google ScholarCross Ref
- J. Vilkamo, T. Lokki, and V. Pulkki. 2009. Directional audio coding: Virtual microphone-based synthesis and subjective evaluation. Journal of the Audio Engineering Society 57, 9, 709--724.Google Scholar
- David R. Wozny, Ulrik R. Beierholm, and Ladan Shams. 2008. Human trimodal perception follows optimal statistical inference. Journal of Vision 8, 3, 1--11.Google ScholarCross Ref
- Shlomit Yuval-Greenberg and Leon Y. Deouell. 2009. The dog’s meow: Asymmetrical interaction in cross-modal object recognition. Experimental Brain Research 193, 4, 603--614.Google ScholarCross Ref
Index Terms
- Reproducing Reality: Multimodal Contributions in Natural Scene Discrimination
Recommendations
Neural Correlates of Perceiving Visual Scenes as "Natural” or "Unnatural” - An fMRI Study with Contrast Modulated Pictures
ISUC '08: Proceedings of the 2008 Second International Symposium on Universal CommunicationUsing functional magnetic resonance imaging, we examined whether distinct activity patterns are elicited by natural and unnatural visual stimuli. Stimuli were black and white pictures portraying various scenes (e.g., cities, country fields); natural ...
Moving sounds enhance the visually-induced self-motion illusion (circular vection) in virtual reality
While rotating visual and auditory stimuli have long been known to elicit self-motion illusions (“circular vection”), audiovisual interactions have hardly been investigated. Here, two experiments investigated whether visually induced circular vection ...
2006 Special Issue: Attention in natural scenes: Neurophysiological and computational bases
How does attention operate in natural scenes? We show that the receptive fields of inferior temporal cortex neurons that implement object representations become small and located at the fovea in complex natural scenes. This facilitates the readout of ...
Comments