Skip to main content
SpringerLink
Log in
Book cover

Cognitive Behavioural Systems pp 304–320Cite as

Effects of Experience, Training and Expertise on Multisensory Perception: Investigating the Link between Brain and Behavior

  • Conference paper

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 7403))

Abstract

The ability to successfully integrate information from different senses is of paramount importance for perceiving the world and has been shown to change with experience. We first review how experience, in particular musical experience, brings about changes in our ability to fuse together sensory information about the world. We next discuss evidence from drumming studies that demonstrate how the perception of audiovisual synchrony depends on experience. These studies show that drummers are more robust than novices to perturbations of the audiovisual signals and appear to use different neural mechanisms in fusing sight and sound. Finally, we examine how experience influences audiovisual speech perception. We present an experiment investigating how perceiving an unfamiliar language influences judgments of temporal synchrony of the audiovisual speech signal. These results highlight the influence of both the listener’s experience with hearing an unfamiliar language as well as the speaker’s experience with producing non-native words.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Landy, M.S., et al.: Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res. 35(3), 389–412 (1995)

    Article  Google Scholar 

  2. Ernst, M.O., Banks, M.S.: Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870), 429–433 (2002)

    Article  Google Scholar 

  3. Green, C.S., Bavelier, D.: Action-video-game experience alters the spatial resolution of vision. Psychol. Sci. 18(1), 88–94 (2007)

    Article  Google Scholar 

  4. Simmons, R.W., Locher, P.J.: Role of extended perceptual experience upon haptic perception of nonrepresentational shapes. Percept. Mot. Skills 48(3 Pt. 1), 987–991 (1979)

    Article  Google Scholar 

  5. Kisilevsky, B.S., et al.: Effects of experience on fetal voice recognition. Psychol. Sci. 14(3), 220–224 (2003)

    Article  Google Scholar 

  6. Atkins, J.E., Fiser, J., Jacobs, R.A.: Experience-dependent visual cue integration based on consistencies between visual and haptic percepts. Vision Res. 41(4), 449–461 (2001)

    Article  Google Scholar 

  7. Jacobs, R.A., Fine, I.: Experience-dependent integration of texture and motion cues to depth. Vision Res. 39(24), 4062–4075 (1999)

    Article  Google Scholar 

  8. Powers III, A.R., Hillock, A.R., Wallace, M.T.: Perceptual training narrows the temporal window of multisensory binding. J. Neurosci. 29(39), 12265–12674 (2009)

    Article  Google Scholar 

  9. Mamassian, P., Goutcher, R.: Prior knowledge on the illumination position. Cognition 81(1), B1-B9 (2001)

    Google Scholar 

  10. Mamassian, P., Landy, M.S.: Interaction of visual prior constraints. Vision Res. 41(20), 2653–2668 (2001)

    Article  Google Scholar 

  11. Mondloch, C.J., et al.: Face perception during early infancy. Psychol. Sci. 10(5), 419–422 (1999)

    Article  Google Scholar 

  12. Turati, C.: Why faces are not special to newborns: An alternative account of the face preference. Current Directions in Psychological Science 13(1), 5–8 (2004)

    Article  Google Scholar 

  13. Hershber, W.: Attached-Shadow Orientation Perceived as Depth by Chickens Reared in an Environment Illuminated from Below. Journal of Comparative and Physiological Psychology 73(3), 407-&. (1970)

    Google Scholar 

  14. Gregory, R.L.: Knowledge in perception and illusion. Philosophical Transactions of the Royal Society of London Series B-Biological Sciences 352(1358), 1121–1127 (1997)

    Article  Google Scholar 

  15. Adams, W.J., Graf, E.W., Ernst, M.O.: Experience can change the ’light-from-above’ prior. Nat. Neurosci. 7(10), 1057–1058 (2004)

    Article  Google Scholar 

  16. Dayan, E., Cohen, L.G.: Neuroplasticity subserving motor skill learning. Neuron 72(3), 443–454 (2011)

    Article  Google Scholar 

  17. May, A.: Experience-dependent structural plasticity in the adult human brain. Trends Cogn. Sci. 15(10), 475–482 (2011)

    Article  Google Scholar 

  18. Pascual-Leone, A., et al.: The plastic human brain cortex. Annu. Rev. Neurosci. 28, 377–401 (2005)

    Article  Google Scholar 

  19. Latinus, M., Crabbe, F., Belin, P.: Learning-induced changes in the cerebral processing of voice identity. Cereb Cortex 21(12), 2820–2828 (2011)

    Article  Google Scholar 

  20. Bentin, S., et al.: Electrophysiological Studies of Face Perception in Humans. J. Cogn. Neurosci. 8(6), 551–565 (1996)

    Article  Google Scholar 

  21. Rossion, B., et al.: Expertise training with novel objects leads to left-lateralized facelike electrophysiological responses. Psychol. Sci. 13(3), 250–257 (2002)

    Article  Google Scholar 

  22. Bukach, C.M., et al.: Does acquisition of Greeble expertise in prosopagnosia rule out a domain-general deficit? Neuropsychologia 50(2), 289–304 (2012)

    Article  Google Scholar 

  23. Tanaka, J.W., Curran, T.: A neural basis for expert object recognition. Psychol. Sci. 12(1), 43–47 (2001)

    Article  Google Scholar 

  24. Busey, T.A., Vanderkolk, J.R.: Behavioral and electrophysiological evidence for configural processing in fingerprint experts. Vision Res. 45(4), 431–448 (2005)

    Article  Google Scholar 

  25. Busey, T.A., Parada, F.J.: The nature of expertise in fingerprint examiners. Psychon. Bull. Rev. 17(2), 155–160 (2010)

    Article  Google Scholar 

  26. Calvo-Merino, B., et al.: Action observation and acquired motor skills: an FMRI study with expert dancers. Cereb Cortex 15(8), 1243–1249 (2005)

    Article  Google Scholar 

  27. Calvo-Merino, B., et al.: Seeing or doing? Influence of visual and motor familiarity in action observation. Curr. Biol. 16(19), 1905–1910 (2006)

    Article  Google Scholar 

  28. Cross, E.S., Hamilton, A.F., Grafton, S.T.: Building a motor simulation de novo: observation of dance by dancers. Neuroimage 31(3), 1257–1267 (2006)

    Article  Google Scholar 

  29. Maguire, E.A., et al.: Navigation-related structural change in the hippocampi of taxi drivers. Proc. Natl. Acad. Sci. U S A 97(8), 4398–4403 (2000)

    Article  Google Scholar 

  30. Woollett, K., Maguire, E.A.: Acquiring "the Knowledge" of London’s layout drives structural brain changes. Curr. Biol. 21(24), 2109–2114 (2011)

    Article  Google Scholar 

  31. Hufner, K., et al.: Structural and functional plasticity of the hippocampal formation in professional dancers and slackliners. Hippocampus 21(8), 855–865 (2011)

    Google Scholar 

  32. Bezzola, L., et al.: Training-induced neural plasticity in golf novices. J. Neurosci. 31(35), 12444–12448 (2011)

    Article  Google Scholar 

  33. Johansen-Berg, H., et al.: Integrity of white matter in the corpus callosum correlates with bimanual co-ordination skills. Neuroimage 36(suppl. 2), T16–T21 (2007)

    Article  Google Scholar 

  34. Tuch, D.S., et al.: Choice reaction time performance correlates with diffusion anisotropy in white matter pathways supporting visuospatial attention. Proc. Natl. Acad. Sci. U S A 102(34), 12212–12217 (2005)

    Article  Google Scholar 

  35. Draganski, B., et al.: Neuroplasticity: changes in grey matter induced by training. Nature 427(6972), 311–312 (2004)

    Article  Google Scholar 

  36. Scholz, J., et al.: Training induces changes in white-matter architecture. Nat. Neurosci. 12(11), 1370–1371 (2009)

    Article  Google Scholar 

  37. Bengtsson, S.L., et al.: Extensive piano practicing has regionally specific effects on white matter development. Nat. Neurosci. 8(9), 1148–1150 (2005)

    Article  Google Scholar 

  38. Bermudez, P., et al.: Neuroanatomical correlates of musicianship as revealed by cortical thickness and voxel-based morphometry. Cereb Cortex 19(7), 1583–1596 (2009)

    Article  Google Scholar 

  39. Gaser, C., Schlaug, G.: Brain structures differ between musicians and non-musicians. J. Neurosci. 23(27), 9240–9245 (2003)

    Google Scholar 

  40. Hutchinson, S., et al.: Cerebellar volume of musicians. Cereb Cortex 13(9), 943–949 (2003)

    Article  Google Scholar 

  41. Halwani, G.F., et al.: Effects of practice and experience on the arcuate fasciculus: comparing singers, instrumentalists, and non-musicians. Front Psychol. 2, 156 (2011)

    Article  Google Scholar 

  42. Imfeld, A., et al.: White matter plasticity in the corticospinal tract of musicians: a diffusion tensor imaging study. Neuroimage 46(3), 600–607 (2009)

    Article  Google Scholar 

  43. Schmithorst, V.J., Wilke, M.: Differences in white matter architecture between musicians and non-musicians: a diffusion tensor imaging study. Neurosci. Lett. 321(1-2), 57–60 (2002)

    Article  Google Scholar 

  44. Schlaug, G., et al.: In vivo evidence of structural brain asymmetry in musicians. Science 267(5198), 699–701 (1995)

    Article  Google Scholar 

  45. Ozturk, A.H., et al.: Morphometric comparison of the human corpus callosum in professional musicians and non-musicians by using in vivo magnetic resonance imaging. J. Neuroradiol. 29(1), 29–34 (2002)

    Google Scholar 

  46. Tervaniemi, M.: Musicians–same or different? Ann. N Y Acad. Sci. 1169, 151–156 (2009)

    Article  Google Scholar 

  47. Aydin, K., et al.: Quantitative proton MR spectroscopic findings of cortical reorganization in the auditory cortex of musicians. AJNR Am. J. Neuroradiol. 26(1), 128–136 (2005)

    Google Scholar 

  48. Elbert, T., et al.: Increased cortical representation of the fingers of the left hand in string players. Science 270(5234), 305–357 (1995)

    Article  Google Scholar 

  49. Hyde, K.L., et al.: Musical training shapes structural brain development. J. Neurosci. 29(10), 3019–3025 (2009)

    Article  Google Scholar 

  50. Hyde, K.L., et al.: The effects of musical training on structural brain development: a longitudinal study. Ann. N Y Acad. Sci. 1169, 182–186 (2009)

    Article  Google Scholar 

  51. Kraus, N., Chandrasekaran, B.: Music training for the development of auditory skills. Nat. Rev. Neurosci. 11(8), 599–605 (2010)

    Article  Google Scholar 

  52. Munte, T.F., Altenmuller, E., Jancke, L.: The musician’s brain as a model of neuroplasticity. Nat. Rev. Neurosci. 3(6), 473–478 (2002)

    Google Scholar 

  53. Bangert, M., Altenmuller, E.O.: Mapping perception to action in piano practice: a longitudinal DC-EEG study. BMC Neurosci. 4, 26 (2003)

    Article  Google Scholar 

  54. Magne, C., Schon, D., Besson, M.: Musician children detect pitch violations in both music and language better than nonmusician children: behavioral and electrophysiological approaches. J. Cogn. Neurosci. 18(2), 199–211 (2006)

    Article  Google Scholar 

  55. Milovanov, R., Tervaniemi, M.: The Interplay between Musical and Linguistic Aptitudes: A Review. Front Psychol. 2, 321 (2011)

    Article  Google Scholar 

  56. Chan, A.S., Ho, Y.C., Cheung, M.C.: Music training improves verbal memory. Nature 396(6707), 128 (1998)

    Article  Google Scholar 

  57. Aleman, A., et al.: Music training and mental imagery ability. Neuropsychologia 38(12), 1664–1668 (2000)

    Article  Google Scholar 

  58. Brochard, R., Dufour, A., Despres, O.: Effect of musical expertise on visuospatial abilities: evidence from reaction times and mental imagery. Brain Cogn. 54(2), 103–109 (2004)

    Article  Google Scholar 

  59. Schmithorst, V.J., Holland, S.K.: The effect of musical training on the neural correlates of math processing: a functional magnetic resonance imaging study in humans. Neurosci. Lett. 354(3), 193–196 (2004)

    Article  Google Scholar 

  60. Besson, M., Chobert, J., Marie, C.: Transfer of Training between Music and Speech: Common Processing, Attention, and Memory. Front Psychol. 2, 94 (2011)

    Article  Google Scholar 

  61. Patel, A.D.: Why would Musical Training Benefit the Neural Encoding of Speech? The OPERA Hypothesis. Front Psychol. 2, 142 (2011)

    Article  Google Scholar 

  62. Wallace, M.T., Stein, B.E.: Sensory and multisensory responses in the newborn monkey superior colliculus. J. Neurosci. 21(22), 8886–8894 (2001)

    Google Scholar 

  63. Wallace, M.T., Stein, B.E.: Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17(7), 2429–2444 (1997)

    Google Scholar 

  64. Wallace, M.T., Stein, B.E.: Cross-modal synthesis in the midbrain depends on input from cortex. J. Neurophysiol. 71(1), 429–4232 (1994)

    Google Scholar 

  65. Jiang, W., Jiang, H., Stein, B.E.: Neonatal cortical ablation disrupts multisensory development in superior colliculus. J. Neurophysiol. 95(3), 1380–1396 (2006)

    Article  Google Scholar 

  66. Haslinger, B., et al.: Transmodal sensorimotor networks during action observation in professional pianists. J. Cogn. Neurosci. 17(2), 282–293 (2005)

    Article  Google Scholar 

  67. Hodges, D.A., Hairston, W.D., Burdette, J.H.: Aspects of multisensory perception: the integration of visual and auditory information in musical experiences. Ann. N Y Acad. Sci. 1060, 175–185 (2005)

    Article  Google Scholar 

  68. Petrini, K., et al.: Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony. Exp. Brain Res. 198(2-3), 339–352 (2009)

    Article  MathSciNet  Google Scholar 

  69. Petrini, K., Russell, M., Pollick, F.: When knowing can replace seeing in audiovisual integration of actions. Cognition 110(3), 432–439 (2009)

    Article  Google Scholar 

  70. Petrini, K., Holt, S.P., Pollick, F.: Expertise with multisensory events eliminates the effect of biological motion rotation on audiovisual synchrony perception. J. Vis. 10(5), 2 (2010)

    Article  Google Scholar 

  71. Petrini, K., et al.: Action expertise reduces brain activity for audiovisual matching actions: an fMRI study with expert drummers. Neuroimage 56, 1480–1492 (2011)

    Article  Google Scholar 

  72. Musacchia, G., et al.: Musicians have enhanced subcortical auditory and audiovisual processing of speech and music. Proc. Natl. Acad. Sci. U S A 104(40), 15894–15898 (2007)

    Article  Google Scholar 

  73. Schulz, M., Ross, B., Pantev, C.: Evidence for training-induced crossmodal reorganization of cortical functions in trumpet players. Neuroreport 14(1), 157–161 (2003)

    Article  Google Scholar 

  74. Lappe, C., et al.: Cortical plasticity induced by short-term unimodal and multimodal musical training. J. Neurosci. 28(39), 9632–9639 (2008)

    Article  Google Scholar 

  75. Schlaug, G., et al.: Effects of music training on the child’s brain and cognitive development. Ann. N Y Acad. Sci. 1060, 219–230 (2005)

    Article  Google Scholar 

  76. Spence, C., Squire, S.: Multisensory integration: maintaining the perception of synchrony. Curr. Biol. 13(13), R519–R521 (2003)

    Article  Google Scholar 

  77. Arrighi, R., Alais, D., Burr, D.: Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. J. Vis. 6(3), 260–268 (2006)

    Article  Google Scholar 

  78. van Wassenhove, V., Grant, K.W., Poeppel, D.: Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45(3), 598–607 (2007)

    Article  Google Scholar 

  79. Vatakis, A., Spence, C.: Audiovisual synchrony perception for music, speech, and object actions. Brain Res. 1111(1), 134–142 (2006)

    Article  Google Scholar 

  80. Vatakis, A., Spence, C.: Audiovisual synchrony perception for speech and music assessed using a temporal order judgment task. Neurosci. Lett. 393(1), 40–44 (2006)

    Article  Google Scholar 

  81. Dixon, N.F., Spitz, L.: The detection of auditory visual desynchrony. Perception 9(6), 719–721 (1980)

    Article  Google Scholar 

  82. Lee, H., Noppeney, U.: Long-term music training tunes how the brain temporally binds signals from multiple senses. Proc. Natl. Acad. Sci. U S A 108(51), E1441–E1450 (2011)

    Article  Google Scholar 

  83. Petrini, K., et al.: The music of your emotions: neural substrates involved in detection of emotional correspondence between auditory and visual music actions. PLoS One 6(4), e19165 (2011)

    Google Scholar 

  84. Petrini, K., McAleer, P., Pollick, F.: Audiovisual integration of emotional signals from music improvisation does not depend on temporal correspondence. Brain Res. 1323, 139–148 (2010)

    Article  Google Scholar 

  85. Hein, G., et al.: Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J. Neurosci. 27(30), 7881–7887 (2007)

    Article  Google Scholar 

  86. Kim, R.S., Seitz, A.R., Shams, L.: Benefits of stimulus congruency for multisensory facilitation of visual learning. PLoS One 3(1), e1532 (2008)

    Google Scholar 

  87. Munhall, K.G., et al.: Temporal constraints on the McGurk effect. Percept. Psychophys 58(3), 351–362 (1996)

    Article  Google Scholar 

  88. Vatakis, A., et al.: Temporal recalibration during asynchronous audiovisual speech perception. Exp. Brain Res. 181(1), 173–181 (2007)

    Article  Google Scholar 

  89. Fain, G.L.: Sensory transduction, 340 p. Sinauer Associates, Sunderland (2003)

    Google Scholar 

  90. King, A.J.: Multisensory integration: strategies for synchronization. Curr. Biol. 15(9), R339–R3941 (2005)

    Article  Google Scholar 

  91. King, A.J., Palmer, A.R.: Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp. Brain Res. 60(3), 492–500 (1985)

    Article  Google Scholar 

  92. Waadeland, C.H.: Strategies in empirical studies of swing groove. Musicologia Norvegica 32, 169–191 (2006)

    Google Scholar 

  93. Jansson, G., Johansson, G.: Visual perception of bending motion. Perception 2(3), 321–326 (1973)

    Article  Google Scholar 

  94. McGurk, H.M., Macdonald, J.: Hearing lips and seeing voices. Nature 264((5588), 746–748 (1976)

    Article  Google Scholar 

  95. Vatakis, A., Spence, C.: Crossmodal binding: evaluating the "unity assumption" using audiovisual speech stimuli. Percept. Psychophys 69(5), 744–756 (2007)

    Article  Google Scholar 

  96. Drake, C., Jones, M.R., Baruch, C.: The development of rhythmic attending in auditory sequences: attunement, referent period, focal attending. Cognition 77(3), 251–288 (2000)

    Article  Google Scholar 

  97. Saygin, A.P., Driver, J., de Sa, V.R.: In the footsteps of biological motion and multisensory perception: judgments of audiovisual temporal relations are enhanced for upright walkers. Psychol. Sci. 19(5), 469–475 (2008)

    Article  Google Scholar 

  98. Aschersleben, G., Prinz, W.: Synchronizing actions with events: the role of sensory information. Percept. Psychophys 57(3), 305–317 (1995)

    Article  Google Scholar 

  99. Miyake, Y., Onishi, Y., Poppel, E.: Two types of anticipation in synchronization tapping. Acta Neurobiol Exp. (Wars), 64(3), 415–426 (2004)

    Google Scholar 

  100. Navarra, J., et al.: Perception of audiovisual speech synchrony for native and non-native language. Brain Res. 1323, 84–93 (2010)

    Article  Google Scholar 

  101. Petrini, K., et al.: Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony. Experimental Brain Research 198(2-3), 339–352 (2009)

    Article  MathSciNet  Google Scholar 

  102. Vatakis, A., Spence, C.: Evaluating the influence of the ’unity assumption’ on the temporal perception of realistic audiovisual stimuli. Acta Psychol. (Amst) 127(1), 12–23 (2008)

    Article  Google Scholar 

  103. Vatakis, A., Ghazanfar, A.A., Spence, C.: Facilitation of multisensory integration by the "unity effect" reveals that speech is special. J. Vis. 8(9),14. 1–14.11 (2008)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, 47405, USA

    Scott A. Love

  2. School of Psychology, University of Glasgow, 58 Hillhead Street, Glasgow, G12 8QB, UK

    Frank E. Pollick

  3. Institute of Ophthalmology, University College London, UK

    Karin Petrini

Authors
  1. Scott A. Love
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frank E. Pollick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karin Petrini
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Psychology, and IIASS, Seconda Università degli Studi di Napoli, Italy

    Anna Esposito

  2. Istituto Nazionale di Geofisica e Vulcanologia, sezione di Napoli Osservatorio Vesuviano, Napoli, Italy

    Antonietta M. Esposito

  3. School of Computing Science, University of Glasgow, Glasgow, UK

    Alessandro Vinciarelli

  4. Laboratory of Acoustics and Speech Communication, Technische Universität Dresden, 01062, Dresden, Germany

    Rüdiger Hoffmann

  5. Dept. of Humanities and Social Sciences, Anatolia College/ACT, P.O. Box 21021, 55510, Pylaia, Greece

    Vincent C. Müller

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Love, S.A., Pollick, F.E., Petrini, K. (2012). Effects of Experience, Training and Expertise on Multisensory Perception: Investigating the Link between Brain and Behavior. In: Esposito, A., Esposito, A.M., Vinciarelli, A., Hoffmann, R., Müller, V.C. (eds) Cognitive Behavioural Systems. Lecture Notes in Computer Science, vol 7403. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34584-5_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-34584-5_27

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-34583-8

  • Online ISBN: 978-3-642-34584-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation