Skip to main content
SpringerLink
Log in
Book cover

International Conference on Artificial Neural Networks

ICANN 2009: Artificial Neural Networks – ICANN 2009 pp 345–355Cite as

Brain Electric Microstate and Perception of Simultaneously Audiovisual Presentation

  • Conference paper

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 5768))

Abstract

Associations between picture and sound form the basis of reading. Learning the correspondences between them is a crucial step in reading acquisition. This study was designed to investigate whether task-related processing of audio and visual features was independent or task-related processing in one modality might influence the processing of the other. The present study employed simultaneous audio-visual stimulus in the oddball paradigm to re-examine the effects of attention on audio, visual and audio-visual perception in the non-musician brain. Electroencephalographic (EEG) was recorded from 28 normal participants. None of them had more than three years of formal musical training and none had any musical training within the past five years. Chinese and Korean subjects were presented with tones (auditory: A), pictures (visual: V), and simultaneous tones and pictures (audio-visual: AV). The neural basis of this interaction was investigated by subtracting the event-related potentials (ERPs) to the A and the V stimuli alone from the ERP to the combined AV stimuli (i.e. interaction = AV - (A+V)). The Korean group showed larger mean interaction amplitude and longer in time than the Chinese group. This reveals that experience influences the early cortical automatic processing of linguistically relevant suprasegmental pitch contour. These results suggest that efficient processing of associations between pictures and sounds relies on neural mechanisms similar to those naturally evolved for integrating audiovisual perception.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Tervaniemi, M., Rytkönen, M., Schröger, E., Ilmoniemi, R., Näätänen, R.: Superior formation of cortical memory traces for melodic patterns in musicians. Learning & Memory 8, 295–300 (2001)

    Article  Google Scholar 

  2. Schröger, E., Tervaniemi, M., Wolff, C., Näätänen, R.: Preattentive periodicity detection in auditory patterns as governed by time and intensity information. Brain Res. Cogn. Brain Res. 4, 145–148 (1996)

    Article  Google Scholar 

  3. Alain, C., Woods, D.L., Ogawa, K.H.: Brain indices of automatic pattern processing. NeuroReport 6, 140–144 (1994)

    Article  Google Scholar 

  4. Alho, K., Tervaniemi, M., Huotilainen, M., Lavikainen, J., Tiitinen, H., Ilmoniemi, R.J., Knuutila, J., Näätänen, R.: Processing of complex sounds in the human auditory cortex as revealed by magnetic brain responses. Psychophysiology 33, 396–375 (1996)

    Google Scholar 

  5. Scherg, M., Vajsar, J., Picton, T.W.: A source analysis of the late human auditory evoked potentials. J. Cogn. Neurosci. 5, 363–370 (1989)

    Google Scholar 

  6. Näätänen, R.: Attention and Brain Function. Erlbaum, Hillsdale (1992)

    Google Scholar 

  7. Nyman, G., Alho, K., Iaurinen, P., Paavilainen, P., Radil, T., Rainikainen, K.: Mismatch negativity (MMN) for sequences of auditory and visual stimuli: evidence for a mechanism specific to the auditory modality. Electroenceph. Clin. Neurophys. 77, 436–444 (1990)

    Article  Google Scholar 

  8. Brattico, E., Winkler, I., Näätänen, R., Paavilainen, P., Tervaniemi, M.: Simultaneous storage of two complex temporal sound patterns in auditory sensory memory. NeuroReport 13, 1747–1751 (2002)

    Article  Google Scholar 

  9. Ritter, W., Deacon, D., Gomes, H., Javitt, D.C., Vaughan Jr., H.G.: The mismatch negativity of event-related potentials as a probe of transient auditory memory: a review. Ear Hear 16, 52–67 (1995)

    Article  Google Scholar 

  10. Tales, A., Newton, P., Troscianko, T., Butler, S.: Mismatch negativity in the visual modality. NeuroReport 10, 3363–3367 (1999)

    Article  Google Scholar 

  11. Näätänen, R.: The role attention in auditory information processing as revealed by event-related potentials and other brain measures of cognitive function. Behav. Brain Sci. 13, 201–288 (1990)

    Article  Google Scholar 

  12. Czigler, I., Balázs, L., Pató, L.G.: Visual change detection: event-related potentials are dependent on stimulus location in humans. Neurosci. Lett. 364(3), 149–153 (2004)

    Article  Google Scholar 

  13. Yucel, G., Pettv, C., McCarthv, G., Belger, A.: Graded visual attention modulates brain response evoked by task-irrelevant auditory pitch changes. J. Cogn. Neurosci. 17(2), 1819–1828 (2005)

    Article  Google Scholar 

  14. Czigler, I., Balazs, L., Winkler, I.: Memory-based detection of task-irrelevant visual changes. Psychophysiology 39, 869–873 (2002)

    Article  Google Scholar 

  15. Heslenfeld, D.J.: Visual mismatch negativity. In: Polish, J. (ed.) Detection of change: event-related potential and fMRI findings, pp. 41–60. Kluwer Academic Publishers, Dordrecht (2003)

    Chapter  Google Scholar 

  16. Cammann, R.: Is there a mismatch negativity (MMN) in the visual modality? Behv. Brain Sci. 13, 234–235 (1999)

    Article  Google Scholar 

  17. Pazo-Alvarez, P., Cadaveira, F., Amenedo, E.: MMN in the visual modality: a review. Biol. Psychology 63, 1999–1236 (2003)

    Google Scholar 

  18. Woods, D.J., Alho, K., Algazi, A.: Intermodal selective attention. I. Effects on event-related potentials to lateralized auditory and visual stimuli. Electroenceph. Clin. Neurophys. 82, 341–355 (1992)

    Article  Google Scholar 

  19. Alho, K., Woods, D.L., Algazi, A., Näätänen, R.: Intermodal selective attention. II. Effects of attentional load on processing of auditory and visual stimuli in central space. Electroenceph. Clin. Neurophys. 82, 356–368 (1992)

    Article  Google Scholar 

  20. Smid, H.G.O.M., Jakob, A., Heinze, H.J.: An event-related brain potential study of visual selective attention to conjunctions of color and shape. Psychophysiology 36, 264–279 (1999)

    Article  Google Scholar 

  21. Wackermann, J., Lehmann, D., Michel, C.M., Strik, W.K.: Adaptive segmentation of spontaneous EEG map series into spatially defined microstates. Int. J. Psychophys. 14, 269–283 (1993)

    Article  Google Scholar 

  22. Lehmann, D., Skrandies, W.: Segmentation of evoked potentials based on spatial field configuration in multichannel recordings. In: McCallum, W.C., Zappoli, R., Denoth, F. (eds.) Cerebral Psychophysiology: Studies in Event-Related Potentials. Electroencephalography and Clinical Neurophysiology (suppl. 38), pp. 506–512 (1986)

    Google Scholar 

  23. Klucharev, V., Möttönen, R., Sams, M.: Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Brain Res. Cog. Brain Res. 18, 65–75 (2003)

    Article  Google Scholar 

  24. Möttönen, R., Schurmann, M., Sams, M.: Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study. Neurosci. Let. 363, 112–115 (2004)

    Article  Google Scholar 

  25. Sams, M., Aulanko, R., Hämäläinen, M., Haru, R., Lounasmaa, O.V., Lu, S.-T.: Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci. Let. 127, 141–145 (1991)

    Article  Google Scholar 

  26. Besle, J., Fort, A., Delpuech, C., Giard, M.H.: Bimodal speech: early suppressive visual effects in human auditory cortex. Euro. J. Neurosci. 20, 2225–2234 (2004)

    Article  Google Scholar 

  27. Calvert, G.A., Thesen, T.: Multisensory integration: methodological approaches and emerging principles in the human brain. J. Phys. Paris 98, 191–205 (2004)

    Article  Google Scholar 

  28. van Wassenhove, V., Grant, K.W., Poeppel, D.: Visual speech speeds up the neural processing of auditory speech. Proc. Natl. Acad. Sci. USA 102, 1181–1186 (2005)

    Article  Google Scholar 

  29. Foxe, J.J., Morocz, I.A., Murray, B.A., Higgins, B.A., Javitt, D.C., Schroeder, C.E.: Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Cog. Brain Res. 10, 77–83 (2000)

    Article  Google Scholar 

  30. Giard, M.H., Peronnet, F.: Audio-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J. Cog. Neurosci. 1, 473–490 (1999)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Music, Mahidol University, Salaya, Nakhonpathom, Thailand

    Wichian Sittiprapaporn

  2. Department of Psychiatry, Seoul National University College of Medicine, Seoul, Korea

    Jun Soo Kwon

  3. Clinical Research Center, Seoul National University Hospital, Seoul, Korea

    Jun Soo Kwon

  4. Brain Korea 21 Human Life Science, Seoul National University, Seoul, Korea

    Jun Soo Kwon

Authors
  1. Wichian Sittiprapaporn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jun Soo Kwon
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Dipartimento di Elettronica, Politecnico di Milano, Piazza L. da Vinci 32, 20133, Milano, Italy

    Cesare Alippi

  2. Department of Electrical and Computer Engineering, University of Cyprus, 75 Kallipoleos Street, 1678, Nicosia, Cyprus

    Marios Polycarpou , Christos Panayiotou  & Georgios Ellinas ,  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Sittiprapaporn, W., Kwon, J.S. (2009). Brain Electric Microstate and Perception of Simultaneously Audiovisual Presentation. In: Alippi, C., Polycarpou, M., Panayiotou, C., Ellinas, G. (eds) Artificial Neural Networks – ICANN 2009. ICANN 2009. Lecture Notes in Computer Science, vol 5768. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04274-4_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-04274-4_36

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-04273-7

  • Online ISBN: 978-3-642-04274-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation