Skip to main content
SpringerLink
Log in

Characterizing the Time-Varying Brain Networks of Audiovisual Integration across Frequency Bands

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Multisensory integration involves multiple cortical regions and occurs at multiple stages with attentional modulation. The structure of network formed by the interactive cortical regions reflects the state of working on a current task and changes continuously with the task processing. In addition, the neural oscillatory responses in various frequency bands are associated with different cognitive functions. Thus, studying topological characteristics of time-varying networks across multiple frequency bands helps to elucidate the mechanism of multisensory integration. Here, we designed an event-related experiment using auditory, visual, and audiovisual stimuli to record electroencephalographic data in both attended and unattended conditions and constructed delta-, theta-, alpha-, and beta-band networks at eight time points post-stimulus. We used graph theory to calculate global properties, nodal out-degree, and their correlation with behavioral performance. The increasing clustering coefficient and global efficiency and decreasing characteristic path length indicated that the brain had optimized the configuration across multiple frequency bands over time to efficiently process audiovisual integration. The differences in global properties and hub distributions showed that each frequency band–specificity system in the brain had a different topological structure, indicating that the networks on each frequency band contributed to various cognitive functions and involved in different stages of audiovisual integration. Our results suggest that differences in cognitive function are, at least partly, due to the different network structures across frequency bands and that the frequency band–specificity systems with different distribution are involved in various stages of audiovisual integration and attention modulation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Wu J, Li Q, Bai O, Touge T. Multisensory interactions elicited by audiovisual stimuli presented peripherally in a visual attention task: a behavioral and event-related potential study in humans. J Clin Neurophysiol. 2009;26(6):407–13.

    Google Scholar 

  2. Ye Z, Russeler J, Gerth I, Munte TF. Audiovisual speech integration in the superior temporal region is dysfunctional in dyslexia. Neuroscience. 2017;356:1–10.

    Google Scholar 

  3. Plank T, Rosengarth K, Song W, Ellermeier W, Greenlee MW. Neural correlates of audio-visual object recognition: effects of implicit spatial congruency. Hum Brain Mapp. 2012;33(4):797–811.

    Google Scholar 

  4. Wang H, Zhang G, Liu B. Influence of auditory spatial attention on cross-modal semantic priming effect: evidence from N400 effect. Exp Brain Res. 2016;235(1):331–9.

    Google Scholar 

  5. Erika-Florence M, Leech R, Hampshire A. A functional network perspective on response inhibition and attentional control. Nature Communications. 2014;5(1):4073.

    Google Scholar 

  6. Hampshire A, Sharp DJ. Contrasting network and modular perspectives on inhibitory control. Trends in Cognitive Sciences. 2015;19(8):445–52.

    Google Scholar 

  7. Elbich DB, Molenaar PCM, Scherf KS. Evaluating the organizational structure and specificity of network topology within the face processing system. Hum Brain Mapp. 2019;40(9):1–15.

    Google Scholar 

  8. Spielberg JM, Miller GA, Heller W, Banich MT. Flexible brain network reconfiguration supporting inhibitory control. Proc Natl Acad Sci. 2015;112(32):10020–5.

    Google Scholar 

  9. Talsma D, Woldorff MG. Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience. 2005;17(7):1098–114.

    Google Scholar 

  10. Xie Y, Xu Y, Bian C, Li M. Semantic congruent audiovisual integration during the encoding stage of working memory: an ERP and sLORETA study. Scientific Reports. 2017;7(1).

  11. Donohue SE, Todisco AE, Woldorff MG. The rapid distraction of attentional resources toward the source of incongruent stimulus input during multisensory conflict. Journal of Cognitive Neuroscience. 2013;25(4):623–35.

    Google Scholar 

  12. Vecchio F, Miraglia F, Marra C, Quaranta D, Vita MG, Bramanti P, et al. Human brain networks in cognitive decline: a graph theoretical analysis of cortical connectivity from EEG data. Journal of Alzheimer’s Disease. 2014;41(1):113–27.

    Google Scholar 

  13. Li F, Chen B, Li H, Zhang T, Wang F, Jiang Y, et al. The time-varying networks in P300: a task-evoked EEG study. IEEE Trans Neural Syst Rehabil Eng. 2016;24(7):725–33.

    Google Scholar 

  14. Li F, Peng W, Jiang Y, Song L, Liao Y, Yi C, et al. The dynamic brain networks of motor imagery: time-varying causality analysis of scalp EEG. Int J Neural Syst. 2018;10:1850016.

    Google Scholar 

  15. Deco G, Tononi G, Boly M, Kringelbach ML. Rethinking segregation and integration: contributions of whole-brain modelling. Nat Rev Neurosci. 2015;16(7):430–9.

    Google Scholar 

  16. Watts DJ, Strogatz SH. Collective dynamics of ‘small-world’ networks. Nature. 1998;393(6684):440–2.

    MATH  Google Scholar 

  17. Sakowitz OW, Quiroga RO, Schürmann M, Başar E. Spatio-temporal frequency characteristics of intersensory components in audiovisually evoked potentials. Cognitive Brain Research. 2005;23(2–3):316–26.

    Google Scholar 

  18. Tavano A, Poeppel D. A division of labor between power and phase coherence in encoding attention to stimulus streams. NeuroImage. 2019;193:146–56.

    Google Scholar 

  19. Lakatos P, Karmos G, Mehta AD, Ulbert I, Schroeder CE. Entrainment of neuronal oscillations as a mechanism of attentional selection. Science. 2008;320(5872):110–3.

    Google Scholar 

  20. Keller AS, Payne L, Sekuler R. Characterizing the roles of alpha and theta oscillations in multisensory attention. Neuropsychologi. 2017;99:48–63.

    Google Scholar 

  21. Dubé C, Payne L, Sekuler R, Rotello CM. Paying attention to attention in recognition memory: Insights from models and electrophysiology. Psychol Sci. 2013;24(12):2398–408.

    Google Scholar 

  22. Spitzer B, Haegens S. Beyond the Status Quo: a role for beta oscillations in endogenous content (re)activation. Eneuro. 2017;4(4):ENEURO.0170-17.2017.

  23. Sakowitz OW, Schürmann M, Başar E. Oscillatory frontal theta responses are increased upon bisensory stimulation. Clin Neurophysiol. 2000;111(5):884–93.

    Google Scholar 

  24. Wyart V, de Gardelle V, Scholl J, Summerfield C. Rhythmic fluctuations in evidence accumulation during decision making in the human brain. Neuron. 2012;76(4):847–58.

    Google Scholar 

  25. Padfield N, Zabalza J, Zhao H, Masero V, Ren J. EEG-based brain-computer interfaces using motor-imagery: techniques and challenges. Sensors. 2019;19(6).

  26. Palmer CE, Auksztulewicz R, Ondobaka S, Kilner JM. Sensorimotor beta power reflects the precision-weighting afforded to sensory prediction errors. NeuroImage. 2019;200:59–71.

    Google Scholar 

  27. Gu L, Yu Z, Ma T, Wang H, Li Z, Fan H. EEG-based classification of lower limb motor imagery with brain network analysis. Neuroscience. 2020. https://doi.org/10.1016/j.neuroscience.2020.04.006.

    Article  Google Scholar 

  28. Kopell N, Kramer MA, Malerba P, Whittington MA. Are different rhythms good for different functions? Frontiers in Human Neuroscience. 2010;4:187.

    Google Scholar 

  29. Cannon J, McCarthy MM, Lee S, Lee J, Börgers C, Whittington MA, et al. Neurosystems: brain rhythms and cognitive processing. Eur J Neurosci. 2014;39(5):705–19.

    Google Scholar 

  30. Snodgrass JG, Vanderwart M. A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology Human Learning and Memory. 1980;6(2):174–215.

    Google Scholar 

  31. Werner S, Noppeney U. Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. J Neurosci. 2010;30(7):2662–75.

    Google Scholar 

  32. Xu P, Xiong XC, Xue Q, Tian Y, Peng Y, Zhang R, et al. Recognizing mild cognitive impairment based on network connectivity analysis of resting EEG with zero reference. Physiol Meas. 2014;35(7):1279–98.

    Google Scholar 

  33. Muraja-Murro A, Mervaala E, Westeren-Punnonen S, Lepola P, Töyräs J, Myllymaa S, et al. Forehead EEG electrode set versus full-head scalp EEG in 100 patients with altered mental state. Epilepsy Behav. 2015;49:245–9.

    Google Scholar 

  34. Wilke C, Ding L, He B. Estimation of time-varying connectivity patterns through the use of an adaptive directed transfer function. IEEE Trans Biomed Eng. 2008;55(11):2557–64.

    Google Scholar 

  35. Tian Y, Xu W, Zhang H, Tam KY, Zhang H, Yang L, et al. The scalp time-varying networks of N170: reference, latency, and information flow. Frontiers in Neuroscience. 2018;12:250.

    Google Scholar 

  36. Schwarz GE. Estimating the dimension of a model. The Annals of Statistics. 1978;6(2):461–4.

    MathSciNet  MATH  Google Scholar 

  37. Stanislaw H, Todorov N. Calculation of signal detection theory measures. Behavior Research Methods, Instruments, & Computers. 1999;31(1):137–49.

    Google Scholar 

  38. Li Q, Wu Y, Yang J, Wu J, Touge T. The temporal reliability of sound modulates visual detection: an event-related potential study. Neurosci Lett. 2015;584:202–7.

    Google Scholar 

  39. Sporns O. The human connectome: a complex network. Ann N Y Acad Sci. 2011;1224(1):109–25.

    Google Scholar 

  40. Wig GS. Segregated systems of human brain networks. Trends in Cognitive Sciences. 2017;21(12):981–96.

    Google Scholar 

  41. Burdette JH, Laurienti PJ, Espeland MA, Morgan A, Telesford O, et al. Using network science to evaluate exercise-associated brain changes in older adults. Frontiers in Aging Neuroscience. 2010;2(23):23.

    Google Scholar 

  42. Ding J-R, An D, Liao W, Li J, Wu G-R, Xu Q, et al. Altered functional and structural connectivity networks in psychogenic non-epileptic seizures. PLoS ONE. 2013;8(5):e63850.

    Google Scholar 

  43. Bassett DS, Bullmore ET. Small-world brain networks revisited. Neuroscientist. 2016;23(5):499–516.

    Google Scholar 

  44. Latora V, Marchiori M. Efficient behavior of small-world networks. Physical Review Letters. 2001;87(19).

  45. Sporns O, Zwi JD. The small world of the cerebral cortex. Neuroinformatics. 2004;2:145–62.

    Google Scholar 

  46. Li Q, Wu J, Touge T. Audiovisual interaction enhances auditory detection in late stage: an event-related potential study. NeuroReport. 2010;21(3):173–8.

    Google Scholar 

  47. Corbetta M, Shulman GL. Control of goal-directed and stimulus-driven attention in the brain. Nat Rev Neurosci. 2002;3(3):201–15.

    Google Scholar 

  48. Duncan J. An adaptive coding model of neural function in prefrontal cortex. Nat Rev Neurosci. 2001;2(11):820–9.

    Google Scholar 

  49. Miller EK, Cohen JD. An integrative theory of prefrontal cortex function. Annu Rev Neurosci. 2001;24(1):167–202.

    Google Scholar 

  50. Payne L, Sekuler R. The importance of ignoring: alpha oscillations protect selectivity. Current Directions in Psychological Science. 2014;23(3):171–7.

    Google Scholar 

  51. Romei V, Gross J, Thut G. On the role of prestimulus alpha rhythms over occipito-parietal areas in visual input regulation: correlation or causation? J Neurosci. 2010;30(25):8692–7.

    Google Scholar 

  52. Scaltritti M, Suitner C, Peressotti F. Language and motor processing in reading and typing insights from beta-frequency band power modulations. Brain Lang. 2020;204:104758.

    Google Scholar 

  53. Kilavik BE, Zaepffel M, Brovelli A, MacKay WA, Riehle A. The ups and downs of beta oscillations in sensorimotor cortex. Exp Neurol. 2013;245:15–26.

    Google Scholar 

  54. Tatti E, Ricci S, Mehraram R, Lin N, George S, Nelson AB, et al. Beta modulation depth is not linked to movement features. Frontiers in Behavioral Neuroscience. 2019;14:13–49.

    Google Scholar 

  55. Fujioka T, Ross XB, Trainor LJ. Beta-band oscillations represent auditory beat and its metrical hierarchy in perception and imagery. J Neurosci. 2015;35(45):15187–98.

    Google Scholar 

  56. Joundi RA, Brittain J-S, Green AL, Aziz TZ, Brown P, Jenkinson N. Persistent suppression of subthalamic beta-band activity during rhythmic finger tapping in Parkinson’s disease. Clin Neurophysiol. 2013;124(3):565–73.

    Google Scholar 

  57. Perfetti B, Moisello C, Landsness EC, Kvint S, Pruski A, Onofrj M, et al. Temporal evolution of oscillatory activity predicts performance in a choice-reaction time reaching task. J Neurophysiol. 2011;105(1):18–27.

    Google Scholar 

  58. Lenartowicz A, Simpson GV, O’Connell SR, Cohen MS. Measurement of neurophysiological signals of ignoring and attending processes in attention control. Journal of Visualized Experiments. 2015;101:e52958.

    Google Scholar 

  59. Gren-’t-Jong T, Woldorff MG. Timing and sequence of brain activity in top-down control of visual-spatial attention. PLoS Biol. 2007;5(1):e12.

    Google Scholar 

  60. Cole MW, Reynolds JR, Power JD, Repovs G, Anticevic A, Braver TS. Multitask connectivity reveals flexible hubs for adaptive task control. Nat Neurosci. 2013;16(9):1348–55.

    Google Scholar 

Download references

Acknowledgments

The authors would like to thank all individuals who participated in the initial experiments from which raw data were collected.

Funding

This work was financially supported by the National Natural Science Foundation of China (grant numbers 61773076 and 61806025) and Jilin Scientific and Technological Development Program (grant numbers 20190302072GX and 20200802004GH).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, 130022, People’s Republic of China

    Yang Xi & Qi Li

  2. School of Computer Science, Northeast Electric Power University, Jilin, 132012, People’s Republic of China

    Yang Xi

  3. Department of Radiology, China-Japan Union Hospital of Jilin University, Changchun, 130022, People’s Republic of China

    Mengchao Zhang & Lin Liu

  4. Graduate School of Natural Science and Technology, Okayama University, Okayama, 700-8530, Japan

    Jinglong Wu

Authors
  1. Yang Xi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mengchao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jinglong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Q.L., Y.X., and J.W. designed research; Q.L., Y.X., M.Z., and L.L. performed research; Y.X., M.Z., and L.L. analyzed the data; Q.L., Y.X., and J.W. wrote the paper.

Corresponding authors

Correspondence to Qi Li or Lin Liu.

Ethics declarations

Ethical Approval

The experimental protocol was approved by the Ethics Committee of Changchun University of Science and Technology (protocol number 201705024). After fully explaining the study, all participants provided written informed consent. All the methods were conducted in accordance with the approved guidelines.

Conflict of Interest

The authors declare no competing financial and/or non-financial interests.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 55 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xi, Y., Li, Q., Zhang, M. et al. Characterizing the Time-Varying Brain Networks of Audiovisual Integration across Frequency Bands. Cogn Comput 12, 1154–1169 (2020). https://doi.org/10.1007/s12559-020-09783-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-020-09783-9

Keywords

Search

Navigation