Skip to main content
SpringerLink
Log in

MMDF-ESI: Multi-Modal Deep Fusion of EEG and MEG for Brain Source Imaging

  • Conference paper
  • First Online:
Brain Informatics (BI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13974))

Included in the following conference series:

Abstract

Electrophysiological Source Imaging (ESI) aims to reconstruct the underlying electric brain sources based on Electroencephalography (EEG) or Magnetoencephalography (MEG) measurements of brain activities. Due to the ill-posed nature of the ESI problem, faithfully recovering the latent brain sources has been a significant challenge. Classical algorithms have primarily focused on the design of regularization terms according to predefined priors derived from neurophysiological assumptions. Deep learning frameworks attempt to learn the mapping between brain source signals and scalp EEG/MEG measurements in a data-driven manner, and have demonstrated improved performance compared to the classical methods. Given that EEG and MEG can be complementary for measuring the tangential and radial electrical signals of the cortex, combining both modalities is believed to be advantageous in improving the reconstruction performance of ESI. However, the fusion of these two modalities for the ESI problem has not been fully explored in existing deep learning frameworks. In this paper, we propose a Multi-Modal Deep Fusion (MMDF) framework for solving the ESI inverse problem, termed as MMDF-ESI. Our framework integrates both EEG and MEG in a deep learning framework with a specially designed squeeze-and-excitation module. This integration of EEG and MEG is conducted at an early phase in the deep learning framework rather than on the final decision level fusion. Our experimental results show that (1) the localization accuracy of MMDF-ESI consistently outperforms that of using a single modality; (2) MMDF-ESI exhibits excellent stability, characterized by significantly smaller error variance in source reconstruction compared to benchmark methods, particularly for sources with larger extended activation areas and under low signal-to-noise ratio (SNR) conditions; (3) the evaluation on a real EEG/MEG dataset demonstrates that MMDF-ESI enables a more concentrated reconstruction, effectively recovering an extended area of underlying source activation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Michel, C.M., Murray, M.M., Lantz, G., Gonzalez, S., Spinelli, L., de Peralta, R.G.: EEG source imaging. Clin. Neurophysiol. 115(10), 2195–2222 (2004)

    Article  Google Scholar 

  2. He, B., Sohrabpour, A., Brown, E., Liu, Z.: Electrophysiological source imaging: a noninvasive window to brain dynamics. Annu. Rev. Biomed. Eng. 20, 171–196 (2018)

    Article  Google Scholar 

  3. Liu, F., Wang, S., Rosenberger, J., Su, J., Liu, H.: A sparse dictionary learning framework to discover discriminative source activations in EEG brain mapping. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  4. Liu, F., Wang, L., Lou, Y., Li, R.-C., Purdon, P.L.: Probabilistic structure learning for EEG/MEG source imaging with hierarchical graph priors. IEEE Trans. Med. Imaging 40(1), 321–334 (2020)

    Article  Google Scholar 

  5. Canuet, L., et al.: Resting-state EEG source localization and functional connectivity in schizophrenia-like psychosis of epilepsy. PloS One 6(11), e27863 (2011)

    Article  Google Scholar 

  6. Gramfort, A., Kowalski, M., Hämäläinen, M.: Mixed-norm estimates for the M/EEG inverse problem using accelerated gradient methods. Phys. Med. Biol. 57(7), 1937 (2012)

    Article  Google Scholar 

  7. Haufe, S., Nikulin, V.V., Ziehe, A., Müller, K.-R., Nolte, G.: Combining sparsity and rotational invariance in EEG/MEG source reconstruction. NeuroImage 42(2), 726–738 (2008)

    Article  Google Scholar 

  8. Hämäläinen, M.S., Ilmoniemi, R.J.: Interpreting magnetic fields of the brain: minimum norm estimates. Med. Biol. Eng. Comput. 32(1), 35–42 (1994)

    Article  Google Scholar 

  9. Pascual-Marqui, R.D., et al.: Standardized low-resolution brain electromagnetic tomography (sLORETA): technical details. Methods Find Exp. Clin. Pharmacol. 24(Suppl D), 5–12 (2002)

    Google Scholar 

  10. Dale, A.M., et al.: Dynamic statistical parametric mapping: combining fMRI and MEG for high-resolution imaging of cortical activity. Neuron 26(1), 55–67 (2000)

    Article  Google Scholar 

  11. Uutela, K., Hämäläinen, M., Somersalo, E.: Visualization of magnetoencephalographic data using minimum current estimates. Neuroimage 10(2), 173–180 (1999)

    Article  Google Scholar 

  12. Liu, F., Rosenberger, J., Lou, Y., Hosseini, R., Jianzhong, S., Wang, S.: Graph regularized EEG source imaging with in-class consistency and out-class discrimination. IEEE Trans. Big Data 3(4), 378–391 (2017)

    Article  Google Scholar 

  13. Bore, J.C., et al.: Sparse EEG source localization using lapps: least absolute lP \((0\,<\,p\,<\,1)\) penalized solution. IEEE Trans. Biomed. Eng. (2018)

    Google Scholar 

  14. Babadi, B., Obregon-Henao, G., Lamus, C., Hämäläinen, M.S., Brown, E.N., Purdon, P.L.: A subspace pursuit-based iterative greedy hierarchical solution to the neuromagnetic inverse problem. NeuroImage 87, 427–443 (2014)

    Article  Google Scholar 

  15. Baillet, S., Mosher, J.C., Leahy, R.M.: Electromagnetic brain mapping. IEEE Signal Process. Mag. 18(6), 14–30 (2001)

    Article  Google Scholar 

  16. Ding, L., He, B.: Sparse source imaging in electroencephalography with accurate field modeling. Hum. Brain Mapp. 29(9), 1053–1067 (2008)

    Article  Google Scholar 

  17. Liu, F., Wan, G., Semenov, Y.R., Purdon, P.L.: Extended electrophysiological source imaging with spatial graph filters. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022, Part I. LNCS, vol. 13431, pp. 99–109. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16431-6_10

    Chapter  Google Scholar 

  18. Haufe, S., Tomioka, R., et al.: Large-scale EEG/MEG source localization with spatial flexibility. Neuroimage 54(2), 851–859 (2011)

    Article  Google Scholar 

  19. Becker, H., et al.: EEG extended source localization: tensor-based vs. conventional methods. NeuroImage 96, 143–157 (2014)

    Article  Google Scholar 

  20. Qin, J., Liu, F., Wang, S., Rosenberger, J.: EEG source imaging based on spatial and temporal graph structures. In: International Conference on Image Processing Theory, Tools and Applications (2017)

    Google Scholar 

  21. Wan, G., Jiao, M., Ju, X., Zhang, Y., Schweitzer, H., Liu, F.: Electrophysiological brain source imaging via combinatorial search with provable optimality. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 10, pp. 12491–12499 (2023)

    Google Scholar 

  22. Hecker, L., Rupprecht, R., van Elst, L.T., Kornmeier, J.: ConvDip: a convolutional neural network for better EEG source imaging. Front. Neurosci. 15, 569918 (2021)

    Article  Google Scholar 

  23. Jiao, M., et al.: A graph Fourier transform based bidirectional LSTM neural network for EEG source imaging. Front. Neurosci. 447 (2022)

    Google Scholar 

  24. Sun, R., Sohrabpour, A., Worrell, G.A., He, B.: Deep neural networks constrained by neural mass models improve electrophysiological source imaging of spatiotemporal brain dynamics. Proc. Natl. Acad. Sci. 119(31), e2201128119 (2022)

    Article  Google Scholar 

  25. Dinh, C., Samuelsson, J.G., Hunold, A., Hämäläinen, M.S., Khan, S.: Contextual meg and EEG source estimates using spatiotemporal LSTM networks. Front. Neurosci. 15, 552666 (2021)

    Article  Google Scholar 

  26. Jiao, M., Xian, X., Ghacibeh, G., Liu, F.: Extended brain sources estimation via unrolled optimization neural network. bioRxiv, pp. 2022–04 (2022)

    Google Scholar 

  27. Ou, W., Hämäläinen, M.S., Golland, P.: A distributed spatio-temporal EEG/MEG inverse solver. NeuroImage 44(3), 932–946 (2009)

    Article  Google Scholar 

  28. Ding, L.: Reconstructing cortical current density by exploring sparseness in the transform domain. Phys. Med. Biol. 54(9), 2683 (2009)

    Article  Google Scholar 

  29. Sohrabpour, A., Yunfeng, L., Worrell, G., He, B.: Imaging brain source extent from EEG/MEG by means of an iteratively reweighted edge sparsity minimization (IRES) strategy. Neuroimage 142, 27–42 (2016)

    Article  Google Scholar 

  30. Zhu, M., Zhang, W., Dickens, D.L., Ding, L.: Reconstructing spatially extended brain sources via enforcing multiple transform sparseness. NeuroImage 86, 280–293 (2014)

    Article  Google Scholar 

  31. Craley, J., Jouny, C., Johnson, E., Hsu, D., Ahmed, R., Venkataraman, A.: Automated seizure activity tracking and onset zone localization from scalp EEG using deep neural networks. PLoS ONE 17(2), e0264537 (2022)

    Article  Google Scholar 

  32. Huang, G., et al.: Electromagnetic source imaging via a data-synthesis-based convolutional encoder-decoder network. IEEE Trans. Neural Netw. Learn. Syst. (2022)

    Google Scholar 

  33. Dassios, G., Fokas, A.S., Hadjiloizi, D.: On the complementarity of electroencephalography and magnetoencephalography. Inverse Probl. 23(6), 2541 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  34. Malmivuo, J.: Comparison of the properties of EEG and MEG in detecting the electric activity of the brain. Brain Topogr. 25, 1–19 (2012)

    Article  Google Scholar 

  35. Fernando Lopes da Silva: EEG and MEG: relevance to neuroscience. Neuron 80(5), 1112–1128 (2013)

    Article  Google Scholar 

  36. Ahlfors, S.P., Han, J., Belliveau, J.W., Hämäläinen, M.S.: Sensitivity of MEG and EEG to source orientation. Brain Topogr. 23, 227–232 (2010)

    Article  Google Scholar 

  37. Ebersole, J.S., Ebersole, S.M.: Combining MEG and EEG source modeling in epilepsy evaluations. J. Clin. Neurophysiol. 27(6), 360–371 (2010)

    Article  Google Scholar 

  38. Aydin, Ü., et al.: Combined EEG/MEG can outperform single modality EEG or MEG source reconstruction in presurgical epilepsy diagnosis. PLoS ONE 10(3), e0118753 (2015)

    Google Scholar 

  39. Lecaignard, F., Bertrand, O., Caclin, A., Mattout, J.: Empirical bayes evaluation of fused EEG-MEG source reconstruction: application to auditory mismatch evoked responses. Neuroimage 226, 117468 (2021)

    Article  Google Scholar 

  40. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 (2015)

  41. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

    Google Scholar 

  42. Gramfort, A., et al.: MNE software for processing MEG and EEG data. Neuroimage 86, 446–460 (2014)

    Article  Google Scholar 

  43. Furong, X., Liu, K., Yu, Z., Deng, X., Wang, G.: EEG extended source imaging with structured sparsity and L1-norm residual. Neural Comput. Appl. 33(14), 8513–8524 (2021)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, USA

    Meng Jiao, Shihao Yang & Feng Liu

  2. Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA

    Shihao Yang, Yevgeniy R. Semenov & Guihong Wan

  3. Department of Computer Science, Western University, London, ON, Canada

    Boyu Wang

  4. Department of Industrial and Systems Engineering, University of Florida, Gainesville, FL, USA

    Xiaochen Xian

Authors
  1. Meng Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shihao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Boyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaochen Xian
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yevgeniy R. Semenov
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guihong Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Feng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Guihong Wan or Feng Liu .

Editor information

Editors and Affiliations

  1. Stevens Institute of Technology, Hoboken, NJ, USA

    Feng Liu

  2. Lehigh University, Bethlehem, PA, USA

    Yu Zhang

  3. Maebashi Institute of Technology, Maebashi, Japan

    Hongzhi Kuai

  4. Boston University, Boston, MA, USA

    Emily P. Stephen

  5. Stevens Institute of Technology, Hoboken, NJ, USA

    Hongjun Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jiao, M. et al. (2023). MMDF-ESI: Multi-Modal Deep Fusion of EEG and MEG for Brain Source Imaging. In: Liu, F., Zhang, Y., Kuai, H., Stephen, E.P., Wang, H. (eds) Brain Informatics. BI 2023. Lecture Notes in Computer Science(), vol 13974. Springer, Cham. https://doi.org/10.1007/978-3-031-43075-6_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43075-6_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43074-9

  • Online ISBN: 978-3-031-43075-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation