Elsevier

Information Fusion

Volume 79, March 2022, Pages 229-247
Information Fusion

Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation

https://doi.org/10.1016/j.inffus.2021.10.018Get rights and content

Highlights

  • Multi-modal bioelectrical signal fusion is reviewed.

  • Rationality and challenge of the acquisition equipment of EEG signal is analyzed.

  • The feasibility and challenges of the stimulus paradigm are analyzed.

  • Advantages, challenges and possible solutions of multi-modal fusion are summarized.

  • Fusion methods, evaluation index, and fusion application are reviewed.

Abstract

Multi-modal fusion combines multiple modal information to overcome the limitation of incomplete information expressed by a single modality, so as to realize the complementarity of modal information and enhance feature representation. Multi-modal medical signal fusion algorithm and extraction equipment play an important role in improving the recognition accuracy of brain diseases. This paper compared the existing data fusion methods and explored the fusion research of multi-modal bioelectrical signals, including: (1) the challenges and shortcomings in the signal acquisition phase are explored from the biological signal acquisition equipment and scene settings; (2) five multi-modal fusion forms are analyzed; (3) the fusion methods and evaluation indexes are briefly reviewed; (4) the research status and challenges of multi-modal fusion in the field of spatial cognitive impairment and biometrics are explored; (5) the advantages and challenges of multi-modal fusion are described. The conclusion of this review is that the research of multimodal medical signal fusion is in the initial stage, and some studies have proved that multi-modal fusion is meaningful for medical research. However, the fusion algorithm and fusion strategy need to be improved. While learning the relatively perfect image fusion algorithm, we need to develop the fusion algorithm and fusion strategy that is suitable for medical signal and strengthen its feasibility in clinical application.

Introduction

Biological signals are usually divided into bioelectrical signals and non-bioelectrical signals, among which bioelectrical signals include ECG, EEG, EMG, EOG, and cell electrical activity. Among them, EOG itself is produced by ion exchange between photoreceptor cells and the retinal pigment epithelium with low frequency and high amplitude, which is mainly used in the study of visual development, abnormal eye movement, and visual function lesions [1], but it is often considered as interference noise in EEG research [2], [3], [4]. EEG is a non-stationary and random weak physiological signal [5,6], which is an effective way to track depression, emotion recognition, and cognitive impairment. However, the application of ECG signals is mainly reflected in the diagnosis and treatment of cardiovascular diseases and cardiovascular and cerebrovascular disease [7,8]. Non-bioelectric signals include blood pressure, respiration, body temperature, and so on. By contrast, bioelectrical signals can reflect the consciousness and functional state of the brain and various organs subjectively [9]. It is widely used in clinical diagnosis and rehabilitation medicine by studying the change of resting potential and action potential, regulating, rehabilitating, and treating various diseases. Therefore, this paper mainly studies and analyzes bioelectrical signals (EEG signals).

Each single-modal biological signal, due to the different acquisition frequency and different scenes, provides multiple characteristics information such as individual physiology, psychology, pathology, and metabolism, which provides the possibility and selectivity for promoting biological mechanism and pathological characterization of the whole or other individuals. In the past few years, many scholars have been committed to improving the training performance of models in a single modal utilizing data preprocessing, feature extraction, model design [10,11], and model optimization [12]. However, it is still not possible to obtain more comprehensive features to supplement the shortcomings of single-mode research. It is well known that simple and crude denoising of other biological signals collected in the study of brain diseases is an undesirable operation. A certain type of disease may be triggered by multiple factors, or the diagnosis and analysis of a certain type of disease rely on multiple evidence for joint decision-making [13,14]. Studies have shown that when only EEG signals are used to identify epilepsy, the probability of normal EEG data of patients is 50% [15]. In recent years, with the change of research direction (from theory to practical application), the cases of combining two or more modes of bioelectrical signals for disease diagnosis are gradually increasing. 30 research methods demonstrate that multi-modal research was superior to single-modal research, and the average accuracy rate was higher than 29.4% of single-modal research [16]. This is due to the heterogeneity of the eigenvector space of each single-mode signal. So, the application of single-mode biological signals has great limitations in the context of clinical application, disease diagnosis, rehabilitation treatment [17], biometrics and so on. Multi-modal fusion aims to reduce the heterogeneity difference between each single modal signal and retain the integrity of the semantic information of each modal representation, which is able to achieve the information complementary between each modal signal, provide a comprehensive explanation for disease detection and rehabilitation treatment, and make it show good performance in classification and eval. Given that the complexity of bioelectrical signals, the noise interference caused by the under-optimization of acquisition equipment in the process of signal acquisition, the difference between subjects caused by the unreasonable setting of acquisition scenes cannot be minimized, and the bottleneck in the research of single-mode electrophysiological signals, this study summarizes the research status on different acquisition equipment (excluding noise interference) and acquisition scene settings of bioelectrical signals (excluding the influence of subjects themselves), as well as the multimodal fusion of medical signals, and discusses the multimodal bioelectrical signal fusion methods, fusion forms, fusion rules, fusion assessment and challenges and new directions in multi-modal fusion issues.

Multi-modal biological signals, composed of multiple complex physiological changes, different psychological states, and behavioral characteristics, have become a pivotal reference in the research fields of mental illness [18], [19], [20], [21], identity authentication [22], [23], [24], cognitive behavior [25,26] and other fields. In a narrow sense, the research of multi-modal bioelectrical signals involves the construction of acquisition devices, or the use of multiple single devices to collect signals of different modes, such as the EEG caps are used to collect EEG signals (including EOG) in the cerebral cortex; or a hybrid acquisition interface, such as a hybrid brain-computer interface, can be used to achieve information fusion and logic sharing. In a broad sense, multi-modal bioelectrical signal acquisition involves different (same) individuals in different (same) scenarios, different frequency bands of the same signal (alpha, beta, delta, theta, and gamma bands of EEG signal), or different periods of the same individual (different bioelectrical signal measurements of the same individual at different times), so that the modal signals are distributed in different subspaces [27,28]. Considering the representation of the same individual [15] or different individuals in different scenes or the same scene comprehensively, although the reliability of multi-modal research [29] has been improved, it also brings great challenges in data acquisition, preprocessing, feature extraction, and fusion.

Most research reviews based on multi-modal fusion are only committed to a specific category, such as deep learning-oriented multi-modal fusion [30] and research from the aspects of modal representation, fusion, transformation, alignment (mainly image fusion). With the rise of brain research and human-computer interaction, a large number of excellent pieces of literature on bioelectrical signals have appeared. However, most of the fusion analysis of bioelectrical signals is sporadic, and researchers only focus on the optimal solution of a problem. In addition, the number of proposals in the area of multi-modal fusion has increased significantly in recent years. Therefore, there is a gap in the current literature that requires a more comprehensive understanding of models and techniques based on multi-modal bioelectrical signal fusion. It is necessary to review the past research hotspots and present the latest research trend of multi-modal bioelectrical signal fusion. Therefore, this paper focuses on the research of EEG and takes the medical field as the research background. It aims to review the literature on the advantages and limitations of bioelectrical signal acquisition equipment and the rationality and limitations of signal acquisition scene setting in recent years from the perspective of multi-modal bioelectric signal fusion. Compared with multi-modal medical image fusion, the preprocessing, feature extraction, and fusion methods of multi-modal bioelectrical signals are discussed. The feasibility scheme of the improved existing data fusion methods is proposed, the current research status in this field is summarized, and the new trends and challenges of bioelectric signals in the fusion process are explored.

The main contributions of this review can be summarized as follows.

(1) Common methods and existing problems of medical image fusion and medical signal fusion are compared and analyzed from the perspectives of fusion types, fusion modes, fusion objectives, fusion results, fusion advantages and disadvantages, and fusion applications.

(2) The methods of preprocessing, feature extraction, and fusion of multimodal bioelectrical signals are reviewed.

(3) The advantages and limitations of bioelectrical signal acquisition devices and the rationality and limitations of signal acquisition scene setting in recent years are analyzed.

(4) The fusion form (multi-modal fusion, multi-biometric fusion, multi-temporal fusion, multi-view fusion, and multi-scale fusion), fusion method, fusion rule, and fusion evaluation index of multi-modal bioelectrical signals are discussed.

(5) The application of multi-modal bioelectrical signals in spatial cognitive impairment and biometrics is briefly discussed.

(6) The advantages and challenges of multi-modal bioelectrical signal fusion are explored, and the new trend of transfer learning in multi-modal bioelectrical signal fusion is analyzed.

This paper is organized as follows. Section 2 briefly described bioelectrical signals (mainly EEG signals), commonly used data fusion methods, as well as the challenges of multi-modal signal fusion. Section 3 introduced the commonly used multi-modal signal acquisition equipment and acquisition scene-setting and analyzed the rationality and deficiency of each acquisition equipment and acquisition scene-setting respectively. Section 4 introduced multi-modal bioelectrical signal fusion forms, fusion rules, fusion methods, and evaluation of fusion results. Section 5 described the research and application of multi-modal bioelectrical signals in spatial cognitive impairment and biological recognition and introduced the urgent scientific problems in each field and the significance of clinical research. Section 6 described multi-modal bioelectrical signal fusion benefits and challenges. Section 7 is the summary of the whole paper.

Section snippets

Data fusion: analysis, challenges

This section analyzes the basic characteristics of the bioelectrical signal, commonly used multi-modal data fusion methods, and the challenges of multi-modal bioelectrical signal fusion methods.

EEG signals acquisition equipment and acquisition scene-setting

This section analyzed and summarized the rationality, urgent problems, and research trends of existing medical signal acquisition equipment and scene based on the challenges of acquisition equipment and scene-setting proposed in 2.3.

Multi-modal bioelectrical signal fusion: methodology

Multi-modal fusion has important theoretical significance and practical application value in human-computer interaction and brain neuroscience, but it is still in the preliminary stage of research in the field of medical data. The main advantage of multi-modal fusion is that it can increase the complementarity between single modalities. On the one hand, multi-modal fusion can realize the complementation of multi-modal information by mining the unique characteristics of single-mode biological

Spatial cognitive impairment

Spatial cognitive ability is a kind of cognitive ability to understand and manipulate the environment, which can be effectively improved through reasonable training. Spatial cognitive ability includes spatial orientation ability, spatial memory ability, spatial observation ability, and spatial thinking ability [213]. Cognitive dysfunction refers to the occurrence of one or several lesions in the promotion, weakening, use, and conversion of cognitive functions so that the normal life of the

Conclusion: findings and future directions

Multi-modal fusion is the integration of information from different modes to strengthen or improve the shortcomings of single-mode data. The advantages of multi-modal fusion can be summarized as follows: (1) for the same task, combining multiple modal features is helpful to obtain more robust prediction; (2) the information is enhanced by using the redundant features among the modes; (3) information complementation among modes is realized; (4) one modal information is used to make up for the

CRediT authorship contribution statement

Jingjing Li: Conceptualization, Methodology, Project administration, Writing – original draft, Writing – review & editing, Data curation, Formal analysis. Qiang Wang: Formal analysis, Investigation, Software, Supervision, Funding acquisition, Resources, Validation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

None.

References (222)

  • R. Gravina et al.

    Multi-sensor fusion in body sensor networks: state-of-the-art and research challenges

    Inf. Fusion

    (2017)
  • Y. Liu et al.

    Deep learning for pixel-level image fusion: recent advances and future prospects

    Inf. Fusion

    (2018)
  • F. Lecaignard et al.

    Empirical Bayes evaluation of fused EEG-MEG source reconstruction: application to auditory mismatch evoked responses

    Neuroimage

    (2021)
  • S. Chen et al.

    Decision-level fusion of Sentinel-1 SAR and Landsat 8 OLI texture features for crop discrimination and classification: case of Masvingo, Zimbabwe

    Heliyon

    (2020)
  • A. Al-Tayyan et al.

    Decision-level fusion for single-view gait recognition with various carrying and clothing conditions

    Image Vis. Comput.

    (2017)
  • A. Salentinig et al.

    2.07 - Data- and decision-level fusion for classification

    Comprehensive Remote Sensing

    (2018)
  • Q. Tao et al.

    Threshold-optimized decision-level fusion and its application to biometrics

    Pattern Recognit.

    (2009)
  • Q. Li et al.

    Medical image fusion using segment graph filter and sparse representation

    Comput. Biol. Med.

    (2021)
  • Y. Li et al.

    Medical image fusion method by deep learning

    Int. J. Cogn. Comput. Eng.

    (2021)
  • E. Pippa et al.

    Data fusion for paroxysmal events’ classification from EEG

    J. Neurosci. Methods

    (2017)
  • R. Mosayebi et al.

    Correlated coupled matrix tensor factorization method for simultaneous EEG-fMRI data fusion

    Biomed. Signal Process. Control

    (2020)
  • S.U. Amin et al.

    Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion

    Future Gener. Comput. Syst.

    (2019)
  • C. Li et al.

    Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition

    Inf. Process. Manag.

    (2020)
  • A.P. James et al.

    Medical image fusion: a survey of the state of the art

    Inf. Fusion

    (2014)
  • H. Hermessi et al.

    Multimodal medical image fusion review: theoretical background and recent advances

    Signal Processing

    (2021)
  • S. Tsuchimoto et al.

    Use of common average reference and large-Laplacian spatial-filters enhances EEG signal-to-noise ratios in intrinsic sensorimotor activity

    J. Neurosci. Methods

    (2021)
  • J.G. Samuelsson et al.

    Spatial fidelity of MEG/EEG source estimates: a general evaluation approach

    Neuroimage

    (2021)
  • T.C. Ferree et al.

    The spatial resolution of scalp EEG

    Neurocomputing

    (2001)
  • K. Kalogianni et al.

    Spatial resolution for EEG source reconstruction—A simulation study on SEPs

    J. Neurosci. Methods

    (2018)
  • A. Molins et al.

    Quantification of the benefit from integrating MEG and EEG data in minimum l2-norm estimation NeuroImage

    (2008)
  • R.W. Homan et al.

    Cerebral location of international 10–20 system electrode placement

    Electroencephal. Clin. Neurophysiol.

    (1987)
  • K.S. Park, J. Heo, Forehead EOG measurement system, has ground electrode positioned on horizontal line of EOG signal...
  • E.C. Menguc et al.

    Ieee, An adaptive noise canceller based on QLMS algorithm for removing EOG artifacts in EEG recordings

  • L.W. Chen et al.

    Method of removing EOG artifacts from EEG based on ICA

  • S. Zahan

    Ieee, Removing EOG artifacts from EEG signal using noise-assisted multivariate empirical mode decomposition

  • C. Kim et al.

    An effective feature extraction method by power spectral density of EEG signal for 2-class motor imagery-based BCI

    Med. Biol. Eng. Comput.

    (2018)
  • Y.R. Wang et al.

    Ieee, feature extracting of weak signal in real-time sleeping EEG with approximate entropy and bispectrum analysis

  • M. Shabaan et al.

    Survey: smartphone-based assessment of cardiovascular diseases using ECG and PPG analysis

    BMC Med. Inform. Decis. Making

    (2020)
  • T.O. Takpor, C.U. Ndujiuba, ECG Biosignal: Vital for detectingcardiovascular diseases,...
  • F. Wang et al.

    Research progress of rehabilitation exoskeletal robot and evaluation methodologies based on bioelectrical signals

  • A. Ray, P.K. Bala, R.J.B.A.I.J. Jain, Utilizing emotion scores for improving classifier performance for predicting...
  • S. Manjunath, A. Nathaniel, J. Druce, S. German, Improving the performance of fine-grain image classifiers via...
  • M. Yu et al.

    Application of PSO-RBF neural network in gesture recognition of continuous surface EMG signals

    J. Intell. Fuzzy Syst.

    (2020)
  • D. Monte, M. Lavasani, A.B.J.N. Manning-Bog, Environmental Factors in Parkinson's Disease, 23 (2002)...
  • M.F. Oliveira et al.

    Optimising cardiovascular care of patients with multiple myeloma

    BMJ

    (2021)
  • H. Zhang et al.

    Detection of coronary artery disease using multi-modal feature fusion and hybrid feature selection

    Physiol. Meas.

    (2020)
  • S. D'Mello et al.

    Consistent but modest: a meta-analysis on unimodal and multimodal affect detection accuracies from 30 studies

  • V.M. Joshi et al.

    IDEA: intellect database for emotion analysis using EEG signal

    J. King Saud Univ. - Comput. Inf. Sci.

    (2020)
  • Z. Yin et al.

    Cross-subject EEG feature selection for emotion recognition using transfer recursive feature elimination

    J. Front. Neurorobotics

    (2017)
  • W. Hang et al.

    Cross-subject EEG signal recognition using deep domain adaptation network

    (2019)
  • Cited by (30)

    • A survey of identity recognition via data fusion and feature learning

      2023, Information Fusion
      Citation Excerpt :

      The multimodal identity recognition system is capable of obtaining other types of biometric features to make up for the deficiency. Then, by fusing different kinds of features [4–8], the influence of external noise can also be effectively reduced to an acceptable extent, while the recognition accuracy can be improved concurrently. Furthermore, the multi-feature system can also improve the anti-counterfeiting ability of the system, so as to improve the security of the system.

    • A feature level image fusion for Night-Vision context enhancement using Arithmetic optimization algorithm based image segmentation

      2022, Expert Systems with Applications
      Citation Excerpt :

      Using different imaging sensors to capture images for the same location helps to get enhanced outputs once they are fused (Singh, Singh, Gehlot, kaur, & Gagandeep, 2022). Multi-modal image fusion is a combination of data acquired from different sensors (Singh et al., 2013, 2020), so that the information finally generated has lesser uncertainties and more information as compared to the individual performance of each sensor (Li & Wang, 2022; Liu & Wang, 2015; Xu, Ma, Jiang, Guo, & Ling, 2020; Zhang, Xu, Tian, Jiang, & Ma, 2021). In order to perform segmentation, OTSU as an objective function to perform segmentation is discussed in the forthcoming section 2.

    View all citing articles on Scopus
    View full text