Elsevier

Neurocomputing

Volume 316, 17 November 2018, Pages 202-209
Neurocomputing

Brief papers
F-score feature selection based Bayesian reconstruction of visual image from human brain activity

https://doi.org/10.1016/j.neucom.2018.07.068Get rights and content

Highlights

  • We provide a new visual image reconstruction model.

  • The proposed reconstruction model is more resistant to noise and more efficient.

  • We have built a variety of visual image reconstruction models and compared them to provide more choices for future research.

Abstract

Decoding perceptual experience from human brain activity is a big challenge in neuroscience. Recent advances in human neuroimaging have shown that it is possible to reconstruct a person’s visual experience based on the retinotopy in the early visual cortex and the multivariate pattern analysis (MVPA) method using functional magnetic resonance imaging (fMRI). Previous researches reconstructed binary contrast-defined images using combination of multi-scale local image decoders in V1, V2 and V3, where contrast for local image bases was predicted from fMRI activity by sparse multinomial logistic regression (SMLR) and other models. However, the precision and efficiency of the visual image reconstruction remain insufficient. Proper feature selection is widely known to be as critical for prediction and reconstruction. Aiming at the shortcomings of existing reconstruction models, we proposed a new model of Bayesian reconstruction based on F-score feature selection (Bayes+F). The results indicate that the proposed Bayes+F model has better reconstruction accuracy and higher efficiency than the SMLR and other models, showing better robustness and noise resistant ability. It can improve the spatial correlation coefficient (Mean  ±  variance: 0.7078  ±  0.2104) and decrease the standard error (Mean  ±  variance: 0.2693  ±  0.0871) between the stimulus and the reconstructed image. Furthermore, the proposed model can reconstruct the images extremely rapid, 100 times faster than SMLR does.

Introduction

Vision is among the important senses of humans and other mammals. However, two basic aspects of visual science have yet to be elucidated. (1) How does our brain respond and encode the visual world? (2) Can we decode and reconstruct our visual perceptual experiences in terms of brain activity? With advancements in single-unit cellular recording, electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), and other techniques, receptive fields, functional columns, and visual pathways associated with the first question have been sufficiently studied [1], [2], [3]. Nevertheless, the second question remains a major challenge in neuroscience. Multi-voxel pattern classification (MVPC) studies have revealed that brain activity patterns generated in the early visual cortex in different mental states can be used to decode different visual perceptions. Haxby et al. first identified distinct patterns of fMRI responses in the ventral temporal cortex for several object categories, including different types of small man-made objects [4]. Kamitani et al. demonstrated that fMRI activity patterns in early visual areas contain detailed orientation and motion information that can reliably predict subjective orientation perception and motion direction [5], [6]. Gerven et al. showed that the orientation and rotation direction of a continuously rotating grating can be accurately decoded by using linear dynamical systems and hidden Markov models [7]. Natural images [8], [9], [10], [11], [12], dynamic movies [13], handwritten letters [14], faces [15], and visual imagery during sleep [16] can also be identified and decoded among numerous candidate images by using visual encoding models.

Visual identification is constrained due to limited candidate image sets or classification categories. As such, categorical constraint-free visual image reconstruction methods should be developed to decode the visual perception. Miyawaki et al. utilized retinotopy in the early visual cortex to accomplish a constraint-free reconstruction of contrast-defined arbitrary visual images from fMRI signals of the human early visual cortex [17]. They proposed sparse multinomial logistic regression (SMLR) by using multi-voxel patterns of fMRI signals and multi-scale visual representation. A stimulus state at each local element is predicted by using a decoder with multi-voxel patterns. The outputs of each local decoder are subsequently combined to reconstruct the presented image.

However, we found that visual images reconstructed via SMLR proposed by Miyawaki et al. contained a lot of noise, and more than 10 h are needed to complete the entire reconstruction process. As a result, a very low reconstruction efficiency is obtained. Recently, Yu Zhan and Sutao Song et al. proposed a support vector machine (SVM) and a naive Bayesian classifier based on independent component analysis (NB-ICA) and improved the time consumption and spatial correlation between a stimulus and the reconstructed image to a certain degree [18], [19]. Nevertheless, better reconstruction methods should be developed to explore potential practical applications. Proper feature selection is widely known to be critical for prediction and reconstruction. Selecting appropriate features and removing irrelevant or redundant features may reduce the noise of reconstructed images, decrease computational complexity, and improve reconstruction efficiency. In this study, an F-score feature selection-based Bayesian model was proposed to reconstruct visual images. The results show that the proposed model is more robust and noise resistant than SMLR and other reconstruction methods, such as SVM and RF with/without F-score feature selection. Furthermore, the proposed model can also reconstruct the images extremely rapid, 100 times faster than SMLR does.

Section snippets

Data sources

The dataset is collected from public data recorded by Miyawaki et al. [17]. Three types of experimental sessions were conducted to record the fMRI responses of the visual cortex: (1) conventional retinotopy mapping session, (2) random image session, and (3) figure image (geometric and alphabet shapes) session. The conventional retinotopy mapping session was used to delineate the borders between visual cortical areas, and to identify the retinotopy map on the flattened cortical surfaces. In the

Reconstruction accuracy

The reconstructing stimulus included geometric shapes and alphabet shapes. We reconstructed the images with the activities of the voxels in V1. In this part, the F-score combined Bayesian method, three representative classification methods [SMLR, random forest (RF), and SVM], and two F-score combined methods (RF+F and SVM+F) were applied to compare their performances and reconstruction efficiencies. The reconstructed images obtained by the six algorithms are shown seperately in Fig. 4. The

Discussion

In this study, an F-score feature selection combined Bayesian model was proposed to improve the reconstruction performance on the basis of Miyawaki’s framework of constraint-free visual image reconstruction. The result showed that the proposed model can be used to reconstruct the images with the highest quality (highest correlation coefficient and lowest standard error) and fastest speed among the six models. F-score method is a basic and simple technique used to determine the distinction

Conclusion

In this paper, aiming at the shortcomings of existing reconstruction models, we proposed an F-score feature selection-combined Bayesian model that can be efficiently used to reconstruct visual images from human brain activities. The contribution of the F-score feature selection algorithm is mainly to reduce the background noise of the reconstructed image and improve the quality of the reconstructed image. The contribution of the Bayesian classifier algorithm is to reduce the time required for

Acknowledgment

This work was supported by National High Technology Development Program (863) of China (2015AA020505), 973 project (2013CB329401), and the National Natural Science Foundation of China (61573080, 91420105, 61773094). The authors declared that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Wei Huang received the M.S. degree in biomedical engineering from University of Electronic Science and Technology of China in 2017. He is currently working toward the Ph.D. degree in biomedical engineering from University of Electronic Science and Technology of China. His research interests include visual cognitive decoding, deep learning, computer vision and functional magnetic resonance imaging.

References (32)

Cited by (16)

  • A neural decoding algorithm that generates language from visual activity evoked by natural images

    2021, Neural Networks
    Citation Excerpt :

    Although the decoding performance of the lower visual areas was lower than that of the higher visual areas, we hold that the lower visual areas still processed the information of visual images. Virtually, the low-level visual areas with a small receptive field might be more suitable to characterize local information such as brightness and contrast, and the high-level visual areas with a large receptive field might be responsible for represent more abstract semantic information (Harvey & Dumoulin, 2011; Huang et al., 2018). This interpretation is consistent with the previous evidence that the method of coding visual information in the visual cortex is a hierarchical processing mode (Horikawa & Kamitani, 2017).

  • Topographic property of backpropagation artificial neural network: From human functional connectivity network to artificial neural network

    2020, Neurocomputing
    Citation Excerpt :

    Studies also have suggested that deep ANN might be an efficient model for understanding biological vision and brain information processing [6–8]. The emergence of FC-fMRI have provided a method to explore the neuromechanism of the human brain [9,10]. It measures the correlation between neural activities of brain regions [11].

  • Integrating chemical similarity and bioequivalence: A pilot study on quality consistency evaluation of dispensing granule and traditional decoction of Scutellariae Radix by a totality-of-the-evidence approach

    2019, Journal of Pharmaceutical and Biomedical Analysis
    Citation Excerpt :

    Herein, a feature selection strategy is proposed to correlate the chemical profiles and biological activities, and further discover the bioactive markers of SR. Selecting a sensitive feature subset and retaining as much of the class discriminatory information as possible have a direct effect on the accuracy of the results [25]. In this case, univariate feature selection method which can analyze every single feature was used to evaluate the correlation (Pearson correlation coefficient (PCC), Maximal information coefficient (MIC), and Lasso correlation coefficient (LCC)) between features and response.

  • An improved predictor for identifying recombination spots based on support vector machine

    2023, Journal of Computational Methods in Sciences and Engineering
View all citing articles on Scopus

Wei Huang received the M.S. degree in biomedical engineering from University of Electronic Science and Technology of China in 2017. He is currently working toward the Ph.D. degree in biomedical engineering from University of Electronic Science and Technology of China. His research interests include visual cognitive decoding, deep learning, computer vision and functional magnetic resonance imaging.

Hongmei Yan received her M.S. and Ph.D. degrees from Chongqing University in 2000, 2003 respectively. She is now a professor in the Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China. Her research interests include visual cognition, eye movements, visual attention, and saliency detection.

Ran Liu received the B.E. degree in biomedical engineering from Sichuan University in 2015. He is currently working toward the M.S. degree in physical biology from University of Electronic Science and Technology of China. His research interests include big data mining in medicine, data analysis and artificial intelligence.

Lixia Zhu received the M.S. degree in biomedical engineering from University of Electronic Science and Technology of China in 2017. Her research interests include frequency specificity, emotion regulation, cognitive science and functional magnetic resonance imaging.

Huangbin Zhang received the B.S. degree in information and computer science from Gannan Normal University in 2014. He is currently working toward the M.S. degree in biomedical engineering from University of Electronic Science and Technology of China. His research interests include deep learning, computer vision and functional magnetic resonance imaging.

Huafu Chen Prof. received the Ph.D. degree in Biomedical Engineering from University of Electronic Science and Technology of China in 2004. He is currently works in the Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China. His research interests include brain imaging and pattern recognition method, functional magnetic resonance imaging, brain connectivity and mental illness brain imaging analysis.

View full text