Brief papersF-score feature selection based Bayesian reconstruction of visual image from human brain activity
Introduction
Vision is among the important senses of humans and other mammals. However, two basic aspects of visual science have yet to be elucidated. (1) How does our brain respond and encode the visual world? (2) Can we decode and reconstruct our visual perceptual experiences in terms of brain activity? With advancements in single-unit cellular recording, electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), and other techniques, receptive fields, functional columns, and visual pathways associated with the first question have been sufficiently studied [1], [2], [3]. Nevertheless, the second question remains a major challenge in neuroscience. Multi-voxel pattern classification (MVPC) studies have revealed that brain activity patterns generated in the early visual cortex in different mental states can be used to decode different visual perceptions. Haxby et al. first identified distinct patterns of fMRI responses in the ventral temporal cortex for several object categories, including different types of small man-made objects [4]. Kamitani et al. demonstrated that fMRI activity patterns in early visual areas contain detailed orientation and motion information that can reliably predict subjective orientation perception and motion direction [5], [6]. Gerven et al. showed that the orientation and rotation direction of a continuously rotating grating can be accurately decoded by using linear dynamical systems and hidden Markov models [7]. Natural images [8], [9], [10], [11], [12], dynamic movies [13], handwritten letters [14], faces [15], and visual imagery during sleep [16] can also be identified and decoded among numerous candidate images by using visual encoding models.
Visual identification is constrained due to limited candidate image sets or classification categories. As such, categorical constraint-free visual image reconstruction methods should be developed to decode the visual perception. Miyawaki et al. utilized retinotopy in the early visual cortex to accomplish a constraint-free reconstruction of contrast-defined arbitrary visual images from fMRI signals of the human early visual cortex [17]. They proposed sparse multinomial logistic regression (SMLR) by using multi-voxel patterns of fMRI signals and multi-scale visual representation. A stimulus state at each local element is predicted by using a decoder with multi-voxel patterns. The outputs of each local decoder are subsequently combined to reconstruct the presented image.
However, we found that visual images reconstructed via SMLR proposed by Miyawaki et al. contained a lot of noise, and more than 10 h are needed to complete the entire reconstruction process. As a result, a very low reconstruction efficiency is obtained. Recently, Yu Zhan and Sutao Song et al. proposed a support vector machine (SVM) and a naive Bayesian classifier based on independent component analysis (NB-ICA) and improved the time consumption and spatial correlation between a stimulus and the reconstructed image to a certain degree [18], [19]. Nevertheless, better reconstruction methods should be developed to explore potential practical applications. Proper feature selection is widely known to be critical for prediction and reconstruction. Selecting appropriate features and removing irrelevant or redundant features may reduce the noise of reconstructed images, decrease computational complexity, and improve reconstruction efficiency. In this study, an F-score feature selection-based Bayesian model was proposed to reconstruct visual images. The results show that the proposed model is more robust and noise resistant than SMLR and other reconstruction methods, such as SVM and RF with/without F-score feature selection. Furthermore, the proposed model can also reconstruct the images extremely rapid, 100 times faster than SMLR does.
Section snippets
Data sources
The dataset is collected from public data recorded by Miyawaki et al. [17]. Three types of experimental sessions were conducted to record the fMRI responses of the visual cortex: (1) conventional retinotopy mapping session, (2) random image session, and (3) figure image (geometric and alphabet shapes) session. The conventional retinotopy mapping session was used to delineate the borders between visual cortical areas, and to identify the retinotopy map on the flattened cortical surfaces. In the
Reconstruction accuracy
The reconstructing stimulus included geometric shapes and alphabet shapes. We reconstructed the images with the activities of the voxels in V1. In this part, the F-score combined Bayesian method, three representative classification methods [SMLR, random forest (RF), and SVM], and two F-score combined methods (RF+F and SVM+F) were applied to compare their performances and reconstruction efficiencies. The reconstructed images obtained by the six algorithms are shown seperately in Fig. 4. The
Discussion
In this study, an F-score feature selection combined Bayesian model was proposed to improve the reconstruction performance on the basis of Miyawaki’s framework of constraint-free visual image reconstruction. The result showed that the proposed model can be used to reconstruct the images with the highest quality (highest correlation coefficient and lowest standard error) and fastest speed among the six models. F-score method is a basic and simple technique used to determine the distinction
Conclusion
In this paper, aiming at the shortcomings of existing reconstruction models, we proposed an F-score feature selection-combined Bayesian model that can be efficiently used to reconstruct visual images from human brain activities. The contribution of the F-score feature selection algorithm is mainly to reduce the background noise of the reconstructed image and improve the quality of the reconstructed image. The contribution of the Bayesian classifier algorithm is to reduce the time required for
Acknowledgment
This work was supported by National High Technology Development Program (863) of China (2015AA020505), 973 project (2013CB329401), and the National Natural Science Foundation of China (61573080, 91420105, 61773094). The authors declared that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Wei Huang received the M.S. degree in biomedical engineering from University of Electronic Science and Technology of China in 2017. He is currently working toward the Ph.D. degree in biomedical engineering from University of Electronic Science and Technology of China. His research interests include visual cognitive decoding, deep learning, computer vision and functional magnetic resonance imaging.
References (32)
- et al.
Neuronal synchronization along the dorsal visual pathway reflects the focus of spatial attention
Neuron
(2008) - et al.
Decoding seen and attended motion directions from activity in the human visual cortex
Curr. Biol.
(2006) - et al.
Dynamic decoding of ongoing perception
NeuroImage
(2011) - et al.
Disentangling visual imagery and perception of real-world objects
Neuroimage
(2012) - et al.
Shared representations for working memory and mental imagery in early visual cortex
Curr. Biol.
(2013) - et al.
A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes
Neuroimage
(2015) - et al.
Reconstructing visual experiences from brain activity evoked by natural movies
Curr. Biol.
(2011) - et al.
Linear reconstruction of perceived images from human brain activity
NeuroImage
(2013) - et al.
Neural portraits of perception: reconstructing face images from evoked brain activity
Neuroimage
(2014) - et al.
Visual image reconstruction from human brain activity using a combination of multiscale local image decoders
Neuron
(2008)
Bayesian reconstruction of multiscale local contrast images from brain activity
J. Neurosci. Methods
Spatial frequency selectivity of cells in macaque visual cortex.
Vis. Res.
A new feature selection method on classification of medical datasets: kernel F-score feature selection
Expert Syst. Appl.
Combination of feature selection approaches with SVM in credit scoring
Expert Syst. Appl.
An improved particle swarm optimization for feature selection
J. Bionic Eng.
Using support vector machines with a novel hybrid feature selection method for diagnosis of erythemato-squamous diseases
Expert Syst. Appl.
Cited by (16)
Style linear k-nearest neighbor classification method
2024, Applied Soft ComputingIdle-state detection in motor imagery of articulation using early information: A functional Near-infrared spectroscopy study
2022, Biomedical Signal Processing and ControlA neural decoding algorithm that generates language from visual activity evoked by natural images
2021, Neural NetworksCitation Excerpt :Although the decoding performance of the lower visual areas was lower than that of the higher visual areas, we hold that the lower visual areas still processed the information of visual images. Virtually, the low-level visual areas with a small receptive field might be more suitable to characterize local information such as brightness and contrast, and the high-level visual areas with a large receptive field might be responsible for represent more abstract semantic information (Harvey & Dumoulin, 2011; Huang et al., 2018). This interpretation is consistent with the previous evidence that the method of coding visual information in the visual cortex is a hierarchical processing mode (Horikawa & Kamitani, 2017).
Topographic property of backpropagation artificial neural network: From human functional connectivity network to artificial neural network
2020, NeurocomputingCitation Excerpt :Studies also have suggested that deep ANN might be an efficient model for understanding biological vision and brain information processing [6–8]. The emergence of FC-fMRI have provided a method to explore the neuromechanism of the human brain [9,10]. It measures the correlation between neural activities of brain regions [11].
Integrating chemical similarity and bioequivalence: A pilot study on quality consistency evaluation of dispensing granule and traditional decoction of Scutellariae Radix by a totality-of-the-evidence approach
2019, Journal of Pharmaceutical and Biomedical AnalysisCitation Excerpt :Herein, a feature selection strategy is proposed to correlate the chemical profiles and biological activities, and further discover the bioactive markers of SR. Selecting a sensitive feature subset and retaining as much of the class discriminatory information as possible have a direct effect on the accuracy of the results [25]. In this case, univariate feature selection method which can analyze every single feature was used to evaluate the correlation (Pearson correlation coefficient (PCC), Maximal information coefficient (MIC), and Lasso correlation coefficient (LCC)) between features and response.
An improved predictor for identifying recombination spots based on support vector machine
2023, Journal of Computational Methods in Sciences and Engineering
Wei Huang received the M.S. degree in biomedical engineering from University of Electronic Science and Technology of China in 2017. He is currently working toward the Ph.D. degree in biomedical engineering from University of Electronic Science and Technology of China. His research interests include visual cognitive decoding, deep learning, computer vision and functional magnetic resonance imaging.
Hongmei Yan received her M.S. and Ph.D. degrees from Chongqing University in 2000, 2003 respectively. She is now a professor in the Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China. Her research interests include visual cognition, eye movements, visual attention, and saliency detection.
Ran Liu received the B.E. degree in biomedical engineering from Sichuan University in 2015. He is currently working toward the M.S. degree in physical biology from University of Electronic Science and Technology of China. His research interests include big data mining in medicine, data analysis and artificial intelligence.
Lixia Zhu received the M.S. degree in biomedical engineering from University of Electronic Science and Technology of China in 2017. Her research interests include frequency specificity, emotion regulation, cognitive science and functional magnetic resonance imaging.
Huangbin Zhang received the B.S. degree in information and computer science from Gannan Normal University in 2014. He is currently working toward the M.S. degree in biomedical engineering from University of Electronic Science and Technology of China. His research interests include deep learning, computer vision and functional magnetic resonance imaging.
Huafu Chen Prof. received the Ph.D. degree in Biomedical Engineering from University of Electronic Science and Technology of China in 2004. He is currently works in the Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China. His research interests include brain imaging and pattern recognition method, functional magnetic resonance imaging, brain connectivity and mental illness brain imaging analysis.