Ziya Yu
ABSTRACT
In recent years, with the development of deep learning, the integration between neuroscience and computer vision has been deepened. In computer vision, it has been possible to generate images from text as well as semantic understanding from images based on deep learning. Here, text refers to human language, and the language that a computer can understand typically requires text to be encoded. In human brain visual expression, it also produces "descriptions" of visual stimuli, that is, the "language" that generates from the brain itself. Reconstruction of visual information is the process of reconstructing visual stimuli from the understanding of human brain, which is the most difficult to achieve in visual decoding. And based on the existing research of visual mechanisms, it is still difficult to understand the "language" of human brain. Inspired by generating images from text, we regarded voxel responses as the "language" of brain in order to reconstruct visual stimuli and built an end-to-end visual decoding model under the condition of small number of samples. We simply retrained a generative adversarial network (GAN) used to generate images from text on 1200 training data (including natural image stimuli and corresponding voxel responses). We regarded voxel responses as semantic information of brain, and sent them to GAN as prior information. The results showed that the decoding model we trained can reconstruct the natural images successfully. It also suggested the feasibility of reconstructing visual stimuli from "brain language", and the end-to-end model was more likely to learn the direct mapping between brain activity and visual perception. Moreover, it further indicated the great potential of combining neuroscience and computer vision.
- COX, D.D. and SAVOY, R.L., 2003. Functional magnetic resonance imaging (fMRI)"brain reading": detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19, 2, 261--270.Google Scholar
Cross Ref
- HAYNES, J.D. and REES, G. Decoding mental states from brain activity in humans. Nature reviews neuroscience 7, 7, 523--534.Google Scholar
- MITCHELL, T.M., SHINKAREVA, S.V., CARLSON, A., CHANG, K.-M., MALAVE, V.L., MASON, R.A., and JUST, M.A., 2008. Predicting human brain activity associated with the meanings of nouns. Science 320, 5880, 1191--1195.Google Scholar
- HUTH, A.G., NISHIMOTO, S., VU, A.T., and GALLANT, J.L., 2012. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 6, 1210--1224.Google ScholarCross Ref
- KAY, K.N., NASELARIS, T., PRENGER, R.J., and GALLANT, J.L., 2008. Identifying natural images from human brain activity. Nature 452, 7185, 352--355.Google Scholar
- NASELARIS, T., PRENGER, R.J., KAY, K.N., OLIVER, M., and GALLANT, J.L., 2009. Bayesian reconstruction of natural images from human brain activity. Neuron 63, 6, 902--915.Google ScholarCross Ref
- THIRION, B., DUCHESNAY, E., HUBBARD, E., DUBOIS, J., POLINE, J.-B., LEBIHAN, D., and DEHAENE, S., 2006. Inverse retinotopy: inferring the visual content of images from brain activation patterns. Neuroimage 33, 4, 1104--1116.Google ScholarCross Ref
- MIYAWAKI, Y., UCHIDA, H., YAMASHITA, O., SATO, M.-A., MORITO, Y., TANABE, H.C., SADATO, N., and KAMITANI, Y., 2008. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60, 5, 915--929.Google ScholarCross Ref
- YARGHOLI, E. and HOSSEIN-ZADEH, G.-A., 2015. Reconstruction of digit images from human brain fMRI activity through connectivity informed Bayesian networks. Journal of Neuroscience Methods.Google Scholar
- YARGHOLI, E. and HOSSEIN-ZADEH, G.-A., 2016. Brain Decoding-Classification of Hand Written Digits from fMRI Data Employing Bayesian Networks. Frontiers in Human Neuroscience 10(07/1302/17/received06/28/accepted), 351. DOI= http://dx.doi.org/10.3389/fnhum.2016.00351.Google Scholar
- YAMINS, D.L. and DICARLO, J.J., 2016. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience 19, 3, 356--365.Google ScholarCross Ref
- VAN GERVEN, M.A., DE LANGE, F.P., and HESKES, T., 2010. Neural decoding with hierarchical generative models. Neural computation 22, 12, 3127--3142. Google Scholar
- HINTON and E., G. Reducing the Dimensionality of Data with Neural Networks. Science 313, 5786, 504--507.Google Scholar
- HAN, K., WEN, H., SHI, J., LU, K.-H., ZHANG, Y., and LIU, Z., 2018. Variational autoencoder: An unsupervised model for modeling and decoding fMRI activity in visual cortex. bioRxiv, 214247.Google Scholar
- SEELIGER, K., G L, U., AMBROGIONI, L., G L T RK, Y., and VAN GERVEN, M., 2018. Generative adversarial networks for reconstructing natural images from brain activity. Neuroimage 181, 775--785.Google ScholarCross Ref
- SHEN, G., DWIVEDI, K., MAJIMA, K., HORIKAWA, T., and KAMITANI, Y., 2018. End-to-end deep image reconstruction from human brain activity. bioRxiv, 272518. DOI= http: //dx. doi. org/10.1101/272518.Google Scholar
- REED, S., AKATA, Z., YAN, X., LOGESWARAN, L., SCHIELE, B., and LEE, H. Generative Adversarial Text to Image Synthesis.Google Scholar
- HORIKAWA, T. and KAMITANI, Y. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications 8, 15037.Google Scholar
- GOODFELLOW, I., POUGET-ABADIE, J., MIRZA, M., XU, B., WARDE-FARLEY, D., OZAIR, S., COURVILLE, A., and BENGIO, Y., 2014. Generative adversarial nets. In Advances in neural information processing systems, 2672--2680. Google Scholar
- IOFFE, S. and SZEGEDY, C., 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.Google Scholar
- WANG, Z., BOVIK, A.C., SHEIKH, H.R., and SIMONCELLI, E.P., 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4, 600--612. Google ScholarDigital Library
Index Terms
- End-to-End Image Reconstruction of Image from Human Functional Magnetic Resonance Imaging Based on the "Language" of Visual Cortex
Recommendations
Localisation and characterisation of auditory perception through functional magnetic resonance imaging
Special issue on engineering and medicine bridging east and westIn the last few years, Functional Magnetic Resonance Imaging (fMRI) has been widely accepted as an effective tool for mapping brain activities in both the neurosensorial and the cognitive field. The present work aims to assess the possibility of using fMRI ...
Investigation of an emotion perception test using functional magnetic resonance imaging
Highlights- The emotion perception index may indicate the mood status of individuals.
- The ...
Abstract Background and ObjectivePatients with mood disorders are known to have an emotion recognition deficit in facial emotion processing. Emotion perception involves two systems of cognitive and affective processes associated ...
Semantic Processing in the Left Inferior Prefrontal Cortex: A Combined Functional Magnetic Resonance Imaging and Transcranial Magnetic Stimulation Study
The involvement of the left inferior prefrontal cortex (LIPC) in phonological processing is well established from both lesion-deficit studies with neurological patients and functional neuroimaging studies of normals. Its involvement in semantic ...
Comments