Abstract
With the development of neuroimaging technology and deep learning methods, neural decoding with functional Magnetic Resonance Imaging (fMRI) of human brain has attracted more and more attention. Neural reconstruction task, which intends to reconstruct stimulus images from fMRI, is one of the most challenging tasks in neural decoding. Due to the instability of neural signals, trials of fMRI collected under the same stimulus prove to be very different, which leads to the poor robustness and generalization ability of the existing models. In this work, we propose a robust brain-to-image model based on cross-domain contrastive learning. With deep neural network (DNN) features as paradigms, our model can extract features of stimulus stably and generate reconstructed images via DCGAN. Experiments on the benchmark Deep Image Reconstruction dataset show that our method can enhance the robustness of reconstruction significantly.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bandettini, P.A.: Twenty years of functional MRI: the science and the stories. Neuroimage 62(2), 575–588 (2012)
Beauchamp, M.S., et al.: Dynamic stimulation of visual cortex produces form vision in sighted and blind humans. Cell 181(4), 774–783 (2020)
Beliy, R., Gaziv, G., Hoogi, A., Strappini, F., Golan, T., Irani, M.: From voxels to pixels and back: self-supervision in natural-image reconstruction from fMRI. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Fang, T., Qi, Y., Pan, G.: Reconstructing perceptive images from brain activity by shape-semantic GAN. Adv. Neural. Inf. Process. Syst. 33, 13038–13048 (2020)
Fernández, E., et al.: Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex. J. Clin. Invest. 131(23) (2021)
Güçlütürk, Y., Güçlü, U., Seeliger, K., Bosch, S., van Lier, R., van Gerven, M.A.: Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 25 (2012)
Mozafari, M., Reddy, L., VanRullen, R.: Reconstructing natural scenes from fMRI patterns using BigBiGAN. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2020)
Naselaris, T., Kay, K.N., Nishimoto, S., Gallant, J.L.: Encoding and decoding in fMRI. Neuroimage 56(2), 400–410 (2011)
Nestor, A., Lee, A.C., Plaut, D.C., Behrmann, M.: The face of image reconstruction: progress, pitfalls, prospects. Trends Cogn. Sci. 24(9), 747–759 (2020)
Poldrack, R.A., Farah, M.J.: Progress and challenges in probing the human brain. Nature 526(7573), 371–379 (2015)
Pulvermüller, F., Tomasello, R., Henningsen-Schomers, M.R., Wennekers, T.: Biological constraints on neural network models of cognitive function. Nat. Rev. Neurosci. 22(8), 488–502 (2021)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
Rakhimberdina, Z., Jodelet, Q., Liu, X., Murata, T.: Natural image reconstruction from fMRI using deep learning: a survey. Front. Neurosci. 15, 795488 (2021)
Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y., van Gerven, M.A.: Generative adversarial networks for reconstructing natural images from brain activity. Neuroimage 181, 775–785 (2018)
Shen, G., Dwivedi, K., Majima, K., Horikawa, T., Kamitani, Y.: End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience 13, 21 (2019)
Shen, G., Horikawa, T., Majima, K., Kamitani, Y.: Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15(1), e1006633 (2019)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Ye, H., Yang, X., Takac, M., Sunderraman, R., Ji, S.: Improving text-to-image synthesis using contrastive learning. arXiv preprint arXiv:2107.02423 (2021)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
Acknowledgements
This work was supported in part by the National Key R &D Program of China 2022ZD0116500; in part by the National Natural Science Foundation of China under Grant 62206284, Grant 62020106015 and Grant 82272072; and in part by the Beijing Natural Science Foundation under Grant J210010.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, C., Du, C., He, H. (2024). RoBrain: Towards Robust Brain-to-Image Reconstruction via Cross-Domain Contrastive Learning. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14449. Springer, Singapore. https://doi.org/10.1007/978-981-99-8067-3_17
Download citation
DOI: https://doi.org/10.1007/978-981-99-8067-3_17
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8066-6
Online ISBN: 978-981-99-8067-3
eBook Packages: Computer ScienceComputer Science (R0)