Abstract
Generic decoding is a challenging problem in visual neural decoding. The existing methods based on generative models ignore the application of prior knowledge, which leads to poor interpretability, and few pay attention to fMRI (functional Magnetic Resonance Imaging) processing. To tackle these problems, a novel framework for generic decoding has been proposed named GD-VAE. GD-VAE is based on Variational Auto-Encoder (VAE) which is capable of meaningful latent space, and contains four modules: feature extractor, feature VAE, Prior Knowledge Network (PKN) and Latent Space Disentangling Network (LSDN). The feature extractors extract features of raw visual and cognitive data, and feature VAE implements decoding with a shared latent space for both modalities. The PKN and LSDN constrain the latent space of VAE with delicate structure, in order to apparently reveal the information in the subspace. Benefiting from these modules, the alignment between visual and cognitive modality can be achieved, and greater interpretability can be acquired. Experiments on Generic Decoding Dataset validate the effectiveness and interpretability of the proposed method.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Akamatsu, Y., Harakawa, R., Ogawa, T., Haseyama, M.: Estimating viewed image categories from fMRI activity via multi-view Bayesian generative model. In: 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), pp. 127–128. IEEE (2019)
Akamatsu, Y., Harakawa, R., Ogawa, T., Haseyama, M.: Brain decoding of viewed image categories via semi-supervised multi-view Bayesian generative model. IEEE Trans. Sig. Process. 68, 5769–5781 (2020)
Akamatsu, Y., Harakawa, R., Ogawa, T., Haseyama, M.: Multi-view Bayesian generative model for multi-subject fMRI data on brain decoding of viewed image categories. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1215–1219. IEEE (2020)
Dieng, A.B., Kim, Y., Rush, A.M., Blei, D.M.: Avoiding latent variable collapse with generative skip models. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2397–2405. PMLR (2019)
Du, C., Du, C., Huang, L., He, H.: Reconstructing perceived images from human brain activities with Bayesian deep multiview learning. IEEE Trans. Neural Netw. Learn. Syst. 30(8), 2310–2323 (2018)
Frome, A., Corrado, G., Shlens, J., et al.: A deep visual-semantic embedding model. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 2121–2129 (2013)
Higashi, T., Maeda, K., Ogawa, T., Haseyama, M.: Estimation of visual features of viewed image from individual and shared brain information based on fMRI data using probabilistic generative model. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1335–1339. IEEE (2021)
Higgins, I., et al.: \(\beta \)-VAE: learning basic visual concepts with a constrained variational framework. In: International Conference on Learning Representations (2017)
Horikawa, T., Kamitani, Y.: Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8(1), 15037 (2017)
Huang, S., Shao, W., Wang, M.L., Zhang, D.Q.: fMRI-based decoding of visual information from human brain activity: a brief review. Int. J. Autom. Comput. 18(2), 170–184 (2021). https://doi.org/10.1007/s11633-020-1263-y
Huang, W., et al.: Long short-term memory-based neural decoding of object categories evoked by natural images. Hum. Brain Mapp. 41(15), 4442–4453 (2020)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders. arXiv preprint arXiv:1511.05644 (2015)
Papadimitriou, A., Passalis, N., Tefas, A.: Visual representation decoding from human brain activity using machine learning: a baseline study. Pattern Recogn. Lett. 128, 38–44 (2019)
Qiao, K., et al.: Category decoding of visual stimuli from human brain activity using a bidirectional recurrent neural network to simulate bidirectional information flows in human visual cortices. Front. Neurosci. 13, 692 (2019)
Rodriguez, E.G.: On disentanglement and mutual information in semi-supervised variational auto-encoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1257–1262 (2021)
Schonfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., Akata, Z.: Generalized zero-and few-shot learning via aligned variational autoencoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8247–8255 (2019)
Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558 (2017)
Tomczak, J., Welling, M.: VAE with a VampPrior. In: International Conference on Artificial Intelligence and Statistics, pp. 1214–1223. PMLR (2018)
Wang, X., Peng, D., Hu, P., Sang, Y.: Adversarial correlated autoencoder for unsupervised multi-view representation learning. Knowl.-Based Syst. 168, 109–120 (2019)
Acknowledgements
This work is supported by the National Science Foundation of China (No. 62088102), China National Postdoctoral Program for Innovative Talents from China Postdoctoral Science Foundation (No. BX2021239).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 IFIP International Federation for Information Processing
About this paper
Cite this paper
Sheng, M., Ma, Y., Chen, K., Zheng, N. (2023). VAE-Based Generic Decoding via Subspace Partition and Priori Utilization. In: Maglogiannis, I., Iliadis, L., MacIntyre, J., Dominguez, M. (eds) Artificial Intelligence Applications and Innovations. AIAI 2023. IFIP Advances in Information and Communication Technology, vol 676. Springer, Cham. https://doi.org/10.1007/978-3-031-34107-6_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-34107-6_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-34106-9
Online ISBN: 978-3-031-34107-6
eBook Packages: Computer ScienceComputer Science (R0)
