skip to main content
10.1145/3394171.3413858acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Brain-media: A Dual Conditioned and Lateralization Supported GAN (DCLS-GAN) towards Visualization of Image-evoked Brain Activities

Published: 12 October 2020 Publication History

Abstract

Essentially, the current concept of multimedia is limited to presenting what people see in their eyes. What people think inside brains, however, remains a rich source of multimedia, such as imaginations of paradise and memories of good old days etc. In this paper, we propose a dual conditioned and lateralization supported GAN (DCLS-GAN) framework to learn and visualize the brain thoughts evoked by stimulating images and hence enable multimedia to reflect not only what people see but also what people think. To reveal such a new world of multimedia inside human brains, we coin such an attempt as "brain-media". By examining the relevance between the visualized image and the stimulation image, we are able to measure the efficiency of our proposed deep framework regarding the quality of such visualization and also the feasibility of exploring the concept of "brain-media". To ensure that such extracted multimedia elements remain meaningful, we introduce a dually conditioned learning technique in the proposed deep framework, where one condition is analyzing EEGs through deep learning to extract a class-dependent and more compact brain feature space utilizing the distinctive characteristics of hemispheric lateralization and brain stimulation, and the other is to extract expressive visual features assisting our automated analysis of brain activities as well as their visualizations aided by artificial intelligence. To support the proposed GAN framework, we create a combined-conditional space by merging the brain feature space with the visual feature space provoked by the stimuli. Extensive experiments are carried out and the results show that our proposed deep framework significantly outperforms the representative existing state-of-the-arts under several settings, especially in terms of both visualization and classification of brain responses to the evoked images. For the convenience of research dissemination, we make the source code openly accessible for downloading at GitHub.

Supplementary Material

MP4 File (3394171.3413858.mp4)
We propose a dual conditioned and lateralization supported GAN framework to learn and visualize the brain thoughts evoked by stimulating images and hence enable multimedia to reflect not only what people see but also what people think. To reveal such a new world of multimedia inside human brains, we coin such an attempt as brain-media. To ensure that such extracted multimedia elements remain meaningful, we introduce a dually conditioned learning technique in the proposed deep framework, where one condition is analyzing EEGs through deep learning to extract a class-dependent and more compact brain feature space utilizing the distinctive characteristics of hemispheric lateralization and brain stimulation, and the other is to extract expressive visual features assisting our automated analysis of brain activities as well as their visualizations aided by artificial intelligence.

References

[1]
Nawfal Al-Hadithi, Ahmed Al-Imam, Manolia Irfan, Mohammed Khalaf, and SaraAl-Khafaji. 2016. The relation between cerebral dominance and visual analytic skills in Iraqi medical students, a cross sectional analysis.Asian Journal of Medical Sciences 7, 6 (Oct. 2016), 47--52. https://doi.org/10.3126/ajms.v7i6.15205
[2]
Koel Das, Barry Giesbrecht, and Miguel P Eckstein. 2010. Predicting variations of perceptual performance across individuals from neural activity using pattern classifiers. Neuroimage51, 4 (2010), 1425--1437.
[3]
Jia Deng, Wei Dong, Richard Socher, Li Jia Li, Kai Li, and Fei Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. 248--255.
[4]
Anupriya Gogna, Angshul Majumdar, and Rabab Ward. 2017. Semi-supervised Stacked Label Consistent Autoencoder for Reconstruction and Analysis of Biomedical Signals. IEEE Transactions on Biomedical Engineering 64, 9 (2017), 2196--2205.
[5]
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2(Montreal, Canada)(NIPS'14). MIT Press, Cambridge, MA, USA, 2672--2680. http://dl.acm.org/citation.cfm?id=2969033.2969125
[6]
Andrea M Green and John F Kalaska. 2011. Learning to move machines with the mind. Trends in neurosciences 34, 2 (2011), 61--75.
[7]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 6626--6637. http://papers.nips.cc/paper/7240-gans-trained-by-a-two-time-scale-update-rule-converge-to-a-local-nash-equilibrium.pdf
[8]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neuralcomputation 9, 8 (1997), 1735--1780.
[9]
J. Jiang, A. Fares, and S. Zhong. 2019. A Context-Supported Deep Learning Framework for Multimodal Brain Imaging Classification.IEEE Transactions on Human-Machine Systems(2019), 1--12. https://doi.org/10.1109/THMS.2019.2904615
[10]
Blair Kaneshiro, Marcos Perreau Guimaraes, Hyung-Suk Kim, Anthony M Norcia,and Patrick Suppes. 2015. A representational similarity analysis of the dynamics of object processing using single-trial EEG classification. Plos one 10, 8 (2015), e0135697.
[11]
Isaak Kavasidis, Simone Palazzo, Concetto Spampinato, Daniela Giordano, and Mubarak Shah. 2017. Brain2Image: Converting Brain Signals into Images. In Proceedings of the 2017 ACM on Multimedia Conference. ACM, 1809--1817.
[12]
Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In2nd International Conference on Learning Representations, ICLR 2014, Banff, AB,Canada, April 14--16, 2014, Conference Track Proceedings. http://arxiv.org/abs/1312.6114
[13]
JP Kulasingham, V Vibujithan, and AC De Silva. 2016. Deep belief networks and stacked autoencoders for the P300 Guilty Knowledge Test. In Biomedical Engineering and Sciences (IECBES), 2016 IEEE EMBS Conference on. IEEE, 127--132.
[14]
Na Lu, Tengfei Li, Xiaodong Ren, and Hongyu Miao. 2017. A deep learning scheme for motor imagery classification based on restricted boltzmann machines. IEEE transactions on neural systems and rehabilitation engineering 25, 6 (2017), 566--576.
[15]
S Makeig, M Westerfield, TP Jung, S Enghoff, J Townsend, E Courchesne, and TJ Sejnowski. 2002. Electroencephalographic sources of visual evoked responses. Science 295 (2002), 690--694.
[16]
Jinyoung Moon, Yongjin Kwon, Kyuchang Kang, Changseok Bae, and Wan ChulYoon. 2015. Recognition of Meaningful Human Actions for Video Annotation Using EEG Based User Responses. In International Conference on Multimedia Modeling. Springer, 447--457.
[17]
Pranay Mukherjee, Abhirup Das, Ayan Kumar Bhunia, and Partha Pratim Roy. 2018. Cogni-Net: Cognitive Feature Learning through Deep Visual Perception. CoRRabs/1811.00201 (2018). arXiv:1811.00201 http://arxiv.org/abs/1811.00201
[18]
Gernot R Muller-Putz and Gert Pfurtscheller. 2008. Control of an electrical prosthesis with an SSVEP-based BCI. IEEE Transactions on Biomedical Engineering 55, 1 (2008), 361--364.
[19]
T. Ogawa, Y. Sasaka, K. Maeda, and M. Haseyama. 2018. Favorite Video Classification Based on Multimodal Bidirectional LSTM. IEEE Access 6 (2018), 61401--61409. https://doi.org/10.1109/ACCESS.2018.2876710
[20]
S. Palazzo, C. Spampinato, I. Kavasidis, D. Giordano, J. Schmidt, and M. Shah. 2020. Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features.IEEE Transactions on Pattern Analysis and Machine Intelligence(2020), 1--1.
[21]
Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano, and Mubarak Shah. 2017. Generative Adversarial Networks Conditioned by Brain Signals. In IEEE International Conference on Computer Vision, ICCV 2017, Venice,Italy, October 22--29, 2017. 3430--3438. https://doi.org/10.1109/ICCV.2017.369
[22]
Alec Radford, Luke Metz, and Soumith Chintala. 2016. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, PuertoRico, May 2--4, 2016, Conference Track Proceedings. http://arxiv.org/abs/1511.06434
[23]
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved Techniques for Training GANs. In Proceedings of the 30th International Conference on Neural Information Processing Systems(Barcelona, Spain)(NIPS'16). Curran Associates Inc., USA, 2234--2242. http://dl.acm.org/citation.cfm?id=3157096.3157346
[24]
Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, FrankHutter, Wolfram Burgard, and Tonio Ball. 2017. Deep learning with convolutional neural networks for EEG decoding and visualization.Human brain mapping 38,11 (2017), 5391--5420.
[25]
C. Spampinato, S. Palazzo, I. Kavasidis, D. Giordano, N. Souly, and M. Shah. 2017. Deep Learning Human Mind for Automated Visual Classification. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4503--4511. https://doi.org/10.1109/CVPR.2017.479
[26]
Sebastian Stober, Avital Sternin, Adrian M Owen, and Jessica A Grahn. 2015. Deep feature learning for EEG recordings. arXiv preprint arXiv:1511.04306(2015).
[27]
Praveen Tirupattur, Yogesh Singh Rawat, Concetto Spampinato, and Mubarak Shah. 2018. ThoughtViz: Visualizing Human Thoughts Using Generative Adversarial Network. In Proceedings of the 26th ACM International Conference on Multimedia(Seoul, Republic of Korea)(MM '18). ACM, New York, NY, USA, 950--958. https://doi.org/10.1145/3240508.3240641
[28]
Jing Wang, Vladimir L. Cherkassky, Ying Yang, Kai min Kevin Chang, Robert Vargas, Nicholas Diana, and Marcel Adam Just. 2016. Identifying thematic roles from neural representations measured by functional magnetic resonance imaging. Cognitive Neuropsychology33, 3--4 (2016), 257--264. https://doi.org/10.1080/02643294.2016.1182480 arXiv:https://doi.org/10.1080/02643294.2016.1182480PMID: 27314175.
[29]
Jun Wang, Eric Pohlmeyer, Barbara Hanna, Yu-Gang Jiang, Paul Sajda, and Shih-Fu Chang. 2009. Brain state decoding for rapid image retrieval. In Proceedings of the 17th ACM international conference on Multimedia. ACM, 945--954.
[30]
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell:Neural Image Caption Generation with Visual Attention. In ICML. 2048--2057. http://proceedings.mlr.press/v37/xuc15.html
[31]
Longhao Yuan and Jianting Cao. 2018. Patients' EEG Data Analysis via Spectrogram Image with a Convolution Neural Network. In Intelligent Decision Technologies 2017, Ireneusz Czarnowski, Robert J. Howlett, and Lakhmi C. Jain (Eds.). Springer International Publishing, Cham, 13--21.
[32]
Dalin Zhang, Lina Yao, Xiang Zhang, Sen Wang, Weitong Chen, Robert Boots, and Boualem Benatallah. 2018. Cascade and Parallel Convolutional Recurrent Neural Networks on EEG-based Intention Recognition for Brain Computer Interface. In AAAI Conference on Artificial Intelligence. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16107
[33]
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Self-Attention Generative Adversarial Networks. In Thirty-sixth International Conference on Machine Learning (ICML).
[34]
Sheng-hua Zhong, Ahmed Fares, and Jianmin Jiang. 2019. An Attentional-LSTM for Improved Classification of Brain Activities Evoked by Images. In Proceedings of the 27th ACM International Conference on Multimedia(Nice, France)(MM '19). ACM, New York, NY, USA, 1295--1303. https://doi.org/10.1145/3343031.3350886
[35]
Yin Zhong and Zhang Jianhua. 2017. Cross-subject classification of mental fatigue by neurophysiological signals and ensemble deep belief networks. In Control Conference (CCC), 2017 36th Chinese. IEEE, 10966--10971.

Cited By

View all
  • (2024)MB2C: Multimodal Bidirectional Cycle Consistency for Learning Robust Visual Neural RepresentationsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681292(8992-9000)Online publication date: 28-Oct-2024
  • (2024)Learning Robust Deep Visual Representations from EEG Brain Recordings2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00738(7538-7547)Online publication date: 3-Jan-2024
  • (2024)Visual-Guided Dual-Spatial Interaction Network for Fine-Grained Brain Semantic DecodingIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.348023273(1-14)Online publication date: 2024
  • Show More Cited By

Index Terms

  1. Brain-media: A Dual Conditioned and Lateralization Supported GAN (DCLS-GAN) towards Visualization of Image-evoked Brain Activities

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      MM '20: Proceedings of the 28th ACM International Conference on Multimedia
      October 2020
      4889 pages
      ISBN:9781450379885
      DOI:10.1145/3394171
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 12 October 2020

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. EEG
      2. GAN
      3. brain-media
      4. deep learning
      5. image generation
      6. regional attention gate
      7. variant bi-directional LSTM

      Qualifiers

      • Research-article

      Funding Sources

      • Natural Science Foundation China (NSFC)
      • The second round of Shenzhen University Research Foundation funding

      Conference

      MM '20
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)62
      • Downloads (Last 6 weeks)4
      Reflects downloads up to 17 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)MB2C: Multimodal Bidirectional Cycle Consistency for Learning Robust Visual Neural RepresentationsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681292(8992-9000)Online publication date: 28-Oct-2024
      • (2024)Learning Robust Deep Visual Representations from EEG Brain Recordings2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00738(7538-7547)Online publication date: 3-Jan-2024
      • (2024)Visual-Guided Dual-Spatial Interaction Network for Fine-Grained Brain Semantic DecodingIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.348023273(1-14)Online publication date: 2024
      • (2024)Learning Exemplar Representations in Single-Trial EEG Category Decoding2024 35th Irish Signals and Systems Conference (ISSC)10.1109/ISSC61953.2024.10603079(1-6)Online publication date: 13-Jun-2024
      • (2024)Image Generation using EEG data: A Contrastive Learning based Approach2024 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE)10.1109/CCECE59415.2024.10667256(794-798)Online publication date: 6-Aug-2024
      • (2024)EEG classification with limited data: A deep clustering approachPattern Recognition10.1016/j.patcog.2024.110934(110934)Online publication date: Aug-2024
      • (2024)Self-supervised cross-modal visual retrieval from brain activitiesPattern Recognition10.1016/j.patcog.2023.109915145(109915)Online publication date: Jan-2024
      • (2024)NeuroDM: Decoding and visualizing human brain activity with EEG-guided diffusion modelComputer Methods and Programs in Biomedicine10.1016/j.cmpb.2024.108213251(108213)Online publication date: Jun-2024
      • (2023)EEG2IMAGE: Image Reconstruction from EEG Brain SignalsICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP49357.2023.10096587(1-5)Online publication date: 4-Jun-2023
      • (2023)Feasibility of decoding visual information from EEGBrain-Computer Interfaces10.1080/2326263X.2023.228771911:1-2(33-60)Online publication date: 7-Dec-2023
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media