skip to main content
10.1145/3343031.3350886acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

An Attentional-LSTM for Improved Classification of Brain Activities Evoked by Images

Published: 15 October 2019 Publication History

Abstract

Multimedia stimulation of brain activities is not only becoming an emerging area for intensive research, but also achieved significant progresses towards classification of brain activities and interpretation of brain understanding of multimedia content. To exploit the characteristics of EEG signals in capturing human brain activities, we propose a region-dependent and attention-driven bi-directional LSTM network (RA-BiLSTM) for image evoked brain activity classification. Inspired by the hemispheric lateralization of human brains, the proposed RA-BiLSTM extracts additional information at regional level to strengthen and emphasize the differences between two hemispheres. In addition, we propose a new attentional-LSTM by adding an extra attention gate to: (i) measure and seize the importance of channel-based spatial information, and (ii) support the proposed RA-BiLSTM to capture the dynamic correlations hidden from both the past and the future in the current state across EEG sequences. Extensive experiments are carried out and the results demonstrate that our proposed RA-BiLSTM not only achieves effective classification of brain activities on evoked image categories, but also significantly outperforms the existing state of the arts.

References

[1]
Nawfal Al-Hadithi, Ahmed Al-Imam, Manolia Irfan, Mohammed Khalaf, and Sara Al-Khafaji. 2016. The relation between cerebral dominance and visual analytic skills in Iraqi medical students, a cross sectional analysis. Asian Journal of Medical Sciences 7, 6 (Oct. 2016), 47--52. https://doi.org/10.3126/ajms.v7i6.15205
[2]
Andreas Antoniades, Loukianos Spyrou, David Martin-Lopez, Antonio Valentin, Gonzalo Alarcon, Saeid Sanei, and Clive Cheong Took. 2017. Detection of Interictal Discharges with Convolutional Neural Networks Using Discrete Ordered Multichannel Intracranial EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering 25, 12 (2017), 2285--2294.
[3]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR abs/1409.0473 (2014). arXiv:1409.0473 http: //arxiv.org/abs/1409.0473
[4]
Pouya Bashivan, Irina Rish, Mohammed Yeasin, and Noel Codella. 2015. Learning representations from EEG with deep recurrentconvolutional neural networks. arXiv preprint arXiv:1511.06448 (2015).
[5]
Mark F Bear, Barry W Connors, and Michael A Paradiso. 2007. Neuroscience: Exploring the brain (3rd ed.). Lippincott Williams and Wilkins. 377--379 pages.
[6]
C Bolduc, A. M. Daoust, E Limoges, C. M. Braun, and R Godbout. 2003. Hemispheric lateralization of the EEG during wakefulness and REM sleep in young healthy adults. Brain and Cognition 53, 2 (2003), 193.
[7]
S Cabral, R. A. Resende, A. C. Clansey, K. J. Deluzio, W. S. Selbie, and A. P. Veloso. 2016. A Global Gait Asymmetry Index. Journal of Applied Biomechanics 32, 2 (2016), 171--177.
[8]
J. H. Cole, Poudel Rpk, D Tsagkrasoulis, Caan Mwa, C Steves, T. D. Spector, and G Montana. 2017. Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker. Neuroimage 163 (2017), 115.
[9]
Koel Das, Barry Giesbrecht, and Miguel P Eckstein. 2010. Predicting variations of perceptual performance across individuals from neural activity using pattern classifiers. Neuroimage 51, 4 (2010), 1425--1437.
[10]
Richard J. Davidson and Nathan A. Fox. 1982. Asymmetrical Brain Activity Discriminates between Positive and Negative Affective Stimuli in Human Infants. Science 218, 4578 (1982), 1235.
[11]
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2018. Universal Transformers. CoRR abs/1807.03819 (2018). arXiv:1807.03819 http://arxiv.org/abs/ 1807.03819
[12]
Hao Dong, Akara Supratak, Wei Pan, Chao Wu, Paul M Matthews, and Yike Guo. 2018. Mixed neural network approach for temporal sleep stage classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering 26, 2 (2018), 324--333.
[13]
Zhen Gao and Shangfei Wang. 2015. Emotion Recognition from EEG Signals by´zLeveraging Stimulus Videos. In Proceedings, Part II, of the 16th Pacific-Rim Conference on Advances in Multimedia Information Processing -- PCM 2015 - Volume 9315. Springer-Verlag, Berlin, Heidelberg, 118--127.
[14]
Z. Gao, X. Wang, Y. Yang, C. Mu, Q. Cai, W. Dang, and S. Zuo. 2019. EEG-Based Spatio-Temporal Convolutional Neural Network for Driver Fatigue Evaluation. IEEE Transactions on Neural Networks and Learning Systems (2019), 1--9. https: //doi.org/10.1109/TNNLS.2018.2886414
[15]
Anupriya Gogna, Angshul Majumdar, and Rabab Ward. 2017. Semi-supervised Stacked Label Consistent Autoencoder for Reconstruction and Analysis of Biomedical Signals. IEEE Transactions on Biomedical Engineering 64, 9 (2017), 2196--2205.
[16]
Stelios K. Hadjidimitriou and Leontios J. Hadjileontiadis. 2012. Toward an EEG-Based Recognition of Music Liking Using Time- Frequency Analysis. IEEE Trans Biomed Eng 59, 12 (2012), 3498--3510.
[17]
Y. Jia, C. Zhou, and M. Motani. 2017. Spatio-temporal autoencoder for feature learning in patient data with missing observations. In 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). 886--890. https://doi.org/10.1109/ BIBM.2017.8217773
[18]
Zhicheng Jiao, Xinbo Gao, Ying Wang, Jie Li, and Haojun Xu. 2018. Deep Convolutional Neural Networks for mental load classification based on EEG data. Pattern Recognition 76 (2018), 582 -- 595. https://doi.org/10.1016/j.patcog.2017.12.002
[19]
Jeffrey S Johnson and Bruno A Olshausen. 2003. Timecourse of neural signatures of object recognition. Journal of Vision 3, 7 (2003), 4. https://doi.org/10.1167/3.7.4 arXiv:/data/journals/jov/932827/jov-3--7--4.pdf
[20]
Blair Kaneshiro, Marcos Perreau Guimaraes, Hyung-Suk Kim, Anthony M Norcia, and Patrick Suppes. 2015. A representational similarity analysis of the dynamics of object processing using single-trial EEG classification. Plos one 10, 8 (2015), e0135697.
[21]
Isaak Kavasidis, Simone Palazzo, Concetto Spampinato, Daniela Giordano, and Mubarak Shah. 2017. Brain2Image: Converting Brain Signals into Images. In Proceedings of the 2017 ACM on Multimedia Conference. ACM, 1809--1817.
[22]
M. K. Kim, M Kim, E Oh, and S. P. Kim. 2013. A review on the computational methods for emotional state estimation from the human EEG. Computational and Mathematical Methods in Medicine,2013,(2013--3--24) 2013, 2 (2013), 573734.
[23]
Stefan Knecht, Michael Deppe, Bianca Dr¨ager, Bobe, Hubertus Lohmann, Ringelstein, and Henningsen. 2000. Language lateralization in healthy right-handers. Brain 123, 1 (2000), 74--81.
[24]
Vernon J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance. 2018. EEGNet: a compact convolutional neural network for EEG-based braincomputer interfaces. Journal of neural engineering 15, 5 (October 2018), 056013. https://doi.org/10.1088/1741--2552/aace8c
[25]
M. Li and B. Lu. 2009. Emotion classification based on gammaband EEG. In 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 1223--1226. https://doi.org/10.1109/IEMBS.2009.5334139
[26]
Xiaowei Li, Rong La, Ying Wang, Junhong Niu, Shuai Zeng, Shuting Sun, and Jing Zhu. 2019. EEG-based mild depression recognition using convolutional neural network. Medical & Biological Engineering & Computing (19 Feb 2019). https://doi.org/10.1007/s11517-019-01959--2
[27]
Matthias M. Mller, Andreas Keil, Thomas Gruber, and Thomas Elbert. 1999. Processing of affective pictures modulates righthemispheric gamma band EEG activity. Clinical Neurophysiology 110, 11 (1999), 1913--1920.
[28]
Jinyoung Moon, Yongjin Kwon, Kyuchang Kang, Changseok Bae, and Wan Chul Yoon. 2015. Recognition of Meaningful Human Actions for Video Annotation Using EEG Based User Responses. In International Conference on Multimedia Modeling. Springer, 447--457.
[29]
Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano, and Mubarak Shah. 2018. Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features. CoRR abs/1810.10974 (2018). arXiv:1810.10974 http://arxiv.org/abs/1810.10974
[30]
R Righart and Gelder B De. 2008. Rapid influence of emotional scenes on encoding of facial expressions: an ERP study. Social Cognitive and Affective Neuroscience 3, 3 (2008), 270.
[31]
C. Spampinato, S. Palazzo, I. Kavasidis, D. Giordano, N. Souly, and M. Shah. 2017. Deep Learning Human Mind for Automated Visual Classification. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4503--4511. https: //doi.org/10.1109/CVPR.2017.479
[32]
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-To-End Memory Networks. In NIPS. 2440--2448. http://papers.nips.cc/paper/5846-end-to-end-memory-networks
[33]
Chuanqi Tan, Fuchun Sun, Wenchang Zhang, Jianhua Chen, and Chunfang Liu. 2017. Multimodal Classification with Deep Convolutional-Recurrent Neural Networks for Electroencephalography. In Neural Information Processing, Derong Liu, Shengli Xie, Yuanqing Li, Dongbin Zhao, and El-Sayed M. El-Alfy (Eds.). Springer International Publishing, Cham, 767--776.
[34]
B. O. Turner, N Marinsek, E Ryhal, and M. B. Miller. 2015. Hemispheric lateralization in reasoning. Ann N Y Acad Sci 1359, 1 (2015), 47--64.
[35]
Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey E. Hinton. 2015. Grammar as a Foreign Language. In NIPS. 2773--2781. http://papers.nips.cc/paper/ 5635-grammar-as-a-foreign-language
[36]
Jun Wang, Eric Pohlmeyer, Barbara Hanna, Yu-Gang Jiang, Paul Sajda, and Shih-Fu Chang. 2009. Brain state decoding for rapid image retrieval. In Proceedings of the 17th ACM international conference on Multimedia. ACM, 945--954.
[37]
P. Wang, A. Jiang, X. Liu, J. Shang, and L. Zhang. 2018. LSTMBased EEG Classification in Motor Imagery Tasks. IEEE Transactions on Neural Systems and Rehabilitation Engineering 26, 11 (Nov 2018), 2086--2095. https://doi.org/10.1109/TNSRE.2018. 2876129
[38]
Jamie Ward. 2006. The student's guide to cognitive neuroscience. Psychology Press. 548 pages.
[39]
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In ICML. 2048--2057. http: //proceedings.mlr.press/v37/xuc15.html
[40]
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. 2016. Stacked Attention Networks for Image Question Answering. In CVPR. 21--29. https://doi.org/10.1109/CVPR. 2016.10
[41]
Y. Yuan, G. Xun, K. Jia, and A. Zhang. 2019. A Multi-View Deep Learning Framework for EEG Seizure Detection. IEEE Journal of Biomedical and Health Informatics 23, 1 (Jan 2019), 83--94. https://doi.org/10.1109/JBHI.2018.2871678
[42]
Dalin Zhang, Lina Yao, Xiang Zhang, Sen Wang, Weitong Chen, Robert Boots, and Boualem Benatallah. 2018. Cascade and Parallel Convolutional Recurrent Neural Networks on EEG-based Intention Recognition for Brain Computer Interface. In AAAI Conference on Artificial Intelligence. https://aaai.org/ocs/ index.php/AAAI/AAAI18/paper/view/16107

Cited By

View all
  • (2025)Mental Workload Assessment Using Deep Learning Models From EEG Signals: A Systematic ReviewIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2024.346075017:1(40-60)Online publication date: Feb-2025
  • (2024)A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion RecognitionMathematics10.3390/math1208118012:8(1180)Online publication date: 15-Apr-2024
  • (2024)Multi-level brain-guided fusion to reinforce spiking neural network in image classificationMultimedia Tools and Applications10.1007/s11042-024-19665-zOnline publication date: 20-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MM '19: Proceedings of the 27th ACM International Conference on Multimedia
October 2019
2794 pages
ISBN:9781450368896
DOI:10.1145/3343031
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. attention-driven lstm
  2. bi-directional computational model
  3. brain activities classification
  4. eeg
  5. region-level information

Qualifiers

  • Research-article

Funding Sources

  • the Shenzhen high-level overseas talents program
  • the National Natural Science Foundation of China
  • the Shenzhen Emerging Industries of the Strategic Basic Research Project under Grant
  • the Inlife-Handnet Open Fund
  • the Natural Science Foundation of Guangdong Province

Conference

MM '19
Sponsor:

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)32
  • Downloads (Last 6 weeks)0
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Mental Workload Assessment Using Deep Learning Models From EEG Signals: A Systematic ReviewIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2024.346075017:1(40-60)Online publication date: Feb-2025
  • (2024)A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion RecognitionMathematics10.3390/math1208118012:8(1180)Online publication date: 15-Apr-2024
  • (2024)Multi-level brain-guided fusion to reinforce spiking neural network in image classificationMultimedia Tools and Applications10.1007/s11042-024-19665-zOnline publication date: 20-Jun-2024
  • (2023)EEG-Based Emotion Recognition via Knowledge-Integrated Interpretable MethodMathematics10.3390/math1106142411:6(1424)Online publication date: 15-Mar-2023
  • (2023)Weakly-Supervised Learning for Fine-Grained Emotion Recognition Using Physiological SignalsIEEE Transactions on Affective Computing10.1109/TAFFC.2022.315823414:3(2304-2322)Online publication date: 1-Jul-2023
  • (2023)Dual Deep Learning for Sleep Classification2023 11th International Japan-Africa Conference on Electronics, Communications, and Computations (JAC-ECC)10.1109/JAC-ECC61002.2023.10479649(190-193)Online publication date: 18-Dec-2023
  • (2023)DM-RE2I: A framework based on diffusion model for the reconstruction from EEG to imageBiomedical Signal Processing and Control10.1016/j.bspc.2023.10512586(105125)Online publication date: Sep-2023
  • (2023)EEG-based visual stimuli classification via reusable LSTMBiomedical Signal Processing and Control10.1016/j.bspc.2023.10458882(104588)Online publication date: Apr-2023
  • (2023)DCAE: A dual conditional autoencoder framework for the reconstruction from EEG into imageBiomedical Signal Processing and Control10.1016/j.bspc.2022.10444081(104440)Online publication date: Mar-2023
  • (2023)Wavelet scattering and scalogram visualization based human brain decoding using empirical wavelet transformInternational Journal of Information Technology10.1007/s41870-023-01213-x15:3(1699-1708)Online publication date: 9-Mar-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media