Skip to main content
SpringerLink
Log in

Brain visual image signal classification via hybrid dilation residual shrinkage network with spatio-temporal feature fusion

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Brain–computer interface (BCI) technology based on electroencephalogram (EEG) has attracted widespread attention, among which interpretation, pattern recognition, and classification of brain activity through EEG are promising researches. However, EEG-based object classification is still confronted with enormous challenges in terms of the performance and interpretability of human brain signals. Accordingly, this paper constructs a novel hybrid dilation residual shrinkage network with Spatio-temporal feature fusion to research brain visual images classification. Inspired by visual attention and brain memory mechanisms, a hybrid dilation residual shrinkage module is designed to obtain the features of interest and reduce noise and redundant information. Then, EEG signals are encoded and stored in terms of the temporal and spatial dimensions, respectively. On the basis of the characteristics of the EEG signals, this work utilizes the gated recurrent unit network to generate temporal features and spatial features are obtained through a 2D hybrid dilation convolution module. Finally, the extracted spatio-temporal features are concatenated and then retrieved. Results indicate that the designed model is usable and effective. The proposed network achieves better classification performance compared with the existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. He, B., Yuan, H., Meng, J., Gao, S.: In Brain–Computer Interfaces, pp. 131–183. Springer (2020)

  2. Basar, M.D., Duru, A.D., Akan, A.: Emotional state detection based on common spatial patterns of EEG. Signal Image Video Process. 14(3), 473–481 (2020)

    Article  Google Scholar 

  3. Jordan, K.G.: Emergency EEG and continuous EEG monitoring in acute ischemic stroke. J. Clin. Neurophysiol. 21(5), 341–352 (2004)

    Google Scholar 

  4. Choubey, H., Pandey, A.: A combination of statistical parameters for the detection of epilepsy and EEG classification using ANN and KNN classifier. Signal Image Video Process. 15(3), 475–483 (2021)

    Article  Google Scholar 

  5. Babiloni, F., et al.: Linear classification of low-resolution EEG patterns produced by imagined hand movements. IEEE Trans. Neural Syst. Rehabil. 8(2), 186–188 (2000)

    Article  Google Scholar 

  6. Zhang, Y., et al.: Sparse bayesian classification of EEG for brain-computer interface. IEEE Trans. Neural Net. Lear. 27(11), 2256–2267 (2015)

    Article  MathSciNet  Google Scholar 

  7. Subasi, A., Gursoy, M.I.: EEG signal classification using PCA, ICA, LDA and support vector machines. Expert Syst. Appl. 37(12), 8659–8666 (2010)

    Article  Google Scholar 

  8. Zheng, W.-L., Lu, B.-L.: Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans. Auton. Mental Dev. 7(3), 162–175 (2015)

    Article  Google Scholar 

  9. Zhao, M., Zhong, S., Fu, X., Tang, B., Pecht, M.: Deep residual shrinkage networks for fault diagnosis. IEEE Trans. Ind. Inform. 16(7), 4681–4690 (2020)

    Article  Google Scholar 

  10. Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling (2014). arXiv:1412.3555

  11. Lei, X., Pan, H., Huang, X.: A dilated CNN model for image classification. IEEE Access 7, 124087–124095 (2019)

    Article  Google Scholar 

  12. Kapoor, A., Shenoy, P., Tan, D.: Combining brain computer interfaces with vision for object categorization, pp. 1–8. IEEE (2008)

  13. Parekh, V., Subramanian, R., Roy, D., Jawahar, C.: An eeg-based image annotation system, 303–313 (Springer, 2017)

  14. Amin, S.U., Muhammad, G., Abdul, W., Bencherif, M., Alsulaiman, M.: Multi-CNN feature fusion for efficient EEG classification, pp. 1–6. IEEE (2020)

  15. Dai, M., Zheng, D., Na, R., Wang, S., Zhang, S.: EEG classification of motor imagery using a novel deep learning framework. Sensors 19(3), 551 (2019)

    Article  Google Scholar 

  16. Spampinato, C. et al.: Deep learning human mind for automated visual classification, pp. 6809–6817 (2017). arXiv:1609.00344

  17. Wang, P., Jiang, A., Liu, X., Shang, J., Zhang, L.: LSTM-based EEG classification in motor imagery tasks. IEEE Trans. Neural Syst. Rehabil. 26(11), 2086–2095 (2018)

    Article  Google Scholar 

  18. Fares, A., Zhong, S., Jiang, J.: Region level bi-directional deep learning framework for EEG-based image classification, pp. 368–373. IEEE (2018)

  19. Zhong, S.-h., Fares, A., Jiang, J.: An attentional-LSTM for improved classification of brain activities evoked by images, pp. 1295–1303 (2019)

  20. Palazzo, S., et al.: Decoding brain representations by multimodal learning of neural activity and visual features. IEEE Trans. Pattern Anal. Mach. Intell. (2020)

  21. Ma, X., Qiu, S., Du, C., Xing, J., He, H.: Improving EEG-based motor imagery classification via spatial and temporal recurrent neural networks, pp. 1903–1906 (2018)

  22. Zheng, X., et al.: Decoding human brain activity with deep learning. Biomed. Signal Proces. 56, 101730 (2020)

    Article  Google Scholar 

  23. Zheng, X., Chen, W.: An attention-based Bi-LSTM method for visual object classification via EEG. Biomed. Signal Process. 63, 102174 (2021)

    Article  Google Scholar 

  24. Xu, G., Ren, T., Chen, Y., Che, W.: A one-dimensional CNN-LSTM model for epileptic seizure recognition using EEG signal analysis. Front. Neurosci. 14, 1253 (2020)

    Article  Google Scholar 

  25. Ramzan, M., Dawn, S.: Fused cnn-lstm deep learning emotion recognition model using electroencephalography signals. Int. J. Neurosci. pp. 1–10 (2021)

  26. Kuruvila, I., Muncke, J., Fischer, E., Hoppe, U.: Extracting the locus of attention at a cocktail party from single-trial EEG using a joint CNN-LSTM model. arXiv preprint arXiv:2102.03957 (2021)

  27. Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  28. Mukkamala, M.C., Hein, M.: Variants of rmsprop and adagrad with logarithmic regret bounds, pp. 2545–2553. ACM (2017)

  29. El-Lone, R., Hassan, M., Kabbara, A., Hleiss, R.: Visual objects categorization using dense eeg: A preliminary study, pp. 115–118. IEEE (2015)

  30. Zheng, X., et al.: Ensemble deep learning for automated visual classification using EEG signals. Pattern Recogn. 102, 107147 (2020)

    Article  Google Scholar 

Download references

Acknowledgements

The research was supported by the National Natural Science Foundation of China under Grant No. 62072468 and the Graduate Innovation Project of China University of Petroleum (East China) under Grant No. YCX2021120.

Author information

Author notes

  1. Guixun Xu and Yanjiang Wang have contributed equally to this work.

Authors and Affiliations

  1. College of Control Science and Engineering, China University of Petroleum (East China), Qingdao, 266580, Shandong Province, China

    Wenhui Guo, Guixun Xu & Yanjiang Wang

Authors
  1. Wenhui Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guixun Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanjiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanjiang Wang.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, W., Xu, G. & Wang, Y. Brain visual image signal classification via hybrid dilation residual shrinkage network with spatio-temporal feature fusion. SIViP 17, 743–751 (2023). https://doi.org/10.1007/s11760-022-02282-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02282-4

Keywords

Search

Navigation