Skip to main content
SpringerLink
Log in

Convolutional neural networks for classification of music-listening EEG: comparing 1D convolutional kernels with 2D kernels and cerebral laterality of musical influence

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper highlights the ability of convolutional neural networks (CNNs) at classifying EEG data listening to different kinds of music without the requirement for handcrafted features. Deep learning architectures presented in this paper include CNN of different depths and different convolutional kernels. Support vector machine (SVM) taking in EEG features describing the frequency spectrum, signal regularity, and cross-channel correlation has been applied for performance comparison with CNN. The best performing CNN model presented in this paper achieves the tenfold cross-validation (CV) binary classification average accuracy of 98.94% (validation) and 97.46% (test), and the tenfold CV three-class classification accuracy of 97.68% (validation) and 95.71% (test). In comparison, the SVM classifier achieves tenfold CV binary classification accuracy of 80.23% (validation). The CNN model presented is able to not only differentiate EEG of subjects listening to music from that of subjects without auditory input, but it is also capable of accurately differentiating the EEG of subjects listening to different music. In the context of designing neural computing models for EEG analysis, this paper shows that decomposing two-dimensional spatiotemporal convolutional kernels into separate one-dimensional spatial and one-dimensional temporal kernels significantly reduces the number of trainable parameters (size) of the model while retaining the classification performance. This finding is useful, especially in designing CNN for memory-critical embedded systems for EEG processing. In neurological aspect, auditory stimulus is found to have altered the EEG pattern of the frontal lobe and the left cerebral hemisphere more than the other brain regions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

The EEG data used in this study may be made available upon reasonable request (e.g. for verification of the results presented in this paper).

References

  1. Labbé E, Schmidt N, Babin J, Pharr M (2007) Coping with stress: the effectiveness of different types of music. Appl Psychophysiol Biofeedback 32(3–4):163–168

    Article  Google Scholar 

  2. Gebauer L, Skewes J, Westphael G, Heaton P, Vuust P (2014) Intact brain processing of musical emotions in autism spectrum disorder, but more cognitive load and arousal in happy vs. sad music. Front Neurosci 8:192. https://doi.org/10.3389/fnins.2014.00192

    Article  Google Scholar 

  3. O’Kelly J, James L, Palaniappan R, Fachner J, Taborin J, Magee WL (2013) Neurophysiological and behavioral responses to music therapy in vegetative and minimally conscious states. Front Hum Neurosci 7:884. https://doi.org/10.3389/fnhum.2013.00884

    Article  Google Scholar 

  4. Zumbansen A, Peretz I, Hébert S (2014) The combination of rhythm and pitch can account for the beneficial effect of melodic intonation therapy on connected speech improvements in broca’s aphasia. Front Hum Neurosci 8:592. https://doi.org/10.3389/fnhum.2014.00592

    Article  Google Scholar 

  5. Särkämö T, Altenmüller E, Rodríguez-Fornells A, Peretz I (2016) Editorial: music, brain, and rehabilitation: emerging therapeutic applications and potential neural mechanisms. Front Hum Neurosci 10:103. https://doi.org/10.3389/fnhum.2016.00103

    Article  Google Scholar 

  6. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:097–1105

    Google Scholar 

  7. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representation (ICLR) 2015. arXiv: 1409.1556v6

  8. Graves A, Fernandez S, Liwicki M (2008) Unconstrained online handwriting recognition with recurrent neural networks. Adv Neural Inf Process Syst 20:577–584

    Google Scholar 

  9. Graves A, Mohamed A, Hinton GE (2013) Speech recognition with deep recurrent neural networks. In: 2013 IEEE international conference on acoustics, speech and signal processing (ICASSP) 2013, pp 6645–6649. https://doi.org/10.1109/ICASSP.2013.6638947

  10. Ng JY, Hausknecht M, Vijayanarasimhan S, Vinyals O, Monga R, Toderici G (2015) Beyond short snippets: deep networks for video classification. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR) 2015, pp 4694–4702. https://doi.org/10.1109/CVPR.2015.7299101

  11. Ren Y, Wu Y (2014) Convolutional deep belief networks for feature extraction of EEG signal. In: 2014 International joint conference on neural networks (IJCNN) 2014, pp 2850–2853. https://doi.org/10.1109/IJCNN.2014.6889383

  12. Behncke J, Schirrmeister RT, Burgard W, Ball T (2017) The signature of robot action success in EEG signals of a human observer: decoding and visualization using deep convolutional neural networks. In: 2018 6th international conference on brain–computer interface (BCI) 2018, pp 1–6. https://doi.org/10.1109/IWW-BCI.2018.8311531

  13. Schirrmeister RT, Springenberg JT, Fiederera LDJ, Glasstetter M, Eggensperger K, Tangermann M, Hutter F, Burgard W, Ball T (2017) Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp 38(11):5391–5420. https://doi.org/10.1002/hbm.23730

    Article  Google Scholar 

  14. Zhang X, Yao L, Sheng QZ, Kanhere SS, Gu T, Zhang D (2018) Converting your thoughts to texts: enabling brain typing via deep feature learning of EEG signals. In: 2018 IEEE international conference on pervasive computing and communications (PerCom) 2018, pp. 1–10

  15. Dose H, Møller JS, Iversen HK, Puthusserypady S (2018) An end-to-end deep learning approach to MI-EEG signal classification for BCIs. Expert Syst Appl 114(2018):532–542. https://doi.org/10.1016/j.eswa.2018.08.031

    Article  Google Scholar 

  16. van Putten MJAM, Olbrich S, Arns M (2018) Predicting sex from brain rhythms with deep learning. Sci Rep 8(1):3069. https://doi.org/10.1038/s41598-018-21495-7

    Article  Google Scholar 

  17. Längkvist M, Karlsson L, Loutfi A (2012) Sleep stage classification using unsupervised feature learning. Adv Artif Neural Syst. https://doi.org/10.1155/2012/107046

    Article  Google Scholar 

  18. Supratak A, Dong H, Wu C, Guo Y (2017) DeepSleepNet: a model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Trans Neural Syst Rehabil Eng 25(11):1998–2008. https://doi.org/10.1109/TNSRE.2017.2721116

    Article  Google Scholar 

  19. Hajinoroozi M, Mao Z, Huang Y (2015) Prediction of driver’s drowsy and alert states from EEG signals with deep learning. In: 2015 IEEE 6th international workshop on computational advances in multi-sensor adaptive processing (CAMSAP) 2015, pp 493–496. https://doi.org/10.1109/CAMSAP.2015.7383844

  20. Moinnereau M, Brienne T, Brodeur S, Rouat J, Whittingstall K, Plourde E (2018) Classification of auditory stimuli from EEG signals with a regulated recurrent neural network reservoir. arXiv: 1804.10322

  21. Stober S, Cameron DJ, Grahn JA (2014) Using convolutional neural networks to recognize rhythm stimuli from electroencephalography recordings. Adv Neural Inf Process Syst 27(NIPS 2014):1449–1457. http://papers.nips.cc/paper/5272-using-convolutional-neural-networks-to-recognize-rhythm-stimuli-from-electroencephalography-recordings.pdf

  22. Phneah SW, Nisar H (2017) EEG-based alpha neurofeedback training for mood enhancement. Australas Phys Eng Sci Med 40(2):325–336. https://doi.org/10.1007/s13246-017-0538-2

    Article  Google Scholar 

  23. Nawaz R, Nisar H, Yap VV (2018) The effect of music on human brain: frequency domain and time series analysis using electroencephalogram. IEEE Access 6:45191–45205. https://doi.org/10.1109/ACCESS.2018.2855194

    Article  Google Scholar 

  24. Delorme A, Makeig S (2004) EEGLAB: an open-source toolbox for analysis of EEG dynamics. J Neurosci Methods 143(1):9–21. https://doi.org/10.1016/j.jneumeth.2003.10.009

    Article  Google Scholar 

  25. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mane D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viegas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2016) TensorFlow: large-scale machine learning on heterogeneous distributed systems. https://arxiv.org/abs/1603.04467

  26. Gramfort A, Luessi M, Larson E, Engemann DA, Strohmeier D, Brodbeck C, Goj R, Jas M, Brooks T, Parkkonen L, Hämäläinen M (2013) MEG and EEG data analysis with MNE-Python. Front Neurosci 7:267. https://doi.org/10.3389/fnins.2013.00267

    Article  Google Scholar 

  27. Gramfort A, Luessi M, Larson E, Engemann DA, Strohmeier D, Brodbeck C, Parkkonen L, Hämäläinen M (2014) MNE software for processing MEG and EEG data. NeuroImage 86:446–460. https://doi.org/10.1016/j.neuroimage.2013.10.027

    Article  Google Scholar 

  28. Jones E, Oliphant E, Peterson P et al (2001) SciPy: open source scientific tools for Python. http://www.scipy.org/

  29. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  30. Amin HU, Mumtaz W, Subhani AR, Saad MNM, Malik AS (2017) Classification of EEG signals based on pattern recognition approach. Front Comput Neurosci 11:103. https://doi.org/10.3389/fncom.2017.00103

    Article  Google Scholar 

  31. Hendel M, Benyettou A, Hendel F (2016) Hybrid self organizing map and probabilistic quadratic loss multi-class support vector machine for mental tasks classification. Inf Med Unlocked 4:1–9. https://doi.org/10.1016/j.imu.2016.09.001

    Article  Google Scholar 

  32. Dutta S, Singh M, Kumar A (2018) Classification of non-motor cognitive task in EEG based brain-computer interface using phase space features in multivariate empirical mode decomposition domain. Biomed Signal Process Control 39:378–389. https://doi.org/10.1016/j.bspc.2017.08.004

    Article  Google Scholar 

  33. Bashivan P, Rish I, Yeasin M, Codella N (2016) Learning representations from EEG with deep recurrent-convolutional neural networks. In: International conference on learning representations (ICLR) 2016. https://arxiv.org/abs/1511.06448

  34. Goodfellow I, Bengio Y, Courville A (2016) Convolutional networks. In: Goodfellow I (ed) Deep learning. MIT Press, London, pp 326–366

    MATH  Google Scholar 

  35. Goodfellow I, Bengio Y, Courville A (2016) Deep feedforward networks. In: Goodfellow I (ed) Deep learning. MIT Press, London, pp 164–223

    MATH  Google Scholar 

  36. Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations (ICLR 2015). https://arxiv.org/abs/1412.6980

  37. Good IJ (1956) Some terminology and notation in information theory. Proce IEE Part C Monogr 103(3):200. https://doi.org/10.1049/pi-c.1956.0024

    Article  MathSciNet  Google Scholar 

  38. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(2014):1929–1958

    MathSciNet  MATH  Google Scholar 

  39. Chang CC, Lin CJ (2011) {LIBSVM}: a library for support vector machines. ACM Trans Intell Syst Technol 2(3):27

    Article  Google Scholar 

  40. Hsu CW, Chang CC, Lin CJ (2003) A practical guide to support vector classification. NTU Department of Computer Science and Information Engineering Web. http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf. Accessed 10 Aug 2018

  41. Zhang Y, Zhou GX, Jin J, Zhao QB, Wang XY, Cichocki A (2016) Sparse Bayesian classification of EEG for brain–computer interface. IEEE Trans Neural Netw Learn Syst 27(11):2256–2267. https://doi.org/10.1109/TNNLS.2015.2476656

    Article  MathSciNet  Google Scholar 

  42. Jin ZC, Zhou GX, Gao DQ, Zhang Y (2018) EEG classification using sparse Bayesian extreme learning machine for brain–computer interface. J Neural Comput Appl 2018:1–9. https://doi.org/10.1007/s00521-018-3735-3

    Article  Google Scholar 

  43. Jiao Y, Zhang Y, Chen X, Yin EW, Jin J, Wang XY, Cichocki A (2018) Sparse group representation model for motor imagery EEG classification. IEEE J Biomed Health Inform 23(2):631–641. https://doi.org/10.1109/JBHI.2018.2832538

    Article  Google Scholar 

  44. Corballis MC (2014) Left brain, right brain: facts and fantasies. PLoS Biol 12(1):e1001767. https://doi.org/10.1371/journal.pbio.1001767

    Article  Google Scholar 

  45. Liu H, Stufflebeam SM, Sepulcre J, Hedden T, Buckner RL (2009) Evidence from intrinsic activity that asymmetry of the human brain is controlled by multiple factors. Proc Natl Acad Sci 106(48):20499–20503. https://doi.org/10.1073/pnas.0908073106

    Article  Google Scholar 

  46. Kushner HI (2011) Retraining left-handers and the aetiology of stuttering: the rise and fall of an intriguing theory. Later Asymmetries Body Brain Cognit 17(6):673–693. https://doi.org/10.1080/1357650X.2011.615127

    Article  Google Scholar 

  47. Crow TJ, Crow LR, Done DJ, Leask S (1998) Relative hand skill predicts academic ability: global deficits at the point of hemispheric indecision. Neuropsychologia 36(12):1275–1282. https://doi.org/10.1016/S0028-3932(98)00039-6

    Article  Google Scholar 

  48. Rodriguez A, Kaakinen M, Moilanen I, Taanila A, McGough JJ, Loo S, Järvelin M-R (2010) Mixed-handedness is linked to mental health problems in children and adolescents. Pediatrics 125(2):e340–e348

    Article  Google Scholar 

  49. Orr KGD, Cannon M, Gilvarry CM, Jones PB, Murray RM (1999) Schizophrenic patients and their first-degree relatives show an excess of mixed-handedness. Schizophr Res 39(3):167–176. https://doi.org/10.1016/S0920-9964(99)00071-7

    Article  Google Scholar 

  50. Siddiqui SV, Chatterjee U, Kumar D, Siddiqui A, Goyal N (2008) Neuropsychology of prefrontal cortex. Indian J Psychiatry 50(3):202–208. https://doi.org/10.4103/0019-5545.43634

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported financially by Universiti Tunku Abdul Rahman Research Fund (UTARRF) (Grant No.: IPSR/RMC/UTARRF/2018-C1/H03) from Universiti Tunku Abdul Rahman, Malaysia.

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Universiti Tunku Abdul Rahman, 31900, Kampar, Malaysia

    Kit Hwa Cheah, Humaira Nisar & Vooi Voon Yap

  2. Department of Electronics Engineering and Institute of Electronics, National Chiao Tung University, Hsinchu, Taiwan

    Chen-Yi Lee

Authors
  1. Kit Hwa Cheah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Humaira Nisar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vooi Voon Yap
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chen-Yi Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Humaira Nisar.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individuals participated in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheah, K.H., Nisar, H., Yap, V.V. et al. Convolutional neural networks for classification of music-listening EEG: comparing 1D convolutional kernels with 2D kernels and cerebral laterality of musical influence. Neural Comput & Applic 32, 8867–8891 (2020). https://doi.org/10.1007/s00521-019-04367-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04367-7

Keywords

Search

Navigation