Abstract
High variability between individual subjects and recording sessions is a known fact about scalp recorded EEG signal. While some do, the majority of the EEG based machine learning studies do not attempt to assess performance of algorithms across recording sessions or across subjects, instead studies use the whole data-set available for training and testing, using an established k-fold cross validation technique and thus missing performance in a real-life setting on an unseen subject. This study primarily aimed to show how important is to have a leave-one-subject-out (LOSO) evaluation done for any scalp recorded EEG based machine learning. This study also demonstrates effectiveness of a Multilayer Perceptron (MLP) in getting good LOSO accuracy from balanced, clean EEG data, without any pre-processing in comparison with traditional machine learning algorithms. The study used data from participants diagnosed with schizophrenia, as well as a group of participants with no known neurological disorder. Classification was done using traditional methods and MLP to classify the participants as belonging to disease or control subjects. Results shows that 85% accuracy on unseen subject was achievable from a clean data-set. MLP is seen to be effective in finding features by which schizophrenia could be detected from clean EEG data. LOSO evaluation done with this proven MLP configuration using carefully and intentionally corrupted data clearly indicate that for disease diagnosis, the k-fold classification result is misleading. Therefore, evaluation of any scalp recorded EEG based disease classification method must use a LOSO style cross-validation.
This work was supported by the National Health and Medical Research Council, the Flinders Medical Centre Foundation, the Clinician’s Special Purpose Fund of the Flinders Medical Centre, and an equipment grant from the Wellcome Trust, London, U.K.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arribas, J.I., Calhoun, V.D., Adali, T.: Automatic Bayesian classification of healthy controls, bipolar disorder, and schizophrenia using intrinsic connectivity maps from fMRI data. IEEE Trans. Biomed. Eng. 57(12), 2850–2860 (2010)
Başar, E., Güntekin, B.: A review of brain oscillations in cognitive disorders and the role of neurotransmitters. Brain Res. 1235, 172–193 (2008)
Boostani, R., Sadatnezhad, K., Sabeti, M.: An efficient classifier to diagnose of schizophrenia based on the EEG signals. Expert Syst. Appl. 36(3), 6492–6499 (2009)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). https://doi.org/10.1023/A:1010933404324
Challis, E., Hurley, P., Serra, L., Bozzali, M., Oliver, S., Cercignani, M.: Gaussian process classification of Alzheimer’s disease and mild cognitive impairment from resting-state fMRI. Neuroimage 112, 232–243 (2015)
Delorme, A., et al.: EEGLAB, SIFT, NFT, BCILAB, and ERICA: new tools for advanced EEG processing. Comput. Intell. Neurosci. 2011, 5–6 (2011)
Fazli, S., Popescu, F., Danóczy, M., Blankertz, B., Müller, K.R., Grozea, C.: Subject-independent mental state classification in single trials. Neural Netw. 22(9), 1305–1312 (2009)
Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y.: Deep Learning, vol. 1. MIT Press, Cambridge (2016)
Grummett, T.: Analytical complexity required for diagnosing bipolar disorder, schizophrenia, and dementia. Ph.D. thesis, College of Science and Engineering, Flinders University, South Australia, December 2018
Jamal, W., Das, S., Oprescu, I.A., Maharatna, K., Apicella, F., Sicca, F.: Classification of autism spectrum disorder using supervised learning of brain connectivity measures extracted from synchrostates. J. Neural Eng. 11(4), 046019 (2014)
Karimi, S., Mijani, A., Talebian, M., Mirzakuchaki, S.: Comparison of the P300 detection accuracy related to the BCI speller and image recognition scenarios. arXiv preprint arXiv:1912.11371 (2019)
Khan, K.A., Shanir, P., Khan, Y.U., Farooq, O.: A hybrid local binary pattern and wavelets based approach for EEG classification for diagnosing epilepsy. Expert Syst. Appl. 140, 112895 (2020)
Merikangas, K.R., et al.: Lifetime and 12-month prevalence of bipolar spectrum disorder in the national comorbidity survey replication. Arch. Gen. Psychiatry 64(5), 543–552 (2007)
Mullen, T., Delorme, A., Kothe, C., Makeig, S.: An electrophysiological information flow toolbox for EEGLAB. Biol. Cybern. 83, 35–45 (2010)
Peters, H., et al.: More consistently altered connectivity patterns for cerebellum and medial temporal lobes than for amygdala and striatum in schizophrenia. Front. Hum. Neurosci. 10, 55 (2016)
Rashid, B., et al.: Classification of schizophrenia and bipolar patients using static and dynamic resting-state fMRI brain connectivity. Neuroimage 134, 645–657 (2016)
Ravi, K., Palaniappan, R.: Leave-one-out authentication of persons using 40 Hz EEG oscillations. In: EUROCON 2005-The International Conference on “Computer as a Tool”, vol. 2, pp. 1386–1389. IEEE (2005)
Sabeti, M., Katebi, S., Boostani, R.: Entropy and complexity measures for EEG signal classification of schizophrenic and control participants. Artif. Intell. Med. 47(3), 263–274 (2009)
Watanabe, T., Kessler, D., Scott, C., Angstadt, M., Sripada, C.: Disease prediction based on functional connectomes using a scalable and spatially-informed support vector machine. Neuroimage 96, 183–202 (2014)
Welch, P.: The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans. Audio Electroacoust. 15(2), 70–73 (1967)
Zhu, M., Jie, N., Jiang, T.: Automatic classification of schizophrenia using resting-state functional language network via an adaptive learning algorithm. In: Medical Imaging 2014: Computer-Aided Diagnosis, vol. 9035, p. 903522. International Society for Optics and Photonics (2014)
Acknowledgements
We thank Prof Michael Baigent and Dr Randall Long (Department of Psychiatry and Flinders University and Medical Centre), Dr Cate Houen (Central Adelaide Local Health Network, SA Psychiatry Training Unit), Dr Emma Whitham and Prof John Willoughby (Department of Neurology, Flinders University and Medical), for their contributions in collecting and classifying the clinical material and providing the clinical background.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kunjan, S. et al. (2021). The Necessity of Leave One Subject Out (LOSO) Cross Validation for EEG Disease Diagnosis. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds) Brain Informatics. BI 2021. Lecture Notes in Computer Science(), vol 12960. Springer, Cham. https://doi.org/10.1007/978-3-030-86993-9_50
Download citation
DOI: https://doi.org/10.1007/978-3-030-86993-9_50
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86992-2
Online ISBN: 978-3-030-86993-9
eBook Packages: Computer ScienceComputer Science (R0)