Abstract
Machine learning and pattern recognition techniques are increasingly adopted in neuroimaging-based neuroscience research. In many applications a classifier is trained on brain data in order to predict a variable of interest. Two leading examples are brain decoding and clinical diagnosis. Brain decoding consists of predicting stimuli or mental states from concurrent functional brain data. In clinical diagnosis it is the presence or absence of a given medical condition that is predicted from brain data. Observing accurate classification is considered to support the hypothesis of variable-related information within brain data. In this work we briefly review the literature on statistical tests for this kind of hypothesis testing problem. We claim that the current approaches to this hypothesis testing problem are suboptimal, do not cover all useful settings, and that they could lead to wrong conclusions. We present a more accurate statistical test and provide examples of its superiority.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Casella, G., Moreno, E.: Assessing Robustness of Intrinsic Tests of Independence in Two-Way Contingency Tables. Journal of the American Statistical Association 104(487), 1261–1271 (2009)
Clémençon, S., Depecker, M., Vayatis, N.: AUC maximization and the two-sample problem. In: Proceedings of NIPS 2009, Advances in Neural Information Processing Systems 22, pp. 360–368. MIT Press (2009)
Cameron Craddock, R., Holtzheimer, P.E., Hu, X.P., Mayberg, H.S.: Disease state prediction from resting state functional connectivity. Magnetic Resonance in Medicine 62(6), 1619–1628 (2009)
Dundar, M., Krishnapuram, B., Bi, J., Bharat Rao, R.: Learning classifiers when the training data is not IID. In: Proceedings of the 20th International Joint Conference on Artifical Intelligence, pp. 756–761. Morgan Kaufmann Publishers Inc., San Francisco (2007)
Gigerenzer, G.: Mindless statistics. Journal of Socio-Economics 33(5), 587–606 (2004)
Golland, P., Fischl, B.: Permutation Tests for Classification: Towards Statistical Significance in Image-Based Studies. In: Taylor, C.J., Noble, J.A. (eds.) IPMI 2003. LNCS, vol. 2732, pp. 330–341. Springer, Heidelberg (2003)
Gretton, A., Borgwardt, K.M., Rasch, M., Schölkopf, B., Smola, A.J.: A Kernel Method for the Two-Sample-Problem. In: NIPS (2006)
Haxby, J.V., Ida Gobbini, M., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P.: Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex. Science 293(5539), 2425–2430 (2001)
Haynes, J.-D., Rees, G.: Decoding mental states from brain activity in humans. Nature Reviews Neuroscience 7(7), 523–534 (2006)
Hubbard, R., Bayarri, M.J.: Confusion Over Measures of Evidence (p’s) Versus Errors (alpha’s) in Classical Statistical Testing. The American Statistician 57(3), 171–178 (2003)
Kass, R.E., Raftery, A.E.: Bayes Factors. Journal of the American Statistical Association 90(430), 773–795 (1995)
Klöppel, S., Stonnington, C.M., Chu, C., Draganski, B., Scahill, R.I., Rohrer, J.D., Fox, N.C., Jack, C.R., Ashburner, J., Frackowiak, R.S.J.: Automatic classification of MR scans in Alzheimer’s disease. Brain 131(3), 681–689 (2008)
Kriegeskorte, N., Kyle Simmons, W., Bellgowan, P.S.F., Baker, C.I.: Circular analysis in systems neuroscience: the dangers of double dipping. Nature Neuroscience 12(5), 535–540 (2009)
Lao, Z., Shen, D., Xue, Z., Karacali, B., Resnick, S.M., Davatzikos, C.: Morphological classification of brains via high-dimensional shape transformations and machine learning methods. NeuroImage 21(1), 46–57 (2004)
Nigam, V.P.P., Graupe, D.: A neural-network-based detection of epilepsy. Neurological Research 26(1), 55–60 (2004)
Olivetti, E., Mognon, A., Greiner, S., Avesani, P.: Brain Decoding: Biases in Error Estimation. In: 1st ICPR Workshop on Brain Decoding (August 2010)
Olivetti, E., Veeramachaneni, S., Avesani, P.: Testing for Information with Brain Decoding. In: IEEE International Workshop on Pattern Recognition in NeuroImaging (May 2011)
Olivetti, E., Veeramachaneni, S., Nowakowska, E.: Bayesian hypothesis testing for pattern discrimination in brain decoding. Pattern Recognition (May 2011)
Pereira, F., Mitchell, T., Botvinick, M.: Machine learning classifiers and fMRI: A tutorial overview. NeuroImage 45(1), 199–209 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Olivetti, E., Greiner, S., Avesani, P. (2012). Induction in Neuroscience with Classification: Issues and Solutions. In: Langs, G., Rish, I., Grosse-Wentrup, M., Murphy, B. (eds) Machine Learning and Interpretation in Neuroimaging. Lecture Notes in Computer Science(), vol 7263. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34713-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-34713-9_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34712-2
Online ISBN: 978-3-642-34713-9
eBook Packages: Computer ScienceComputer Science (R0)