Abstract
Measuring the performance of a classifier properly is important to determine which classifier to use for an application domain. The comparison is not straightforward since different experiments may use different datasets, different class categories, and different data distribution, thus biasing the results. Many performance (correctness) measures have been described to facilitate the comparison of classification results. In this paper, we provide an overview of the performance measures for multiclass classification, and list the qualities expected in a good performance measure. We introduce a novel measure, probabilistic accuracy (Pacc), to compare multiclass classification results and make a comparative study of several measures and our proposed method based on different confusion matrices. Experimental results show that our proposed method is discriminative and highly correlated with accuracy compared to other measures. The web version of the software is available at http://sprite.cs.uah.edu/perf/.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Cumbaa, C., Jurisica, I.: Protein crystallization analysis on the world community grid. Journal of Structural and Functional Genomics, 61–69 (2010)
Espndola, R.P., Ebecken, N.F.F.: On extending F-measure and G-mean metrics to multi-class problems. In: Data Mining, Text Mining and their Business Applications (2005)
Özgür, A., Özgür, L., Güngör, T.: Text categorization with class-based and corpus-based keyword selection. In: 20th Inter. Conf. on Comp. and Inf. Sci., pp. 606–615 (2005)
Landgrebe, T., Duin, R.: Efficient multiclass roc approximation by decomposition via confusion matrix perturbation analysis. IEEE Trans on Patttern Analysis and Machine Intelligence 30, 810–822 (2008)
Rees, G.S., Wright, W.A., Greenway, P.: Roc method for the evaluation of multi-class segmentation classification algorithms with infrared imagery (2002)
Yang, B.: The extension of the area under the receiver operating characteristic curve to multi-class problems, vol. 2, pp. 463–466 (2009)
Wei, J., Yuan, X., Hu, Q., Wang, S.: A novel measure for evaluating classifiers. Expert Systems with Applications 37, 3799–3809 (2010)
Gorodkin, J.: Comparing two k-category assignments by a k-category correlation coefficient. Computational Biology and Chemistry 28, 367–374 (2004)
Sokolova, M., Lapalme, G.: A systematic analysis of performance measures for classification tasks. Inf. Process. Manage. 45, 427–437 (2009)
Baldi, P., Brunak, S., Chauvin, Y., Andersen, C.A.F., Nielsen, H.: Assessing the accuracy of prediction algorithms for classification: An overview (2000)
Garca, S., Herrera, F.: An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons. Journal of Machine Learning Research 9, 2677–2694 (2009)
Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006)
Perner, P.: How to interpret decision trees? In: Perner, P. (ed.) ICDM 2011. LNCS, vol. 6870, pp. 40–55. Springer, Heidelberg (2011)
Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46 (1960)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sigdel, M., Aygün, R.S. (2013). Pacc - A Discriminative and Accuracy Correlated Measure for Assessment of Classification Results. In: Perner, P. (eds) Machine Learning and Data Mining in Pattern Recognition. MLDM 2013. Lecture Notes in Computer Science(), vol 7988. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39712-7_22
Download citation
DOI: https://doi.org/10.1007/978-3-642-39712-7_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39711-0
Online ISBN: 978-3-642-39712-7
eBook Packages: Computer ScienceComputer Science (R0)