PToPI: A Comprehensive Review, Analysis, and Knowledge Representation of Binary Classification Performance Measures/Metrics

Abstract

Although few performance evaluation instruments have been used conventionally in different machine learning-based classification problem domains, there are numerous ones defined in the literature. This study reviews and describes performance instruments via formally defined novel concepts and clarifies the terminology. The study first highlights the issues in performance evaluation via a survey of 78 mobile-malware classification studies and reviews terminology. Based on three research questions, it proposes novel concepts to identify characteristics, similarities, and differences of instruments that are categorized into ‘performance measures’ and ‘performance metrics’ in the classification context for the first time. The concepts reflecting the intrinsic properties of instruments such as canonical form, geometry, duality, complementation, dependency, and leveling, aim to reveal similarities and differences of numerous instruments, such as redundancy and ground-truth versus prediction focuses. As an application of knowledge representation, we introduced a new exploratory table called PToPI (Periodic Table of Performance Instruments) for 29 measures and 28 metrics (69 instruments including variant and parametric ones). Visualizing proposed concepts, PToPI provides a new relational structure for the instruments including graphical, probabilistic, and entropic ones to see their properties and dependencies all in one place. Applications of the exploratory table in six examples from different domains in the literature have shown that PToPI aids overall instrument analysis and selection of the proper performance metrics according to the specific requirements of a classification problem. We expect that the proposed concepts and PToPI will help researchers comprehend and use the instruments and follow a systematic approach to classification performance evaluation and publication.

Appendices

Appendix 1: Instrument Abbreviation and Name List

The list of performance instrument abbreviations (symbols) in alphabetic order per level per instrument category and their names and alternative names are given below. We suggest using the first full name (not the one in square braces) to standardize the terminology in the classification context.

PERFORMANCE MEASURES (29 measures)

(Canonicals: 11 measures: base measures and 1st level measures).

Base Measures (BM) (4 measures):

FN: False Negatives, FP: False Positives, TN: True Negatives, TP: True Positives.

1st Level Measures (7 measures):

N: Negatives, P: Positives, ON: Outcome Negatives, OP: Outcome Positives,

FC: False Classification, TC: True Classification, Sn: Sample Size.

2nd Level Measures (11 measures):

BIAS: Bias, CKc: Cohen's Kappa Chance, DET: Determinant,

DPR: D Prime, IMB: (Class) Imbalance, LRN: Negative Likelihood Ratio, LRP: Positive Likelihood Ratio, NER: Null Error Rate, NIR: No Information Rate (non-information rate), PREV: Prevalence, SKEW: (Class) Skew.

Probabilistic error/loss measures (2 measures):

LogLoss (binary cross-entropy), MRAE (MdRAE / GMRAE): Mean (Median/Geometric Mean) Relative Absolute Error.

3rd Level Measures (5 measures):

DP: Discriminant Power, HC: Class Entropy, HO: Outcome Entropy, LIFT: Lift, OR: Odds Ratio.

PERFORMANCE METRICS (28 metrics)

Base Metrics (14 metrics):

ACC: Accuracy (efficiency, rand index), CRR: (Correct) Rejection Rate, DR: Detection Rate, FDR: False Discovery Rate, FNR: False Negative Rate (miss rate), FOR: False Omission Rate (imprecision), FPR: False Positive Rate (fall-out), HOC: Joint Entropy, MCR: Misclassification Rate, Zero–One Loss (normalized), MI: Mutual Information, NPV: Negative Predictive Value, PPV: Positive Predictive Value (precision, confidence), TNR: True Negative Rate (inverse recall, specificity), TPR: True Positive Rate (recall, sensitivity, hit rate, recognition rate).

1st Level Metrics (13 metrics):

Confusion-matrix derived metrics (8 metrics): BACC: Balanced Accuracy (strength), CK: Cohen's Kappa (Heidke skill score, quality index), F1: F metric (F-score, F-measure, positive specific agreement), (Fm: F-metrics for all weights, F2, F0.5, and Fβ: F metric with weight 2, 0.5 and β), G: G metric (G-mean, Fowlkes-Mallows index), INFORM: Informedness (Youden’s index, delta P', Peirce skill score), MARK: Markedness (delta P, Clayton skill score, predictive summary index), nMI: Normalized Mutual Information, wACC: Weighted Accuracy.

Graphical metrics (2 metrics): AUCROC: Area-Under-ROC-Curve (ROC: Receiver Operating Curve) (GINI), AUCPR: Area-Under-Precision–Recall Curve.

Probabilistic error/loss measures (3 metrics): MSE: Mean Squared Error (Brier score), MAE/MdAE/MxAE Mean/Median/Maximum Absolute Error, RMSE: Root Mean Square Error, nsMAPE: Normalized Symmetric Mean Absolute Percentage Error.

2nd Level Metric (1 metric):

MCC: Matthews Correlation Coefficient (Phi correlation coefficient, Cohen’s index, Yule phi).

Appendix 2: Performance Instrument Equations

The equations of performance instruments are listed below as a complete reference. Equations (with complements and duals if any) are provided in high-level and/or canonical forms. Equivalent forms are also provided for some instruments.

Measures’ Equations (underlined numbered as in PToPI)
$TP$	(B.1)	$FP$	(B.2)	$FN$	(B.3)	$TN$	(B.4)
$P=TP+FN$	(B.5)	$N=TN+FP$	(B.6)	$OP=TP+FP$	(B.7)	$ON=TN+FN$	(B.8)
$TC=TP+TN$			(B.9)	$FC=FP+FN$			(B.10)
$Sn=TP+FP+FN+TN=P+N=OP+ON=TC+FC$							(B.11)
$PREV=\frac{P}{Sn}={BIAS}^{*}$	(B.12)	$NER=\frac{N}{Sn}=\overline{PREV }$	(B.13)	$SKEW =N:P$	(B.14)	$IMB =\frac{\mathrm{max}\left(P, N\right)}{\mathrm{min}\left(P, N\right)}$	(B.15)
$NIR=\frac{\mathrm{max}\left(P, N\right)}{Sn}$	(B.16)	$BIAS=\frac{OP}{Sn}={PREV}^{*}$	(B.17)	$DPR={\rm Z}\left(TPR\right)-{\rm Z}\left(FPR\right)$			(B.18)
$LRP=\frac{TPR}{FPR}=\frac{TP\cdot N}{FP.P}$			(B.19)	$LRN=\frac{FNR}{TNR}=\frac{FN\cdot N}{TN.P}$			(B.20)
$DET=TP\cdot TN-FP\cdot FN$			(B.21)	$CKc=\frac{P\cdot OP+N\cdot ON}{{Sn}^{2}}$			(B.22)
$HC =-\sum_{m=PREV,1-PREV}m{\mathrm{log}}_{2}m$			(B.23)	$HO =-\sum_{m=BIAS,1-BIAS}m{\mathrm{log}}_{2}m$			(B.24)
$LIFT=\frac{TPR}{BIAS}=\frac{TP\cdot Sn}{P\cdot OP}$			(B.25)	$OR=\frac{LRP}{LRN}=\frac{TPR\cdot TNR}{FPR\cdot FNR}=\frac{TP\cdot TN}{FP\cdot FN}$			(B.26)
$DP=\frac{\sqrt{3}}{\pi }\mathrm{log}\frac{LRP}{LRN}=\frac{\sqrt{3}}{\pi }\mathrm{log}\frac{TPR\cdot TNR}{FPR\cdot FNR}=\frac{\sqrt{3}}{\pi }\mathrm{log}\frac{TP\cdot TN}{FP\cdot FN}$							(B.27)
Metrics’ Equations (numbered as in PToPI)
$TPR=\frac{TP}{P}={PPV}^{*}$	(B.1)	$FNR=\frac{FN}{P}=\overline{TPR }$	(B.2)	$FPR=\frac{FP}{N}=\overline{TNR }$	(B.3)	$TNR=\frac{TN}{N}={NPV}^{*}$	(B.4)
$PPV=\frac{TP}{OP}={TPR}^{*}$	(B.5)	$FDR=\frac{FP}{OP}=\overline{PPV }$	(B.5)	$FOR=\frac{FN}{ON}=\overline{NPV }$	(B.7)	$NPV=\frac{TN}{ON}={TNR}^{*}$	(B.8)
$ACC =\frac{TC}{Sn}$	(B.9)	$MCR =\frac{FC}{Sn}=\overline{ACC }$	(B.10)	$DR=\frac{TP}{Sn}$	(B.11)	$CRR=\frac{TN}{Sn}$	(B.12)
$HOC =-\sum_{m=TP,FP,FN,TN}\frac{m}{Sn}{\mathrm{log}}_{2}\frac{m}{Sn}$							(B.13)
$\begin{aligned} MI &=\frac{TP}{Sn}{\mathrm{log}}_{2}\frac{TP/Sn}{PREV\cdot BIAS}+\frac{FP}{Sn}{\mathrm{log}}_{2}\frac{FP/Sn}{\left(1-PREV\right)\cdot BIAS}\\ &\quad +\frac{FN}{Sn}{\mathrm{log}}_{2}\frac{FN/Sn}{PREV\cdot (1-BIAS)}+\frac{TN}{Sn}{\mathrm{log}}_{2}\frac{TN/Sn}{(1-PREV)\cdot (1-BIAS)} \end{aligned}$							(B.14)
$INFORM=TPR+TNR-1=\frac{TP\cdot N+TN\cdot P-P\cdot N}{P\cdot N}=\frac{TP\cdot N+TN\cdot P}{P\cdot N}-1={MARK}^{*}$							(B.15)
$BACC =\frac{TPR+TNR}{2}=\frac{TP\cdot N+TN\cdot P}{2\cdot P\cdot N}$			(B.16)	$G=\sqrt[2]{TPR\cdot TNR}=\sqrt{\frac{TP\cdot TN}{P\cdot N}}$			(B.17)
$wACC=w\cdot TPR+\left(1-w\right)\cdot TNR \text{ where } \textit{w} \text{ is in (0, 1)}$							(B.18)
$MARK=PPV+NPV-1=\frac{TP\cdot ON+TN\cdot OP-OP\cdot ON}{OP\cdot ON}=\frac{TP\cdot ON+TN\cdot OP}{OP\cdot ON}-1={INFORM}^{*}$							(B.19)
$CK=\frac{ACC-CKc}{1-CKc}=\frac{2(TP\cdot TN-FP\cdot FN)}{P\cdot ON+N\cdot OP}=\frac{DET}{(P\cdot ON+N\cdot OP)/2}$ Correction: CK is undefined (NaN) due to the zero division by zero (0/0) in case of P = 0 and OP = 0 (TP = Sn) or N = 0 and ON = 0 (TN = Sn). Therefore, CK should be 1 (one) for these cases							(B.20)
${F}_{1}=\frac{2PPV\cdot TPR}{PPV+TPR}=\frac{2TP}{P+OP}=\frac{2TP}{2TP+FC}$							(B.21)
${F}_{\upbeta }=\frac{\left(1+{\upbeta }^{2}\right)PPV\cdot \mathrm{TPR}}{{\upbeta }^{2}PPV+TPR}=\frac{\left(1+{\upbeta }^{2}\right)TP}{\left(1+{\upbeta }^{2}\right)TP+{\upbeta }^{2}FN+FP}$							(B.21.1)
${F}_{0.5}=\frac{1.25PPV.TPR}{0.25PPV+TPR}=\frac{1.25TP}{1.25TP+0.25FN+FP}$							(B.21.2)
${F}_{2}=\frac{5PPV.TPR}{4PPV+TPR}=\frac{5TP}{5TP+4FN+FP}$							(B.21.3)
nMI variants:
$nMI=\frac{MI}{f(HO,HC,HOC)}$			(B.22)	$nMI={nMI}_{ari}=\frac{MI}{(HO+HC)/2}$			(B.22.1)
${nMI}_{geo}=\frac{MI}{\sqrt[2]{HO\cdot HC}}$			(B.22.2)	${nMI}_{joi}=\frac{MI}{HOC}$			(B.22.3)
${nMI}_{min}=\frac{MI}{{\text{min}}(HO,HC)}$			(B.22.4)	${c}{nMI}_{max}=\frac{MI}{{\text{max}}(HO,HC)}$			(B.22.5)
$\begin{aligned} MCC & = \sqrt {INFORM \cdot MARK} = \sqrt {TPR \cdot TNR \cdot PPV \cdot NPV} - \sqrt {FPR \cdot FNR \cdot FDR \cdot FOR} \\ & \quad = \frac{{TP/Sn - PREV \cdot BIAS}}{{\sqrt {PREV \cdot BIAS \cdot \left( {1 - PREV} \right) \cdot \left( {1 - BIAS} \right)} }} = \frac{{TP \cdot TN - FP \cdot FN}}{{\sqrt {P \cdot OP \cdot N \cdot ON} }} = \frac{{DET}}{{\sqrt {P \cdot OP \cdot N \cdot ON} }} \\ \end{aligned}$							(B.23)
Graphical Performance Metrics (numbered with ‘g’ prefix)
AUCROC: area-under-ROC-curve (TPR versus TNR)							(B.g1)
$GINI=2AUCROC-1$							(B.g1.1)
AUCPR: area-under-PR- curve (PPV versus TPR)							(B.g2)
Probabilistic Error/Loss Base Equations (numbered with ‘p’ prefix)
(Summary Functions)
${e}_{i}={c}_{i}-{p}_{i}$	(B.pi)	${\%e}_{i}=\frac{{e}_{i}}{{c}_{i}}$	(B.pii)	${\Delta c}_{i}={c}_{i}-\overline{c }$	(B.piii)	${sym\_e}_{i}=\frac{{e}_{i}}{\left|{c}_{i}\right|+\left|{p}_{i}\right|}$	(B.pvi)
${rel\_e}_{i}=\frac{{e}_{i}}{{\Delta c}_{i}}$			(B.piv)	${sca\_e}_{i}=\frac{{e}_{i}}{\underset{i=2, \ldots ,{ Sn}}{\mathrm{mean}}\left|{c}_{i}-{c}_{i-1}\right|}$			(B.pv)
${c}_{i}\in \{0, 1\}$: ground-truth class label for ith example (0 for negative, 1 for positive class), In LogLoss; ${p}_{i}\in [0, 1]$ scores produced by a model C for each of Sn examples, In others: ${p}_{i}=P\left({p}_{i}=1\right|{ x}_{i})=C({x}_{i})$: predicted class membership score where ${p}_{i}\ge \theta$ outcome is positive otherwise the outcome is negative (decision threshold $\theta \; \mathrm{in} \; [0, 1]$). $\overline{c }$ is the arithmetic mean of class labels
Probabilistic Error/Loss Measures (‘x’ in equation numbers mean “not a proper binary-classification instrument and excluded in PToPI”)
$LogLoss=-\frac{1}{Sn}\sum_{i}^{Sn}{c}_{i}{\mathrm{log}}_{2}{p}_{i}+{(1-c}_{i}){\mathrm{log}}_{2}(1-{p}_{i})$							(B.p1)
(Relative absolute/squared error measures)
$MRAE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\left|{rel\_e}_{i}\right|$			(B.p2.1)	$MdRAE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{median}}\left|{rel\_e}_{i}\right|$			(B.p2.2)
$GMRAE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{geomean}}\left|{rel\_e}_{i}\right|$			(B.p2.3)	$RAE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{sum}}\left|{rel\_e}_{i}\right|$			(B.p2.3x)
$RSE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{sum}}{{rel\_e}_{i}}^{2}$							(B.p2.4x)
(Squared error measures, continued from MSE, RMSE, and MdSE squared error metrics below)
$SSE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{sum}}{{e}_{i}}^{2}$			(B.p1.4x)	$nMSE \;\text{v1}=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\frac{{{e}_{i}}^{2}}{\overline{c }\cdot \overline{p} }$			(B.p1.5x.1)
$nMSE \;\text{v2}=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\frac{{{e}_{i}}^{2}}{\mathrm{var}(c)}$			(B.p1.5x.2)	$nMSE \;\text{v3}=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\frac{{{e}_{i}}^{2}}{\underset{j=1, \ldots ,{ Sn}}{\mathrm{mean}}{{\Delta c}_{j}}^{2}}$			(B.p1.5x.3)
$nMSE \;\text{v4}=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\frac{{{e}_{i}}^{2}}{\underset{j=1, \ldots ,{ Sn}}{\mathrm{mean}}{{c}_{j}}^{2}}$			(B.p1.5x.4)	$nMSE \;\text{v5}=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\frac{{{e}_{i}}^{2}}{{c}_{i}\cdot {p}_{i}}$			(B.p1.5x.5)
Probabilistic Error Metrics
$ME=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}{e}_{i}$							(B.p0x)
(Squared error metrics)
$MSE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}{{e}_{i}}^{2}$			(B.p1)	$RMSE=\sqrt{\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}{{e}_{i}}^{2}}$			(B.p1.1)
(Absolute error metrics)
$MAE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\left|{e}_{i}\right|$			(B.p2.1)	$MdAE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{median}}\left|{e}_{i}\right|$			(B.p2.2)
$MxAE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{max}}\left|{e}_{i}\right|$			(B.p2.2)	$GMAE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{geomean}}\left|{e}_{i}\right|$			(B.p2.4x)
(Percentage error metrics)
$MPE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\%{e}_{i}$	(B.p4.1x)	$MAPE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\left|\%{e}_{i}\right|$	(B.p4.1x)	$MdAPE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{median}}\left|\%{e}_{i}\right|$			(B.p4.1x)
$RMSPE=\sqrt{\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\%{{e}_{i}}^{2}}$			(B.p4.4x)	$RMdSPE=\sqrt{\underset{i=1, \ldots ,{ Sn}}{\mathrm{median}}\%{{e}_{i}}^{2}}$			(B.p4.5x)
(Symmetric percentage error metrics)
$sMAPE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\left|sym\_\mathrm{\%}{e}_{i}\right|$			(B.p3.0x)	$nsMAPE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}\left|\frac{sym\_\mathrm{\%}{e}_{i}}{2}\right|$			(B.p3)
$nsMdAPE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{median}}\left|\frac{sym\_\mathrm{\%}{e}_{i}}{2}\right|$							(B.p3.1x)
Probabilistic Error Metrics (Absolute scaled errors for time-series forecasting, not applicable for binary classification)
$MASE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}sca\_{e}_{i}$			(B.px.1)	$MdASE=\underset{i=1, \ldots ,{ Sn}}{\mathrm{median}}sca\_{e}_{i}$	(B.px.2)	$RMSSE=\sqrt{\underset{i=1, \ldots ,{ Sn}}{\mathrm{mean}}sca\_{{e}_{i}}^{2}}$	(B.px.3)
Inter/intra-model complexity criteria based on probabilistic error metrics (k: number of model parameters)
$AIC=2k-2{ln}MSE$			(B.i)	$BIC= \textit{kln}Sn-2\textit{ln}MSE$			(B.ii)

Appendix 3: (Online) PToPI: Periodic Table of Performance Instruments (Full View)

The proposed binary-classification performance instruments exploratory table for a total of 57 performance instruments is provided online at https://github.com/gurol/ptopi as in two files: PToPI.xlsx spreadsheet file and ‘Fig. C.1.png’ high-resolution image file). The full view (Fig. C.1) presents all the information such as canonical or high-level dependency equations. See the legend in Table 6 for the design elements used in PToPI.

Appendix 4: Case Study (Performance Evaluation in Android Mobile-Malware Classification) Selection Methodology

The case study described in “Case study: performance evaluation in android mobile-malware classification” surveys 78 academic studies about Android malware classification from 2012 to 2018. The references are given in online Table E.1. Additional to 35 symposia, conference, and journal articles published that had already been reviewed by us, 43 articles were included using the following methodology:

Selecting the relevant journal articles by searching the IEEE academic database with having "((Android and malware) and (accuracy or precision or "True Positive" or "False Positive") and (Classification OR Detection))" words in the articles’ title, abstract, or body on 27 March 2018.

Selecting the relevant conference/journal articles by searching Google Scholar by matching the same keywords above and reviewing the first ten related articles per year from 2012 to 2018 in May 2018, excluding the patents.

Among the relevant surveyed studies, all the articles were included in performance evaluation terminology findings where available. For other statistics, only the related studies were included, as specified in Appendix 5.

Appendix 5: (Online) References of the Surveyed Studies and the Detailed Results of the Case Study in “Case study: performance evaluation in android mobile-malware classification”.

The detailed data and results are provided online at https://doi.org/10.17632/5c442vbjzg.3 via the Mendeley Data platform. Besides, the online Table E.1, which is provided at (AppendixE_Table_E1.pdf) at https://www.github.com/gurol/ptopi, lists the references of the surveyed studies selected by the methodology described in Appendix 4 above.