BenchMetrics Prob: benchmarking of probabilistic error/loss performance evaluation instruments for binary classification problems

Abstract

Probabilistic error/loss performance evaluation instruments that are originally used for regression and time series forecasting are also applied in some binary-class or multi-class classifiers, such as artificial neural networks. This study aims to systematically assess probabilistic instruments for binary classification performance evaluation using a proposed two-stage benchmarking method called BenchMetrics Prob. The method employs five criteria and fourteen simulation cases based on hypothetical classifiers on synthetic datasets. The goal is to reveal specific weaknesses of performance instruments and to identify the most robust instrument in binary classification problems. The BenchMetrics Prob method was tested on 31 instrument/instrument variants, and the results have identified four instruments as the most robust in a binary classification context: Sum Squared Error (SSE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE, as the variant of MSE), and Mean Absolute Error (MAE). As SSE has lower interpretability due to its [0, ∞) range, MAE in [0, 1] is the most convenient and robust probabilistic metric for generic purposes. In classification problems where large errors are more important than small errors, RMSE may be a better choice. Additionally, the results showed that instrument variants with summarization functions other than mean (e.g., median and geometric mean), LogLoss, and the error instruments with relative/percentage/symmetric-percentage subtypes for regression, such as Mean Absolute Percentage Error (MAPE), Symmetric MAPE (sMAPE), and Mean Relative Absolute Error (MRAE), were less robust and should be avoided. These findings suggest that researchers should employ robust probabilistic metrics when measuring and reporting performance in binary classification problems.

Appendices

Appendix A: Preliminaries

Classification and Binary Classification: Classification is a specific problem in machine learning at which a classifier (i.e. a computer program) improves its performance through learning from experience. In a supervised approach, the experience is gained by providing labeled examples (i.e. training dataset) of one or more classes with common properties or characteristics. In a binary classification or two-class classification, a classifier separates an example given into two classes. The classes are named positive (e.g., malicious software or spam) and negative (e.g., benign software or non-spam) in general.

Classification Performance and Confusion Matrix: The performance of the trained classifier (i.e. to what degree it predicts the labels of known examples) is then improved or evaluated on different labeled examples (i.e. validation or test datasets). At this stage, the classifier is supposed to be ready to predict the class of additional unknown or unlabeled instances. The binary classification performances in training, validation, or test datasets are presented by a confusion matrix, also known as a “2 × 2 contingency table” or “four-fold table”, (i.e. the number of correct and incorrect classification per positive and negative classes).

1.1 Confusion-matrix-derived instruments

Confusion-matrix-derived instruments are a convenient, familiar, and frequently used instrument category. Along with well-known metrics such as accuracy (ACC), true positive rate (TPR), and F1, other specific metrics such as Cohen’s Kappa (CK) [27] and Mathews Correlation Coefficient (MCC) [28] have been used in the evaluation of crisp classifiers that assign instances to either a positive (value: one) or negative (value: zero) class absolutely (also known as “hard label”) [24, 55]. The performance measured by these instruments can be interpreted as.

External: They present observed results without an explicit connection to internal design parameters. The classifier is modeled with a single/final optimum configuration (i.e. a model threshold).

Production-ready: They provide an estimate of the classifier’s performance in a production environment for the intended problem domain when compared to other classifiers.

Kinetic: They represent a classifier’s performance summarizing a specific application of the samples of a dataset (e.g., the ACC or MSE values that are measured for the first run or iteration in k-fold cross-validation).

1.2 Graphical-based instruments

Graphical-based performance instruments are not based on a single instance of confusion-matrix elements yielded from a specific application. Instead, they list a classifier’s performance panorama by varying a decision threshold (i.e. full operating range of a classifier) in terms of metric pairs that involve trade-off (e.g., x: FPR and y: TPR for ROC, receiver-operating-characteristic) [1]. A graph is used to visualize the variance and the area under the curve provides a single value (e.g., AUCROC, Area-Under-ROC-Curve) to summarize the variance [66]. These instruments represent the classifiers’ internal capability designed with different possible settings and provide insight into the classifiers’ potential during model development. However, since a classifier is eventually deployed with a single decision threshold in a production environment, a confusion-matrix-derived (e.g., ACC) and/or a probabilistic error/loss instrument (e.g., MSE) should be used and reported to represent the final performance. A graphical-based instrument (e.g., AUCPR) can also be included to show the classifiers' potential when used with different decision thresholds.

Appendix B

2.1 Probabilistic error/loss instruments’ equations

See Table 10.

Table 10 Instruments’ equations categorized into performance measures and metrics and instrument subtypes

Appendix C

3.1 Probabilistic error instruments aggregation and error function frequency distribution

Table 11 lists the frequency distribution of aggregation (g) and error (e_i) functions described in Table 1. The most used aggregation functions are mean and square(d) mean and error functions (shown in underlined) are absolute and percentage.

Table 11 Aggregation function and error function (underlined) frequencies

Appendix D

4.1 Introduction to BenchMetrics Prob calculator and simulation tool

The BenchMetrics Prob, depicted in Fig. 2, is a spreadsheet-based tool that is designed to prepare cases for evaluating the robustness of probabilistic error/loss instruments. The tool can be accessed online at https://github.com/gurol/BenchMetricsProb. The user interface of the tool is divided into nine parts:

I. Class label/prediction score values settings

II. Ground truth/prediction input method settings

III. Synthetic dataset instances

IV. Hypothetical classifier predictions

V. Classification examples/outputs/confusions

VI. Confusion matrix and other measures

VII. Performance metrics/measure results

VIII. Different error function results

IX. Probabilistic error/loss performance instrument results

Part I. Class label/prediction score values settings

The first part of the tool allows the users to define the class label and prediction score values. By default, the tool is set up for conventional binary classification, as shown in Fig. 5a below, where the minimum prediction score (min(p_i)) for the negative class is set to 0 and the maximum prediction score (max(p_i)) for the positive class is set to 1.

The decision threshold (class-decision boundary) is set to the middle of [0, 1] prediction score interval. However, the user can change these values by editing the cells with blue text/background color. For example, if the users want to avoid division-by-zero errors, they can set min(p_i) for the negative class to 1, max(p_i) for the positive class to 2, and θ = 1.5.

Part II. Ground truth/prediction input method settings

In the second part of the tool, shown in Fig. 5b, the users can choose between “manual” or “random” input methods for the ground truth and prediction values. If the users select the random input method, the tool will generate values according to the class label and prediction score values defined in Part I above. The user can also set the classifier’s prediction by adjusting TPR and/or TNR to be above a given value. If the users set these values to 0.5, the tool will generate purely random values. Setting a higher value made the classifier predict stratified random values. Additionally, the users can specify the number of samples (Sn) by defining the starting (e.g., 21) and ending (e.g., 40) row numbers in the sheet. Note that refreshing the sheet using the SHIFT and F9 shortcut keys will change the random values.

Fig. 5

a Class label/prediction score values settings (screenshot of BenchMetrics Prob—Part I) b Ground truth/prediction input method settings (screenshot of BenchMetrics Prob—Part II)

Part III. Synthetic dataset instances

Part II of the BenchMetrics Prob tool, shown in Fig. 6, generates the synthetic dataset instances for evaluation. The dataset instances (“i”) are numbered sequentially in the first column, starting from 1. In the cells with blue text/background colors in the second column, users can manually enter the class labels for each instance when the ground truth input method is selected as “Manual”. The values should be either 0 or 1 for default binary classification problems. When the ground truth input method is set to “Random”, automatically generated random class labels are displayed in the third column. These values should not be changed by the users. The total number of instances generated can be changed in Part II by adjusting the "Number of samples (Sn)" parameter. The formulas should be pasted to the rows for the new instances.

Fig. 6

Synthetic dataset instances (screenshot of BenchMetrics Prob—Part III)

Part IV. Hypothetical classifier predictions

The same approach in Part III is applied to the predictions of the hypothetical classifier per corresponding synthetic dataset instances. Figure 7 shows the predictions along with the dataset instances where p_i values are either manually entered in the fourth column (shown in blue text/background) or automatically generated in the last column. Note that the tool takes the columns according to current “random”/“manual” settings shown in Fig. 5b.

Fig. 7

Hypothetical classifier predictions (screenshot of BenchMetrics Prob – Part IV)

Part V. Classification examples/outputs/confusions

Having generated synthetic dataset examples and corresponding hypothetical classifier’s prediction outputs, Part IV summarizes ground truth, predictions, and confusion status. Figure 8 shows all possible confusions (e.g., the first instance is “positive” but predicted as “outcome negative” so that the instance is classified as “false negative”).

Fig. 8

Classification examples/outputs (screenshot of BenchMetrics Prob—Part VII)

Part VI. Confusion matrix and other measures

Part VI shown in Fig. 9 provides the confusion matrix and other measures based on the matrix (total of 15 measures). The values summarize the current case’s classification performance as a crisp classifier.

Fig. 9

Confusion matrix and other measures (screenshot of BenchMetrics Prob—Part V)

Part VII. Performance metrics/measure results

In Part VII, shown in Fig. 10, the tool provides various performance metrics and measures derived from the confusion matrix, with a total of 21 instruments. The zero–one loss metrics described in Sect. 2.4 are shown in red text color. In addition, two measures of classifier model complexity are calculated based on the number of model parameters (k). These measures are:

Akaike Information Criterion (AIC): A measure of the quality of a model that penalizes models for the number of parameters used. Lower AIC values indicate a better model fit.

Bayesian Information Criterion (BIC): Similar to AIC, BIC also penalizes models for the number of parameters used. However, BIC has a stronger penalty for model complexity than AIC. Lower BIC values indicate a better model fit.

The AIC and BIC values can be used to compare different models and select the one with the best fit.

Fig. 10

Performance metrics/measure results (screenshot of BenchMetrics Prob – Part VI)

Part VIII. Error function results

Part VIII shows the results of error function based on class labels (c_i) and prediction scores (p_i), as shown in Fig. 11. Probabilistic error/loss instruments summarize those errors listed in rows into a single figure (in Part IX below) according to their aggregation functions.

Part IX. Probabilistic error/loss performance instrument results

Part IX, shown in Fig. 12, lists the results of probabilistic performance instruments for the predictions made on dataset instances. The instruments are grouped into subtypes (shown in black background text).

Note that the current instrument outputs, including confusion-matrix-based ones and the configuration setting, are listed in the sixth row in a separate worksheet (‘simulation cases’). You can copy and paste the row into another row to create a simulation case for your analysis. The benchmarking results were already provided in this sheet per seven cases.

Fig. 11

Different error function results (screenshot of BenchMetrics Prob—Part VIII)

Fig. 12

Probabilistic error/loss performance instrument results (screenshot of BenchMetrics Prob—Part IX)