Robustification of Gaussian Bayes Classifier by the Minimum β-Divergence Method

Abstract

The goal of classification is to classify new objects into one of the several known populations. A common problem in most of the existing classifiers is that they are very much sensitive to outliers. To overcome this problem, several author’s attempt to robustify some classifiers including Gaussian Bayes classifiers based on robust estimation of mean vectors and covariance matrices. However, these type of robust classifiers work well when only training datasets are contaminated by outliers. They produce misleading results like the traditional classifiers when the test data vectors are contaminated by outliers as well. Most of them also show weak performance if we gradually increase the number of variables in the dataset by fixing the sample size. As the remedies of these problems, an attempt is made to propose a highly robust Gaussian Bayes classifiers by the minimum β-divergence method. The performance of the proposed method depends on the value of tuning parameter β, initialization of Gaussian parameters, detection of outlying test vectors, and detection of their variable-wise outlying components. We have discussed some techniques in this paper to improve the performance of the proposed method by tackling these issues. The proposed classifier reduces to the MLE-based Gaussian Bayes classifier when β → 0. The performance of the proposed method is investigated using both synthetic and real datasets. It is observed that the proposed method improves the performance over the traditional and other robust linear classifiers in presence of outliers. Otherwise, it keeps equal performance.

Appendix

Figures 3a–b and 4a–b show the performance results of the proposed Bayesian robust LDA and QDA both under different conditions. Figure 3a shows the plots of training (left) and test (right) MER against the common differences between the mean components of two mean vectors with equal covariance matrices (V₁ = V₂) for the data structure D_1B of p = 15 variables respectively, in absence outliers, while Fig. 3b shows the plots of training (left) and test (right) MER against the common differences between the mean components of two mean vectors with equal covariance matrices (V₁ = V₂) for the data structure D_1B of p = 15 variables respectively, in presence of 15% outlying data vectors. We observed that the proposed LDA shows smaller test MER than the proposed QDA in both absence and presence of outliers. Figure 4a shows the plots of training (left) and test (right) MER against the common differences between the mean components of two mean vectors with unequal covariance matrices (V₁ ≠ V₂) for the data structure D_1B of p = 15 variables respectively, in absence outliers, while Fig. 4b shows the plots of training (left) and test (right) MER against the common differences between the mean components of two mean vectors with unequal covariance matrices (V₁ ≠ V₂) for the data structure D_1B of p = 15 variables respectively, in presence of 15% outlying data vectors. We observed that the proposed QDA shows smaller test MER than the proposed LDA in both absence and presence of outliers. Thus the proposed LDA shows better performance than the proposed QDA in the case of constant covariance matrices, while the proposed QDA shows better performance than the proposed LDA in the case of inconstant covariance matrices.

Fig. 3

Plot of misclassification error rate (MER) against the common differences between the mean components of two mean vectors with equal covariance matrices for the data structure D_1B of p = 15 variables. a Training (left) and test (right) MER in absence of outliers. b Training (left) and test (right) MER in presence of 15% outlying vectors under THCM in both training and test datasets

Fig. 4

Plot of misclassification error rate (MER) against the common differences between the mean components of two mean vectors with unequal covariance matrices for the data structure D_1B of p = 15 variables. a Training (left) and test (right) MER in absence of outliers. b Training (left) and test (right) MER in presence of 15% outlying vectors under THCM in both training and test datasets

Table 4 The average values of miss-classification error (MER) based on 500 simulated datasets of type D₂ of p = 5, 15, 50, and 75 variables in presence of outliers are computed by the classical, FSA, MCD-A, MCD-B, MCD-C, OGK, and proposed methods, where m = 3, n_j = 100;j = 1, 2,3, and the difference among the mean vectors of m are τ

However, it should be to be mentioned here that the performance of the proposed method also depends on the cutoff/threshold value (δ) of β_j-weights for outlier detection as discussed in Section 2.3.1. Figure 5 shows the plots of MER (Top) and ROC curve (Bottom) to investigate the performance of the proposed threshold/cutoff value (δ) of β_j-weight function in a comparison of the cutoff value used by Mollah et al. (2010) for outlier detection with the data structure D_1B of p = 15 variables. We observe that the proposed cutoff value produces smaller MER and larger TPR (true positive rate) by the proposed classifiers.

Fig. 5

Plots of (Top) misclassification error rate (MER) and (Bottom) ROC curve to investigate the performance of threshold/cutoff value (δ) of β_j-weight function for outlier detection with the data structure D_1B of p = 15 variables

Fig. 6

ROC curves with the average values of TPR and FPR based on 500 simulated datasets of types D_1A and D_1B in absence/presence of outliers by the classical MLE, FSA, MCD-A, MCD-B, MCD-C, OGK, and proposed methods. a ROC curve with p = 5 (left) and p = 15 (right) variables in absence of outliers. b ROC curve with p = 5 (left) and p = 15 (right) variables in presence of 15% outlying vectors under THCM in the test datasets. c ROC curve with p = 5 (left) and p = 15 (right) variables in presence of 15% outlying vectors under THCM in both training and test datasets

Fig. 7

Plot for misclassification error rate (MER) against common mean vector differences for data structure D_2B. a Training (left) and test (right) MER in presence of 50% outlying vectors under THCM in the test datasets only. b Training (left) and test (right) MER in presence of 50% outlying vectors under THCM in both training and test datasets

Fig. 8

Some important results for the proposed method with the simulated data structure D₂ = {D_2A, D_2B} and the real fish catch dataset. a Plot of β_j-weight for each data point of each group/class in presence of outliers. b The tuning parameter β_j selection by cross-validation (CV) using the measure which is defined in Eq. 17. (i) Box plot for all estimated β_j with all simulated datasets in the absence (left) and presence (right) of outliers. (ii) Plot of $D_{{\beta _{j}}_{0}}(\beta _{j})$ against β_j (17) based on CV results for β_j selection from real fish catch dataset. (iii) Plot of CV results for β_j selection from real fish catch dataset in the presence 10% outlying vectors under THCM