On optimal pairwise linear classifiers for normal distributions: the d-dimensional case

Abstract

We consider the well-studied pattern recognition problem of designing linear classifiers. When dealing with normally distributed classes, it is well known that the optimal Bayes classifier is linear only when the covariance matrices are equal. This was the only known condition for classifier linearity. In a previous work, we presented the theoretical framework for optimal pairwise linear classifiers for two-dimensional normally distributed random vectors. We derived the necessary and sufficient conditions that the distributions have to satisfy so as to yield the optimal linear classifier as a pair of straight lines.

In this paper we extend the previous work to d-dimensional normally distributed random vectors. We provide the necessary and sufficient conditions needed so that the optimal Bayes classifier is a pair of hyperplanes. Various scenarios have been considered including one which resolves the multi-dimensional Minsky’s paradox for the perceptron. We have also provided some three-dimensional examples for all the cases, and tested the classification accuracy of the corresponding pairwise-linear classifier. In all the cases, these linear classifiers achieve very good performance. To demonstrate that the current pairwise-linear philosophy yields superior discriminants on real-life data, we have shown how linear classifiers determined using a maximum-likelihood estimate (MLE) applicable for this approach, yield better accuracy than the discriminants obtained by the traditional Fisher's classifier on a real-life data set. The multi-dimensional generalization of the MLE for these classifiers is currently being investigated.

Introduction

The problem of finding linear classifiers has been the study of many researchers in the field of pattern recognition (PR) [3], [17], [18], [19]. Linear classifiers are very important because of their simplicity when it concerns implementation, and their classification speed. Various schemes to yield linear classifiers are reported in the literature such as Fisher’s approach [3], [4], [5], the perceptron algorithm (the basis of the back propagation neural network learning algorithms) [6], [7], [8], [9], piecewise recognition models [10], random search optimization [11], removal classification structures [12], adaptive linear dimensionality reduction [13] (which outperforms Fisher's classifier for some data sets), and linear constrained distance-based classifier analysis [14] (an improvement to Fisher's approach designed for hyperspectral image classification). All of these approaches suffer from the lack of optimality, and thus, although they do determine linear classification functions, the classifier is not optimal.

Apart from the results reported in [1], [15], in statistical PR, the Bayesian linear classification for normally distributed classes involves a single case. This traditional case is when the covariance matrices are equal [16], [17], [18]. In this case, the classifier is a single straight line (or a hyperplane in the d-dimensional case) completely specified by a first-order equation.

In [1], [15], we showed that although the general classifier for two-dimensional normally distributed random vectors is a second-degree polynomial, this polynomial degenerates to be either a single straight line or a pair of straight lines. Thus, we have found the necessary and sufficient conditions under which the classifier can be linear even when the covariance matrices are not equal. In this case, the classification function is a pair of first-order equations, which are factors of the second-order polynomial (i.e. the classification function). When the factors are equal, the classification function is given by a single straight line, which corresponds to the traditional case when the covariance matrices are equal.

Some examples of pairwise-linear classifiers for two and three-dimensional normally distributed random vectors can be found in Ref. [3, pp. 42–43]. By studying these, the reader should observe that the existence of such classifiers was known. The novelty of our results are the conditions for pairwise-linear classifiers, and the demonstration that these, in their own right, lead to superior linear classifiers.

In this paper, we extend these conditions for d-dimensional normal random vectors, where d>2. We assume that the features of an object to be recognized are represented as a d-dimensional vector which is an ordered tuple X=[x₁…x_d]^T characterized by a probability distribution function. We deal only with the case in which these random vectors have a jointly normal distribution, where class ω_i has a mean M_i and covariance matrix $\text{Σ}{}_{\mathrm{i}}\text{,}\text{i=1,2}$.

Without loss of generality, we assume that the classes ω₁ and ω₂ have the same a priori probability, 0.5, in which case, the classifier is given by

$$\text{log}\text{|Σ}{}_2\text{|}\text{|Σ}{}_1\text{|}\text{−(X−M}{}_1\text{)}{}^{\mathrm{<mtext>T</mtext>}}\text{Σ}{}_1{}^{\mathrm{-1}}\text{(X−M}{}_1\text{)+}\text{(X−M}{}_2\text{)}{}^{\mathrm{<mtext>T</mtext>}}\text{Σ}{}_2{}^{\mathrm{-1}}\text{(X−M}{}_2\text{)=0.}$$

When Σ₁=Σ₂, the classification function is linear [3], [18], [19]. For the case when Σ₁ and Σ₂ are arbitrary, the classifier results in a general equation of second degree which results in the classifier being a hyperparaboloid, a hyperellipsoid, a hypersphere, a hyperboloid, or a pair of hyperplanes. This latter case is the focus of our present study.

The results presented here have been rigorously tested. In particular, we present some empirical results for the cases in which the optimal Bayes classifier is a pair of hyperplanes. It is worth mentioning that we tested the case of Minsky's paradox [20] on randomly generated samples, and we have found that the accuracy is very high even though the classes are significantly overlapping.