Reduced Multi-class Contour Preserving Classification

Abstract

This research presents the augmentation of the original contour preserving classification technique to support multi-class data and to reduce the number of synthesized vectors, called multi-class outpost vectors (MCOVs). The technique has been proven to function on both synthetic-problem data sets and real-world data sets correctly. The technique also includes three methods to reduce the number of MCOVs by using minimum vector distance selection between fundamental multi-class outpost vectors and additional multi-class outpost vectors to select only MCOVs located at the decision boundary between consecutive classes of data. The three MCOV reduction methods include the FF-AA reduction method, the FA-AF reduction method, and the FAF-AFA reduction method. An evaluation has been conducted to show the reduction capability, the contour preservation capability, and the levels of classification accuracy of the three MCOV reduction methods on both non-overlapping and highly overlapping synthetic-problem data sets and highly overlapping real-world data sets. For non-overlapping problems, the experimental results present that the FA-AF reduction method can partially reduce the number of MCOVs while preserving the contour of the problem most accurately and obtaining similar levels of classification accuracy as when the whole set of MCOVs is used. For highly overlapping problems, the experimental results present that the FF-AA reduction method can partially reduce the number of MCOVs while preserving the contour of the problem most accurately and obtaining similar levels of classification accuracy as when the whole set of MCOVs is used.

Appendix

1.1 Runtime Complexity: Multi-class Contour Preserving Classification

The runtime complexity of MCOV Generation Algorithm ($T_{mcov(n)}$) comprises the runtime complexity of FMCOV Generation Algorithm ($T_{fmcov(n)}$) and the runtime complexity of AMCOV Generation Algorithm ($T_{amcov(n)}$) as described in Eq. (19).

$$\begin{aligned} T_{mcov(n)} = T_{fmcov(n)} + T_{amcov(n)}. \end{aligned}$$

(19)

where n is the number of input vectors in all classes.

The FMCOVs and AMCOVs are generated from the algorithm presented in Algorithm 1 using the Euclidean distance function described in Eq. (24) on a set of input vectors. The runtime complexity of Euclidean distance is described in Eq. (25).

The runtime complexity of the FMCOV Generation Algorithm ($T_{fmcov(n)}$) is described in Eq. (20).

$$\begin{aligned} T_{fmcov(n)}= & {} O\left( \sum _{z=1}^{c} (n_{z} \times \left( \sum _{z=1}^{c} (n_{z}) - n_{z})\right) \times T_{d(p,q)}\right) .\nonumber \\= & {} O\left( \sum _{z=1}^{c} \left( n_{z} \times (cn - n_{z})\right) \times d\right) .\nonumber \\= & {} O\left( c(c-1)n^2 \times d\right) .\nonumber \\= & {} O\left( c^2n^2\right) .\nonumber \\= & {} O\left( n^2\right) . \end{aligned}$$

(20)

where $n_{z}$ is the number of input vectors in class z, n is the number of input vectors in all classes, c is the number of classes of data, p and q are two points in different classes, d is the number of dimensions of the input vector, k is a constant and $T_{d(p,q)}$ is the runtime complexity of Euclidean distance function between points p and q.

The runtime complexity of the AMCOV Generation Algorithm ($T_{amcov(n)}$) is described in Eq. (21).

$$\begin{aligned} T_{amcov(n)}= & {} O\left( \sum _{z=1}^{c} n_{z} \times T_{d(p,q)}\right) .\nonumber \\= & {} O(cn \times d).\nonumber \\= & {} O(n). \end{aligned}$$

(21)

where $n_{z}$ is the number of input vectors in class z, n is the number of input vectors in all classes, c is the number of classes of data, p and q are two points in different classes, d is the number of dimensions of the input vector and $T_{d(p,q)}$ is the runtime complexity of Euclidean distance function between points p and q.

As a result, the runtime complexity of the MCOV Generation Algorithm ($T_{mcov(n)}$) described in Eq. (19) is formulated as Eq. (22).

$$\begin{aligned} T_{mcov(n)}= & {} T_{fmcov(n)} + T_{amcov(n)}.\nonumber \\= & {} O(n^2 + n).\nonumber \\= & {} O(n^2). \end{aligned}$$

(22)

where n is the number of input vectors in all classes.

1.2 Runtime Complexity: Reduced Multi-class Contour Preserving Classification

The runtime complexity of the RMCOV Generation Algorithm ($T_{rmcov(m+n)}$) comprises the runtime complexity of the RFMCOV Generation Algorithm ($T_{rfmcov(m)}$) and the runtime complexity of RAMCOV Generation Algorithm ($T_{ramcov(n)}$) as described in Eq. (23).

$$\begin{aligned} T_{\textit{rmcov}(m+n)} = T_{\textit{rfmcov}(m)} + T_{\textit{ramcov}(n)}. \end{aligned}$$

(23)

where m is the number of FMCOVs in all classes and n is the number of AMCOVs in all classes.

Since there are three MCOV reduction methods for selecting FMCOVs and AMCOVs as shown in Sect. 3.3, the runtime complexity of RFMCOV Generation Algorithm and the runtime complexity of RAMCOV Generation Algorithm of each reduction method is different from any other reduction methods. “FF-AA Reduction Method” section explains the runtime complexity of the FF-AA reduction method. “FA-AF Reduction Method” section explains the runtime complexity of the FA-AF reduction method. “FAF-AFA Reduction Method” section explains the runtime complexity of the FAF-AFA reduction method.

1.2.1 FF-AA Reduction Method

This section presents the Runtime Complexity of the FF-AA Reduced Multi-class Outpost Vector Generation Algorithm: FOV vs FOV, AOV vs AOV.

The FF-AA RFMCOVs and FF-AA RAMCOVs are selected from the algorithm presented in Algorithm 3 using the Euclidean distance function described in Eq. (24) on a set of FMCOVs and a set of AMCOVs. The runtime complexity of Euclidean distance is described in Eq. (25).

The runtime complexity of the FF-AA RFMCOV Generation Algorithm ($T_{rfmcov(m)}$) is described in Eq. (26).

$$\begin{aligned} d(p,q)= & {} \sqrt{\left( p_{1}-q_{1}\right) ^2 + \left( p_{2}-q_{2}\right) ^2 + \cdots + \left( p_{f}-q_{f}\right) ^2}.\nonumber \\= & {} \sqrt{\sum _{i=1}^{f} \left( p_{i}-q_{i}\right) ^2}. \end{aligned}$$

(24)

where p and q are two points in Euclidean space, $p_i$ and $q_i$ are the value of the feature i of input vectors p and q, respectively and f is the number of features of the point.

$$\begin{aligned} T_{d(p,q)}= & {} O\left( \sqrt{\sum _{i=1}^{d} \left( p_{i}-q_{i}\right) ^2}\right) .\nonumber \\= & {} O(d). \end{aligned}$$

(25)

where p and q are two points in Euclidean space, $p_i$ and $q_i$ are the value of the dimension i of input vectors p and q, respectively and d is the number of dimensions of the point.

$$\begin{aligned} T_{rfmcov(m)}= & {} O\left( \sum _{z=1}^{c} (m_{z} \times \left( \sum _{z=1}^{c} (m_{z}) - m_{z})\right) \times T_{d(p,q)}\right) .\nonumber \\= & {} O\left( \sum _{z=1}^{c} \left( m_{z} \times (cm - m_{z})\right) \times d\right) .\nonumber \\= & {} O\left( c(c-1)m^2 \times d\right) .\nonumber \\= & {} O\left( m^2\right) . \end{aligned}$$

(26)

where $m_{z}$ is the number of FMCOVs in class z, m is the number of all FMCOVs, c is the number of classes of data, p and q are two points in different classes, d is the number of dimensions of the input vector, k is a constant and $T_{d(p,q)}$ is the runtime complexity of Euclidean distance function between points p and q.

The runtime complexity of the FF-AA RAMCOV Generation Algorithm ($T_{ramcov(n)}$) is described in Eq. (27).

$$\begin{aligned} T_{ramcov(n)}= & {} O\left( \sum _{z=1}^{c} (n_{z} \times \left( \sum _{z=1}^{c} (n_{z}) - n_{z})\right) \times T_{d(p,q)}\right) .\nonumber \\= & {} O\left( \sum _{z=1}^{c} \left( n_{z} \times (cn - n_{z})\right) \times d\right) .\nonumber \\= & {} O\left( c(c-1)n^2 \times d\right) .\nonumber \\= & {} O\left( n^2\right) . \end{aligned}$$

(27)

where $n_{z}$ is the number of AMCOVs in class z, n is the number of all AMCOVs, c is the number of classes of data, p and q are two points in different classes, d is the number of dimensions of the input vector, k is a constant and $T_{d(p,q)}$ is the runtime complexity of Euclidean distance function between points p and q.

As a result, the runtime complexity of the FF-AA Reduced Multi-class Outpost Vector Generation Algorithm ($T_{rmcov(n)}$) described in Eq. (23) is formulated as Eq. (28).

$$\begin{aligned} T_{rmcov(m+n)}= & {} T_{rfmcov(m)} + T_{ramcov(n)}.\nonumber \\= & {} O\left( m^2 + n^2\right) . \end{aligned}$$

(28)

where m is the number of all FMCOVs and n is the number of all AMCOVs.

Since the number of FMCOVs and the number of AMCOVs are generally equal to the number of input vectors, m is generally equal to n as well. When m is equal to n, the runtime complexity of FF-AA reduction method will be equal to $O(m^2)$.

1.2.2 FA-AF Reduction Method

This section presents the Runtime Complexity of the FA-AF RMCOV Generation Algorithm: FOV vs AOV, AOV vs FOV.

The FA-AF RFMCOVs and FA-AF RAMCOVs are selected from the algorithm presented in Algorithm 4 using the Euclidean distance function described in Eq. (24) on a set of FMCOVs and a set of AMCOVs. The runtime complexity of Euclidean distance is described in Eq. (25).

The runtime complexity of the FA-AF RFMCOV Generation Algorithm ($T_{rfmcov(m)}$) is described in Eq. (29).

$$\begin{aligned} T_{rfmcov(m)}= & {} O\left( \sum _{z=1}^{c} (m_{z} \times \left( \sum _{z=1}^{c} (n_{z}) - n_{z})\right) \times T_{d(p,q)}\right) .\nonumber \\= & {} O\left( \sum _{z=1}^{c} (m_{z} \times (cn - n_{z})) \times d\right) .\nonumber \\= & {} O(c(c-1)mn \times d).\nonumber \\= & {} O(mn). \end{aligned}$$

(29)

where $m_{z}$ is the number of FMCOVs in class z, $n_{z}$ is the number of AMCOVs in class z, m is the number of all FMCOVs, n is the number of all AMCOVs, c is the number of classes of data, p and q are two points in different classes, d is the number of dimensions of the input vector, k is a constant and $T_{d(p,q)}$ is the runtime complexity of Euclidean distance function between points p and q.

The runtime complexity of the FA-AF RAMCOV Generation Algorithm ($T_{ramcov(n)}$) is described in Eq. (30).

$$\begin{aligned} T_{ramcov(n)}= & {} O\left( \sum _{z=1}^{c} (n_{z} \times \left( \sum _{z=1}^{c} (m_{z}) - m_{z})\right) \times T_{d(p,q)}\right) .\nonumber \\= & {} O\left( \sum _{z=1}^{c} (n_{z} \times (cm - m_{z})) \times d\right) .\nonumber \\= & {} O(c(c-1)nm \times d).\nonumber \\= & {} O(nm).\nonumber \\ \end{aligned}$$

(30)

where $m_{z}$ is the number of FMCOVs in class z, $n_{z}$ is the number of AMCOVs in class z, m is the number of all FMCOVs, n is the number of all AMCOVs, c is the number of classes of data, p and q are two points in different classes, d is the number of dimensions of the input vector, k is a constant and $T_{d(p,q)}$ is the runtime complexity of Euclidean distance function between points p and q.

As a result, the runtime complexity of the FA-AF RMCOV Generation Algorithm ($T_{rmcov(n)}$) described in Eq. (23) is formulated as Eq. (31).

$$\begin{aligned} T_{rmcov(m+n)}= & {} T_{rfmcov(m)} + T_{ramcov(n)}.\nonumber \\= & {} O(mn + nm).\nonumber \\= & {} O(mn). \end{aligned}$$

(31)

where m is the number of all FMCOVs and n is the number of all AMCOVs.

Since the number of FMCOVs and the number of AMCOVs are generally equal to the number of input vectors, m is generally equal to n as well. When m is equal to n, the runtime complexity of FA-AF reduction method will be equal to $O(m^2)$.

1.2.3 FAF-AFA Reduction Method

This section presents the Runtime Complexity of the FAF-AFA RMCOV Generation Algorithm: FOV vs FOV+AOV, AOV vs AOV+FOV.

The FAF-AFA RFMCOVs and FAF-AFA RAMCOVs are selected from the algorithm presented in Algorithm 5 using the Euclidean distance function described in Eq. (24) on a set of FMCOVs and a set of AMCOVs. The runtime complexity of Euclidean distance is described in Eq. (25).

The runtime complexity of the FAF-AFA RFMCOV Generation Algorithm ($T_{rfmcov(m)}$) is described in Eq. (32).

$$\begin{aligned} T_{rfmcov(m)}= & {} O\left( \sum _{z=1}^{c} \left( m_{z} \times \left( (\sum _{z=1}^{c} (n_{z}) - n_{z}) + \left( \sum _{z=1}^{c} (m_{z}) - m_{z}\right) \right) \right) \times T_{d(p,q)}\right) .\nonumber \\= & {} O\left( \sum _{z=1}^{c} \left( m_{z} \times \left( \left( cn - n_{z}\right) + \left( cm - m_{z}\right) \right) \times d\right) \right) .\nonumber \\= & {} O\left( (c(c-1)mn) + (c(c-1)m^2) \times d\right) .\nonumber \\= & {} O\left( \left( c^2mn + c^2m^2\right) \times d\right) .\nonumber \\= & {} O\left( mn + m^2\right) . \end{aligned}$$

(32)

where $m_{z}$ is the number of FMCOVs in class z, $n_{z}$ is the number of AMCOVs in class z, m is the number of all FMCOVs, n is the number of all AMCOVs, c is the number of classes of data, p and q are two points in different classes, d is the number of dimensions of the input vector, k is a constant and $T_{d(p,q)}$ is the runtime complexity of Euclidean distance function between points p and q.

The runtime complexity of the FAF-AFA RAMCOV Generation Algorithm ($T_{ramcov(n)}$) is described in Eq. (33).

$$\begin{aligned} T_{ramcov(n)}= & {} O\left( \sum _{z=1}^{c} \left( n_{z} \times \left( \left( \sum _{z=1}^{c} (n_{z}) - n_{z}\right) + \left( \sum _{z=1}^{c} (m_{z}) - m_{z}\right) \right) \right) \times T_{d(p,q)}\right) .\nonumber \\= & {} O\left( \sum _{z=1}^{c} (n_{z} \times ((cn - n_{z}) + (cm - m_{z})) \times d\right) .\nonumber \\= & {} O\left( \left( c(c-1)n^2\right) + (c(c-1)nm) \times d\right) .\nonumber \\= & {} O\left( \left( c^2n^2 + c^2nm\right) \times d\right) .\nonumber \\= & {} O\left( n^2 + nm\right) . \end{aligned}$$

(33)

where $m_{z}$ is the number of FMCOVs in class z, $n_{z}$ is the number of AMCOVs in class z, m is the number of all FMCOVs, n is the number of all AMCOVs, c is the number of classes of data, p and q are two points in different classes, d is the number of dimensions of the input vector, k is a constant and $T_{d(p,q)}$ is the runtime complexity of Euclidean distance function between points p and q.

As a result, the runtime complexity of the FAF-AFA RMCOV Generation Algorithm ($T_{rmcov(n)}$) described in Eq. (23) is formulated as Eq. (34).

$$\begin{aligned} T_{rmcov(m+n)}= & {} T_{rfmcov(m)} + T_{ramcov(n)}.\nonumber \\= & {} O\left( \left( mn + m^2\right) + \left( n^2 + nm\right) \right) .\nonumber \\= & {} O\left( m^2 + n^2\right) . \end{aligned}$$

(34)

where m is the number of all FMCOVs and n is the number of all AMCOVs.

Since the number of FMCOVs and the number of AMCOVs are generally equal to the number of input vectors, m is generally equal to n as well. When m is equal to n, the runtime complexity of FAF-AFA reduction method will be equal to $O(m^2)$. All three MCOV reduction methods have identical runtime complexity.

1.3 Experiment: Support Vector Machine (SVM)

The experiments in Sects. 4.1, 4.2, and 4.3 are conducted on a feed-forward neural network to show the practicality of the three MCOV reduction methods in terms of MCOV reduction capability and generalization on a non-overlapping four-class synthetic-problem data set, eight highly overlapping two-class synthetic-problem data sets from the ELENA Project, and ten real-world data sets from both UCI machine learning repository and ELENA Project, respectively. This section additionally presents a comparison on the generalization among the three MCOV reduction methods and the Support Vector Machine (SVM) [22] on these data sets. For references, the characteristics of a non-overlapping four-class synthetic-problem data set are described in the first paragraph of Sect. 4.1, the characteristics of eight highly overlapping two-class synthetic-problem data sets from the ELENA Project are presented in Table 4, the characteristics of ten real-world data sets from both UCI machine learning repository and ELENA Project are presented in Table 8. The experiments are conducted with a LIBSVM [35] on MATLAB R2014b using the following parameters:

• SVM Type = C-SVC (multi-class classification)

• Kernel Function = Radial Basis Function

• Degree in Kernel Function = 3

• Gamma in Kernel Function = 1

• Coefficient 0 in Kernel Function = 0

• Cost of C-SVC = 1

• Weight of C-SVC = 1

• Cache Memory Size = 100 MB

• Tolerance of Termination Criterion = 0.001

• Shrinking Heuristics = 1

• Probability Estimates = 0

Table 12 Misclassification rates of three MCOV reduction methods and Support Vector Machine on both synthetic and real-world data sets

Table 12 presents the misclassification rates of the three MCOV reduction methods and SVM on both synthetic and real-world data sets. The experimental results from a feed-forward neural network on ORG+FF-AA, ORG+FA-AF, and ORG+FAF-AFA data sets are taken from Tables 2, 5, 6, 10, and 11, respectively. The experiment result from the method that yields the lowest misclassification rate is highlighted in bold. For non-overlapping synthetic-problem data set having various population sizes, the three MCOV reduction methods yield significantly lower misclassification rates than the SVM. For overlapping synthetic-problem data sets, the FF-AA reduction method yields slightly lower misclassification rates than the SVM, and much lower misclassification rates than FA-AF and FAF-AFA reduction methods. More precisely, it yields the lowest misclassification rates. For overlapping real-world data sets, the three MCOV reduction methods generally yield lower misclassification rates than the SVM. It is noticeable that the misclassification rates of SVM on Statlog (Landsat Satellite), Forest Cover Type, Pen-based Recognition of Handwritten Digits, Optical Recognition of Handwritten Digits, and Satimage data sets are significantly large. SVM might need to be fine-tuned to be able to classify these data sets more accurately, or it may not be suitable for these data sets.

Based on the results from these experiments, it can be concluded that the three MCOV reduction methods outperform a Support Vector Machine on both synthetic and real-world data sets.