Stochastic nonparallel hyperplane support vector machine for binary classification problems and no-free-lunch theorems

In this paper, the binary classification problem is considered and its solution is proposed as the formulated classification model, based on genetic algorithm (GA) and nonparallel hyperplane support vector machine (NHSVM), termed as stochastic nonparallel hyperplane support vector machine (SNHSVM). As GA provably violates the non-revisiting condition of the no-free-lunch theorems for optimization (NFLO), then SNHSVM have the natural property that NFLO do not apply to it. All the experiments are performed in a scenario in which no-free-lunch theorems for machine learning (NFLM) do not apply on all the compared machines. The hypothesis is that in such a scenario some classifier can perform better than others. The experiments are performed on the real world UCI datasets and the SNHSVM is compared with the state of art support vector based classifiers with performance measure as accuracy. SNHSVM achieves the highest accuracy in 100% of the cases and the Friedman test confirms the better performance of SNHSVM on all of the datasets used. These results validate the hypothesis empirically while apart from SNHSVM the NFLM floats up for the other compared classifiers.

I am highly thankful to Mr. Arvind Upadhyay (IES-IPSA, India), the complete computer science department IES-IPSA and Dr. D. H. Wolpert for their helpful suggestions. I am heartily indebted to the anonymous reviewers whose invaluable comments pushed me beyond my thinking limits.

Authors and Affiliations

1. Institute of Engineering and Science, IPS Academy, Rajendra Nagar A.B. Road, Indore, 452012, India

   Ashish Sharma

Corresponding author

Correspondence to Ashish Sharma.

Conflict of interest

The author declares no conflict of interest.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix 1

Here, I analyze the GA as presented in Algorithm-I mathematically and prove the theorems that are used in the paper.

Then, consider the optimization problem (this is same as (16)):

where \(X\) is a bounded and compact measurable space. \(f\left( x \right)\) is called the objective function and the set \(S_{OPT}\) is the set of all \(x\) such that for each \(x \in S_{OPT}\), \(f\left( x \right)\) is maximum of \(f\), formally:

where \(f_{max}\) is the maximum value of \(f\). In this analysis it is assumed that \(S_{OPT}\) is not empty and contains at least one element. Let \(\phi \left( \cdot \right)\) is a measure on \(X\), then \(\phi \left( {S_{OPT} } \right) = 0\) as \(S_{OPT}\) contains a few elements usually, so for the ease of the analysis it is assumed that \(S_{OPT - \varepsilon } = \left\{ {x;\left| {f\left( x \right) - f_{max} } \right| \le \varepsilon } \right\}\) is the set that I mean by \(S_{OPT}\) henceforth, where \(\varepsilon\) is a small positive.

The problem (24) or (16) are same and is solved with the genetic algorithm GA which was described previously in the paper (as Algorithm-I) and now for that GA a mathematical model is presented for its convergence and performance analysis. The GA will be modeled as a Markov Chain next.

A Markov chain is a random process that is completely described by its state transition probability matrix. Therefore, to describe a Marko chain one needs to specify its states and the state transition probabilities. Let, GA operates on the binary strings (solutions) each having the fixed length \(\left( l \right)\). Then, the total number of possible strings is \(r = 2^{l}\). In what follows italic p stands for probability, while non-italic p stands for population (the context also clear this distinction). Let, the set of all possible strings is \(\varOmega \left( { = X} \right)\) and \(\varOmega^{n}\) is the set of all possible n-string sets, i.e. let \(\text p = \left\{ {x_{1} ,x_{2} \ldots x_{n} ;x_{i} \in \varOmega \;for\;all\;1 \le i \le n} \right\},\) then \(\varOmega^{n} = \left\{ {\text p;\text p \subseteq \varOmega } \right\}\). The set p is called a population having a size \(n\) and the number of possible populations is N. Let \(\text p\;is\;represented\;as\;\varvec{x}\;\left( {bold\;\varvec{x}} \right)\) and the fitness of x is defined as:

The set of all populations that contains the global optimal string is defined as:

Definition 1

Let Z is a \({\text{r}} \times N\) matrix (rows of Z are all possible strings and columns of Z are all possible populations) defined as follows:\(Z = \left[ {z_{i,j} } \right]\), where \(z_{i,j}\) is the number of occurrences of string i in population j.

It is clear that some enumeration of all possible strings and populations must exist so that Z is well defined. One can agree on any kind of possible enumerations. One interpretation is based on Z as follows: the strings are numbered from \(0\) to \(r - 1\) and let \(p_{i}\) and \(p_{j}\) be any two populations as column vector of Z then \(p_{i}\) comes before \(p_{j}\) if interpreting the transpose of \(p_{i}\) and \(p_{j}\) as integer’s yields \(p_{i} < p_{j} .\)

Definition 2

The number of possible populations is

Definition 3

The states of the Markov chain are the N possible populations.

Definition 4

The transition probabilities are given by an \(N \times N\) transition probability matrix \(\left( {\text{Q}} \right)\) in which an entry \(Q_{ij}\) gives the probability that the kth generation (population) will be \({\text{p}}_{j}\) given that the \(k - 1{\text{st}}\) generation is \({\text{p}}_{i}\).

\(Q_{ij}\) can be calculated as follows with respect to \(p_{i} \left( y \right)\), where \(p_{i} \left( y \right)\) is the probability of producing string \(y\) in the \(i + 1{\text{th}}\) generation assuming the currect generation is ith having population \({\text{p}}_{i}\). Let the next generation is \({\text{p}}_{j}\) having \(z_{y,j}\) occurrences of string y and the probability of such occurrences of string y is (almost) given by \(\left\{ {p_{i} \left( y \right)} \right\}^{{z_{y,j} }}\). We can fill the n vacancies for strings in a population (\({\text{say}}\;{\text{p}}_{j}\)) as follows:

The binomial coefficient gives us the number of ways of choosing \(z_{0,j }\) occurrences of string 0 as:

After that there are \(n - z_{0,j}\) vacancies remains to be filled, so for string 1 the number of ways of choosing \(z_{1,j}\) occurrences of it is:

Continuing as above for all the strings the total combination is:

So, \(Q_{ij}\) can be written as a multinomial distribution with parameters \(n, p_{i} \left( 0 \right), \ldots , p_{i} \left( {r - 1} \right)\) as:

Definition 5

(Model-I) [40] The Markov chain model for GA is described by \(Q_{ij}\) whose states are as defined in Definition 3.

Now, it can be proved that the GA can break the non-revisiting condition of NFLO as Theorem 1.

Theorem 1

The GA as defined in Definition 5 have a positive probability of breaking the non-revisiting condition of NFLO.

Proof

According to the above discussion and the definition of \(Q_{ij}\) the probability of repeating a string y (in the next generation) is given by \(p_{i} \left( y \right)\) and moreover the probability that there are \(z_{y,j}\) occurrences of string y (in the next generation) is given by \(\left\{ {p_{i} \left( y \right)} \right\}^{{z_{y,j} }}\). Note that it can’t be 0 otherwise \({\text{Q}}_{\text{ij}}\) itself becomes 0 (and the GA will never progress) which is impossible.

Model-I as defined in Definition 5 explicitly contains \(p_{i} \left( y \right)\) which forces to incorporate definitions of genetic operators (mutation, crossover and selection) which I can do but I want to be as general as possible so a Markov chain model independent of any such assumptions of genetic operators will be very useful. Such a model is defined next.

Let \(\xi \left( t \right)\), \(\xi_{M} \left( t \right),\) \(\xi_{C} \left( t \right)\), \(\xi_{S} \left( t \right)\) and \(\xi \left( {t + 1} \right)\) are random variables on \(\varOmega^{n}\) defined as follows:

1. (i)

   \(\xi \left( t \right)\) is the population state at time t.

2. (ii)

   \(\xi_{M} \left( t \right)\) is the population state after the application of mutation operator.

3. (iii)

   \(\xi_{C} \left( t \right)\) is the population state after the application of crossover operator.

4. (iv)

   \(\xi_{S} \left( t \right)\) is the population state after the application of selection operator.

5. (v)

   \(\xi \left( {t + 1} \right) = \xi_{S} \left( t \right)\).

In the GA as described in Algorithm-I all the genetic operators (mutation, crossover and section) are invariant in time, therefore the transition probability functions are also invariant in time.

Definition 6

GA operators are defined as their respective transition probability function as follows:Let \(A \subseteq \varOmega^{n}\)

1. (i)

   Mutation transition probability function defines the mutation operator as follows:

   $$p_{M} \left( {\varvec{x},A} \right) = p_{M} \left( {\xi_{M} \left( t \right) \in A |\xi \left( t \right) = \varvec{x}} \right)$$
2. (ii)

   Crossover transition probability function defines the crossover operator as follows:

   $$p_{C} \left( {\varvec{x},A} \right) = p_{C} \left( {\xi_{C} \left( t \right) \in A |\xi_{M} \left( t \right) = \varvec{x}} \right)$$
3. (iii)

   Selection transition probability function defines the selection operator as follows:

   $$p_{S} \left( {\varvec{x},A} \right) = p_{S} \left( {\xi_{S} \left( t \right) \in A |\xi_{C} \left( t \right) = \varvec{x}} \right)$$

Definition 7

(Model-II) [47] The GA is defined mathematically as the Markov chain \(\left\{ {\xi \left( t \right);t \in Z^{ + } } \right\}\) with the state space \(\varOmega^{n}\) and the transition probability given by Kolmogorov–Chapman equation as:

Note that (25) and (26) are basically the same. Models I and II are partial models of the GA (described in Algorithm-I) the complete model is defined in Definition 10.

Theorem 2

The Markov chains of model-I and model-II are ergodic.

Proof

This fact directly follows from the inclusion of the mutation operator in the chains as because of mutation, it is possible to go from every state to every state. For model-I, I assume the same definition of mutation operator as for model-II as defined in Definition 6.

Theorem 3

In an ergodic Markov chain the expected transition time from any state to any other is finite irrespective of the states.

Proof

This is Theorem 5 in [48].

Next, two properties of great importance are defined:

Definition 8

1. (i)

   Property 1 (□1): Maintain the best solution found so far over time.

2. (ii)

   Property 2 (□2): It must be possible to go from any state of the Markov chain to any other state.

□1 says that the best solution found so far must be preserved so that it will be not lost. □1 holds in GA by forcing the requirement on GA operators that they will not lose the best solution. □2 is basically the accessibility property saying that all the states must be accessible starting from any state. This property makes the set \(S_{OPT}^{n}\) accessible. In model-II (and therefore in model-I) □2 holds because of Theorem 2 and if □1 is forced to hold then the Markov chain of model-II becomes absorbing as the states containing the global optimal solutions \(S_{OPT}^{n}\) becomes absorbing. The following result says that all the states can reach the optimal state(s) and in an expected finite time.

Theorem 4

In Model-II satisfying □1 and □2 it is possible to reach the global optimal set \(S_{OPT}^{n}\) and the expected transition time from any initial state to any state in \(S_{OPT}^{n}\) is finite.

Proof

By Theorem 2 the Model-II is basically ergodic and by Theorem 3 in model-II the expected time to go from one state to another is finite irrespective of the states. □1 makes the states in \(S_{OPT}^{n}\) absorbing which means if the Markov chain enters any of the states in \(S_{OPT}^{n}\) then it will not leave \(S_{OPT}^{n}\). □1 does not change other states so it is still possible to go from any state to any state in \(S_{OPT}^{n}\) in a finite time which is the basic property of model-II by Theorems 2 and 3. In fact now the model-II with □1 and □2 becomes an absorbing Markov chain. Therefore, \(S_{OPT}^{n}\) is reachable in an expected finite time from any initial state.

Definition 9

(Model-III): Model-III is defined as Model-II obeying properties □1 and □2.

Definition 10

(GA): The GA as described in Algorithm-I is mathematically defined by model-III.

Now, the definition of convergence is needed so that it can be defined for the GA. Then,

Definition 11

Let \(A \subseteq \varOmega^{n}\) is any measurable set and \(\upmu_{1}\) and \(\upmu_{2}\) are probability measures defined on \(\varOmega^{n}\) (which is a measurable space) then the total variation distance between \(\upmu_{1}\) and \(\upmu_{2}\) is defined as:

Definition 12

Let π is a probability distribution on \(\varOmega^{n}\), then it is invariant for the Markov chain \(\left\{ {\xi \left( t \right);t = 1,2, \ldots } \right\}\) having transition function \(p\left( {\varvec{x},A} \right)\) if and only if

for all measurable sets \(A \subseteq \varOmega^{n}\).

Definition 13

Let \(\left\{ {\xi \left( t \right);t \in Z^{ + } } \right\}\) is a Markov chain with the transition function \(p\left( {\varvec{x},A} \right)\) and \(\upmu_{t}\) is the probability distribution of \(\xi \left( t \right)\). If there is an invariant probability distribution π for the Markov chain such that \(\mathop {\lim }\limits_{{{\text{t}} \to \infty }} \left| {\left| {\upmu_{\text{t}} -\uppi} \right|} \right| = 0,\) then \(\upmu_{t}\) is said to converge to π and if there exists a set \(A \subseteq \varOmega^{n}\) such that \(\uppi\left( {\text{A}} \right) = 1\) then \(\upmu_{t}\) is said to converge to the set A globally.

Theorem 5

Assume \(\varOmega^{n}\) is a bounded and compact measurable space. The Markov chain \(\left\{ {\xi \left( t \right);t \in Z^{ + } } \right\}\) given by Model-III converges and there exists some positive \(\delta > 0\) and some time \(t_{0} > 0\) and an invariant probability distribution π for the Markov chain such that,

and starting from any initial distribution \(\xi \left( 1 \right) = \varvec{x}\) it holds for all time \(t > 0\)

where \(\mu_{t}\) is the probability distribution of \(\xi \left( t \right)\).

Proof

This is Theorem 5 in [47].

Theorem 6

SNHSVM converges globally.

Proof

This directly follows from the definition of the SNHSVM and Theorem 5. Specifically, SNHSVM is a combination of the GA and NHSVM. But, NHSVM is presented to GA as (19) in which the objective function is (18) which is an example of the problem (16) or (24) for which Theorem 5 says that the GA will converge globally. This completes the proof.

Appendix 2

Here, I demonstrate how the accuracy is calculated for each classification model that is compared in the paper. The reader must recall the scenario of the experiments i.e. \(\underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle\cdot}$}}{{\dot{\text{S}} }}3\). In \(\underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle\cdot}$}}{{\dot{\text{S}} }}3\) the important point with respect to experiments is that each dataset \({\text{D}}\) is divided into two disjoint sets \({\text{D}}_{1}\) and \({\text{D}}_{2}\), and training is performed on \({\text{D}}_{1}\) and testing is performed on \({\text{D}}_{2}\). For statistical fairness this division is done 10 times differently for each dataset and the average accuracy is recorded. That is, a dataset is divided into different parts exactly like the tenfold cross validation (for training and testing purposes), but the difference here is that for each fold the respective model is run and the accuracy is recorded. Then, the accuracy is averaged (summed up for each fold and divided by 10) to get the values as recorded in Table 3 (in linear case) and 4 (in non-linear case).

This is not a tenfold cross validation in which a model with same parameters setting is run on all tenfold, rather in the experimental setting of this paper a model is run on each fold, i.e. optimizing parameters for each fold. To avoid confusion I will call a fold a section. The example of the calculation of SNHSVM accuracy on Heart-c dataset in the linear case below will clear the above picture.

All the accuracies for all the models are rounded off to two decimal places in a standard way so the average accuracy of SNHSVM on heart-c dataset in the linear case is \(85.39\% \pm 07.64\)

Reprints and permissions