A study of statistical techniques and performance measures for genetics-based machine learning: accuracy and interpretability

Abstract

The experimental analysis on the performance of a proposed method is a crucial and necessary task to carry out in a research. This paper is focused on the statistical analysis of the results in the field of genetics-based machine Learning. It presents a study involving a set of techniques which can be used for doing a rigorous comparison among algorithms, in terms of obtaining successful classification models. Two accuracy measures for multi-class problems have been employed: classification rate and Cohen’s kappa. Furthermore, two interpretability measures have been employed: size of the rule set and number of antecedents. We have studied whether the samples of results obtained by genetics-based classifiers, using the performance measures cited above, check the necessary conditions for being analysed by means of parametrical tests. The results obtained state that the fulfillment of these conditions are problem-dependent and indefinite, which supports the use of non-parametric statistics in the experimental analysis. In addition, non-parametric tests can be satisfactorily employed for comparing generic classifiers over various data-sets considering any performance measure. According to these facts, we propose the use of the most powerful non-parametric statistical tests to carry out multiple comparisons. However, the statistical analysis conducted on interpretability must be carefully considered.

A genetic algorithms in classification

Here we will give a wider description of all the methods employed in our work, regarding their main components, structure and operation of each one of them.

For more details about the methods explained here, please refer to the corresponding references.

Pitts-GIRLA algorithm. The Pittsburgh genetic interval rule learning algorithm (Pitts-GIRLA) (Corcoran and Sen 1994) is a GBML method which makes use of the Pittsburgh approach in order to perform a classification task. Two real variables indicate the minimum and maximum value of the attribute, where a “don’t care” condition may occur if the maximum value is lower than the minimum value.

This algorithm employs three different operators: modified simple (one point) crossover, creep mutation and simple random mutation.

XCS algorithm. XCS (Wilson 1995) is a LCS that evolves online a set of rules that describe the feature space accurately. In the following we will present in detail the different components of this algorithm:

• 1. Interaction with the environment: In keeping with the typical LCS model, the environment provides as input to the system a series of sensory situations σ(t) ∈ {0,1}^L, where L is the number of bits in each situation. In response the system executes actions α(t) ∈ {a ₁,…,a _n } upon the environment. Each action results in a scalar reward ρ(t).

• 2. A classifier in XCS: XCS keeps a population of classifiers which represent its knowledge about the problem. Each classifier is a condition-action-predic-tion rule having the following parts: the condition C ∈ {0,1,#}^L, the action A ∈ {a ₁,…,a _n } and the prediction p. Furthermore, each classifier keeps certain additional parameters such as the prediction error ε, the fitness F, the experience exp, the time stamp ts, the action set size as and the numerosity.

• 3. The different sets: There are four different sets that need to be considered in XCS: the population [P], the match set [M], the action set [A] and the previous action set [A ₋₁].

The result of this algorithm is that the knowledge is represented by a set of rules or classifiers with a certain fitness. When classifying unseen examples, each rule that matches the input votes according its prediction and fitness. The most voted class is chosen to be the output.

GASSIST algorithm. Genetic Algorithms based claSSIfier sySTem (GASSIST) (Bacardit and Garrell 2007) is a Pittsburgh style classifier system based on GABIL (De Jong et al. 1993) from where it has taken the semantically correct crossover operator. The main features of this classifier system are presented as follows:

• 1. General operators and policies

  • Matching strategy The matching process follows a “if ... then ... else if ... then ...” structure, usually called decision lists (Rivest 1987).

  • Mutation operators When an individual is selected for mutation a random gene is chosen inside its chromosome to be mutated.

• 2. Control of the individuals length: This control is achieved using two different operators:

  • Rule deletion: This operator deletes the rules of the individuals that do not match any training example.

  • Selection bias using the individual size: Tournament selection is used, where the criterion of the tournament is given by an operator called “hierarchical selection”, defined as follows:

    • If |accuracy _a –accuracy _b | < threshold then:

      • If length _a  < length _b then a is better than b.

      • If length _a  > length _b then b is better than a.

      • If length _a  = length _b then we will use the general case.

    • Otherwise, we use the general case: we select the individual with higher fitness.

• 3. Knowledge representations

  • Rule Representations for symbolic or discrete attributes: It uses the GABIL (De Jong et al. 1993) representation for this kind of attributes.

  • Rule Representations for real-valued attributes For GASSIST-ADI, the representation is based on the Adaptive Discretization Intervals rule representation (Bacardit and Garrell 2003; Bacardit 2004).

HIDER algorithm. HIerarchical DEcision Rules (HIDER) (Aguilar-Ruiz et al. 2000), produces a hierarchical set of rules, which may be viewed as a Decision List. In order to extract the rule-list a real-coded GA is employed in the search process. The elements of this procedure are described below.

• 1. Coding: Each rule is represented by an individual (chromosome), where two genes define the lower and upper bounds of the rule attribute.

• 2. Algorithm: The algorithm is a typical sequential covering GA. It chooses the best individual of the evolutionary process, transforming it into a rule which is used to eliminate data from the training file (Venturini 1993).

  Initially, the set of rules R is empty, but in each iteration a rule is included in R. In each iteration, the training file is reduced, eliminating those examples that have been covered by the description of the rule r, independently of its class.

The GA main operators are defined in the following:

• (a) Initialization: First, an example is randomly selected from the training file for each individual of the population. Afterwards, an interval to which the example belongs is obtained.

• (b) Crossover: The crossover works as follows: let [l ^j _i , u ^j _i ] and [l ^k _i , u ^k _i ] be the intervals of two parents, j and k, for the same attribute i. From these parents one child is generated by selecting values that satisfy the expression: l ∈ [min(l ^j _i , l ^k _i ), max(l ^j _i , l ^k _i )] and u ∈ [min(u ^j _i , u ^k _i ),max(u ^j _i , u ^k _i )].

• (c) Mutation: a small value is subtracted or added, depending on whether it is the lower or the upper boundary, respectively.

• (d) Fitness function: The fitness function f considers a two-objective optimization, trying to maximize the number of correctly classified examples and to minimize the number of errors.