Enhanced ensemble-based classifier with boosting for pattern recognition

doi:10.1016/j.amc.2017.04.019

Applied Mathematics and Computation

Volume 310, 1 October 2017, Pages 1-14

https://doi.org/10.1016/j.amc.2017.04.019 Get rights and content

Highlights

•
Optimization of training sets – irrelevant items elimination.
•
Ensembles of neural-networks-based classifiers – a sloppy adaptation.
•
Methods of the classifiers diversity enhancing – doubling, shuffling and input filters.

Abstract

The aim of the article is a proposal of a classifier based on neural networks that will be applicable in machine digitization of incomplete and inaccurate data or data containing noise for the purpose of their classification (pattern recognition). The article is focused on the possibility of increasing the efficiency of the algorithms via their appropriate combination, and particularly increasing their reliability and reducing their time demands. Time demands do not mean runtime, nor its development, but time demands of applying the algorithm to a particular problem domain. In other words, the amount of professional labour that is needed for such an implementation. The article aims at methods from the field of pattern recognition, which primarily means various types of neural networks. The proposed approaches are verified experimentally.

Introduction

A classifier may be regarded as a computer based agent, which can perform a classification task. Classifiers can be divided into two categories [3]: rule-based classifiers and soft computing based classifiers.

Rule-based classifiers are generally constructed by the designer, where the designer defines rules for the interpretation of detected inputs.

This is in contrast to soft-computing based classifiers, where the designer only creates a basic framework for the interpretation of data. The training algorithms within such systems are responsible for the generation of rules for the correct interpretation of data.

In practice, there are often used soft-computing classifiers that use rule-based methods for preprocessing inputs before their own classification. Such classifiers are a combination of both approaches and their activities can be divided into two steps.

Selection of key features. Input data is preprocessed by algorithms, which extract the key features from the input objects. For feature extraction, there is no general rule. Their choice is related to a given application and it depends on the type of data.

Own classification. Features extracted from objects are presented to the classifier for classification.

A typical classification scheme is shown in Fig. 1, where two extreme situations may occur.

Ideal preprocessing gives us absolute control over the classifier logic. The output of this preprocessing is the number of the class, which the input object belongs to. The task is thus solved during the first step and the use of a classifier is not necessary.

Ideal classifier gives us a better possibility of adaptation to a new problem area. In this case, a classifier, which is sufficiently “intelligent” is used in order that all input objects are correctly ranked without any preprocessing, i.e., it is able to independently deduce all key features of the input objects.

The article aims at proposing and developing such a classifier, which is able to suppress weak points of the selected algorithms.

•
Linear neural classifiers provide poor diversity. The algorithm of the linear neural classifiers is almost deterministic. This property makes it virtually impossible to utilize the Hebbian network as a weak classifier in the AdaBoost. Therefore, the outcome of the article would be to develop some diversity-enhancing method, which would work with the linear classifiers.
•
Adaptation process of the backpropagation neural network is very slow. As the AdaBoost is designed to utilize a high number of weak classifiers and profit from their diversity, therefore the outcome of the article would be to exploit the backpropagation's capabilities in some less time consuming way.

This paper contains a summary of adjustments for linear and multi-layer neural network that we have proposed. Our approach is based on the idea that it is more efficient to create a number of imperfectly adapted networks, which are smaller in their topology than one perfectly adapted sophisticated network. It also includes experimental studies that have verified impacts of these adjustments. The proposed optimization and adjustments concerned both the process of adaptation and preparation of patterns.

Section snippets

Proposal of enhanced classifier

Fig. 2 shows the proposed improvements, wherein each of them works with all neural network based classifiers.

•
Optimization of training sets – irrelevant items elimination [5].
•
Ensembles of neural-networks-based classifiers – sloppy adaptation [12].
•
Methods of the classifiers diversity enhancing [11].

In addition, for experimental purposes the original software Neurotask [6] was designed and created to identify different types of patterns. It is a framework to support plug-ins for different types

Optimization of training sets – irrelevant items elimination

We noticed an unexpected behavior of classifier during our experiments with adaptation aimed at pattern recognition. It inspired us to study the influence of learning patterns “shape” on ability of neural network to adapt properly. The aim of the classifier was to test the ability of Hebbian networks to learn the fundamental features of patterns.

When analyzing the behavior, we used two sets of simpler patterns, sets R1 (Fig. 3(a)) and R2 (Fig. 3(b)). Hebb network was not able to learn set R1.

Ensembles of neural-networks-based classifiers – a sloppy adaptation

A sloppy neural network adaptation means a weak adaptation, because neural networks are only partially adapted. We have used Hebb network and backpropagation networks with the hyperbolic tangent activation function in configuration of 5 and 20 hidden neurons in our experimental study. We tested 1000 instances of each network. Figs. 6–8 show training errors as well as generalization errors during the first 50 iterations. The first iteration is marked TRN MAX, TRN AVG, and TRN MIN, which

Methods of the classifiers diversity enhancing

In the following experimental study, we focused on methods of increasing the diversity of classifiers. The experiment was performed on the MNIST database of handwritten digits [8]. We used neural networks with the following adaptation rules: Adaline, delta rule, Hebb rule, perceptron, and backpropagation. Each neural network represents a classifier with n inputs. Each proposed ensemble contains a set of m neural networks. Each of such a neural-networks-based classifier is able to recognize one

Boosting of neural networks over MNIST data

In our experimental study, we used two different types of neural networks. Hebb network and backpropagation network. All the neural networks used the winner-takes-all strategy for output neurons when worked in an active mode. We used a slightly modified Hebb rule with the identity activation function, i.e. the input value to the neuron is considered as its output value. We used backpropagation networks with the hyperbolic tangent activation function in configuration of 8 and 50 hidden neurons

Conclusion

We have proposed possible ways of improving the existing algorithms for classification, which was pushed towards a greater simplicity and universality.

We have proposed, developed, and mathematically and experimentally proved the method of input patterns optimization based on the elimination of irrelevant input vector items. This method demonstrably improves the performance of the neural networks. This method also reduces the input vectors length, which means that it also reduces the space and

Acknowledgments

The research described here has been financially supported by the University of Ostrava grant SGS07/PrF/2017.

References (13)

O. Fink et al.
Quantifying the reliability of fault classifiers
Inf. Sci.
(2014)
R. Ñanculef et al.
Training regression ensembles by sequential target correction and resampling
Inf. Sci.
(2012)
L.V. Fausett
Fundamentals of Neural Networks
(1994)
Y. Freund et al.
Experiments with a new boosting algorithm
T. Iwakura et al.
An adaboost using a weak-learner generating several weak hypotheses for large training data of natural language processing
IEEJ Trans. Electron. Inf. Syst.
(2010)
V. Kocian et al.
Optimization of training sets for Hebbian-learning based classifiers

There are more references available in the full text version of this article.

Cited by (7)

Towards asymmetric uncertainty modeling in designing General Type-2 Fuzzy classifiers for medical diagnosis
2021, Expert Systems with Applications
Citation Excerpt :
The general flowchart is illustrated in Fig. 5, but more details can be found in (Kennedy & Eberhart, 1995; Sengupta et al., 2018). This sampling methodology is widely used in different approaches for improving the classification accuracy, for example in different Bagging architectures (Lin et al., 2019; Moral-García et al., 2020; Roshan & Asadi, 2020), or in a variety of Boosting architectures (Kuncheva et al., 2002; Rodríguez & Maudes, 2008; Volna & Kotyrba, 2017). This sampling consists on generating M subsets of a dataset of N elements (Fig. 6), and is usually used for generating M weak classifiers and after that performing aggregation of the results (Bagging architecture).
One of the most studied application areas of intelligent systems is the classification area, and this is because classification covers a wide range of real-world problems. Some examples are fault-diagnosis, image segmentation, medical diagnosis, among others. In most cases, the intelligent systems designed for the solution of this kind of problems are based on supervised learning, which is based on learning how to classify with previous datasets for finding relations between the inputs and outputs. The main focus of the present paper is the supervised generation of general type-2 fuzzy classifiers with a new strategy for modeling data uncertainty. The proposed methodology includes a mix of concepts, such as the use of embedded type-1 membership functions, statistical concepts such as the quartiles, and nature inspired optimization methods. The classifiers generated with the proposed methodology are compared with respect to other general type-2 fuzzy classifiers based on symmetric uncertainty to evaluate their performance, in this way obtaining interesting results for medical diagnosis with benchmark data sets.
HBoost: A heterogeneous ensemble classifier based on the Boosting method and entropy measurement
2020, Expert Systems with Applications
Citation Excerpt :
Afterward, by applying an evolutionary algorithm called Estimation of Bayesian Network Algorithm –EBNA– have found the best combination of the base classifiers. An enhanced ensemble-based classifier with Boosting method is presented in (Volna & Kotyrba, 2017). The authors focused on some tasks to construct an enhanced classifier.
In recent years, ensemble classifiers have attracted a lot of attention in the field of machine learning. The main challenges with these classifiers are 1) to select the base classifiers and 2) to combine the outputs. The key point for an ensemble to be successful is the diversity and accuracy of the base classifiers. This paper proposes a heterogeneous Boosting-based ensemble classifier (HBoost) which is inspired by Boosting algorithm and aims at increasing the diversity by recruiting distinct learning algorithms. In this approach, for each learning algorithm, several classifiers are generated by using the Boosting method and a matrix of heterogeneous classifiers is formed. Since all generated classifiers may not be appropriate for generating the final output, the subset of those that make the most diverse and accurate classifiers is selected and the rest of the classifiers are eliminated. Finally, the classifiers in the subset are combined based on the weight assigned to each classifier. HBoost is a self-configured algorithm i.e., unlike existing methods, it does not require a human expert to specify the type of base classifiers manually. It can also automatically determine the optimum number of base classifiers which should be combined. For evaluating the HBoost, two evaluation metrics have been used: Accuracy and Geometric Mean which are the most popular metrics to measure the performance of a classifier. We compare the proposed approach in three different scenarios 1) all the base classifiers are in the ensemble, 2) four traditional ensemble methods including Bagging, Boosting, Stacking, and StackingC, 3) two state-of-the-art approaches in the literature including Random Forest and Classifier Subset Selection (CSS). HBoost is performed on 20 datasets from the UCI repository and experimental results reveal how adequate the HBoost is in comparison to other approaches.
Improved discrete artificial fish swarm algorithm combined with margin distance minimization for ensemble pruning
2019, Computers and Industrial Engineering
Citation Excerpt :
Over the last decade, ensemble learning has been a very popular research topic. It has been applied in many fields including pattern recognition (Volna & Kotyrba, 2017), machine learning (Zhai, Zang, & Zhou, 2018), and data mining (Murray, Agard, & Barajas, 2017). There are two main steps with respect to ensemble pruning: the first step is that a collection of classifiers is generated, and the other one is that these classifiers are aggregated using a strategy to solve a given classification or regression task (Onan, Korukoğlu, & Bulut, 2017).
Ensemble pruning aims to achieve a good result in classification using a smaller size of classifiers by finding the optimal sub-ensemble. Diversity and accuracy of classifiers are widely recognized to be two key factors for a successful ensemble. There is a tradeoff between the diversity and the accuracy of classifiers, which makes the ensemble perform the best. Existing ensemble pruning approaches always find the tradeoff using diversity measures or heuristic algorithms separately. Those pruning approaches based on diversity measures cannot exactly find the tradeoff; Those algorithms based on heuristic algorithms are not also to exhaustively search for it extracted from an initial pool of classifiers with a medium-scale or large-scale size. To address the issue, Improved Discrete Artificial Fish swarm algorithm combined with Margin distance minimization for Ensemble Pruning (IDAFMEP) is proposed using a combination of diversity measure and heuristic algorithm. First, the classifiers in a constructed initial pool are pre-pruned using Margin Distance Minimization (MDM), which can downsize the classifiers who perform badly, and markedly alleviate the computational complexity of ensemble pruning. Second, the final ensemble is efficiently achieved from the retaining classifiers after pre-pruning based on MDM using the proposed Improved discrete Artificial Fish Swarm Algorithm (IDAFSA). Experimental results on 29 datasets from the UCI Machine Learning Repository demonstrate that IDAFMEP can achieve better results than the original ensemble and other state-of-the-art pruning approaches, and that its validity and effectiveness. It provides a new research idea for ensemble pruning.
An advance extended binomial GLMBoost ensemble method with synthetic minority over-sampling technique for handling imbalanced datasets
2023, International Journal of Electrical and Computer Engineering
Enhancing extended belief rule-based systems for classification problems using decomposition strategy and overlap function
2022, International Journal of Machine Learning and Cybernetics
Background and Theory
2021, SpringerBriefs in Applied Sciences and Technology

View all citing articles on Scopus

View full text

Enhanced ensemble-based classifier with boosting for pattern recognition

Highlights

Abstract

Introduction

Section snippets

Proposal of enhanced classifier

Optimization of training sets – irrelevant items elimination

Ensembles of neural-networks-based classifiers – a sloppy adaptation

Methods of the classifiers diversity enhancing

Boosting of neural networks over MNIST data

Conclusion

Acknowledgments

Inf. Sci.

Inf. Sci.

Fundamentals of Neural Networks

Experiments with a new boosting algorithm

An adaboost using a weak-learner generating several weak hypotheses for large training data of natural language processing

IEEJ Trans. Electron. Inf. Syst.

Optimization of training sets for Hebbian-learning based classifiers