Problems in pattern classification in high dimensional spaces: behavior of a class of combined neuro-fuzzy classifiers

doi:10.1016/S0165-0114(01)00180-4

Fuzzy Sets and Systems

Volume 128, Issue 1, 16 May 2002, Pages 15-33

https://doi.org/10.1016/S0165-0114(01)00180-4 Get rights and content

Abstract

The designer of a pattern classification system is often faced with the following situation: finite sets of samples, from the various classes, are available along with a large set of measurements, or features, to be computed from the patterns. In pattern recognition literature, several investigations have demonstrated the relationship between dimensionality, sample size and recognition accuracy. The ultimate goal of this paper is to study the generalization error of statistical (1_NN, 4_NN), fuzzy (FCM, FKCN), neural (MLPNN, RBFNN) and neuro-fuzzy classifiers in high dimensional spaces. Computational complexity of classification algorithms in high dimensional spaces is also discussed. Our experimental results show the robustness of fuzzy, neural, and neuro-fuzzy classifiers to the curse of dimensionality.

References (79)

R. Bellman et al.
Abstraction and pattern classification
J. Math. Anal. Appl.
(1966)
B. Duerr et al.
A combination of statistical and syntactical pattern recognition applied to classification of unconstrained handwritten numerals
Pattern Recognition
(1980)
A.K. Jain et al.
Dimensionality and sample size considerations in pattern recognition practice
A.K. Jain et al.
On the optimum number of features in the classification of multivariate Gaussian data
Pattern Recognition
(1978)
M.S. Kamel et al.
New algorithms for solving the fuzzy clustering problem
Pattern Recognition
(1994)
F. Kimura et al.
Handwritten numerical recognition based on multiple algorithms
Pattern Recognition
(1991)
Y. Lu et al.
Fuzzy integration of classification results
Pattern Recognition Society
(1997)
E. Mandler
Schuermann, Combining the classification result of independent classifiers based on the Dempster shafer theory of evidences
A.F.R. Rahman et al.
An evaluation of multiexpert configurations for the recognition of handwritten numerals
Pattern Recognition
(1998)
S. Raudys
Evolution and generalization of a single neuron, I. SLP as seven statistical classifiers
Neural Networks
(1998)

S. Raudys

Evolution and Generalization of a single neuron. II. Complexity of statistical classifiers and sample size consideration

Neural Networks

(1998)

S.J. Raudys et al.

Small sample size problems in designing artificial neural networks

G. Rogova

Combining the results of several neural network classifiers

Neural Networks

(1994)

E.H. Ruspini

A new approach to clustering

Inform. and Control

(1969)

C.Y. Suen et al.

Building a new generation of handwriting recognition systems

Pattern Recognition Lett.

(1993)

E.C.K. Tsao et al.

Fuzzy Kohonen clustering networks

Pattern Recognition

(1994)

K. Tumer et al.

Analysis of decision boundaries in linearly combined neural classifiers

Pattern Recognition

(1996)

F. Wyman et al.

A comparison of asymptotic error rate expansions for the sample linear discriminant function

Pattern Recognition

(1990)

K. Abend et al.

Comments on the mean recognition accuracy of statistical pattern recognition

IEEE Trans. Inform. Theory

(1969)

D.C. Allais

The problem of too many measurements in pattern recognition

IEEE Internat. Con. Rec.

(1966)

J.A. Barnett, Computational methods for a mathematical theory of evidence, in: Proceedings of the Seventh IJCAI,...

R. Bellman

Adaptive Control Processes: A Guided Tour

(1961)

J.C. Bezdek, Fuzzy mathematics in pattern classification, Ph.D. Dissertation, Center for Applied Mathematics, Cornell...

J.C. Bezdek

Pattern Recognition with Fuzzy Objective Function Algorithms

(1981)

C.M. Bishop

Neural Networks for Pattern Recognition

(1995)

B. Chandrasekaran

Independence of measurements and the mean recognition accuracy

IEEE Trans. Inform. Theory

(1971)

B. Chandrasekaran et al.

Quantization Complexity and Independent Measurements

IEEE Trans. Comput.

(1974)

S.B. Cho et al.

Multiple network fusion using fuzzy logic

IEEE Trans. Neural Networks

(1995)

P.A. Devijver et al.

Pattern Recognition: A Statistical Approach

(1982)

R.O. Duda et al.

Pattern Classification and Scene Analysis

(1973)

R.P.W. Duin, Superlearning capabilities of neural networks, Proceedings of the Eighth Scandinavian Conference on Image...

J.C. Dunn

A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters

J. Cybernet.

(1973)

J. Elmalek, A.M. Alimi, R. Tourki, Evolution of the generalization error of fuzzy classifiers with the size of the...

J. Elmalek, A.M. Alimi, R. Tourki, Effect of the feature vector size on the generalization error: the case of MLPNN and...

K. Fukunaga

Introduction to Statistical Pattern Recognition

(1990)

K. Fukunaga et al.

Bias of nearest neighbor estimates

IEEE Trans. Pattern Anal. Mach. Intell.

(1987)

P.D. Gader, D. Hepp, B. Forester, T. Peurach, Pipelined systems for recognition of handwritten digits in USPS Zip...

J. Ghosh, S. Beck, C.C. Chu, Evidence combination techniques for robust classification of short-duration oceanic...

Y. Hamamoto et al.

On the behavior of artificial neural network classifiers in high-dimensional spaces

IEEE Trans. Pattern Anal. Mach. Intell.

(1996)

Cited by (11)

An experimental study of one- and two-level classifier fusion for different sample sizes
2011, Pattern Recognition Letters
Citation Excerpt :
If defining the best strategy as the one which results in the fastest convergence of the classification error zero, the best strategy was shown to depend on the distribution of the base classifier prediction errors, that is, average is the best for normal distributions, the maximum rule performs best for uniform distributions whereas the median combiner outperforms the other two for Cauchy distributions. In pattern recognition literature, it is widely believed that the choice of the best classifier depends on the data, the classifier complexity and the training set size (Malek et al., 2002; Raudys and Duin, 1998; Skurichina and Duin, 1999). Thus, it is essential to take these factors into consideration when designing and selecting proper classifiers to handle a practical problem.
Due to the wide variety of fusion techniques available for combining multiple classifiers into a more accurate classifier, a number of good studies have been devoted to determining in what situations some fusion methods should be preferred over other ones. However, the sample size behavior of the various fusion methods has hitherto received little attention in the literature of multiple classifier systems. The main contribution of this paper is thus to investigate the effect of training sample size on their relative performance and to gain more insight into the conditions for the superiority of some combination rules.
A large experiment is conducted to study the performance of some fixed and trainable combination rules for executing one- and two-level classifier fusion for different training sample sizes. The experimental results yield the following conclusions: when implementing one-level fusion to combine homogeneous or heterogeneous base classifiers, fixed rules outperform trainable ones in nearly all cases, with only one exception of merging heterogeneous classifiers for large sample size. Moreover, the best classification for any considered sample size is generally achieved by a second level of combination (namely, utilizing one fusion rule to further combine a set of ensemble classifiers with each of them constructed by fusing base classifiers). Under these circumstances, it seems that adopting different types of fusion rules (fixed or trainable) as the combiners for two levels of fusion is appropriate.
Hierarchical genetic algorithm with new evaluation function and bi-coded representation for the selection of features considering their confidence rate
2011, Applied Soft Computing Journal
Citation Excerpt :
Without the use of accurate and highly consistent systems [4,5,16], classical procedures are usually subject to human errors and misinterpretations especially when those problems cannot be correctly or completely defined. In the area of pattern recognition, many machine learning based techniques have been developed to construct classifiers or predictive models from real world data [8,19,27]. Pattern classification is a procedure that teaches a classifier with a predefined set of features and samples from different classes.
In this paper, we propose a new feature selection method based on a hierarchical genetic algorithm (GA) with a new evaluation function and a bi-coded representation. The hierarchical GA with homogeneous and heterogeneous population is used to minimize the computational load and to accelerate convergence speed. The fitness function is designed to find the solution that both maximizes the recognition rate and minimizes the feature set size. Each solution candidate is represented by two chromosomes which lengths are identical to the number of available features. The first binary chromosome represents the presence of features in the solution candidate; the second represents the confidence rates of features, which are used to assign different weights to features during the classification procedure and to achieve more accurate classifier. The proposed method is tested using five databases and is shown to outperform many commonly used feature selection algorithms.
Implementation of artificial intelligence for diagnosis of the faults of motor-fan system
2019, Journal of Advanced Research in Dynamical and Control Systems
Spread control for huge data fuzzy learning
2017, Advances in Intelligent Systems and Computing
Performance evaluation of FMIG clustering using fuzzy validity indexes
2015, Soft Computing
Big data clustering validity
2014, 6th International Conference on Soft Computing and Pattern Recognition, SoCPaR 2014

View all citing articles on Scopus

View full text

Problems in pattern classification in high dimensional spaces: behavior of a class of combined neuro-fuzzy classifiers

Abstract

J. Math. Anal. Appl.

Pattern Recognition

Pattern Recognition

Pattern Recognition

Pattern Recognition

Pattern Recognition Society

Pattern Recognition

Neural Networks

Neural Networks

Neural Networks

Inform. and Control

Pattern Recognition Lett.

Pattern Recognition

Pattern Recognition

Pattern Recognition

Comments on the mean recognition accuracy of statistical pattern recognition

IEEE Trans. Inform. Theory

The problem of too many measurements in pattern recognition

IEEE Internat. Con. Rec.

Adaptive Control Processes: A Guided Tour

Pattern Recognition with Fuzzy Objective Function Algorithms

Neural Networks for Pattern Recognition

Independence of measurements and the mean recognition accuracy

IEEE Trans. Inform. Theory

Quantization Complexity and Independent Measurements

IEEE Trans. Comput.

Multiple network fusion using fuzzy logic

IEEE Trans. Neural Networks

Pattern Recognition: A Statistical Approach

Pattern Classification and Scene Analysis

A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters

J. Cybernet.

Introduction to Statistical Pattern Recognition

Bias of nearest neighbor estimates

IEEE Trans. Pattern Anal. Mach. Intell.

On the behavior of artificial neural network classifiers in high-dimensional spaces

IEEE Trans. Pattern Anal. Mach. Intell.