Adaptive neural tree exploiting expert nodes to classify high-dimensional data

Abstract

Classification of high dimensional data suffers from curse of dimensionality and over-fitting. Neural tree is a powerful method which combines a local feature selection and recursive partitioning to solve these problems, but it leads to high depth trees in classifying high dimensional data. On the other hand, if less depth trees are used, the classification accuracy decreases or over-fitting increases. This paper introduces a novel Neural Tree exploiting Expert Nodes (NTEN) to classify high-dimensional data. It is based on a decision tree structure, whose internal nodes are expert nodes performing multi-dimensional splitting. Any expert node has three decision-making abilities. Firstly, they can select the most eligible neural network with respect to the data complexity. Secondly, they evaluate the over-fitting. Thirdly, they can cluster the features to jointly minimize redundancy and overlapping. To this aim, metaheuristic optimization algorithms including GA, NSGA-II, PSO and ACO are applied. Based on these concepts, any expert node splits a class when the over-fitting is low, and clusters the features when the over-fitting is high. Some theoretical results on NTEN are derived, and experiments on 35 standard data show that NTEN reaches good classification results, reduces tree depth without over-fitting and degrading accuracy.

Introduction

High-dimensional data classification causes some significant challenges in basic classifiers, including curse of dimensionality and over-fitting, which makes them impractical and insufficient to large sets of real problems (Shi, Liu, Qi, & Wang, 2018). One of the imposing solutions to these challenges is the dimension reduction (Shi et al., 2018). In addition, data partitioning could help to achieve simpler classifiers in each partition (Castro, Georgiopoulos, Demara, & Gonzalez, 2005). Neural trees by combining a local feature selection and recursive partitioning, could be helpful in this case. Moreover, neural trees by borrowing the fast training phase and the good generalization ability of the decision trees and the strong classification ability of neural networks, lead to powerful classifiers, even in high dimensional data (Federer & Zylberberg, 2018). There exist different neural trees with good achievements in different problems like Perceptron tree (Utgoff, 1989), neural tree with linear discriminant (Rani, Kumar, Micheloni, & Foresti, 2013), decision tree with bounded error (Saettler, Laber, & Pereira, 2017), balanced neural tree (Micheloni, Rani, Kumar, & Foresti, 2012), neural tree with multi-dimensional split (Maji, 2008), generalized neural tree (Foresti & Micheloni, 2002), omnivariate neural tree (Yildiz & Alpaydin, 2001) and neural trees with knowledge transferring (Abpeykar & Ghatee, 2018). In addition, some neural trees achieved significant results in dealing with high dimensional data including CART (Castro et al., 2005), SAINT (Federer & Zylberberg, 2018), adaptive high-order neural tree (Foresti & Dolso, 2004), decision forest of RBF networks (Abpeykar, Ghatee, & Zare, 2019) and neural trees with P2P and SC knowledge transferring (Abpeykar & Ghatee, 2019). By the way there are some dilemmas, which existing neural trees do not consider all of them jointly:

• Accurate classification of high dimensional feature space leads to more depth trees, thus achieving less depth neural trees require more complex computations at each node (Foresti and Micheloni, 2002, Micheloni et al., 2012, Rani et al., 2013).

• Multi-dimensional split is considered in some neural tree models (Foresti and Dolso, 2004, Maji, 2008, Micheloni et al., 2012, Ojha et al., 2017), but no one analyzes the data complexity for splitting, which leads to more accurate classification in neural tree’s node.

• The data partition of each neural tree node is considered as a sub-problem (Utgoff, 1989). Each sub-problem needs an eligible neural network, which leads to accurate and less depth trees (Foresti and Dolso, 2004, Micheloni et al., 2012). There are some studies in hybrid neural trees with different MLPs, but there is not any neural tree with expert nodes. By applying some if–then rules, these nodes can be extended to select an eligible neural network with respect to data complexity.

• Because of huge amount of redundancy in high-dimensional feature spaces, the neural tree faces with over-fitting. Feature clustering by using expert nodes can be used to reduce this problem.

NTEN is characterized by three main novelties, which consider the mentioned dilemmas. These novelties are: (1) multi-dimensional split of feature space, (2) selection of the most appropriate neural network for each local training set and (3) feature clustering in expert nodes in case of over-fitting by applying metaheuristic optimization algorithms. In the third novelty different algorithms are applied for feature clustering, like Genetic Algorithm (GA) (Silva Filho, Souza, & Prudêncio, 2016), Non-dominated Sorted Genetic Algorithm (NSGA-II) (Deb, Pratap, Agarwal, & Meyarivan, 2002), Particle Swarm Optimization (PSO) (Silva Filho et al., 2016) and Ant Colony Optimization (ACO) (Silva Filho et al., 2016). In traditional decision trees, feature selection at each node is done on the basis of entropy (Altınçay, 2007), Gini Index (Hady, Schwenker, & Palm, 2010) and miss classification errors (Chen & Hung, 2009). Data split based on these methods, leads to deep trees on data with high-dimensional feature spaces and needs huge computation. In these cases, a multi-dimensional split could be helpful (Foresti and Dolso, 2004, Foresti and Micheloni, 2002, Rani et al., 2013). NTEN applies multi-dimensional split in its expert nodes and chooses features with the least volume of overlap region in such a way it leads to less depth trees without degradation in the classification accuracy. By selecting features with low volume of overlap region between classes, each neural network trains a feature space with low complexity, therefore it allows computing good boundaries between classes and splitting the samples more confidently. In addition, it leads to child nodes with less overlap volume too. The assignment of the features to the most appropriate neural network based on the data complexity of the local training set (LTS) enhances the classification performance. Finally, when NTEN faces with over-fitting, the expert node clusters the features as a solution for over-fitting (Brownlee, 2016, Cano, 2013, Castro et al., 2005, Cestnik, 1987, Chen and Hung, 2009, Coello and Lechuga, 2002, Cong et al., 2017, Deb et al., 2002, Demšar, 2006, Detrano et al., 1989, Diaconis and Efron, 1983, Evett and Ernest, 1987, Federer and Zylberberg, 2018, Fisher, 1936, Fonollosa et al., 2016, Fonollosa et al., 2014, Fontenla-Romero et al., 2010, Foresti and Dolso, 2004, Foresti and Micheloni, 2002, Foresti and Pieroni, 1998, Frey and Slate, 1991, Gorman and Sejnowski, 1988, Guvenir et al., 1997, Güvenir et al., 1998, Guyon and Elisseeff, 2003, Guyon et al., 2008, Hady et al., 2010, Hall et al., 2009, Ho and Basu, 2002, Ho et al., 2006, Hong and Yang, 1991, Horton and Nakai, 1996, Leondes, 2001, Lipowski and Lipowska, 2012, Ma et al., 2009, Maji, 2008, Marler and Arora, 2004, Mesejo et al., 2016, Michalski et al., 1986, Micheloni et al., 2012, Morán-Fernández et al., 2017, Noordewier et al., 1991, Ojha et al., 2017, Peng et al., 2005, Penrose, 1946, Rani et al., 2015, Rani et al., 2013, Robert, 2014, Saettler et al., 2017). Since redundancy could cause over-fitting in high-dimensional data (Cong et al., 2017), it is also minimized in the clusters.

This paper is organized as follows. Section 2 describes the related works. In Section 3, the NTEN model is presented. Section 4 introduces expert nodes of the NTEN. Theoretical aspects are mentioned in Section 5. Experimental results on high dimensional data are presented in Section 6. Section 7 concludes the paper.