A feed-forward network for input that is both categorical and quantitative

Abstract

The data on which a multi-layer perceptron (MLP) is to be trained to approximate a continuous function may have inputs that are categorical rather than numeric or quantitative such as color, gender, race, etc. A categorical variable causes a discontinuous relationship between an input variable and the output. A MLP, with connection matrices that multiply input values and sigmoid functions that further transform values, represents a continuous mapping in all input variables. A MLP therefore requires that all inputs correspond to numeric, continuously valued variables and represents a continuous function in all input variables. The way that this problem is usually dealt with is to replace the categorical values by numeric ones and treat them as if they were continuously valued. However, there is no meaningful correspondence between the continuous quantities generated this way and the original categorical values. Another approach is to encode the categorical portion of the input using 1-out-of-n encoding and include this code as input to the MLP.

The approach in this paper is to segregate categorical variables from the continuous independent variables completely. The MLP is trained with multiple outputs; a separate output unit for each of the allowed combination of values of the categorical independent variables. During training the categorical value or combination of categorical values determines which of the output units should have the target value on it, with the remaining outputs being ‘do not care’. Three data sets were used for comparison of methods. Results show that this approach is much more effective than the conventional approach of assigning continuous variables to the categorical features. In case of the data set where there were several categorical variables the method proposed here is also more effective than the 1-out-of-n input method.

Introduction

Often the data on which a multi-layer perceptron (MLP) is to be trained to approximate a continuous function has inputs that are categorical rather than numeric or quantitative. Examples of categorical variables are gender, race, region, type of industry, different types of species of fish, etc. A categorical variable causes a discontinuous relationship between an input variable and the output. A quantitative variable on the other hand is one that assumes numerical values corresponding to the points on a real line, e.g. the kilowatt-hours of electricity used per day or the number of defects in a product. An example of a data set containing a combination of quantitative and categorical data is the ‘Boston housing data’ data set (Harrison & Rubinfeld, 1978) which is used for training here. In this case there are two categorical independent variables: one is the Charles River dummy variable (=1 if tract bounds river; 0 otherwise) and the other is the index of accessibility to radial highways. Another example is the data used for predicting the age of abalone. In this case one of the independent variables is categorical with values of M for male, F for female and I for infantile.

Categorical variables may be broken up into two types: ordinal and non-ordinal. Ordinal variable values may be ordered. A non-ordinal categorical variable will be a label of a category without ordering property as is true in the case of the abalone data where no ordering can be placed on the values M, F and I. The Charles River dummy variable above also has this property.

A MLP (Werbos, 1974, Rumelhart et al., 1986, Wasserman, 1993) with connection matrices that multiply input values and sigmoid functions that further transform values, however, represents a continuous mapping in all input variables. A MLP requires that all inputs correspond to numeric, continuously valued variables and represents a continuous function in all input variables. One way of dealing with the problem of categorical variables, which seems intuitively unacceptable, is to replace the categorical values by numeric ones and treat them as if they were continuously valued. Thus if the set of values for a categorical variable was the set {red, blue, green} the values red=1, green=2 and blue=3 would have been used. However, there is no meaningful correspondence between the continuous quantities generated this way and the original categorical values. This type of encoding imposes an ordering on the values that does not exist. To ensure the generalization capability of a neural network, the data should be encoded in a form that allows for interpolation. Therefore categorical variables should be distinguished from the continuous independent variables. One way of dealing with this is to use 1-out-of-n coding on inputy, e.g. for the three colours we get, red=(100), green=(010) and blue=(001). This means one input unit for each categorical value. In this paper, however, the encoded values are not input to the MLP but are used to select 1-out-n output units. This is somewhat similar but not quite the same as the more drastic solution of having a separate network for each combination of categorical values.

Bishop, 1994, Bishop, 1995 introduces a new class of neural, network models obtained by combining a conventional neural network with a mixture density model. The complete system is called a Mixture Density Network. This network can in theory represent arbitrary conditional probability distributions in the same way that a conventional neural network can represent arbitrary functions. The neural network component of the MDN has three groups of output units for each of the Gaussian kernel functions. These units output a mean, variance and multiplier for each conditional probability density in the mixture of densities. The multiplier for including each density in the mixture is the probability that a particular density out of the mixture is applicable. Each combination of categorical values could have its own density.

Lee and Lee (2001) also address the problem of multi-value regression estimation with neural network architecture. They confine the multi-regression problems to those mapping vectors to a scalar with a single input vector mapping to several scalars. They propose a modular network approach such that each module handles only a single output rather than a set of outputs. A decision then has to be made as to which modules, since the number of outputs for a single input varies, produce the correct output values for the given input.

The remainder of the paper is organized as follows. It commences with a discussion of some neural network approaches. Next follows a description of a neural network construction and its amended training algorithm. This is followed by the results of simulations that demonstrate the validity of the approach suggested in this paper. Finally we have the summary including a comparison of the neural network approach to the statistical approach based on using indicator variables, a generalization of the approach in this paper and a description of future work.