Kernel-based mixture models for classification

Abstract

A generative model for classification based on kernels and mixtures of univariate Gamma distributions is introduced. It models the point distances to cluster centroids in the transformed Hilbert space associated with the inner product induced by the kernel. The distances are readily computed using the kernel trick. Nested within this kernel-based Gamma mixture model (KMM) are two special cases corresponding to the kernel-based mixture of exponentials and the kernel-based mixture of spherical Gaussians. The Akaike information criterion is used to select an appropriate parsimonious type-of-mixture model for the data at hand. A powerful classification rule based on the knowledge of all point distances to every class centroid is developed based on this model. The flexibility in the choice of the kernel and the probabilistic nature of a mixture distribution makes KMM appealing for modeling and inference. A comparison with other popular classification methods shows that this model is very efficient when handling high dimensional data.

Appendix

The data sets used in Sect. 5 were:

• The well-known Iris data set (Anderson 1935). It consists of 150 four-dimensional instances associated with each of the three Iris types Setosa, Versicolor and Virginica.

• The Yeast data set (Nakai and Kanehisa 1991, 1992). It contains 1,459 proteins described by five variables: mcg, gvh, alm, mit and vac. They correspond to different prediction scores. The eight different classes are possible protein localizations: cytosolic or cytoskeletal, nuclear, mitochondrial, membrane protein with no N-terminal signal, membrane protein with uncleaved signal, membrane protein with cleaved signal, extracellular and vacuolar. This is a modification of the original data set, which has dimension eight and ten different classes.

• The Bupa Liver Disorder data set (Asuncion and Newman 2007). It consists of 345 male individuals described by six variables. The first five variables are blood test results thought to be sensitive to liver disorders. The last variable measures the alcoholic beverage intake per day. The two classes indicate whether or not the patient presents a liver disorder.

• The Car Evaluation data set (Bohanec and Rajkovic 1988). In this data set, 1,728 cars are described by six variables: buying price, maintenance price, number of doors, person capacity, luggage boot size, and safety of the car. The cars are classed in four groups: unacceptable, acceptable, good, and very good.

• The Olive Oil data set (Forina and Armanino 1982). It contains 572 samples of olive oil described by their content in eight fatty acids. Nine different types of oil origins are represented, namely South-Apulia, North-Apulia, Calabria, Sicily, Inland-Sardinia, Coast-Sardinia, Umbria, East-Liguria and West-Liguria.

• The Pima Indians Diabetes data set (Smith et al. 1988). In this data set, 768 persons are described by $8$ variables (Number of times pregnant, Plasma glucose concentration, blood pressure, Triceps skin fold thickness, 2-h serum insulin, Body mass index, Diabetes pedigree function and Age). The two classes indicate whether a person is tested positive for diabetes.

• The Glass data set (Asuncion and Newman 2007). In this data set, 214 glasses are described by nine variables (refractive index, sodium, magnesium, aluminium, silicon, potassium, calcium, barium and iron). There are seven classes corresponding all to a type of glass.

• The two Wine Quality data sets (Cortez et al. 2009). The two data sets are related to red and white variants of the Portuguese “Vinho Verde” wine. The 1,599 red and 4,898 white wines are described by eleven physico-chemical variables (fixed acidity, volatile acidity, citric acid, residual sugar, chlorides, free sulfur dioxide, total sulfur dioxide, density, pH, sulphates, alcohol). The different classes correspond to a quality score between 0 and 10 resulting in six classes for the red wine and seven for the white wine.

• The Yeast cell cycle data set (Schliep et al. 2005) describes the expression of $386$ genes along $17$ time points. This data set is in fact an excerpt of a larger data set (Spellman et al. 1998). The genes have each a peak expression at one among five different phases producing thus five different subgroups.

• The image segmentation data set SEG (Asuncion and Newman 2007). In this data set, 2,310 regions of $3\times 3$ pixels are taken from seven outdoor images. These regions are described by $19$ variables and are classified among seven different classes: brickface, sky, foliage, cement, window, path and grass.

• The waveform data set WAV (Breiman et al. 1984). In this data set, 5,000 instances of waves are obtained by combining $2$ of $3$ base waves, leading to three classes, and by adding noise. The number of variables is equal to $21$.

• The SPECT Heart data set (Kurgan et al. 2001). In this data set, $267$ patients are described by $22$ binary feature variables synthesizing information on collected images for each patient. There are two classes corresponding to either normal or abnormal patients.

• The Wisconsin Data Breast Cancer (Wdbc) (Mangasarian et al. 1995). The features are computed from digitized images of a fine needle aspirate (FNA) of a breast mass taken from $569$ patients with or without cancer. The features describe characteristics of the cell nuclei present in the image. Ten real-valued features are computed for each cell nucleus. The mean, standard error, and “worst” or largest (mean of the three largest values) of these ten features were computed for each image.

• The Satellite data set (Asuncion and Newman 2007). The database consists of the multi-spectral values of pixels in $3\times 3$-neighborhoods in a satellite image. There are four different wave lengths so that the total dimension is $36$. There are 4,435 and 2,000 pixels in the training and the testing set, respectively. The six different classes correspond to six different area types: red soil, cotton crop, grey soil, damp grey soil, soil with vegetation stubble and very damp grey soil.

• The colon tumor data set (Alon et al. 1999) contains $62$ samples coming from biopsies from tumors and normal tissues. The samples are described by 2,000 genes whose expression is observed through oligonucleotide arrays.

• The ALL/AML data set (Golub et al. 1999) contains $72$ samples coming either from acute lymphoblastic leukemia (ALL) or acute myelogenous leukemia (AML). The samples are described by the expression of 7,130 genes monitored by DNA microarrays.

• The lung cancer data set (Gordon et al. 2002) contains $181$ samples from malignant pleural mesothelioma (MPM) and adenocarcinoma (ADCA) from the lung. Each sample is described by the expression level of 12,533 genes.

• The all seven cancer data set (Yeoh et al. 2002) contains seven subgroups corresponding to six gene rearrangements, namely of T-ALL, hyperdiploid with $>$50 chromosomes, BCR-ABL, E2A-PBX1, TEL-AML1 and MLL. The last subgroup was identified in the same study as being distinct from the first known six. In total, there are $215$ samples described by the expression of 12,559 genes.

• The MLL data set (Armstrong et al. 2002) contains $72$ samples described by the expression of 12,583 genes. The samples are divided into three classes: ALL (acute lymphoblastic leukemia), MLL (MLL translocation leukemia) and AML (acute myelogenous leukemia).

• The ovarian data set (Petricoin et al. 2002) contains $253$ proteomic spectra of $91$ normal (control) patients and $162$ ovarian cancers. All of them are described by 15,154 molecular mass per charge intensities.

• The sm data set contains a protein alignment description of $102$ so-called Sm protein (Wicker et al. 2001). They are divided among eight subgroups found by the Secator method (Wicker et al. 2001). Although this clustering has been found by an automatic method, it has been validated biologically (Wicker et al. 2001).

• The nucrecept data set has been kindly provided by Jean-Marie Wurtz and Jérome Fagart in the same aforementioned study (Wicker et al. 2001) and were divided into $16$ biologically meaningful subgroups.