Restructuring forward step of MARS algorithm using a new knot selection procedure based on a mapping approach

Abstract

In high dimensional data modeling, Multivariate Adaptive Regression Splines (MARS) is a popular nonparametric regression technique used to define the nonlinear relationship between a response variable and the predictors with the help of splines. MARS uses piecewise linear functions for local fit and apply an adaptive procedure to select the number and location of breaking points (called knots). The function estimation is basically generated via a two-stepwise procedure: forward selection and backward elimination. In the first step, a large number of local fits is obtained by selecting large number of knots via a lack-of-fit criteria; and in the latter one, the least contributing local fits or knots are removed. In conventional adaptive spline procedure, knots are selected from a set of all distinct data points that makes the forward selection procedure computationally expensive and leads to high local variance. To avoid this drawback, it is possible to restrict the knot points to a subset of data points. In this context, a new method is proposed for knot selection which bases on a mapping approach like self organizing maps. By this method, less but more representative data points are become eligible to be used as knots for function estimation in forward step of MARS. The proposed method is applied to many simulated and real datasets, and the results show that it proposes a time efficient forward step for the knot selection and model estimation without degrading the model accuracy and prediction performance.

Appendices

Appendix A

1.1 Mathematical formulations for data generation

• $P_{1}. f(x) = \sum \limits _{i=1}^7 {\left[ {\left( {\ln \left( {x_i -2} \right) } \right) ^{2}+\left( {\ln \left( {10-x_i } \right) } \right) ^{2}} \right] -\left( {\prod \limits _{i=1}^7 {x_i } } \right) ^{2}} 2.1\le x_i \le 9.9$

• $P_{2}. f(x) = \sum \limits _{j=1}^{10} {\exp ({x_j})\left( {c_j +x_j -\ln \left( {\sum \limits _{k=1}^{10} {\exp \left( {x_k}\right) }} \right) } \right) }$,

• $\hbox {cj} = -6.089, -17. 164, -34.054, -5.914, -24.721, -14.986, -24.100, -10.708, -26.662, -22.179$

• $P_{3}. f(x) = x_1^2 +x_2^2 +x_1 x_2 -14x_1 -16x_2 +\left( {x_3 -10} \right) ^{2}-4\left( {x_4 -5} \right) ^{2}+\left( {x_5 -3} \right) ^{2} +2\left( {x_6 -1} \right) ^{2} +5x_7^2 +7\left( {x_8 -11} \right) ^{2}+2\left( {x_9 -10} \right) ^{2}+\left( {x_{10} -7} \right) ^{2}+45$

• $P_{4}. f(x) = \sin \left( {\pi x_1 /12} \right) \cos \left( {\pi x_2 /16} \right)$

• $P_{5}. f(x) = \left( {x_1 -1} \right) ^{2}+\left( {x_1 -x_2 } \right) ^{2}+\left( {x_2 -x_3 } \right) ^{4}$

• $P_{6}. f(x) = 5.3578547x_2^2 +0.8356891x_1 x_3 +37.293239x_1 -40792.141$

• $P_{8}. f(x) = 3(1-x)^{2}\exp \left( {-x^{2}-(y+1)^{2}} \right) -10\left( {x/5-x^{3}-y^{5}} \right) \exp \left( {-x^{2}-y^{2}} \right) -1/{3\exp \left( {-(x+1)^{2}-y^{2}} \right) +2x}$

Appendix B

1.1 Grid plots of mathematical functions

Appendix C

1.1 Root mean squared error (RMSE)

RMSE indicates the grossly inaccurate estimates. The smaller RMSE, the better it is.

$$\begin{aligned} RMSE=\sqrt{\frac{1}{n}\sum {\left( {y_i -\hat{{y}}_i } \right) ^{2}}}. \end{aligned}$$

where, $n$ is the number of observation, $y_i$ is an $i$th observed response value and $\hat{{y}}_i$ is the $i$th fitted response.

1.2 Adjusted-multiple coefficient of determination (Adj-R$^{2})$:

The value of coefficient of determination ($\hbox {R}^{2})$ indicates how much variation in response is explained by the model. $\hbox {Adj-R}^{2}$ is penalized form of $\hbox {R}^{2}$ with respect to the number of predictors in model. The higher the $\hbox {Adj-R}^{2}$, the better the model fits the data.

$$\begin{aligned} Adj-R^{2}=1-\left( \frac{\sum {\left( {y_i -\hat{{y}}_i} \right) ^{2}}}{\sum {\left( {y_i -\bar{{y}}_i } \right) ^{2}}} \right) \left( {\frac{n-1}{n-p-1}} \right) , \end{aligned}$$

where $\left( {n-p-1} \right) \ne 0;\, p$ is the number of terms in the model, $y_i$ is an $i$th observed response value; $\bar{{y}}_i$ is the mean response and $\hat{{y}}_i$ is the $i$th fitted response.

1.3 Stability

$$\begin{aligned} \min \left\{ {\frac{MR_{TR} }{MR_{TE} },\frac{MR_{TE} }{MR_{TR} }} \right\} , \end{aligned}$$

where $MR_{TR}$ and $MR_{TE}$ represents the performance measures (RMSE or $\hbox {Adj-R}^{2})$ for the test and training data sets, respectively. The model whose stability measure is close to one represents a stable model.

1.4 Generalized cross validation (GCV)

$$\begin{aligned} GCV(M)=\frac{1}{n}\frac{\sum _{i=1}^n {(y_i -\hat{{f}}_M (\mathbf{x}_i ))^{2}} }{(1-{P(M)}/n)^{2}}, \end{aligned}$$

where, $y_i$ is the $i$th observed response value; $\hat{{f}}_M (\mathbf{x}_i )$ is the fitted response value obtained for the $i$th observed predictor vector $\mathbf{x}_i =(x_{i,1},\ldots ,x_{i,p} )^{T}\;(i=1,\ldots ,n), \quad n$ is the number of data points; $M$ represents the maximum number of BFs in the model.

The value $P(M)$ is the effective number of parameters which is a penalty measure for complexity. Here, $P(M)=r+dN,$ where $r$ is the number of linearly independent BFs in the model, and $N$ is the number of knots selected in the forward process. Note that if the model is additive then $d$ is taken to be two; if the model is an interaction model then $d= 3$.

Appendix D

1.1 Red Wine quality

The dataset is related to red type of the Portuguese “Vinho Verde” wine [8]. The inputs include objective tests (e.g. PH values) and the output is based on sensory data (median of at least 3 evaluations made by wine experts). Input variables (based on physicochemical tests) are fixed acidity, volatile acidity, citric acid, residual sugar, chlorides, free sulfur dioxide, total sulfur dioxide, density, pH, sulphates, and alcohol. There is no missing attribute value in data.

1.2 Concrete compressive strength data set

Concrete is the most important material in civil engineering. The concrete compressive strength is a highly nonlinear function of age and ingredients. These ingredients include cement, blast furnace slag, fly ash, water, super plasticizer, coarse aggregate, and fine aggregate [50]. There is no missing attribute value in data.

1.3 Communities and crime

The data combines socio-economic data from the 1990 US Census, law enforcement data from the 1990 US LEMAS survey, and crime data from the 1995 FBI UCR. The variables included in the dataset involve the community, such as the percent of the population considered urban, and the median family income, and involving law enforcement, such as per capita number of police officers, and percent of officers assigned to drug units. The per capita violent crimes variable was calculated using population and the sum of crime variables considered violent crimes in the United States: murder, rape, robbery, and assault. It should be noted here that the Communities and Crime dataset is reduced in both size and scale to be appropriate for our analysis.

1.4 Auto-Mpg data

The data concerns city-cycle fuel consumption in miles per gallon to be predicted in terms of 3 multi-valued discrete and 5 continuous attributes including cylinders, displacement, horsepower, weight, acceleration, model year and origin [40]. Data has 6 missing values.

1.5 Parkinsons telemonitoring

This dataset is composed of a range of biomedical voice measurements from 42 people with early-stage Parkinson’s disease recruited to a six-month trial of a telemonitoring device for remote symptom progression monitoring. The recordings were automatically captured in the patient’s homes. Columns in the table contain subject number, subject age, subject gender, time interval from baseline recruitment date, motor UPDRS, total UPDRS, and 16 biomedical voice measures. Each row corresponds to one of 5,875 voice recording from these individuals. The main aim of the data is to predict the motor and total UPDRS scores (‘motor_UPDRS’ and ‘total_UPDRS’) from the 16 voice measures [45]. It should be noted here that the Parkinsons dataset is reduced in both size and scale to be appropriate for our analysis and the “motor UPDRS” variable is used as dependent variable.

1.6 PM10

The data are a subsample of 500 observations from a data set that originates in a study where air pollution at a road is related to traffic volume and meteorological variables, collected by the Norwegian Public Roads Administration. The response variable consists of hourly values of the logarithm of the concentration of PM10 (particles), measured at Alnabru in Oslo, Norway, between October 2001 and August 2003 [2].

1.7 Energy load

The data includes hourly electricity load of different regions in Netherlands which is recorded per day between 2008 and 2010. The average electricity load per day is aimed to be predicted by considering the past average values of energy demand. In the analysis, 1730 record is used to predict the daily average energy load.