Second Order Training of a Smoothed Piecewise Linear Network

Abstract

In this paper, we introduce a smoothed piecewise linear network (SPLN) and develop second order training algorithms for it. An embedded feature selection algorithm is developed which minimizes training error with respect to distance measure weights. Then a method is presented which adjusts center vector locations in the SPLN. We also present a gradient method for optimizing the SPLN output weights. Results with several data sets show that the distance measure optimization, center vector optimization, and output weight optimization, individually and together, reduce testing errors in the final network.

Appendix

1.1 Calculations for Distance Measure Optimization

Taking the gradient of the error in Eq. (12) with respect to the distance measure weight change element $e_{b}(v)$,

$$\begin{aligned} \frac{\partial E}{\partial e_{b}\left( v \right) } = - \frac{2}{N_{{\mathrm {v}}}}\sum _{p = 1}^{N_{{\mathrm {v}}}}{\sum _{i = 1}^{M}{\left[t_{p}\left( i \right) - y_{p}\left( i \right) \right]\frac{\partial y_{p}\left( i \right) }{\partial e_{b}\left( v \right) }}} \end{aligned}$$

(36)

where

$$\begin{aligned} \frac{\partial y_{p}\left( i \right) }{\partial e_{b}\left( v \right) } = \sum _{k = 1}^{K}{\frac{\partial \theta \left( k \right) }{\partial e_{b}\left( v \right) } y_{\text {pk}}(i)} \end{aligned}$$

Here, we represent $d_{p}\left( k \right)$ with $d_{k}$ to improve readability.

$$\begin{aligned}&\frac{\partial \theta \left( k \right) }{\partial e_{b}\left( v \right) } = \frac{D\frac{\partial d_{k}^{- a}}{\partial e_{b}\left( v \right) } - d_{k}^{- a}\sum _{m = 1}^{K}\frac{\partial d_{m}^{- a}}{\partial e_{b}\left( v \right) }}{D^{2}}\\&\frac{\partial d_{k}^{- a}}{\partial e_{b}\left( v \right) } = - \frac{a}{d_{k}^{a + 1}}\frac{\partial d_{k}}{\partial e_{b}\left( v \right) }\\&\frac{\partial d_{k}}{\partial e_{b}\left( v \right) } = \left( x_{p}\left( v \right) - m_{k}\left( v \right) \right) ^{2} \end{aligned}$$

The elements of the Gauss–Newton Hessian matrix ${\mathbf {H}}_{b}$ are calculated as

$$\begin{aligned} h_{b}\left( u,v \right) = \frac{\partial ^{2}E}{\partial e_{b}\left( u \right) \ \partial e_{b}(v)} = \frac{2}{N_{{\mathrm {v}}}}\sum _{p = 1}^{N_{{\mathrm {v}}}}{\sum _{i = 1}^{M}{\frac{\partial y_{p}\left( i \right) }{\partial e_{b}\ (u)} \frac{\partial y_{p}\left( i \right) }{\partial e_{b}\ (v)}}} \end{aligned}$$

(37)

1.2 Calculations for Center Vector Optimization

The gradient of the SPLN error from Eq. (12) with respect to the uth cluster’s center vector element ${\mathbf {m}}_{u}(v)$ is calculated as:

$$\begin{aligned} g_{m}\left( u,v \right) = \frac{\partial E}{\partial m_{u}\left( v \right) } = - \frac{2}{N_{{\mathrm {v}}}}\sum _{p = 1}^{N_{{\mathrm {v}}}}{\sum _{i = 1}^{M}{\left[t_{p}\left( i \right) - y_{p}\left( i \right) \right]\frac{\partial y_{p}\left( i \right) }{\partial m_{u}(v)}}} \end{aligned}$$

(38)

where

$$\begin{aligned}&\frac{\partial y_{p}\left( i \right) }{\partial m_{u}(v)} = \sum _{k = 1}^{K}{\frac{\partial \theta \left( k \right) }{\partial m_{u}\left( v \right) } \ y_{\text {pk}}(i)}\\&\frac{\partial \theta \left( k \right) }{\partial m_{u}\left( v \right) } = \frac{\delta \left( u - k \right) \ D\frac{\partial d_{u}^{- a}}{\partial m_{u}\left( v \right) } - d_{k}^{- a}\frac{\partial d_{u}^{- a}}{\partial m_{u}\left( v \right) }}{D^{2}}\\&\frac{\partial d_{u}^{- a}}{\partial m_{u}\left( v \right) } = \frac{2 \; a}{d_{u}^{a + 1}} \ b(v) \left( x_{p}\left( v \right) - m_{u}\left( v \right) \right) \end{aligned}$$

The elements of the Gauss–Newton Hessian matrix ${\mathbf {H}}_{m}$ are given as

$$\begin{aligned} h_m\left( u,v \right) = \frac{\partial ^{2}E}{\partial z_{m}\left( u \right) \ \partial z_{m}(v)} = \frac{2}{N_{{\mathrm {v}}}}\sum _{p = 1}^{N_{{\mathrm {v}}}}{\sum _{i = 1}^{M}{\frac{\partial y_{p}\left( i \right) }{\partial z_{m}(u)} \frac{\partial y_{p}\left( i \right) }{\partial z_{m}(v)}}} \end{aligned}$$

(39)

and the gradient of the error with respect to the learning factor elements

$$\begin{aligned} g_{zm}\left( u \right) = \frac{\partial E}{\partial z_{m}\left( u \right) } = - \frac{2}{N_{{\mathrm {v}}}}\sum _{p = 1}^{N_{{\mathrm {v}}}}{\sum _{i = 1}^{M}{\left[t_{p}\left( i \right) - y_{p}\left( i \right) \right]\frac{\partial y_{p}\left( i \right) }{\partial z_{m}\left( u \right) }}} \end{aligned}$$

(40)

where

$$\begin{aligned}&\frac{\partial y_{p}\left( i \right) }{\partial z_{m}\left( u \right) } = \sum _{k = 1}^{K}{\frac{\partial \theta \left( k \right) }{\partial z_{m}\left( u \right) } y_{\text {pk}}(i)}\\&\frac{\partial \theta \left( k \right) }{\partial z_{m}(u)} = \frac{\delta \left( u - k \right) \, D\frac{\partial d_{u}^{- a}}{\partial z_{m}(u)} - d_{k}^{- a}\frac{\partial d_{u}^{- a}}{\partial z_{m}(u)}}{D^{2}}\\&\frac{\partial d_{u}^{- 1}}{\partial z_{m}(u)} = \frac{2 \, a}{d_{u}^{a + 1}}\sum _{v = 1}^{N}{b(v) \left( x_{p}\left( v \right) - m_{u}\left( v \right) \right) g_{m}(u,v)} \end{aligned}$$

1.3 Description of Datasets

(1), (2) Red and White wine quality data sets The two datasets are related to red and white variants of the Portuguese “Vinho Verde” wine [12]. Due to privacy and logistic issues, only physicochemical (inputs) and sensory (the output) variables are available (e.g. there is no data about grape types, wine brand, wine selling price, etc.).

(3) Twod data set This training file is used in the task of inverting the surface scattering parameters from an inhomogeneous layer above a homogeneous half space, where both interfaces are randomly rough. The data file contains 2768 patterns. It has eight inputs and seven outputs [14, 15].

(4) Three spirals data This is a synthetic data set used in [8], which consists of three two-dimensional spirals, each labelled for a different class. It was converted to a regression problem by decoding the classes as binary outputs. It is available at [47].

(5) Oh7 data This data set is given in [41]. The training set contains VV and HH polarization at L-band $30^{\circ }$, $40^{\circ }$, C-band $10^{\circ }$, $30^{\circ }$, $40^{\circ }$, $50^{\circ }$, $60^{\circ }$, and X-band $30^{\circ }$, $40^{\circ }$, $50^{\circ }$ along with the corresponding unknowns rms surface height, surface correlation length, and volumetric soil moisture content in g / cubic cm. The file has 20 inputs, 3 outputs and 10,453 training patterns.

(6) Melting point data This data set comes from [26] where a robust and general model is developed for the prediction of melting points. It has a diverse set of 4401 examples of compounds with 202 descriptors that capture molecular physicochemical and other graph-based properties. It is included in the R package QSARdata [28].

(7) Concrete data This data set predicts compressive strength of high performance concrete from its components and age. It comprises of eight inputs, the first seven being the quantities of cement, slag, fly ash, water, superplasticizer, coarse aggregate, and fine aggregate in kg/m³. The eighth input is age in days. The output variable is the compressive strength in MPa [56].