Bivariate Nonisotonic Statistical Regression by a Lookup Table Neural System

Abstract

Linear data regression is a fundamental mathematical tool in engineering and applied sciences. However, for complex nonlinear phenomena, the standard linear least-squares regression may prove ineffective, hence calling for more involved data modeling techniques. The current research work investigates in particular nonlinear statistical regression of bivariate data that do not exhibit a monotonic dependency. The current contribution proposes a neural-network-based data processing method, termed data monotonization, followed by neural isotonic statistical regression. Such data monotonization processing is performed by means of an adaptive neural network that learns its nonlinear transfer function from the training set. The artificial neural system that performs data monotonization is implemented through a lookup table (LUT), which entails few computationally inexpensive algebraic operations to adapt and to compute the output from the input data-set. A number of learning rules to adapt such LUT-based neural system are introduced and compared in order to elucidate their relative merits and drawbacks.

Appendix: Bivariate Isotonic Regression Method

The regression model to infer has a nonlinear structure described by relationship $y=f(x)$, where $x\in {\mathcal {X}}\subseteq {{{\mathcal {R}}}}$ denotes the input random variable, having probability density function $p_x(\cdot )$, and $y\in {\mathcal {Y}}\subseteq {{{\mathcal {R}}}}$ denotes the output random variable, having probability density function $p_y(\cdot )$.

With the assumption that the regression model be strictly monotonic, namely $f'(x)>0$ or $f'(x)<0$, $\forall x\in {\mathcal {X}}$, the input–output distributions and the regression model may be shown to stay in the relationship:

• Positive-slope regression model: $f(x)=P_y^{-1}(P_x(x))$,

• Negative-slope regression model: $f(x)=P_y^{-1}(1-P_x(x))$,

where the symbol $P_y^{-1}(\cdot )$ denotes the inverse of the cumulative distribution function $P_y(\cdot )$ pertaining to the y random variable and the symbol $P_x(\cdot )$ denotes the cumulative distribution function pertaining to the x random variable. In effect, the above transformations make sure that the distribution $p_x(\cdot )$ is transformed into the distribution $p_y(\cdot )$ according to the law of measure invariance of probability density functions [28].

The above regression method does not depend explicitly on the data, but on the probability density/cumulative functions obtained from each data-set separately.

To develop a fully numerical bivariate isotonic statistical regression method, the following ingredients are of use: a suitable numerical estimation method for cumulative density functions and a suitable format for function representation/handling (with particular emphasis on numerical function inversion). In order to put the above regression equations into effect, a representation of the quantities of interest based on lookup tables was chosen.

A real-valued lookup table with N entries is represented by a pair ${\mathrm {LUT}}=({x},{y})$, where ${x}\in {{{\mathcal {R}}}}^{N}$ and ${y}\in {{{\mathcal {R}}}}^{N}$. The entries $x_k$ of vector ${x}$ and the entries $y_k$ of vector ${y}$, with $k\in \{1,\ \ldots ,\ N\}$, are paired and provide a point-wise description of an arbitrarily shaped function. In order to handle the lookup tables for statistical regression purpose, the following operations are of use:

• Histogram computation: In order to numerically approximate the probability distribution of a data-set ${\mathcal {D}}$, a histogram operator is made use of. The histogram-computation operation is denoted by $({x},{y})=\mathrm {hist}({\mathcal {D}})$. The constructed lookup table is built up as follows: $x_k$ equals the value of the k th bin center, and $y_k$ equals the number of data points falling in the k th bin.

• Cumulative-sum computation: On the basis of a lookup table $({x},{y})$, a new lookup table $({x},{v})=\mathrm {csum}({x},{y})$ is constructed, where array ${v}$ contains the cumulative sum of the entries of array ${y}$, possibly normalized in order to approximate the numerical integration of the function represented by the lookup table $({x},{y})$. In its unnormalized version, the cumulative sum is described by $v_1=y_1$ and $v_k=v_{k-1}+y_k$ for $2\le k\le N$.

• Function/table interpolation: Interpolation may be invoked to make computations with lookup tables on other points in the domain. In the present context, it is necessary to preserve the monotonicity of an approximated function; therefore, linear interpolation only is made use of. Denote by ${\mathcal {D}}$ the x-coordinate point-set, where the function represented by a lookup table $({x},{y})$ needs to be interpolated. The interpolation operation may be denoted by ${\mathcal {I}}=\mathrm {interp}({x},{y},{\mathcal {D}})$, where the set ${\mathcal {I}}$ contains the interpolated y-values corresponding to the x-values in the set ${\mathcal {D}}$. Because of the hypotheses of monotonicity of the model underlying the data, a set-type representation is equivalent to an ordered-list representation.

• Function/table inversion: If a function is given a point-wise representation by the help of a lookup table $({x},{y})$, then its inverse function may be easily given a point-wise representation by the lookup table $({y},{x})$, namely function inversion is equivalent to swapping lookup table’s arguments. Apparently, therefore, function inversion in the context of lookup table representation is a computationally costless operation.

The bivariate isotonic regression method proposed in [15] consists of the following steps.

First, it is necessary to estimate the probability density functions of data within ${\mathcal {X}}$ and ${\mathcal {Y}}$ data-sets. They may be numerically estimated—up to scale factors—by:

$$\begin{aligned} ({x},{p}_x)=\mathrm {hist}({\mathcal {X}}),\ ({y},{p}_y)=\mathrm {hist}({\mathcal {Y}}). \end{aligned}$$

(4.1)

The numerical cumulative distribution functions of the ${\mathcal {X}}$ and ${\mathcal {Y}}$ data-sets are estimated by numerical integration of the numerical probability density functions, which is achieved by the help of the cumulative-sum operator applied to lookup tables $({x},{p}_x)$ and $({y},{p}_y)$:

$$\begin{aligned} ({x},{P}_x)=\mathrm {csum}({x},{p}_x),\ ({y},{P}_y)=\mathrm {csum}({y},{p}_y). \end{aligned}$$

(4.2)

If the statistical model has negative slope, the lookup table $({x},{P}_x)$ should be replaced by $({x},1-{P}_x)$ in what follows.

For regression purpose, the regression model needs to be evaluated on an ordered set of x-points denoted here as an array $\hat{{x}}$. The array $\hat{{x}}$ consists of R points equally spaced over the range of interest for the x-variable.

The last step consists in numerically evaluating the quantity $P_x(\cdot )$ over the points in $\hat{{x}}$ and then in evaluating the function $P_y^{-1}(\cdot )$ over the values of $P_x(\hat{{x}})$, namely:

$$\begin{aligned} \mathcal {P}_x=\mathrm {interp}({x},{P}_x,\hat{{x}}),\ \hat{{y}}=\mathrm {interp}({P}_y,{y},\mathcal {P}_x). \end{aligned}$$

(4.3)

In the second of equations (4.3), the inverse function $P_y^{-1}(\cdot )$ appears through the swapped lookup table of function $P_y(\cdot )$.

The pair $(\hat{{x}},\hat{{y}})$ provides a lookup table representation for the nonlinear bivariate isotonic regression model $f(\cdot )$.

Full details about the above-summarized statistical isotonic regression technique are available in [15].