Artificial Neural Networks with Random Weights for Incomplete Datasets

Abstract

In this paper, we propose a method to design Neural Networks with Random Weights in the presence of incomplete data. We present a method, under the general assumption that the data is missing-at-random, to estimate the weights of the output layer as a function of the uncertainty of the missing data estimates. The proposed method uses the Unscented Transform to approximate the expected values and the variances of the training examples after the hidden layer. We model the input data as a Gaussian Mixture Model with parameters estimated via a maximum likelihood approach. The validity of the proposed method is empirically assessed under a range of conditions on simulated and real problems. We conduct numerical experiments to compare the performance of the proposed method to the performance of popular, parametric and non-parametric, imputation methods. By the results observed in the experiments, we conclude that our proposed method consistently outperforms its counterparts.

Appendix: Unscented Transform (UT)

Given a D-dimensional random variable X, we are interested in estimating statistical moments of $\psi$, which results from an application of a non-linear function $h(\cdot )$ to X. These values could be obtained via standard sampling procedures or numerical integration methods. However, such procedures can be computationally intensive and depend on many factors such as proper initialization, stop criteria, etc. The Unscented Transform (UT) provides a scheme to estimate the moments of $\psi$ using a small set of deterministically chosen samples, referred to as sigma points (SPs), from the space of X.

There are different possible ways to choose the SPs. A common approach is to use a symmetric set of $S = 2D + 1$ SPs as described in Eqs. (32) to (34).

$$\begin{aligned} \gamma _1&= {\mathbb {E}}[X]&\quad&\end{aligned}$$

(32)

$$\begin{aligned} \gamma _s&= \gamma _1 + \left[ \sqrt{D \, \varSigma _{}}\right] _{s-1}&\quad&\forall \, 1 < s \le D + 1 \end{aligned}$$

(33)

$$\begin{aligned} \gamma _s&= \gamma _1 - \left[ \sqrt{D \, \varSigma _{}}\right] _{s - (D + 1)}&\quad&\forall \, D+1 < s \le 2D + 1 \end{aligned}$$

(34)

where $\left[ \sqrt{D \, \varSigma _{}}\right] _s$ denotes the s-th row of the matrix square root of $D \, \varSigma _{}$, which is the covariance matrix $\varSigma _{}$ of X.

Given the SPs and a set of weights $\{k_s\}^S_{s=1} \subset {\mathbb {R}}$, we can approximate the moments of $\psi$ using a simple set of rules. For instance ${\mathbb {E}}[\psi ]$ and $\mathrm {Cov}(\psi )$ can then be approximated using the following equations:

$$\begin{aligned}&\delta _s \leftarrow h \left( \gamma _s\right) \quad \forall \, 1 \le s \le S, \end{aligned}$$

(35)

$$\begin{aligned}&{\mathbb {E}}[\psi ] \approx \sum \limits _{s=1}^S k_s \delta _s \end{aligned}$$

(36)

$$\begin{aligned}&\mathrm {Cov}({\psi }) \approx \sum \limits _{s=1}^S k_s \left( \delta _s - {\mathbb {E}}[\psi ]\right) \left( \delta _s - {\mathbb {E}}[{\psi }]\right) ^T. \end{aligned}$$

(37)

Although there is no restriction on their sign, the weights $k_1, \ldots , k_S$ must respect the convexity constraint.

$$\begin{aligned} \sum \limits _{s=1}^S k_s = 1, \end{aligned}$$

(38)

to provide an unbiased estimate [22]. In this paper, we set $k_1 = k_2 = \cdots = k_S = 1/S$.