A study on random weights between input and hidden layers in extreme learning machine

Abstract

Extreme learning machine (ELM), as an emergent technique for training feed-forward neural networks, has shown good performances on various learning domains. This paper investigates the impact of random weights during the training of ELM. It focuses on the randomness of weights between input and hidden layers, and the dimension change from input layer to hidden layer. The direct motivation is to verify as to whether during the training of ELM, the randomly assigned weights exert some positive effects. Experimentally we show that for many classification and regression problems, the dimension increase caused by random weights in ELM has a performance better than the dimension increase caused by some kernel mappings. We assume that via the random transformation, output-samples are more concentrate than input-samples which will make the learning more efficient.

Appendix

1.1 A Kernel function and its determinant

1.1.1 A.1 Definition of Kernel function

Definition 1

(Kernel function) Suppose that $\mathbf{X}$ is a subset of $\mathbf{R}^n.$ The function $k(\mathbf{x}_i,\mathbf{x}_j)$ defined on $\mathbf{X}\times\mathbf{X}$ is called a kernel function if there exists a mapping ${\phi:\mathbf{X}\to\mathbb{H},\mathbf{x}\to\phi(\mathbf{x}), }$ such that $k(\mathbf{x}_i,\mathbf{x}_j)=\langle\phi(\mathbf{x}_i),\phi(\mathbf{x}_j)\rangle,$ where ${\mathbb{H}}$ denotes a Hilbert space, and $\langle,\rangle$ denotes the inner product of ${\mathbb{H}}.$

1.1.2 A.2 Mercer Theorem

Theorem 1

(Mercer Theorem) Suppose that χ is a compact set of $\mathbf{R}^n.$ For any symmetric function $k(\mathbf{x}_i,\mathbf{x}_j)$ defined on χ × χ, the necessary and sufficient condition for it being the inner product of a feature space is that: for any $\phi(\mathbf{x})\neq0$ and $\int \phi^2(\mathbf{x})d\mathbf{x}< \infty,$ there has $\int\int k(\mathbf{x},\mathbf{x}_i)\phi(\mathbf{x})\phi(\mathbf{x}_i)d\mathbf{x} d\mathbf{x}_i>0.$

1.1.3 A.3 Definition of Gram matrix (Kernel matrix)

Definition 2

Given a function ${k:\chi\times\chi\to \mathbb{K}}$ (where χ is a compact set of $\mathbf{R}^n,$ ${\mathbb{K}}$ is the mapped set) and patterns $\mathbf{x}_1,\ldots,\mathbf{x}_N\in\chi,$ the N × N matrix $\mathbf{K}$ with elements $\mathbf{K}_{i,j}:=k(\mathbf{x}_i,\mathbf{x}_j)$ is called the Gram matrix (or Kernel matrix) of k with respect to $\mathbf{x}_1,\ldots,\mathbf{x}_N.$

1.1.4 A.4 Property of Kernel function

Theorem 2

Suppose that χ is a compact set of $\mathbf{R}^n.$ For the continuous and symmetric function $k(\mathbf{x}_i,\mathbf{x}_j)$ defined on χ × χ, the necessary and sufficient condition for it being a kernel function is that: the Gram Matrix of k with respect to any $\mathbf{x}_1,\ldots,\mathbf{x}_N\in\chi$ is positive semi-definite.