Multilayer Perceptrons with Banach-Like Perceptrons Based on Semi-inner Products – About Approximation Completeness

Abstract

The paper reconsiders multilayer perceptron networks for the case where the Euclidean inner product is replaced by a semi-inner product. This would be of interest, if the dissimilarity measure between data is given by a general norm such that the Euclidean inner product is not longer consistent to that situation. We prove mathematically that the universal approximation completeness is guaranteed also for those networks where the used semi-inner products are related either to uniformly convex or to reflexive Banach-spaces. Most famous examples of uniformly convex Banach spaces are the spaces $L_{p}$ and $l_{p}$ for $1<p<\infty$. The result is valid for all discriminatory activation functions including the sigmoid and the ReLU activation.

A. Engelsberger—Supported by an ESF PhD grant.

Appendix

In this appendix we give some useful definitions regarding SIPs and Banach spaces, which are used in the text as well as some basic statements and remarks.

Definition 23

A Banach space $\mathcal {B}$ is denoted as strictly convex iff for $\textbf{x},\textbf{y}\ne 0$ with $\left\| \textbf{x}\right\| +\left\| \textbf{y}\right\| =\left\| \textbf{x}+\textbf{y}\right\|$ we can always conclude that $\textbf{x}=\lambda \textbf{y}$ for some $\lambda >0$.

Lemma 24

A Banach space $\mathcal {B}$ with SIP $\left[ \cdot ,\cdot \right]$ is strictly convex iff for $\textbf{x},\textbf{y}\ne 0$ with $\left[ \textbf{x},\textbf{y}\right] =\left\| \textbf{x}\right\| \cdot \left\| \textbf{y}\right\|$ we can always conclude that $\textbf{x}=\lambda \textbf{y}$ for some $\lambda >0$.

Proof

The proof can be found in [7].    $\square$

The following definition for the uniform convexity was introduced in [4]:

Definition 25

A Banach space $\mathcal {B}$ is denoted as uniformly convex iff for each $\varepsilon >0$ exists a $\delta \left( \varepsilon \right) >0$ such that if $\left\| \textbf{x}\right\| =\left\| \textbf{y}\right\| =1$ with $\left\| \textbf{x}-\textbf{y}\right\| >\varepsilon$ then $\frac{\left\| \left( \textbf{x}+\textbf{y}\right) \right\| }{2}<1-\delta \left( \varepsilon \right)$ is valid.

Definition 26

A Banach space $\mathcal {B}$ with SIP $\left[ \cdot ,\cdot \right]$ is denoted as continuous iff

$$ \Re \left\{ \left[ \textbf{x},\textbf{y}+\lambda \textbf{x}\right] \right\} \underset{\lambda \rightarrow 0}{\longrightarrow }\Re \left\{ \left[ \textbf{x},\textbf{y}\right] \right\} $$

is valid for $\lambda \in \mathbb {R}$. The space is uniformly continuous iff this limit is approached uniformly.

Definition 27

A Banach space $\mathcal {B}$ is denoted as reflexive iff the mapping $J:\mathcal {B}\rightarrow \mathcal {B}^{**}=\left( \mathcal {B}^{*}\right) ^{*}$ is surjective, where the star indicates the dual space.

Theorem 28

Let $\mathcal {B}$ be a Banach space. Then a necessary and sufficient condition for $\mathcal {B}$ to be reflexive is that for every $f\in \mathcal {B}^{*}$ exists an SIP $\left[ \cdot ,\cdot \right]$ and an element $\textbf{y}\in \mathcal {B}$ with $f\left( \textbf{x}\right) =\left[ \textbf{x},\textbf{y}\right]$ for all $\textbf{x}\in \mathcal {B}$. If $\mathcal {B}$ is strictly convex then $\textbf{y}$ is unique.

Proof

The proof can be found in [6, Theorem 2].    $\square$

Definition 29

A Banach space $\mathcal {B}$ is denoted as smooth iff for each $\textbf{x}\in \mathcal {B}$ with $\left\| \textbf{x}\right\| =1$ there exists a linear functional $f_{\textbf{x}}\in \mathcal {B}^{*}$ with $f_{\textbf{x}}\left( \textbf{x}\right) =\left\| f_{\textbf{x}}\right\|$. The existence of $f_{\textbf{x}}$ is guaranteed by the Hahn-Banach-Theorem.