Constructive lower bounds on model complexity of shallow perceptron networks

Abstract

Limitations of shallow (one-hidden-layer) perceptron networks are investigated with respect to computing multivariable functions on finite domains. Lower bounds are derived on growth of the number of network units or sizes of output weights in terms of variations of functions to be computed. A concrete construction is presented with a class of functions which cannot be computed by signum or Heaviside perceptron networks with considerably smaller numbers of units and smaller output weights than the sizes of the function’s domains. A subclass of these functions is described whose elements can be computed by two-hidden-layer perceptron networks with the number of units depending on logarithm of the size of the domain linearly.

Appendix

Proof Proof of Lemma 1

Choose an expression of g ∈ P _d (X)as g(z) = sign(a ⋅ z + b),where $z= (x,y) \in \mathbb {R}^{d_{1}} \times \mathbb {R}^{d_{2}}$,$a \in \mathbb {R}^{d} = \mathbb {R}^{d_{1}} \times \mathbb {R}^{d_{2}}$,and $b \in \mathbb {R}$.Let a _l and a _r denote the left and the right part, resp, of a, i.e., a _{l i} = a _i for $i=1,\dots , d_{1}$ and a _{r i} = a _{d 1} + i for $i=1,\dots ,d_{2}$.Then, sign(a ⋅ z + b) = sign(a _l ⋅ x + a _r ⋅ y + b).Let ρ and κ be permutations of the set $\{1, \dots ,n\}$such that $a_{l} \cdot x_{\rho (1)} \leq a_{l} \cdot x_{\rho (2)}\leq {\dots } \leq a_{l} \cdot x_{\rho (n)}$and $a_{r} \cdot y_{\kappa (1)} \leq a_{r} \cdot y_{\kappa (2)} \leq {\dots } \leq a_{r} \cdot y_{\kappa (n)}$.

Denote by M(g)^∗the matrix obtained from M(g)by permuting its rows and columns by ρ and κ,resp. It follows from the definition of the permutations ρ and κ that each row and each column of M(g)^∗starts with a (possibly empty) initial segment of − 1’sfollowed by a (possibly empty) segment of + 1’s. □

Proof Proof of Theorem 5

By Theorem 2,

$$ \| f_{M} \|_{P_{d}(X)} \geq \frac{\| f_{M}\|^{2}}{\sup_{g \in P_{d}(X)} |\langle f_{M}, g\rangle|}{}={}\frac{n^{2}}{\sup_{g \in P_{d}(X)} |\langle f_{M}, g\rangle}|. $$

(5)

The inner product of f _M with g is equal tothe sum of entries of the matrices M and M(g),i.e., $\langle f_{M}, g\rangle = {\sum }_{i,j}^{n} M_{i,j} M(g)_{i,j}$,and so it is invariant under permutations of rows and columns performed simultaneously on both matrices M and M(g).

Thus, without loss of generality, we can assume that each row and each column of M(g)starts with a (possibly empty) initialsegment of − 1’s followed by a (possiblyempty) segment of + 1’s. Otherwise, wereorder rows and columns in both matrices M(g)and M applying permutations from Lemma 1.

To estimate $\langle f_{M}, g\rangle = {\sum }_{i,j=1}^{n} M_{i,j} M(g)_{i,j},$we define apartition of the matrix M(g)into a family of submatrices such that each submatrix from the partition of M(g)has either all entriesequal to − 1or all entriesequal to + 1(see Fig. 1). Weconstruct the partition of M(g)as a union of sequence of families of submatrices (possibly some of them empty)

$$\mathcal{P}(g,k) = \{P(g,k,1), \dots, P(g,k,2^{k})\}, \quad k=1, \dots, \lceil \log_{2} n\rceil, $$

definedrecursively. To construct it, we also define an auxiliary sequence of families of submatrices

$$\mathcal{Q}(g,k)= \{Q(g,k,1), \dots, Q(g,k,2^{k})\},\! \quad k=1, \dots, \lceil \log_{2} n\rceil, $$

such thatfor each k,

$$\{P(g,k,1), \dots, P(g,k,2^{k}), Q(g,k,1), \dots, Q(g,k,2^{k})\} $$

is a partition of thewhole matrix M(g).

First, we define $\mathcal {P}(g,1)= \{P(g,1,1), P(g,1,2)\}$and $\mathcal {Q}(g,1)= \{Q(g,1,1), \ Q(g,1,2)\}$. Let r _1,1and c _1,1be such that thesubmatrix P(g, 1, 1)of M(g)formed by the entriesfrom the first r _1,1rowsand the first c _1,1columnsof M(g)has all entriesequal to − 1and thesubmatrix P(g, 1, 2)by theentries from the last r _1,2 = n − r _1,1rows and the last c _1,2 = n − c _1,1of M(g)has all entriesequal to + 1. Let Q(g, 1, 1)be the submatrixformed by the last r _1,2 = n − r _1,1rows and the first c _1,1columns of M(g)and Q(g, 1, 2)be the the submatrixformed by the first r _1,2rows and the last c _1,2 = n − c _1,1columns. So {P(g, 1, 1),P(g, 1, 2),Q(g, 1, 1),Q(g, 1, 2)}isa partition of M(g)into four submatrices (see Fig. 1).

Now, assume that the families $\mathcal {P}(g,k)$and $\mathcal {Q}(g,k)$are constructed.To define $\mathcal {P}(g,k+1)$and $\mathcal {Q}(g,k+1)$, we divideeach of 2^ksubmatrices Q(g,k,j),$j=1, \dots , 2^{k}$into foursubmatrices: P(g,k + 1, 2j − 1), P(g,k + 1, 2j), Q(g,k + 1, 2j − 1), and Q(g,k + 1, 2j)such that each of thesubmatrices P(g,k + 1, 2j − 1)has allentries equal to − 1and eachof the submatrices P(g,k + 1, 2j)hasall entries equal to + 1.

Iterating this construction at most $\lceil \log _{2} n \rceil$times, we obtain a partition of M(g)formedby the union of families of submatrices $\mathcal {P}(g,k)$.It follows from the construction that for each k, the sum of the numbers of rows$\{r_{k,t} \, | \, t=1, \dots , 2^{k}\}$and the sum of thenumbers of columns $\{ c_{k,t} \, | \,t=1, \dots , 2^{k}\}$of these submatrices satisfy

$$\sum\limits_{t=1}^{2^{k}} r_{k,t} =n \quad \text{and} \quad \sum\limits_{l=1}^{2^{k}} c_{k,t} =n. $$

Let $\mathcal {P}(k) = \{ P(k,1), \dots , P(k,2^{k})\}$bethe family of submatrices of M formed by the entries from the same rows and columns as corresponding submatrices from the family$\mathcal {P}(g,k) = \{ P(g,k,1), \dots , P(g,k,2^{k})\}$of submatricesof M(g).

To derive an upper bound on |〈f _M ,g〉|,we express it as

$$\begin{array}{@{}rcl@{}} \left|\langle f_{M}, g\rangle\right| &=& \left|{\sum}_{i,j}^{n} M_{i,j} {M(g)}_{i,j} \right|\\ &=&\left|\sum\limits_{k=1}^{\lceil \log_{2} n \rceil}\sum\limits_{t=1}^{2^{k}} {P(k,t)}_{i,j} \, {P(g,k,t)}_{i,j} \right|. \end{array} $$

(6)

As all the matrices P(k,t)are submatrices of the Hadamard matrix M, by the Lindsay Lemma 2 for each submatrix P(k,t),

$$\left | \sum\limits_{i=1}^{r_{k,t}} \sum\limits_{j=1}^{c_{k,t}} {P(k,t)}_{i,j} \right | \leq \sqrt{ n \,r_{k,t} \, c_{k,t} }. $$

All the matrices P(g,k,t)have all entrieseither equal to 1 or all entries equal to − 1.Thus,

$$\left |\sum\limits_{i=1}^{r_{k,t}} \sum\limits_{j=1}^{c_{k,t}} {P(k,t)}_{i,j} \, {P(g,k,t)}_{i,j} \right | \leq \sqrt{ n \,r_{k,t} \, c_{k,t}}. $$

As for all k, ${\sum }_{t=1}^{2^{k}} r_{k,t} =n$and ${\sum }_{t=1}^{2^{k}} c_{k,t} =n$, weobtain by the Cauchy-Schwartz inequality

$$\sum\limits_{t=1}^{2^{k}} \sqrt{r_{k,t} \, c_{k,t}} \leq n. $$

Thus, foreach k,

$$\sum\limits_{t=1}^{2^{k}} | {P(k,t)}_{i,j}\, {P(g,k,t)}_{i,j} | \leq \sum\limits_{t=1}^{2^{k}} \sqrt{n \, r_{k,t} \, c_{k,t}} \leq n \sqrt{n}.$$

Hence, by (6),

$$\left|\langle f_{M}, g\rangle \right | \leq \sum\limits_{k=1}^{\lceil \log_{2} n \rceil} \left| \sum\limits_{t=1}^{2^{k}} {P(k,t)}_{i,j}\, {P(g,k,t)}_{i,j} \right| \leq n \sqrt{n}\, \lceil \log_{2} n\rceil.$$

So by (5),

$$\| f_{M} \|_{P_{d}(X)} \geq \frac{n^{2}}{n \,\sqrt{n} \, \lceil \log_{2} n \rceil } \geq \frac{ \sqrt{n}}{\lceil \log_{2} n\rceil}$$

□

Proof Proof of Theorem 6

Any 2^k × 2^kSylvester-Hadamardmatrix S(k)is equivalent tothe matrix M(k)with rows andcolumns indexed by vectors u,v ∈{0, 1}^kand entries

$$M(k)_{u,v} = -1^{u \cdot v} $$

(see, e.g., [33]). Thus, without lossof generality, we can assume that S(k) _u,v = −1^u⋅v(otherwise, we permute rows and columns).

To represent the function $h_{k}: \{0,1\}^{k} \times \{0,1\}^{k} \to \{-1,1\}$by a two-hidden-layer network, we first define k Heaviside perceptrons from the first hidden layer. Choose any bias b ∈ (1, 2)and defineinput weights $c^{i} = (c^{i,l},c^{i,r}) \in \mathbb {R}^{k} \times \mathbb {R}^{k}$,$i=1, \dots , k$, as$c^{il}_{j}= 1$and$c^{ir}_{j} =1$when j = i, otherwise$c^{il}_{j}= 0$and$c^{ir}_{j} =0$. So for an inputvector x = (u,v) ∈{0, 1}^k ×{0, 1}^k, the output y _i (x)of the i-th perceptron in thefirst hidden layer satisfies y _i (x) = 𝜗(c ⁱ ⋅ x − b) = 1if and only if both u _i = 1and v _i = 1;otherwise, y _i (x)isequal to zero.

Let w = (w ₁,…,w _k )be suchthat w _j = 1for all$j =1, \dots , k$. In the second hidden layer,define k perceptrons by z _j (y) := 𝜗(w ⋅ y − j + 1/2).Finally, for all $j=1, \dots , k$,let the j-th unit from the second hidden layer be connected with one linear output unit with the weight (−1)^j.

The two-hidden-layer network obtained in this way computes the function${\sum }_{j=1}^{k} (-1)^{j} \vartheta (w \cdot y(x) -j +1/2)$, where y _i (x) = 𝜗(c ⁱ ⋅ x − b), i.e., it computesthe function ${\sum }_{j=1}^{k} (-1)^{j} \vartheta \left ({\sum }_{i=1}^{d/2} \vartheta (c^{i} \cdot x -b) -j +1/2\right ) = h_{k}(x) = h_{k}(u,v) = -1^{u \cdot v}$. □