One Analog Neuron Cannot Recognize Deterministic Context-Free Languages

Abstract

We analyze the computational power of discrete-time recurrent neural networks (NNs) with the saturated-linear activation function within the Chomsky hierarchy. This model restricted to integer weights coincides with binary-state NNs with the Heaviside activation function, which are equivalent to finite automata (Chomsky level 3), while rational weights make this model Turing complete even for three analog-state units (Chomsky level 0). For an intermediate model $\alpha$ANN of a binary-state NN that is extended with $\alpha \ge 0$ extra analog-state neurons with rational weights, we have established the analog neuron hierarchy 0ANNs $\subset$ 1ANNs $\subset$ 2ANNs $\subseteq$ 3ANNs. The separation 1ANNs $\subsetneqq$ 2ANNs has been witnessed by the deterministic context-free language (DCFL) $L_\#=\{0^n1^n\,|\,n\ge 1\}$ which cannot be recognized by any 1ANN even with real weights, while any DCFL (Chomsky level 2) is accepted by a 2ANN with rational weights. In this paper, we generalize this result by showing that any non-regular DCFL cannot be recognized by 1ANNs with real weights, which means (DCFLs $\setminus$ REG$)\,\subset \,($2ANNs $\setminus$ 1ANNs), implying 0ANNs $=$ 1ANNs $\cap$ DCFLs. For this purpose, we show that $L_\#$ is the simplest non-regular DCFL by reducing $L_\#$ to any language in this class, which is by itself an interesting achievement in computability theory.

Research was done with institutional support RVO: 67985807 and partially supported by the grant of the Czech Science Foundation GA19-05704S.