Inductive Bias in Recurrent Neural Networks

Abstract

The use of prior knowledge to train neural networks for better performance has attracted increased attention. Initial domain theories exists for many machine learning applications. In both, feed forward and recurrent neural networks, algortihms for encoding prior knwoledge has been constructed. We propose a heuristic for determining the strength of the prior knowledge (inductive bias) for recurrent neural networks encoded with a DFA as initial domain knowledge. Our heuristic uses gradient information in weight space in the direction of the prior knowledge to enhance performance. Tests on known benchmark problems demonstrate that our heuristic reduces training time, on average, by 30% compared to a random choice of the strength of the inductive bias. It also achieves, on average, near perfect generalization for that specific choice of the inductive bias.