The Bayesian brain model has been proposed as a possible simplified view of the inner workings of the human brain. According to this view, the brain is a prediction machine with an internal model of the world which can be improved by comparing the generated predictions to sensory observations. Knowledge about expected events (prior predictions) are combined with observations (sensory input) into posterior beliefs, helping us to infer what is happening in the environment. While it is commonly acknowledged that this integration is optimally performed in a Bayesian way, the effects of Bayesian inference on the developmental process are less well investigated. In this study, we propose a computational framework which combines Bayesian inference with recurrent neural network training. We demonstrate that learning in this framework proceeds in a human-like manner in that the system is able to appropriately weight sensory input and prior predictions depending on their reliability which increases during development. As a result, during the course of learning, the model gradually switches from relying on sensory information to a stronger reliance on own predictions and becomes more robust against disturbances in the environment.