In this paper we introduce a kernel-based recursive least-squares (KRLS) algorithm that is able to track nonlinear, time-varying relationships in data. To this purpose we first derive the standard KRLS equations from a Bayesian perspective (including a principled approach to pruning) and then take advantage of this framework to incorporate forgetting in a consistent way, thus enabling the algorithm to perform tracking in non-stationary scenarios. In addition to this tracking ability, the resulting algorithm has a number of appealing properties: It is online, requires a fixed amount of memory and computation per time step and incorporates regularization in a natural manner. We include experimental results that support the theory as well as illustrate the efficiency of the proposed algorithm.