In this article, we perform continual reinforcement learning (CRL) experiments using two benchmarks that rely on a physically simulated robot with a differential drive. CRL is a flavor of reinforcement learning (RL) in which environments themselves are taken to be non-stationary, which we model here by abrupt changes in observations and rewards, termed tasks. In particular, we demonstrate that deep Q-learning (DQN) with experience replay suffers from conflicting objectives in such a scenario: on the one hand, the replay buffers must be large to prevent catastrophic forgetting (CF), but, on the other hand, they should be small to enable rapid learning of new statistics. Taking inspiration from the domain of supervised continual learning (CL), we generalize common learning and forgetting measures in CL and show that they are highly indicative of typical effects in CRL: forgetting, intransigence, and, surprisingly, backwards transfer, i.e., the improvement of policies for past tasks by current learning.