Offline–online RL is a new framework for RL. It can use historical data in offline training phase and improve the policy performance in online training phase. However, the existing offline–online RL algorithms cannot handle the $Q$-value overestimation caused by distribution shift in offline training phase and the policy performance degradation in the conversion from offline to online learning phases. The algorithm we propose namely O2AC can solve above problems effectively. Experimental results show that our algorithm exceeds the current state-of-the-art offline–online RL algorithms.

Offline–online reinforcement learning (RL) can effectively address the problem of missing data (commonly known as transition) in offline RL. However, due to the effect of distribution shift, the performance of policy may degrade when an agent moves from offline to online training phases. In this article, we first analyze the problems of distribution shift and policy performance degradation in offline–online RL. Then, in order to alleviate these problems, we propose a novel RL algorithm offline–online actor–critic (O2AC) algorithm. In O2AC, a behavior clone constraint term is introduced into the policy objective function to address the distribution shift in offline training phase. In addition, in online training phase, the influence of the behavior clone constraint term is gradually reduced, which alleviates the policy performance degradation. Experiments show that O2AC outperforms existing offline–online RL algorithms.