Integrating Policy Reuse with Learning from Demonstrations for Knowledge Transfer in Deep Reinforcement Learning

Abstract

Transfer learning (TL) assisted deep reinforcement learning (DRL) has attracted much attention in recent years, which aims to enhance reinforcement learning performance by leveraging prior knowledge from past learned tasks. However, it still remains challenging to conduct positive knowledge transfer when the target tasks are dissimilar to the source tasks, e.g., the source and target tasks possess diverse environmental dynamics. Taking this cue, this paper presents an attempt to explore TL in DRL across tasks with heterogeneous dynamics towards enhanced reinforcement learning performance. In particular, we propose to combine policy reuse and learning from demonstrations for knowledge transfer in DRL. It allows multiple learned policies in separate source tasks to adaptively fuse to generate a teacher policy for the target task, which will be further used for knowledge transfer via learning from demonstrations to boost the learning process of the target DRL agent. To evaluate the performance of our proposed method, comprehensive empirical studies have been conducted on continuous control tasks, i.e., Reacher and HalfCheetah. The obtained results show that the proposed method is superior in contrast to recently proposed algorithms in terms of both accumulated reward and training computational cost.