Adapting the mastered manipulation skill to novel objects is still challenging for robots. Recent works have attempted to endow the robot with the ability to adapt to unseen tasks by leveraging meta-learning. However, these methods are data-hungry in the training phase, which limits their application in the real world. In this paper, we propose Meta-Residual Policy Learning (MRPL) to reduce the cost of policy learning and adaptation. During meta-training, MRPL accelerates the learning process by focusing on the residual task-shared knowledge that is hard to be embedded in the hand-engineered controller. During testing, MRPL achieves fast adaptation on similar unseen tasks through fusing task-specific knowledge in the demonstration with task-shared knowledge in the learned policy. We conduct a series of simulated and real-world peg-in-hole tasks to evaluate the proposed method. The experimental results demonstrate that MRPL outperforms prior methods in robot skill adaptation. Code for this work is available at https://github.com/Bartopt/code4MRPL.