Model-free algorithms in Reinforcement Learning (RL) are known to be a powerful learning tool and have performed well in solving complex issues. However, RL training results are often poor when the reward function is sparse or misleading in short term. In this paper, we propose a physics informed intrinsic reward function to assist the agent to overcome this difficulty. We evaluate the proposed intrinsic reward method on different types of actor-critic (AC) algorithms. The experimental results show noticeable improvement.