The mobility load balancing (MLB) in self-organizing networks (SONs) is designed to automatically resolve the mismatch between network resource distribution and network traffic demand. In this paper, we propose an off-policy deep reinforcement learning (DRL) based MLB framework to balance the load distribution among all the cells. Our main contribution is three-fold. First, we propose to use off-policy RL with multiple behavior policies to autonomously learn the optimal MLB policy without any prior knowledge over the underlying wireless environments. Second, we propose a corresponding DRL-based MLB model by using deep neural networks as the function approximators to improve the generalization ability over complex system states. Third, we propose an asynchronous parallel learning framework for MLB to improve the training efficiency in a collaborative manner. Experimental results show that our proposed DRL-based MLB model can outperform the existing approaches considerably.