Human whole-body pose estimation is a challenging task since the model needs to learn more keypoints than the body-only case. To meet the needs of real-time performance while maintaining accuracy is also a hard issue in whole-body pose estimation due to the learning capability of lightweight networks. In order to solve the above problems to a large extent, we propose a light whole-body pose estimation method with an optimized training strategy. The model is designed based on bottom-up architecture as a base network followed by a refinement network. We propose a two-stage training process, which learns rough features in the first stage and then improves estimation precision in the second stage. An online data augmentation procedure is proposed in the second stage to improve refinement performance. We also introduce a separate learning refinement structure that fine-tunes for body, foot, and hand part independently. Experimental results show that our method improves over 8%–10% average precision compared with other lightweight state-of-the-art approaches in the whole-body pose estimation task, with nearly a quarter (25%) size of model parameters saved.