Human motion prediction (HMP) predicts future human pose sequences given the past ones. HMP has recently attracted attention in computer vision and the robotics domain as it helps machines understand human behavior, plan target actions and optimize interaction strategies. Existing methods for HMP are based on either purely physics-based models or statistical models. However, each of these methods has its shortcomings. The physics-based techniques are complex and error-prone, while the statistical methods require a large amount of data and lack physical consistency. To overcome their limitations, we propose a physics-infused neural network (PIMNet), which combines both physics-based and statistical methods. We first computed the contact forces and joint torques for each pose using the physics-based human dynamical model. Then they are fed into an Encoder-Decoder machine learning architecture to predict future ones. In this way, PIMNet simultaneously obtains computational efficiency and physical consistency. Extensive experimental results on Human 3.6M show that the proposed PIMNet could accurately predict human motion in both short-term and long-term scope. It achieves better or comparable prediction accuracy than the state-of-the-art, even using a basic LSTM as the machine learning model.