This paper proposes an efficient and lightweight model called PoseHMR to address the interference of irrelevant image features and the issues of model inefficiency in 3D human body mesh reconstruction. PoseHMR uses a transformer-decoder architecture and obtains holistic and regional prior constraints about human posture, which serve as signals for the model throughout the process of human mesh reconstruction. To filter out the useless features extracted from the image, the self-attention module is guided by holistic prior constraints to focus on the area where the human body is located. Likewise, the cross-attention module is guided by regional prior constraints to focus on key sampling points around vertices. Furthermore, after generating key query areas by regional prior constraints, a heuristic fine-tuning strategy is applied to refine the local human mesh effectively. Our model is evaluated on mainstream Human3.6M and 3DPW datasets and achieves a state-of-the-art result with fewer parameters. The codes are available at https://github.com/Sookiep/Pose-HMR.