Penalty prediction is one of the main tracks of legal judgement prediction (LJP) which is to apply artificial intelligence methods to predicting the court’s judgement based on case descriptions. It is still far from effective, as the elements affecting the penalty are numerous but sparsely distributed in case descriptions, which may make them covered by noise in contexts. Allocating reasonable attention to these elements related to the penalty is the key to improving the effect of the penalty prediction. To this end, we propose a novel model called MHA-AR which learns to focus on the key elements by a modulated hierarchical attention mechanism and a legal attribute recognition subtask. Besides benefiting the prediction accuracy, it also improves the reliability and credibility of predictions for users by providing comprehensible legal attributes related to the penalty. A series of experiments conducted on the real-world datasets demonstrate the correctness of our assumptions and the superiority of MHA-AR. The implementation of our proposed model will be available at https://github.com/realcatking/penaltyprediction.