Recent deep-learning-based face hallucination methods have achieved great success. Due to the parameter sharing characteristics of convolutional neural network, most existing deep-learning-based methods essentially use the same kernel for different regions of the entire face image in a convolution layer. This scheme of treating the face image as a whole will lead to the neglect of important facial details. To address this problem, we design a novel heatmap-aware convolution with spatially variant kernels rather than a spatially sharing kernel in the standard convolution to recover different regions. Based on this, we propose a heatmap-aware pyramid face super-resolution network (HaPSR) that embeds our heatmap-aware convolution into a two-branch network for both face super-resolution and facial heatmap estimation. The facial heatmap estimation branch can not only be used as an auxiliary to regularize face super-resolution reconstruction, but also provide an important basis for spatially variant kernels. Quantitative and qualitative experimental results demonstrate that our method outperforms state-of-the-arts.