Deep neural networks based on multi-scale features have achieved great success in image inpainting. Existing methods usually learn features at different scale levels to fill holes gradually. However, they ignore the relationship between these features, which results in inconsistency. A new image inpainting model based on the U-Net structure is proposed in this paper, which considers the consistency between multi-scale features. With the proposed Ladderlike Feature Consistent Module (LFCM), we ensure the consistency of multi-scale features, which is achieved by referring to other-scale features when filling specific-scale features. An attention mechanism called the Multi-level Attention Mechanism (MAM) is proposed to focus on feature relationships at different scale levels, further refining the filled features. Furthermore, the High-Frequency Residual Discriminator (HFRD) is proposed to improve high-frequency texture details. And the Attention Reusing Module (ARM) is also introduced in every encoder feature to pre-fill the missing regions. Experiments on human faces (CelebA-HQ), street view (Paris StreetView), and natural landscapes (Places2) datasets demonstrate that our proposed inpainting model generates higher quality results in comparison to the existing state-of-the-art methods.