Image inpainting aims to restore the missing regions of corrupted images plausibly. Recent techniques of image inpainting handle it as a pure generation problem and utilize large-scale training data to learn semantic and texture information to fill the missing area. However, generative networks frequently distort local structures or introduce visible artifacts while filling large missing regions. To address this issue, we propose SLTFill, an image inpainting method that uses a set of reference images which share scene contents with the corrupted image to guide the synthesis of missing regions. A key challenge addressed by our approach is aligning the transformations of the reference images with the context of the corrupted image and making sure they are consistent in spatial structure and light space. Qualitative and quantitative experiments demonstrate the state-of-the-art performance of our approach on the image inpainting task.