PCA-Based Recognition for Efficient Inpainting

Abstract

We present a technique for efficiently constructing a “clean” texture map of a partially occluded building facade from a series of images taken by a moving camera. After a robust registration procedure, building regions blocked by trees, signs, people, and other foreground objects are automatically inferred via the median absolute deviation of colors from different source images mapping to the same mosaic pixels. In previous work we extended an existing non-parametric inpainting algorithm for filling such holes to incorporate spatiotemporal appearance and motion cues in order to correctly replace the outlier pixels of the texture map. In contrast to other inpainting techniques that perform an exhaustive search over the image, in this work we introduce a principal components-based method that learns to recognize patches that locally adhere to the properties of the building being mapped, resulting in a significant performance boost with results of indistinguishable quality. Results are demonstrated on sequences where previous stitching and inpainting algorithms fail.