Residual map-based defect detection methods have been widely concerned in the field of prints, because of their superiority in detecting defects of arbitrary shapes and extreme aspect ratios by taking advantage of pixel-level comparison, especially in detecting the tiny defects. However, residual map-based methods require extremely high structural similarity between images, so it is important to eliminate the structural differences by warping the tested image to template image via the displacement field. In practice, weak textures, large displacement, and deformation widely existed in the nonflat print images, which increased the difficulty of obtaining a satisfied displacement field. Inspired by the PatchMatch, our approach, adaptive coarse-to-fine PatchMatch (ACPM), blends an adaptive patch matching strategy with the coarse-to-fine scheme for displacement field estimation. We generate adaptive size patches according to the texture richness of local regions and propagate displacement information based on morphological operations, so the matching correspondences can be more reliable and accurate. Furthermore, the displacement field is interpolated from the matching result of ACPM and refined by the variational model to tackle nonrigid deformation (called ACPMFlow). Compared to the state-of-the-art method DeepFlow, ACPMFlow can achieve comparable performance with $20\times$ speed and obtain an excellent performance in the detection of tiny defects on nonflat prints, which is more suitable for real-time inspection.