Hyper-Resolution: Image detail reconstruction through parametric edges

Abstract

Hyper-Resolution, a new technique for super-resolution reconstruction of images, is based on matching low-resolution target image details to their high-resolution counterparts from an image database. Central to the algorithm is a novel transform of image content from the orthogonal pixel space to a parametric space structured around edges. This approach offers improved quality, more flexibility and significantly faster performance than previous work in the field. Implementation strategies for achieving this efficiency are carefully outlined. The algorithm is evaluated by controlled assessment, qualitative evaluation, and applications to facial detail reconstruction and identification. The algorithm is finally analyzed through the comparison with alternative techniques.

Introduction

In this age of high-resolution imaging, researchers constantly strive for more pixels well beyond the limits of current image capture technology. As such, techniques for accurately and efficiently reconstructing high-resolution image details are in great demand. Numerous applications benefit from these techniques, including photographic enlargement, scalable texture mapping (image-based rendering), forensics and surveillance imaging, restoration of fine artwork, diagnosis via higher-resolution medical images, and video processing with high-definition TV reconstruction.

Research on resolution enhancement has been conducted for over two decades [1]. The most traditional method, interpolation [2], [3], estimates missing data between sampled pixels by using the surrounding pixels. Unfortunately, interpolation results in blurred images lacking distinct edges and fine textures. While many improvements on interpolation have been proposed [4], [5], it is still mathematically impossible for any interpolation algorithm to reconstruct subpixel detail above the spatial frequency of the original image. This fundamental problem has given rise to a number of so-called super-resolution techniques [6], [7], [8], [9], [10]: if one “target” image does not contain enough information to reconstruct the desired level of detail, we can use multiple lower-resolution “source” images of the same or similar subjects (captured from different perspectives) to reconstruct the missing data points in a single high-resolution frame.

Registration-based super-resolution relies on multiple images captured by a precisely moving camera. Based on the camera's motion vector, we can accurately solve for missing samples at subpixel intervals to reconstruct a super-resolved image [1], [6], [8], [9], [10], [11]. Recently, training-based super-resolution algorithms have been explored in [12], [13], [14], [15], [16], [17], [18]. These techniques rely on the premise that we can reconstruct visually plausible high-resolution detail for one low-resolution target image based on the pattern recognition of “similar” details in a potentially large database of high-resolution source images. The most closely matching details and textures from the source images (typically organized as a set of small image patches) are then substituted in place of their low-resolution counterparts in the target image to create a super-resolution image.

In this area, Hertzmann et al. [15] developed a training-based approach called “image analogies”, where fine details are learned and predicted between images. Baker and Kanade [14] devised a so-called reconstruction (recognition based reconstruction) algorithm in which local features in the low-resolution target image are recognized and then enhanced using parent structure vector matching. This method produced impressive results on human faces, but still lacks robustness in certain illumination conditions. Most of training-based techniques require a specific model adapted to the subject material of the target images and cannot be used on arbitrary images. Example-based super-resolution, developed by Freeman et al. [12], gets around this problem by using a database of small pixel patches from many training images to reconstruct plausible high-frequency detail in enlarged images; the matching is performed by correlating these small patches (e.g., $7\times 7$ pixels). Most recently, an image hallucination approach by using primal sketch priors was developed by Sun et al. [19]. The image resolution is enhanced to a very high quality. This method relies on the good match of the primal sketch priors between target image and source image, its robustness is a major concern for further applications. Motivated by the existing work mentioned above, we have developed a new technique to attack this problem.

Image sharpness and detail are often judged by the quality of an image's strong edges. However, most existing super-resolution algorithms work on either the pixel or pixel patch level. This raises the obvious question: if image detail is in edges by definition, why match in the pixel domain to begin with? This observation leads us to a novel idea: instead of matching patches in the spatial domain [12], we can first transform each image into a new parametric vector space structured by the image's edges. Our detail “patches” are then composed of only the texture details sampled on and around image edges, with coordinates relative to these edges. Note that several most recent work reported in [20], [21], [22] show that it is very promising to use feature-based texture or silhouette map to increase the texture rendering quality. The impressive results were presented in the reported work, however, only the simple-texture images with salient object contours were tested. We propose a new algorithm to trace the image contents around edges using the texture matching approach so that more complex image details are expected to be reconstructed.

This proposed approach is called Hyper-Resolution, or HyperRes. In our algorithm, edge detection is used to obtain the edges of all input images. After fitting each edge to a parametric curve representation, we use a coordinate system transform to sample points along and normal to the edge, thus forming a “parametric map”. Each parametric map, which is invariant across affine transforms of its corresponding edge, is decomposed into a hierarchy of smaller segments; these segments are then entered as keys in a database. To reconstruct the high-resolution version of a given target edge, we locate similar edge segments in the database, project the matching source edge data onto the curvature of the appropriate target edges, and add this high-resolution data back into the original low-resolution target image. While edge structured imaging has been described before [20], [21], [22], [23], [24], our technique differs in that an image is described as the textures surrounding the edges rather than as a superposition of contour edges themselves.

The HyperRes process, as shown in Fig. 1, is logically split into two parts. The assimilation phase transforms both source and target images to their parametric edge representations and adds them to a database, while the reconstruction phase maps appropriate high-resolution edge details from the database onto the target image to reconstruct it at a higher resolution than its original pixels provided. The HyperRes algorithm is detailed in Section 2. After we provide experimental results and assessments in Section 3, we will discuss its applications to enhancing face feature detection and identification. Our algorithm's advantages and weaknesses are analyzed in Section 4. Finally, concluding remarks and future work are given in Section 5.