Anytime similarity measures for faster alignment

Abstract

Image alignment refers to finding the best transformation from a fixed reference image to a new image of a scene. This process is often optimizing a similarity measure between images, computed based on the image data. However, in time-critical applications state-of-the-art methods for computing similarity are too slow. Instead of using all the image data to compute similarity, one could use only a subset of pixels to improve the speed, but often this comes at the cost of reduced accuracy. These kinds of tradeoffs between the amount of computation and the accuracy of the result have been addressed in the field of real-time artificial intelligence as deliberation control problems. We propose that the optimization of a similarity measure is a natural application domain for deliberation control using the anytime algorithm framework. In this paper, we present anytime versions for the computation of two common image similarity measures: mean squared difference and mutual information. Off-line, we learn a performance profile specific to each measure, which is then used on-line to select the appropriate amount of pixels to process at each optimization step. When tested against existing techniques, our method achieves comparable quality and robustness with significantly less computation.

Introduction

The need to align, or register, two images is one of the basic problems of computer vision. It can be defined as the task of finding the spatial mapping that places elements in one image into meaningful correspondence with elements in a second image. It is essential for data fusion tasks in medical imaging [1] and remote sensing (e.g. [2]). It is also widely applied in tracking and automatically mosaicking photographs [3].

One of the most straightforward and widely used approaches is referred to as direct image alignment. It works by defining a similarity measure, $D$, as a function of a reference image, and a template image warped by a transformation with some parameters, $\mathit{\varphi }$. The computation of $D$ typically requires examining all pixels in each image. The alignment problem becomes that of finding the values of $\mathit{\varphi }$ that maximize the chosen similarity measure. A number of optimization techniques for smooth functions, such as gradient descent, have been used for this problem, and provide good solutions on a wide range of image types. However, these approaches can be slow, which reduces their usefulness in time-sensitive applications such as real-time video registration (e.g. [4]) and medical image registration during surgery (e.g. [5]). It is possible to increase the speed of processing by using only a subset of the pixels to compute $D$, but this can easily lead to a reduction in accuracy and reliability. Determining the size of the subset to obtain good performance is typically done in an ad-hoc fashion, or using heuristics which are applicable only to certain domains. Furthermore, since a different number of pixels may be needed at different stages in the optimization, a fixed subset is necessarily a compromise.

The process of intelligently dealing with tradeoffs such as this one between computational speed and accuracy is referred to as deliberation control. Deliberation control methods rely on two key components: algorithms that support partial evaluation, and knowledge about how those algorithms perform after different amounts of computation. The anytime algorithms [6], [7] are a, class of algorithms supporting partial evaluation, which provide a solution when run for any length of time. The solution quality is guaranteed to improve with the amount of computation performed. In this paper, we propose to use a deliberation control framework using anytime algorithms to arrive at a principled solution to the speed vs. accuracy trade-off in this problem. The first step is an off-line training process to learn the properties of the similarity measure under consideration, in terms of accuracy vs. computation time, by analysing image pairs for which the transformation parameters are known. Given a new pair of images to align, we then use this knowledge to determine the number of pixels that need to be considered at each step of the optimization. In this paper, we explore the effectiveness of this approach using two common similarity measures, mean squared difference and mutual information, and a gradient descent optimizer. We tested the algorithm on several types of images: images of everyday scenes, multimodal medical images and earth observation data (i.e. Landsat and Radarsat images). In all cases, using a deliberation control approach is faster than computing the transformation using all the image data and gives more reliable results than simply performing the optimization using an arbitrary, fixed, percentage of the pixels.

The remainder of this paper is organized as follows. In Section 2 we review the image alignment problem and in Section 3 we review methods of deliberation control using anytime algorithms. The details of how deliberation control has been implemented in the context of image alignment are given in Section 4. Finally, Sections 5 Experiments, 6 Conclusions and future work describe our experimental setup, results and conclusions.