Image dense matching based on region growth with adaptive window

Abstract

A new and efficient dense matching algorithm is presented based on region growth, which can be applied to a wide range of image pairs including those with large disparity or without rectification. Firstly, some points in the image pair are matched using a new two-level matching method. These points are taken as the seeds from which corresponding relations propagate towards other regions of the images under two strategies. By introducing the statistic model of disparity distribution within the window, modified SSD is produced and adopted as the cost function. The size of the template window is adaptive with textures within it and the size of the search window is changed in inverse proportion to the confidence coefficient. The algorithm has been tested with real stereo images and the results demonstrate its accuracy and efficiency.

Introduction

Stereo matching is the process of finding corresponding points in two or more images. It is one of the most important and challenge subjects in computer science, especially in computer vision. So it has been the focus of attention in this field for a long time. To solve this problem, some constraints are often introduced:

• Epipolar constraint. For any point in the left image, its matching point in the right image must lie on the corresponding epipolar line.

• Similarity constraint. Corresponding points are assumed to have similar intensity or color. So intensity is the main information used in stereo matching.

• Continuity constraint. Surfaces of objects are assumed to be smooth, which means its disparity varies continuously. Region growth is a simple and effective way of using this constraint to solve the problem of dense matching.

Traditional dense matching methods are window-based matching, which compares intensity similarity of neighboring pixels within a window between images to decide whether the center points of the windows are a pair of corresponding points. In this approach, the selection of an appropriate window size is critical to achieve a smooth and detailed disparity map. The optimal choice of window size depends on the local amount of variation in texture and disparity. In general, a smaller window is desirable to avoid unwanted smoothing. In areas of low texture, however, it does not contain intensity variation enough to achieve reliable matching. On the other hand, when the disparity varies within the window, intensity values may not correspond due to projective distortion. These are the main problems in window-based stereo methods.

Many other techniques have also been investigated extensively in recent years. Epipolar lines are assumed to be image rows in image pairs after rectification. Matching after rectification reduces the searching area from two-dimension window to one-dimension epipolar line, which simplifies the search process. But the rectification depends on a good estimation of the epipolar geometry. Or the mean and covariance of the corresponding epipolar line must be computed to determine the epipolar envelopes. An adaptive window method (Kanade and Okutomi, 1994) is proposed to remedy the main problems in window-based stereo methods mentioned above. The window size and shape are iteratively changed based on the local variation of the intensity and current depth estimates. Though it improved the matching result greatly, it is extremely computationally expensive. Attempts at using dynamic programming for solving stereo matching problem (Ohta and Kanade, 1985) use edges as the basic primitives. A generalization of the dynamic programming approach transforms the stereo correspondence problem into a maximum-flow problem (Roy and Cox, 1998). Once solved, the minimum-cut associated to the maximum-flow yields a disparity surface for the whole image at once. This global approach provides a more accurate and coherent disparity map than the traditional line-by-line stereo. A cooperative iterative algorithm (Zitnick and Kanade, 1999) and a volumetric iterative approach (Zitnick and Kanade, 1998) use global constraints to find a dense disparity map. By introducing global constraints and adopting iterative algorithm, a fairly good dense disparity map is obtained and its computational complexity is less than that of the adaptive window method. But both the amount of memory needed and the complexity of computation are still high especially when the disparities are large. And rectification must be applied to the initial image pair before matching. The maximum-flow method mentioned above also suffers from the same problem. Dense matching method based on region growing techniques (Lhuillier, 1998; Lhuillier and Quan, 1999) has shown good performances, but the method developed so far can only be applied to globally textured images. Propagation is forbidden in regions that are smooth. To cope with these problems, we investigate the following algorithm. This algorithm can deal with large camera motions and a wide range of images without rectification even if parts of the images are less textured.

In our region growth algorithm, seed points are matched using a new two-level matching method. Firstly, edges in stereo image pair are extracted using edge detection operator. Cost functions are calculated within a large template window in the two edge images to measure their edge similarity. Thus the position of the matching point is restricted within a small region after this coarse-level matching based on the contours of the images. Then intensity similarity is measured between the image pair within a small template window using intensity information and the position of the corresponding point is localized accurately. This two-level matching method combines area-based method with feature-based method. That is, edges are taken as features to guide the intensity matching. So multiple solutions are eliminated and an accurate and reliable result is obtained for the seed points. This is very important for the region growth algorithm since it is its foundation. After seed points have been matched, corresponding relations “grow” from the seeds towards other regions of the images under the continuity constraint. Choosing an appropriate cost function and modifying it according to the statistical model of disparity distribution within the window can both reduce the computation time and increase the accuracy. The size of the template window is adaptive in the process of region growth according to the textures within it and the size of the search window is changed in inverse proportion to the confidence coefficient. All of these new techniques improved the performance of region growth algorithms developed so far.

In this paper, the two-level matching method for the seed points is firstly discussed in Section 2. The new dense matching algorithm based on region growth is developed in Section 3. Section 4 provides experimental results with real stereo images, which demonstrate the effectiveness and accuracy of the algorithm. Finally, some concluding comments are given in Section 5.