Relaxation by Hopfield network in stereo image matching

Abstract

This paper outlines a relaxation approach using the Hopfield neural network for solving the global stereovision matching problem. The primitives used are edge segments. The similarity, smoothness and uniqueness constraints are transformed into the form of an energy function whose minimum value corresponds to the best solution of the problem. We combine two methods: (a) optimization/relaxation[1]and (b) relaxation merit[2]with the above three constraints mapped in an energy function. The main contribution is made (1) by applying a learning strategy in the similarity constraint and (2) by introducing specific conditions to overcome the violation of the smoothness constraint and to avoid the serious problem arising from the required fixation of a disparity limit. So, we improve the stereovision matching process. A better performance of the proposed method is illustrated with a comparative analysis against a classical relaxation method.

Introduction

A major portion of the research efforts of the computer vision community has been directed towards the study of the three-dimensional (3-D) structure of the objects using machine analysis of images.3, 4, 5Analysis of video images in stereo has emerged as an important passive method for extracting the 3-D structure of a scene.

Following the Barnard and Fischler[6]terminology, we can view the problem of stereo analysis as consisting of the following steps: image acquisition, camera modeling, feature acquisition, image matching, depth determination and interpolation. The key step is that of image matching, that is, the process of identifying the corresponding points in two images that are cast by the same physical point in 3-D space. This paper is devoted solely to this problem.

The basic principle involved in the recovery of depth using passive imaging is triangulation. In stereopsis the triangulation needs to be achieved with the help of only the existing environmental illumination. Hence, a correspondence needs to be established between features from two images that correspond to some physical feature in space. Then, provided the position of centers of projection, the effective local length, the orientation of the optical axis, and the sampling interval of each camera are known, the depth can be established using triangulation.[7]

A review of the state-of-the-art-of stereovision matching allows us to distinguish two sorts of techniques broadly used in this discipline, area-based and feature-based.2, 4, 8Area-based stereo techniques use correlation between brightness (intensity) patterns in the local neighborhood of a pixel in one image with brightness patterns in the local neighborhood of the other image9, 10, 11, 12, 13where the number of possible matches becomes high, while feature-based methods use sets of pixels with similar attributes, normally, either pixels belonging to edges14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25or the corresponding edges themselves.2, 5, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36These latter methods lead to a sparse depth map only, leaving the rest of the surface to be reconstructed by interpolation; but they are faster than area-based methods, because there are many fewer points (features) to be considered.[4]We select a feature-based method as justified below.

Due to the nature of the stereovision system, there are intrinsic and extrinsic factors affecting it: (a) extrinsic, in a practical stereovision system, the left and right images are obtained at different positions/angles; (b)intrinsic, the stereovision system is equipped with two different physical cameras (i.e. with different components), which are always placed at the same relative position (left and right). A systematic noise appears for each one.

As a result of the above-mentioned factors the corresponding features in both images may display different values. This may lead to incorrect matches. Thus, it is very important to find features in both images which are unique or independent of possible variation in the images.[37]This research paper uses a feature-based method where edge segments are to be matched, because our experiment has been carried out in an artificial environment where the edge segments are abundant. Such features have been studied in terms of reliability8, 38and robustness[37]and, as mentioned before, have also been used in previous stereovision matching works. This fact justifies our choice of features, although they may be too local. Four average attribute values (module and direction gradient, variance and Laplace) are computed for each edge segment, as we will see latter.

Our stereo correspondence problem can be defined in terms of finding pairs of true matches, namely, pairs of edge segments in two images that are generated by the same physical edge segment in space. These true matches satisfy some competing constraints, generally three, proposed by Marr and Poggio:21, 22(1) uniqueness, each edge segment in an image should be matched to a unique edge segment in the other image; (2) similarity, matched edge segments have similar local properties or attributes; (3) smoothness, disparity values in a given neighborhood change smoothly, except at a few depth discontinuities.

We, subsequently, introduce the very important disparity concept. Assume a stereo pair of edge segments, where a segment belongs to the left image and the other to the right one. If we superpose the right image over the left one, for example, the two edge segments of the stereo pair appear horizontally displaced. Following horizontal (epipolar) lines from top to bottom in the superimposed images, we determine the common length of the two edge segments from the parallelogram formed by the set of epipolar lines containing a point of the left edge segment and other of the right one. We compute the disparity value for the pair of edge segments as the average displacement between points belonging to both edge segments along the common length.

Following the above-introduced constraints, we can say that the similarity and smoothness constraints are associated to local and global matching processes, respectively. The major difficulty of stereo processing arises due to the need to make global correspondences. A local edge segment in one image may match equally well with a number of edge segments in the other image (this problem is compounded by the fact that the local matches are not perfect due to the above-mentioned extrinsic and intrinsic factors). These ambiguities in local matches can only be resolved by considering sets of local matches globally. Hence, to make global correspondences given a pair of edge segments, we consider a set of neighboring edge segments, where a bound on the disparity range defines the neighborhood. Relaxation is a technique commonly used to find the best matches globally and it refers to any computational mechanism that employs a set of locally interacting parallel processes, one associated with each image unit, that in an iterative fashion update each unit’s current labeling in order to achieve a globally consistent interpretation of image data.39, 40

We, next, perform a review of relaxation approaches in stereovision matching. Relaxation labeling is a technique (proposed by Rosenfeld et al.4, 5, 11developed to deal with uncertainty in sensory data interpretation systems and to find the best matches. It uses contextual information as an aid in classifying a set of interdependent objects by allowing interactions among the possible classifications of related objects. In the stereo paradigm the problem involves assigning unique labels (or matches) to a set of features in an image from a given list of possible matches. So, the goal is to assign each feature (edge segment) a value corresponding to disparity in a manner consistent with certain predefined constraints.

Two main relaxation processes can be distinguished in stereo correspondence: optimization based and probabilistic/merit based. In the optimization-based processes, stereo correspondence is carried out by minimizing an energy function which is formulated from the applicable constraints. It represents a mechanism for the propagation of constraints among neighboring match features for the removal of ambiguity of multiple stereo matches in an iterative manner. The optimal solution is ground state, that is, the state (or states) of the lowest energy. In the probabilistic/merit-based processes, the initial probabilities/merits, established from a local stereo correspondence process and computed from similarity in the feature values, are updated iteratively depending on the matching probabilities/merits of neighboring features and also from the applicable constraints. The following papers use a relaxation technique: (a) probabilistic/merit5, 13, 16, 17, 20, 21, 24, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56(b) optimization through a Hopfield neural network.11, 13, 23, 28, 57We use (b).

These systems impose global constraints, i.e. similarity, uniqueness and smoothness. The smoothness constraint requires, as stated, to fix a bound on the disparity range for any feature. As a result, they give good performance on the images shown. In our experiments, we have found complex images with highly repetitive structures (features) or with objects very close to the cameras and occlusions violating the smoothness constraint. Hence, setting a disparity limit is a difficult task, because the global correspondence, for a given pair of features, is based on how well their neighboring pairs of features match. The neighborhood is a concept related strongly to the disparity limit. Therefore, global correspondence for complex images is the difficult issue with which this paper is mainly concerned. We propose an approach for solving the edge-segment stereovision matching problem, in which we have made good use of existing global matching strategies, where the similarity, uniqueness and continuity are applied as a result of the combination of two methods (we have followed a very important conclusion of Pavlidis[58]: (1) the Yu and Tsai (YT)[1]optimization relaxation process, implemented by a Hopfield neural network; (2) the Minimum Differential Disparity (MDD) algorithm of Medioni and Nevatia.[2]The main contribution of this paper is made (1) by introducing specific conditions during the relaxation process to overcome the violation of the smoothness constraint and (2) to avoid the problem arising from the fixation of the disparity limit. We have also applied a learning strategy in the similarity constraint. So, our method works properly even if the images are complex. We have chosen the Hopfield neural network because the convergence of the relaxation process is guaranteed and has been used in previous stereovision matching works with success.11, 13, 23, 25, 28, 57Additionally, this computational model has a massively parallel execution capability, as we are only worried with the method effectiveness (from the viewpoint of the percentage of matching successes), this is a possibility not implemented in our work.

The paper is organized as follows: in Section 2the stereovision optimization relaxation matching schema and the relationship to Hopfield neural networks are described. In Section 3we summarize the proposed matching procedure. The performance of the method is illustrated in Section 4, where a comparative study against an existing global relaxation matching method is carried out. The necessity of a relaxation matching strategy against a local matching technique is also made clear. Finally in Section 5, there is a discussion of some related topics.