Integration of multiresolution image segmentation and neural networks for object depth recovery

Abstract

A novel technique for three-dimensional depth recovery based on two coaxial defocused images of an object with added pattern illumination is presented. The approach integrates object segmentation with depth estimation. Firstly segmentation is performed by a multiresolution based approach to isolate object regions from the background given the presence of blur and pattern illumination. The segmentation has three sub-procedures: image pyramid formation; linkage adaptation; and unsupervised clustering. These maximise the object recognition capability while ensuring accurate position information. For depth estimation, lower resolution information with a strong correlation to depth is fed into a three-layered neural network as input feature vectors and processed using a Back-Propagation algorithm. The resulting depth model of object recovery is then used with higher resolution data to obtain high accuracy depth measurements. Experimental results are presented that show low error rates and the robustness of the model with respect to pattern variation and inaccuracy in optical settings.

Introduction

Depth measurement is one of the most important tasks in many computer vision applications, including three dimensional object recognition, scene interpretation, post inspection and part manipulation. Three-dimensional (3-D) positional information restoration can be obtained using various techniques, among which depth from defocus (DFD) methods have the advantage that they require only two co-axial images obtained with different optical settings. DFD avoids the missing part and correspondence problems that occur with stereo. Despite these merits, DFD shares one inherent weakness with stereo and motion techniques in that it requires the scene to contain natural or projected textures. In the work presented here we used projected texture (active illumination).

Several researchers have developed accurate, dense depth estimations from defocused images in the past decade. The DFD method, originally developed by Pentland [1], uses the relative defocus in two images taken with different camera settings to determine scene structures. Many other techniques have followed, and these fall into the two main categories of Fourier and spatial domain based modelling. Subbarao has successively presented depth models for both domains [2]. Nayar et al. [3] give a precise blur analysis in the frequency domain using focus operators as models. They considered both actively and passively illuminated scenes [4]. Furthermore, they proposed telecentric optics [5] to achieve magnification invariance under changes in the focus setting. Their technique employed a small bank of broadband rational filters [6] able to handle arbitrary textures. The method computed efficiently and produced accurate results even for weak textures. Ghita and Whelan [7] reported a practical DFD implementation based on simple filters and a striped illumination pattern. They later use this algorithm in a bin picking application [8].

Recently, the techniques of artificial neural networks (ANNs), an empirical modelling in the spatial domain, have been applied to the DFD problem. ANNs have the properties of robustness and adaptation to approximate any non-linear function, so the stringent requirements for optical settings have been reduced compared to the earlier techniques. Tsai [9] proposed an algorithm to estimate the amount of blur from a single camera, in which the blur is calculated using a moment-preserving technique. The ANNs are only used to compensate for certain depth errors. Pham and Aslantas [10] have presented a technique employing a multi-layer perceptron (MLP) network to compute distances from derivative images of blurred edges. The theory of the MLP is described by Pinkus [11]. In addition, Jong and Huang [12] have explored the Radical basis Function (RBF) neural network for blur scale detection of the point-spread function (PSF). Presently, there are few ANN based approaches to DFD oriented object recovery in the literature. The main problems for ANN based depth models are to build robust, accurate depth estimates with reasonably small networks, and the trade-off between the amount of pre-processed input data required and the efficiency achieved by the training procedure. In this paper, a novel ANN based approach for depth measurement is reported that simplifies model architecture and improves model performance. We have integrated image segmentation with neural network learning, to solve depth recovery by a two-stage procedure, in which two-dimensional (2-D) object segmentation is followed by 3-D depth model formation. The first stage can be viewed as data pre-processing before the depth modelling stage. A multiresolution scheme, used for edge detection in [13], [14], was applied at the first stage with the objectives of reducing the data needed to form the depth model in the later stage, and to provide a reliable segmentation for the pattern-based image. Firstly, the data from one defocused image is processed to form a multiresolution pyramid, in which the subsequent levels have progressively lower image resolution, but preserve the essential depth information in the similarity measures between parent–child nodes at the neighbouring levels. Only one image is required in this stage as a telecentric lens was employed to eliminate any magnification of objects between images. Finally an unsupervised fuzzy clustering was applied at a working level, defined in Section 3.4, to produce isolated object regions. In the depth estimation stage, a depth model in a three-layered neural network, whose architecture is determined by depth feature extractions, was generated using a back-propagation algorithm with the training data derived from the previous stage at a low resolution level and camera calibration data. The basic framework of our approach is shown in Fig. 1. Firstly, the two defocused grey level images with the projected illumination pattern are segmented to decompose the scene into distinct meaningful regions. The resulting data are derived from the object regions at a low resolution in order to ease the burden of network learning and to reduce the uncertainty in object detection. To estimate the 3-D information, reliable feature vectors from the first stage, that are to be input to the nodes of the neural network, are selected to provide useful data related to depth. Finally, the ANN model is generated to perform the object depth recovery. After building the ANN model, it can be used to calculate the depth of objects in unseen or partly seen images, that is images that were not part of the training set. This work is the first reported in the literature to use neural networks to calculate the depth from a pair of defocused images. It is also novel in using an object detection stage followed by the depth recovery stage. Experimental results with different illumination patterns are demonstrated to show the effectiveness of the approach.

This paper is organized as follows: Section 2 overviews the theory of DFD and the derivation of the depth formulae. Section 3 gives a detailed illustration of the multiresolution techniques for 2-D object detection using the image segmentation and boundary forming algorithms on blurred images. Section 4 presents a MLP solution to object recovery using the pre-processed images and depth related features. Section 5 demonstrates the experimental results and includes discussion on the depth accuracy and the effect of varying the illumination pattern. The paper is concluded in Section 6.