A pointwise smooth surface stereo reconstruction algorithm without correspondences

doi:10.1016/j.imavis.2012.06.003

Image and Vision Computing

Volume 30, Issue 9, September 2012, Pages 619-629

https://doi.org/10.1016/j.imavis.2012.06.003 Get rights and content

Abstract

This paper describes an algorithm for 3D reconstruction of a smooth surface with a relatively dense set of self-similar point features from two calibrated views. We bypass the usual correspondence problem by triangulating a point in space from all pairs of features satisfying the epipolar constraint. The surface is then extracted from the resulting point cloud by taking advantage of the statistical and geometric properties of the point distribution on the surface. Results are presented for computer simulations and for a laboratory experiment on a silicon gel phantom used in a breast cancer screening project.

Graphical abstract

Highlights

► 3D surface reconstruction from 2D point sets without correspondences. ► Surface extraction from space of all potential epipolar reconstructions. ► Application to breast cancer screening.

Introduction

Why write another article on yet another method for 3D reconstruction from digital image pairs? This article addresses a specific type of problem that may be encountered in some applications for which standard techniques are not appropriate or are difficult to use. The problem is that of reconstructing in 3D a relatively dense set of specific points on a smooth surface from two calibrated views, but in a situation where solving the correspondence problem is difficult due to a high degree of self-similarity between the image features.

The motivation for this research came from a problem in medical imaging. In the DIET (Digital Image-based Elasto-Tomography) project [1], it is necessary to reconstruct in 3D the motion of a large set of points on a human breast surface as it is being mechanically vibrated. The motion of specific points on the surface is required, rather than a description of the surface motion as a whole, because this point motion is required to solve an inverse problem for the 3D distribution of internal elasticity of the breast tissue.

Because in this type of application, specific feature points are required that can be tracked as the surface deforms, volume-based methods [2], [3], [4] based on visual hulls that do not measure the location of trackable individual surface points are not appropriate. Likewise, state of the art stereo methods such as graph cuts [5], [6] and belief propagation [5], [7] that are based on Markov Random Fields are also inappropriate, as they are concerned with creating a 3D reconstruction of a scene by computing a pixel-by-pixel depth map, and once again do not compute the precise locations of trackable individual surface points. For the DIET problem small artificial fiducials are randomly applied to the breast surface. For practical reasons these features are essentially identical in appearance, and hence difficult to correspond using standard interest-point detectors [8], [9], [10]. Dense stereo methods [11], [12] relying on accurate correspondence of a number of key features are therefore also not well-suited to this application. Dellaert et. al. [13] have developed an algorithm for computing feature point correspondences without any a priori information by computing the maximum likelihood estimate of the scene and cameras using an EM algorithm. This method is potentially a promising alternative approach, however significant work would be required to modify the algorithm to incorporate large numbers of occluded features and an arbitrary number of world features.

The types of application envisioned for the algorithm presented herein are those where a (relatively) dense set of self-similar points on a smooth surface needs to be reconstructed in 3D from fully calibrated cameras. Another application might be in surveying, for example, where landmarks such as trees or plants are the feature points, and the goal is to reconstruct the topography on which they lie from multiple aerial views.

Section snippets

Camera model

The computer vision notation used follows that of Hartley and Zisserman [14], a standard reference for multiple view geometry. In this paper, cameras are fully calibrated, and a 3 × 4 camera projection matrix P is assumed to be of the form $P = K [\begin{array}{c} R & t \end{array}]$ where $R \in R^{3 \times 3}$ is a rotation matrix, $t \in R^{3}$ a 3-vector, and $K \in R^{3 \times 3}$ an upper triangular matrix representing the internal parameters of the camera. The projection, in homogeneous coordinates, from world coordinates X = (X, Y, Z, W)^┬ to image coordinates u = (u, v, w)^┬ is

Problem Definition

Let $S \subset R^{3}$ be the surface in Euclidean space to be reconstructed and let $X = \{X_{1}, X_{2}, \dots, X_{N}\}$ be the set of feature points on $S$ . The feature points are points on the surface that can be reliably extracted and localised in images, but are indistinguishable in appearance from one another. Make the following two assumptions about $S$ :

Assumption 1 Smoothness

The surface $S$ has continuous directional derivatives, and the maximum principal curvature over the entirety of $S$ is bounded by some known constant κ

Assumption 2 Feature distribution

The feature points are

Test surface

A number of experiments were performed using simulated data. For a test surface, the function $\begin{array}{l} z = α c o s (3 π) x s i n (3 π y), & x, y \in [- 0.5, 0.5] \end{array}$ was used. Points were randomly generated on the surface by choosing x and y from U(− 0.5, 0.5) and generating the corresponding z values. This gives a slight irregularity to the surface, but this effect is not large. This function has maximum principal curvature given by $κ_{max} = 9 π^{2} α$

Implentation Details

All code in this section was written purely in MATLAB and experiments were run on a single 1.2

Discussion

The results in the case study were produced with a MATLAB implementation of the algorithm, executed on a 2.53 GHz dual-core laptop with 4 GB RAM. The MATLAB version was R2011a in a 64-bit Linux environment. With this implementation, the full surface reconstruction procedure for the DIET system takes around 11 seconds, not including the extraction of the point features by threshold segmentation. There are numerous optimisations, not implemented here, that could be made to improve the efficiency of

Acknowledgements

Thanks to the Tertiary Education Commission (TEC) for providing funding through a Top Achiever Doctoral Scholarship for RGB. Thanks also to the anonymous referees for constructive reviews and for suggesting references and improvements to the content and presentation.

References (18)

A. Peters et al.
Digital image-based elasto-tomography: first experiments in surface based mechanical property estimation of gelatine phantoms
JSME Int. J.
(2005)
A. Laurentini
The visual hull concept for silhouette-based image understanding
IEEE Trans. Pattern Anal. Mach. Intell.
(1994)
W. Matusik et al.
Polyhedral visual hulls for real-time rendering
M. Li et al.
Hardware-accelerated visual hull reconstruction and rendering
M. Tappen et al.
Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters
V. Kolmogorov et al.
Multi-camera scene reconstruction via graph cuts
J. Sun et al.
Stereo matching using belief propagation
D. Lowe
Distinctive image features from scale-invariant keypoints
Int. J. Comput. Vision
(2004)
S. Smith et al.
SUSANA new approach to low level image processing
Int. J. Comput. Vision
(1997)

There are more references available in the full text version of this article.

Cited by (11)

Silicone phantom validation of breast cancer tumor detection using nominal stiffness identification in digital imaging elasto-tomography (DIET)
2018, Biomedical Signal Processing and Control
Citation Excerpt :
The cameras and actuator are synchronized by strobing a set of LEDs at specific phases in the cycle. Consecutive sets of 2D images are converted into a 3D description to measure the surface displacement of each reference point between frames [21,28]. The DIET system is intended to be compact and can be easily deployed, event in remote areas, where people have little access to the breast cancer screen facilities [27].
Cancerous tumors exhibit considerably higher stiffness than healthy breast tissues. This research develops a hysteresis loop analysis detection method for breast cancer diagnosis, which is validated using silicone phantom tests to provide an initial proof of concept of the overall DIET diagnostic approach.
Steady state sinusoidal displacements induced by mechanical actuation are captured using the DIET system for 4 silicone breast phantoms (1 homogeneous “healthy”; 3 with 10–20 mm stiffer inclusion “tumors” at different locations). Hysteresis loops for each measured reference point on the breast surface (N≈14,500/phantom) are constructed from the measured displacements and a calculated mass normalized restoring force. The distribution of nominal elastic stiffness over breast surface is estimated using a hypothesis test and regression analysis. A small motion threshold, Δd, is used to reduce errors in identified elastance and its values determined from sensitivity analysis.
Sensitivity analysis shows displacement reconstruction errors from very small motions can be eliminated using a threshold Δd = 6 μm to clearly reduce errors in identified nominal elastance. Results for all 4 phantoms indicate stiffer inclusions induce a greater number of reference points with stiffness ≥2 × higher in the segment with the inclusion than the nominal stiffness in the “healthy” segments in the x, y and total net 2D displacement directions. Therefore, the presence of a stiffer “tumor” inclusion is automatically detected and located by comparing the nominal stiffness to a calculated threshold, k_t. Inclusion segments are finally localized algorithmically using an inclusion indicator, K_index.
The overall results validate the capability of the proposed method to accurately detect and locate a stiff inclusion in typical, representative silicone breast phantoms without misidentifying other regions or a healthy no-inclusion phantom in the non-invasive DIET system. The methods are entirely generalizable to human clinical data from a DIET system, and the results justify such clinical trials.
Nominal Stiffness Identification for Tumor Detection of Women Breast in a Digital Image Elasto Tomography (DIET) System
2017, IFAC-PapersOnLine
This research develops a hysteresis loop analysis (HLA) based method for breast cancer diagnosis in a Digital Imaging Elasto-tomography (DIET) system. Dynamic displacements induced by mechanical actuation for 4 silicone breast phantoms (1 homogeneous healthy, 3 with 10-20mm stiffer inclusion “tumors”) are captured using the DIET system. Hysteresis loops for each measured reference point across the breast surface are reconstructed using the measured displacement and a calculated mass normalized restoring force. The distribution of the nominal elastic stiffness over the breast surface is estimated using an F-type hypothesis test and regression analysis. A higher stiffness is identified in the region of the inclusions, which is at least 2x greater than the nominal stiffness in other areas of the phantom. Thus, the method accurately detects and locates the inclusion in typical, representative silicone breast phantoms without misidentifying other regions or a healthy no-inclusion phantom. The overall results show the capability of the proposed method, based on the identification of the local nominal stiffness over the phantom surface as an efficient index, to accurately detect inclusion presence and location in a rapid, objective fashion using the non-invasive DIET approach.
Sensitivity Analysis for Stiffness Identification Using a DIET Breast Cancer Screening System
2017, IFAC-PapersOnLine
Cancerous tumors exhibit significantly higher stiffness than healthy breast tissues. a hysteresis loop analysis method is used to identify the nominal elastic stiffness in different horizontal directions for breast cancer diagnosis in a Digital Imaging Elasto-tomography (DIET) screening system. The sensitivity of the method to measurement errors from the DIET system is first evaluated using a homogeneous healthy phantom with different camera angles. The identification method is then validated against 4 phantoms with stiffer inclusions of 10-20mm diameter over a range of actuation frequencies. Results show displacement reconstruction errors from very small motion can be eliminated by using a threshold ∆d to significantly reduce errors in identified nominal elastane. The stiffer inclusion induced a greater number of reference points with higher stiffness in the inclusion segment than other segments in x and y directions. The presence of a stiffer “tumor” inclusion is detected by comparing the nominal stiffness to an automatically defined threshold, k_t. Inclusion segments can then be localized using an inclusion indicator, K_index. The overall results validate the method is capable of providing robust estimations of nominal stiffness over a realistic range of actuation frequencies, and inclusion sizes and angle for the DIET concept.
A Surface Vibration-based Method for Tumor Detection of Women Breast in a DIET System
2017, Procedia Engineering
This paper presents a hysteresis loop analysis (HLA) method based on dynamic force-displacement relationship to detect and localize cancerous tumors that exhibit significantly higher stiffness than soft healthy tissues in women breast. Steady state sinusoid displacements induced by mechanical actuation for 2 silicone phantoms (1 homogeneous “healthy”; 1 with 20mm stiffer inclusion “tumor”) are captured using a non-invasive Digital Image Elasto Tomography (DIET) system. Hysteresis loops for each measured reference point across the phantom surface are reconstructed using the measured displacement and a calculated mass normalized restoring force. The distribution of the elastic nominal stiffness over the breast surface is estimated using an F-type hypothesis test and regression analysis. The sensitivity of the method to measurement errors from the DIET system is evaluated using a homogeneous healthy phantom with different camera angles. Results show displacement reconstruction errors from very small motion can be eliminated by using a threshold ∆d to significantly reduce errors in identified nominal elastance. The stiffer inclusion induced a greater number of reference points with higher stiffness in the inclusion segment than other segments in the x and y directions, while a consistent distribution of stiffness for the homogeneous control phantom was identified. The overall results show the capability of the method to accurately detect and locate the inclusion in typical, representative silicone breast phantoms without misidentifying other regions or a healthy no-inclusion phantom.
Structural health monitoring of tissue mechanics for non-invasive diagnosis of breast cancer
2018, At-Automatisierungstechnik
Finite element modelling and validation for breast cancer detection using digital image elasto-tomography
2018, Medical and Biological Engineering and Computing

View all citing articles on Scopus

^☆: This paper has been recommended for acceptance by Daniel Rueckert, PhD.

View full text

Image and Vision Computing

A pointwise smooth surface stereo reconstruction algorithm without correspondences☆

Abstract

Graphical abstract

Highlights

Introduction

Section snippets

Camera model

Problem Definition

Test surface

Implentation Details

Discussion

Acknowledgements

Digital image-based elasto-tomography: first experiments in surface based mechanical property estimation of gelatine phantoms

JSME Int. J.

The visual hull concept for silhouette-based image understanding

IEEE Trans. Pattern Anal. Mach. Intell.

Polyhedral visual hulls for real-time rendering

Hardware-accelerated visual hull reconstruction and rendering

Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters

Multi-camera scene reconstruction via graph cuts

Stereo matching using belief propagation

Distinctive image features from scale-invariant keypoints

Int. J. Comput. Vision

SUSANA new approach to low level image processing

Int. J. Comput. Vision

Silicone phantom validation of breast cancer tumor detection using nominal stiffness identification in digital imaging elasto-tomography (DIET)

Nominal Stiffness Identification for Tumor Detection of Women Breast in a Digital Image Elasto Tomography (DIET) System

Sensitivity Analysis for Stiffness Identification Using a DIET Breast Cancer Screening System

A Surface Vibration-based Method for Tumor Detection of Women Breast in a DIET System

Structural health monitoring of tissue mechanics for non-invasive diagnosis of breast cancer

Finite element modelling and validation for breast cancer detection using digital image elasto-tomography