Reconstruction of Deformation from Depth and Color Video with Explicit Noise Models

Abstract

Depth sensors like ToF cameras and structured light devices provide valuable scene information, but do not provide a stable base for optical flow or feature movement calculation because the lack of texture information makes depth image registration very challenging. Approaches associating depth values with optical flow or feature movement from color images try to circumvent this problem, but suffer from the fact that color features are often generated at edges and depth discontinuities, areas in which depth sensors inherently deliver unstable data. Using deformation tracking as an application, this article will discuss the benefits of Analysis by Synthesis (AbS) while approaching the tracking problem and how it can be used to:

• exploit the complete image information of depth and color images in the tracking process,

• avoid feature calculation and, hence, the need for outlier handling,

• regard every measurement with respect to its accuracy and expected deviation.

In addition to an introduction to AbS based tracking, a novel approach to handle noise and inaccuracy is proposed, regarding every input measurement according to its accuracy and noise characteristics. The method is especially useful for time of flight cameras since it allows to take the correlation between pixel noise and the measured amplitude into account. A set of generic and specialized deformation models is discussed as well as an efficient way to synthesize and to optimize high dimensional models. The resulting applications range from real-time deformation reconstruction methods to very accurate deformation retrieval using models of 100 dimensions and more.