Direct Model-Based Tracking of 3D Object Deformations in Depth and Color Video

Abstract

The tracking of deformable objects using video data is a demanding research topic due to the inherent ambiguity problems, which can only be solved using additional assumptions about the deformation. Image feature points, commonly used to approach the deformation problem, only provide sparse information about the scene at hand. In this paper a tracking approach for deformable objects in color and depth video is introduced that does not rely on feature points or optical flow data but employs all the input image information available to find a suitable deformation for the data at hand. A versatile NURBS based deformation space is defined for arbitrary complex triangle meshes, decoupling the object surface complexity from the complexity of the deformation. An efficient optimization scheme is introduced that is able to calculate results in real-time (25 Hz). Extensive synthetic and real data tests of the algorithm and its features show the reliability of this approach.

Appendices

Appendix A: Occlusion Handling

The ability of the algorithm to cope with holes in the input images (see Sect. 3.2) allows to implement a straight forward occlusion handling: pixel, supposed to occlude the actual object can be removed from the input data without leaving unfeasible data to the main tracking algorithm. Given the current movement of each vertex and under the assumption that it’s speed will at maximum be doubled in the current frame, it is possible to calculate a 3D region and a corresponding image area for each vertex, in which it (or its projection respectively) can be found in the current frame. Given the color error f _c for the vertices, we assume that most of the vertices have a color fit twice as high as the color error or lower.

These assumptions allow to formulate a two step occlusion classifier. In a first step, each pixel in the depth and color image is assigned one of these states:

• (1) Not classified

• (2) Depth and color are in the vicinity of at least one vertex

• (3) Only the depth value is in the vicinity of an object vertex but the color value does not fit

• (4) The pixel seems to be part of an occluding object

Starting with every pixel set to state (1), for every vertex v in V and every pixel [x,y] in the vicinity (as defined above) of the projection of v, the following rules are processed:

• If [x,y] is within the three dimensional vicinity of v according to its recent movement and the color of v is in the vicinity of the color at [x,y], then set the pixel state of [x,y] to (2).

• If the state of [x,y] is not (2) and [x,y] is within the three dimensional vicinity of v but the color of v is not in the vicinity of the color at [x,y], then set the pixel state of [x,y] to (3).

• If the state of [x,y] is not (2) the depth value of [x,y] is outside the vicinity of v such that it is in front of v, then set the pixel state of [x,y] to (4).

This procedure computes a classification into certain object pixels (2), uncertain pixels (3) and certain occlusion pixels (4). In a second step, every pixel classified as (3) in the vicinity of a pixel classified as (4) is also set to (4). Finally, every pixel classified as (4) is removed from the input data.

Appendix B: Training of Color/Depth Weights

A rather clumsy property of Jordt and Koch (2011) are the manually chosen weights for color and depth errors. Because of the novel formulation of (10) and (13), the depth error function does not have to be weighted, so the color weight λ _c (see Eq. (13)) is the only hyper parameter in this method. The selection of color/depth weights is a general problem that appears in the fusion of different errors from various domains to one fitness value, due to the lack of a common metric. Without additional information, a depth error value can not be compared to an error value in the color space. A fused error function simply adding up these error values in their current domain is likely to disregard one of the domains at hand.

A common tool to handle this problem is to define a certainty measure in each domain via the variance of a Gaussian distribution (Kim et al. 2009) or a cost function derived from it (Bartczak and Koch 2009). Once, every color and depth measurement is equipped with a cost or a distribution, a sound solution can be formulated by calculating the maximum likelihood of the given measurements or the cost minimum respectively. Although based on statistics, it can be shown that in the end an optimum calculated that way is equivalent to the minimum of the weighted squared error functions, given the correct weight.

Though it is possible to define reasonable variances for the Kinect color and depth measurements, the input data in the experiments of Sect. 6 show that it is more appropriate to weight color and depth information depending on the input data rather than defining static variances. Hence, we will adapt the weight λ _c according to the input data.

f _c (⋅) and f _d (⋅) both have a linear characteristic in respect to the number of “falsely matched” pixels and both functions yield 0 for a perfect fit and perfect input data. That means, that for the perfect deformation P ^∗, the error caused by noise in the depth and color image is f _c (P ^∗) and f _d (P ^∗). Following the principle of calculating a maximum likelihood based on variances calculated by these noise ratios, the resulting weight λ _c should equal $\frac {f_{d}(P^{*})}{f_{c}(P^{*})}$.

Although this definition can be statistically deduced, it is based on the assumption of P ^∗ being the perfect match, i.e. for every frame it is assumed that the preceding frames matched correctly. So in a situation as depicted in Fig. 7 (left column), in which the color information is valuable whereas the depth information is rather useless, f _c (P ^∗) is likely to yield increased error values, even for a good solution P ^∗, and f _d (⋅) remains constantly low. This would cause λ _c to decrease, leading to a fit much less influenced by the color, causing λ _c to decrease further. A similar example for an increasing λ _c can be found in Fig. 18, depicting the tracking of a white object on white background.

To counter this behavior, a second aspect is considered in addition to the noise levels: The information content of the error values. Due to the optimization process, f has already been evaluated in the vicinity of P ^∗. Let $\mathcal{P}$ be the set of individuals used by CMA-ES to calculate P ^∗ and let

$$ \begin{array}{@{}l} \displaystyle\mathrm{inf}_d(\mathcal{P}) := \sum_{P \in\mathcal{P}} \frac {f_d(P)}{\vert P \vert f_d(P^*)};\\[6mm] \displaystyle\mathrm{inf}_c(\mathcal{P}) := \sum _{P \in\mathcal{P}} \frac{f_c(P)}{\vert P \vert f_c(P^*)}. \end{array} $$

(15)

The resulting inf _c and inf _d are a measure for how valuable the error values were in determining the currently best fit in respect to their “noise levels”. Hence, the ratio $\frac {\mathrm{inf}_{c}}{\mathrm{inf}_{d}}$ is a measure on how valuable the color error is in relation to the depth error. Hence the color weight is updated after each iteration, such that:

$$ \lambda_c \rightarrow\frac{n-1}{n} \lambda_c + \frac{1}{n} \frac {\mathrm{inf}_c}{\mathrm{inf}_d}, $$

(16)

for n being the number of the current frame.