A non-parametric depth modification model for registration between color and depth images

Abstract

Despite its most popularity among all depth cameras in the computer vision applications, the Microsoft Kinect sensor suffers from low depth accuracy. In this work we propose a novel non-parametric depth modification model to improve the depth accuracy of the Kinect sensor by iteratively registering depth images and color images. In particular, we first establish a coarse correspondence based on the feature descriptor of the canny edge at each iteration, and estimate the fine correspondence using an $L_2E$ algorithm. We utilize the non-parametric Gaussian mixture model to replace the Gaussian single model and build the regularization term to constrain the correlations between functions. Then, based on the correspondence results, the depth data are corrected and optimized. Extensive experiments have been performed to verify the effectiveness of the proposed approach, and the results have demonstrated that our method is able to greatly enhance the depth accuracy of the Kinect sensor compared with baseline methods.

Appendix

1.1 Proof of Eq. (3)

We have $F(\epsilon ) = \sum _{k = 1}^{4}w_k \phi (\epsilon |0,\sigma _{k}^2)$, where $\phi (\epsilon |0,\sigma _{k}^2) =\phi (\epsilon |0,\sigma _k^2) = \frac{1}{\sqrt{2\pi }\sigma _k}e^{(-\frac{\epsilon ^2}{2\sigma _k^2})}$, and $w_k$ denotes the possibility of a match point belonging to a match function. Suppose $\phi _1 = \phi (\epsilon |0,\sigma _1^2), \phi _2 = \phi (\epsilon |0,\sigma _2^2), \phi _3 = \phi (\epsilon |0,\sigma _3^2), \phi _4 = \phi (\epsilon |0,\sigma _4^2)$. We assume $M = \int (F(\epsilon ))^2d\epsilon$, and then

$$\begin{aligned} M&=\int \left( \sum _{k = 1}^{4}w_k \phi (\epsilon |0,\sigma _k^2)\right) ^2d\epsilon \nonumber \\&= \sum _{k = 1}^{4}\int w_k^2\phi _k^2d\epsilon + 2\sum _{i,j=1,i\ne j}^{4}w_iw_j\int \phi _i\phi _jd\epsilon \nonumber \\&=\sum _{k=1}^4\frac{w_k^2}{2^d(\pi \sigma _k)^{\frac{d}{2}}} + 2\sum _{i,j=1,i\ne j}^{4}w_iw_j\int \phi _i\phi _jd\epsilon . \end{aligned}$$

(12)

According to Wand and Jones (1994), we can get the following formulation:

$$\begin{aligned} M&=\sum _{k=1}^4\frac{w_k^2}{2^d(\pi \sigma _k)^{\frac{d}{2}}} +2\sum _{i,j=1,i\ne j}^{4}w_iw_j \phi (0|0,\sigma _i^2+\sigma _j^2).\nonumber \\&=\sum _{k=1}^4\frac{w_k^2}{2^d(\pi \sigma _k)^{\frac{d}{2}}} + A. \end{aligned}$$

(13)

where A is a constant, and we can omit it. Then we can get the following formulation:

$$\begin{aligned} \int (F(\epsilon ))^2d\epsilon&=\sum _{k=1}^4\frac{w_k^2}{2^d(\pi \sigma _k)^{\frac{d}{2}}}. \end{aligned}$$

(14)

1.2 Vector-valued reproducing Kernel Hilbert space

We review the basic theory of vector-valued reproducing kernel Hilbert space here, and for further details and references please see Carmeli et al. (2006), Micchelli and Pontil (2005).

Let X be a set, i.e.$X \subseteq {\mathbb {R}}^P$, Y a real Hilbert space with the inner product (norm) $\langle \cdot ,\cdot \rangle ,(\Vert \cdot \Vert )$, i.e.$y \subseteq {\mathbb {R}}^D$, and H a Hilbert space with the inner product (norm) $\langle \cdot ,\cdot \rangle _H,(\Vert \cdot \Vert _H)$, where $P=D=$2 or 3 for point matching problem. Note that a norm can be induced by an inner product, i.e.$\forall f \in H,\Vert f\Vert _H = \root \of {\langle f,f \rangle _H}$. And a Hilbert space is a real or complex inner product space that is also a complete metric space with respect to the distance function induced by the inner product. Thus a vector-valued RKHS can be defined as follows.

Definition 1

A Hilbert space H is an RKHS if the evaluation maps $ev_x:H \rightarrow Y(i.e.,ev_x(f)= f(x))$ are bounded; i.e.$\forall x \in X$ and there exists a positive constant $C_x$ such that

$$\begin{aligned} \Vert ev_x(f)\Vert = \Vert f(x)\Vert \le C_x\Vert f\Vert _H, \quad \forall f \in H. \end{aligned}$$

(15)

A reproducing kernel $\varGamma :X \times X \rightarrow B(Y)$ is then defined as $\varGamma (x,x'):=ev_xev_{x'}^*$, where B(Y) is the Banach space of bounded linear operators (i.e., $\varGamma (x,x'),\forall x,x' \in X)$ on Y, i.e.$B(Y) \subseteq {\mathbb {R}}^{D\times D}$, and $ev_x^*$ is the adjoint of $ev_x$. We have the following two properties about the RKHS and kernel.

Remark 1

The kernel $\varGamma$ reproduces the value of a function $f\in H$ at a point $x \in X$. Indeed, given $\forall x \in X$ and $y \in Y$, we have $ev_x^*y=\varGamma (\cdot ,x)y$, so that $\langle f(x),y \rangle = \langle f,\varGamma (\cdot ,x)y\rangle _H$.

Remark 2

An RKHS defines a corresponding reproducing kernel. Conversely, a reproducing kernel defines a unique RKHS.

More specifically, for any $M \in {\mathbb {N}}, \lbrace x_i \rbrace _{i=1}^M \subseteq X$, and a reproducing kernel $\varGamma$, a unique RKHS can be defined by considering the completion of the space:

$$\begin{aligned} H_M=\left\{ \sum _{i=1}^M\varGamma (\cdot ,x_i)c_i:c_i \in Y\right\} , \end{aligned}$$

(16)

with respect to the norm induced by the inner product

$$\begin{aligned} \langle f,g \rangle _H=\sum _{i,j=1}^M \langle \varGamma (x_j,x_i)c_i,d_j\rangle , \quad \forall f,g \in H_M, \end{aligned}$$

(17)

where $f = \sum _{i=1}^M \varGamma (\cdot ,x_i)c_i$ and $g = \sum _{j=1}^M \varGamma (\cdot ,x_j)d_j$.