A novel hardware plane fitting implementation and applications for bionic vision

Abstract

This paper presents a high-speed real-time plane fitting implementation on a field-programmable gate array (FPGA) platform. A novel hardware-based least squares algorithm fits planes to patches of points within a depth image captured using a Microsoft Kinect v2 sensor. The validity of a plane fit and the plane parameters are reported for each patch of 11 by 11 depth pixels. The high level of parallelism of operations in the algorithm has allowed for a fast, low-latency hardware implementation on an FPGA that is capable of processing depth data at a rate of 480 frames per second. A hybrid hardware–software end-to-end system integrates the hardware solution with the Kinect v2 sensor via a computer and PCI express communication link to a Terasic TR4 FPGA development board. We have also implemented two proof-of-concept object detection applications as future candidates for bionic vision systems. We show that our complete end-to-end system is capable of running at 60 frames per second. An analysis and characterisation of the Kinect v2 sensor errors has been performed in order to specify logic precision requirements, statistical testing of the validity of a plane fit, and achievable plane fitting angle resolution.

Appendix: Derivation of resultant error models of normalised plane parameters

Having found the mean and standard deviation of depth value error (Sect. 3.1), a model for the error in the parameters of our plane fitting algorithm can be developed. As shown in Sect. 2 which discusses our plane fitting implementation, for a plane with the form $z = Au + Bv + C$, the parameters A, B, and C can be found using the simplified equations from (6). The z values are assumed to be mutually independent normal random variables and so the expectation, variance and standard deviation of the parameters can be found as follows. A, B and C can be represented as a linear combinations of $z_i$:

$$\begin{aligned} A= & {} \frac{1}{\sum _{i=1}^{m}u_i^2}\left( \sum _{i=1}^{m}u_iz_i\right) \nonumber \\ B= & {} \frac{1}{\sum _{i=1}^{m}v_i^2}\left( \sum _{i=1}^{m}v_iz_i\right) \\ C= & {} \frac{1}{n}\sum _{i=1}^{m}z_i\nonumber \end{aligned}$$

(47)

Therefore, assuming $z_i = (z_\mathrm{true} + e_i )$, where $e_i\sim {}N(0,\sigma _z)$, all estimates of A, B, and C will have zero mean errors and variances as follows:

$$\begin{aligned}&\sigma _A^2 = \left[ \frac{1}{\sum _{i=1}^{m}u_i^2}\right] ^2\left( \sum \limits _{i=1}^{m}u_i^2\sigma _i^2\right) = \frac{\sigma _z^2}{\sum _{i=1}^{m}u_i^2}\nonumber \\&\sigma _B^2 = \left[ \frac{1}{\sum _{i=1}^{m}v_i^2}\right] ^2\left( \sum \limits _{i=1}^{m}v_i^2\sigma _i^2\right) = \frac{\sigma _z^2}{\sum _{i=1}^{m}v_i^2} \nonumber \\&\sigma _C^2 = \frac{1}{n^2}\sum \limits _{i=1}^{m}\sigma _i^2 = \frac{\sigma _z^2}{n}\nonumber \\&\mathrm{and}~\mathrm{standard}~\mathrm{deviations}~\sigma _A,~\sigma _B~\mathrm{and}~\sigma _C \end{aligned}$$

(48)

From Eq. (9) the standard deviations of A and B can be expressed in terms of k as follows:

$$\begin{aligned} \sigma _A= & {} \frac{\sigma _z}{k\sqrt{6}}\nonumber \\ \sigma _B= & {} \frac{\sigma _z}{k\sqrt{6}}\nonumber \\ \sigma _C= & {} \frac{\sigma _z}{3} \end{aligned}$$

(49)

From Eq. (49) it is apparent that the standard deviation of errors scales inversely with patch size. Therefore larger patch sizes reduce errors. From Eq. (19) we can see that $\alpha$ and $\beta$ are scaled versions of A and B and so the variances follow as scaled versions of Eq. (48):

$$\begin{aligned}&\sigma _\alpha ^2 = \mathrm{Var}[-AF] = F^2\frac{\sigma _z^2}{\sum _{i=1}^{m}u_i^2} = F^2\frac{\sigma _z^2}{6k^2}\nonumber \ \\&\sigma _\beta ^2 = \mathrm{Var}[-BF] = F^2\frac{\sigma _z^2}{\sum _{i=1}^{m}v_i^2} = F^2\frac{\sigma _z^2}{6k^2}\\&\mathrm{and}~\mathrm{standard}~\mathrm{deviations}~\sigma _\alpha ~\mathrm{and}~\sigma _\beta \nonumber \end{aligned}$$

(50)

To find the variance of $\gamma$ we need to expand Eq. (19) as follows, so that all correlated terms are grouped.

$$\begin{aligned} \gamma= & {} C + Au_c + Bv_c \nonumber \\= & {} \frac{\sum _{i=1}^{m}z_i}{n} + u_c\frac{\sum {(\mathrm{column}\,3)} - \sum {(\mathrm{column}\,1)}}{30}\nonumber \\&+\, v_c\frac{\sum {(\mathrm{row}\,3)} - \sum {(\mathrm{row}\,1)}}{30} \nonumber \\= & {} z_1\left( \frac{1}{9} + \frac{u_c}{30} - \frac{v_c}{30}\right) + z_2\left( \frac{1}{9} - \frac{v_c}{30}\right) + z_3\left( \frac{1}{9} + \frac{u_c}{30} - \frac{v_c}{30}\right) \nonumber \\&+\, z_4\left( \frac{1}{9} - \frac{u_c}{30}\right) + z_5\left( \frac{1}{9}\right) + z_6\left( \frac{1}{9} + \frac{u_c}{30}\right) \nonumber \\&+\, z_7\left( \frac{1}{9} - \frac{u_c}{30} + \frac{v_c}{30}\right) + z_8\left( \frac{1}{9} + \frac{v_c}{30}\right) \nonumber \\&+\, z_9\left( \frac{1}{9} - \frac{u_c}{30} + \frac{v_c}{30}\right) \end{aligned}$$

(51)

Now if we choose the worst case values for $u_c$ and $v_c$ of 256 and 212, respectively, then the variance can be expressed as follows:

$$\begin{aligned} \sigma _\gamma ^2= & {} \sigma _z^2\left( \frac{-697}{45}\right) ^2 + \sigma _z^2\left( \frac{-313}{45}\right) ^2 + \sigma _z^2\left( \frac{71}{45}\right) ^2 \nonumber \\&+\, \sigma _z^2\left( \frac{-379}{45}\right) ^2 + \sigma _z^2\left( \frac{1}{9}\right) ^2 + \sigma _z^2\left( \frac{389}{45}\right) ^2 \nonumber \\&+\, \sigma _z^2\left( \frac{-61}{45}\right) ^2 + \sigma _z^2\left( \frac{323}{45}\right) ^2 + \sigma _z^2\left( \frac{707}{45}\right) ^2 \nonumber \\= & {} \sigma _z^2\left( \frac{33149}{45}\right) \approx (27.14\sigma _z)^2 \end{aligned}$$

(52)

For the variance of $\kappa$, we use the following approximation:

$$\begin{aligned}&\mathrm{For}~\mathrm{some}~f = aX^b \nonumber \\&\sigma _f^2\approx (abX^{b-1}\sigma _X)^2 = \left( \frac{fb\sigma _X}{X}\right) ^2 \end{aligned}$$

(53)

Giving us the following variance for $\kappa$:

$$\begin{aligned} \sigma _\kappa ^2\approx (2C\sigma _C)^2 = \left( \frac{2C\sigma _z}{3}\right) ^2 \end{aligned}$$

(54)

From Eqs. (50, 52, 54) we can now define standard deviations for the four plane parameters $\alpha$, $\beta$, $\gamma$ and $\kappa$:

$$\begin{aligned} \sigma _\alpha= & {} F\frac{\sigma _z}{k\sqrt{6}}\nonumber \\ \sigma _\beta= & {} F\frac{\sigma _z}{k\sqrt{6}} \\ \sigma _\gamma= & {} \sigma _z\sqrt{\frac{33149}{45}} \approx 27.14\sigma _z \nonumber \\ \sigma _\kappa\approx & {} \frac{2C\sigma _z}{3}\nonumber \end{aligned}$$

(55)

To find the variances of the normalised plane parameters $\alpha _n$, $\beta _n$, $\gamma _n$ and $\kappa _n$, we can use the following approximation:

$$\begin{aligned}&\mathrm{For}~\mathrm{some}~f = \frac{X}{Y}\nonumber \\&\sigma _f^2\approx f^2\left[ \left( \frac{\sigma _X}{X}\right) ^2 + \left( \frac{\sigma _Y}{Y}\right) ^2 - 2\frac{\sigma _{XY}}{XY}\right] \end{aligned}$$

(56)

We need to define standard deviation equations for $\lambda$ first. From Eq. (22), we know that $\lambda = |\alpha | + |\beta | + |\gamma |$. Therefore there are eight different cases for evaluating variance, as each of $\alpha$, $\beta$, $\gamma$ could have a positive or negative result. It can be shown, however, that there are only four unique cases, and furthermore we are only interested in the best and worst case variances. This leaves the two following cases for $\lambda$:

$$\begin{aligned} \lambda _1= & {} (\alpha ) + (\beta ) + (\gamma ) = -FA -FB + (C + Au_c + Bv_c)\nonumber \\ \lambda _2= & {} (\alpha ) + (\beta ) + (-\gamma ) = -FA -FB - (C + Au_c + Bv_c)\nonumber \\ \end{aligned}$$

(57)

Using a similar method as was done for $\sigma _{\gamma }^2$ in Eqs. (51, 52), we can find the following variance equations for the two cases of $\lambda$:

$$\begin{aligned} \sigma _{\lambda _1}^2= & {} \frac{{\sigma _z^2}53749}{225} \approx (15.456\sigma _z)^2 \nonumber \\ \sigma _{\lambda _2}^2= & {} \frac{{\sigma _z^2}1081477}{225} \approx (69.329\sigma _z)^2 \end{aligned}$$

(58)

We also need to find the covariances of each of the plane parameters $\alpha$, $\beta$, $\gamma$ and $\kappa$ with the scaling factor $\lambda$. This can be done through the products of each, removing the non-correlated cross-terms. The following covariances can be defined:

$$\begin{aligned}&\mathrm{For}~\mathrm{best}~\mathrm{case}~\mathrm{of}~\sigma _{\lambda _1} = 15.456\sigma _z, \nonumber \\&\sigma _{\alpha \lambda _1} = \frac{1342\sigma _z^2}{5} \approx 16.38^2\sigma _z^2 \nonumber \\&\sigma _{\beta \lambda _1} = \frac{9394\sigma _z^2}{25} \approx 19.38^2\sigma _z^2 \\&\sigma _{\gamma \lambda _1} = -\frac{91187\sigma _z^2}{225} \approx -20.13^2\sigma _z^2 \nonumber \\&\sigma _{\kappa \lambda _1} = \frac{\sigma _z^3}{81}\nonumber \end{aligned}$$

(59a)

$$\begin{aligned}&\mathrm{For}~\mathrm{worst}~\mathrm{case}~\mathrm{of}~\sigma _{\lambda _2} = 69.329\sigma _z,\nonumber \\&\sigma _{\alpha \lambda _2} = \frac{37942\sigma _z^2}{25} \approx 38.96^2\sigma _z^2 \nonumber \\&\sigma _{\beta \lambda _2} = \frac{35258\sigma _z^2}{25} \approx 37.55^2\sigma _z^2\\&\sigma _{\gamma \lambda _2} = -\frac{422677\sigma _z^2}{225} \approx -43.34^2\sigma _z^2 \nonumber \\&\sigma _{\kappa \lambda _2} = -\frac{\sigma _z^3}{81}\nonumber \end{aligned}$$

(59b)

Substituting Eqs. (55, 58, 59a, 59b) into Eq. (56) gives the following equations for the variances of the normalised plane parameters $\alpha _n$, $\beta _n$, $\gamma _n$ and $\kappa _n$:

$$\begin{aligned}&\mathrm{For}~\mathrm{best}~\mathrm{case}~\mathrm{of}~\sigma _{\lambda _1} = 15.456\sigma _z,\nonumber \\&\sigma _{\alpha _n}^2 \approx \sigma _z^2\left( \frac{\alpha }{\lambda _1}\right) ^2\left[ \frac{29.88^2}{\alpha ^2} + \frac{15.45^2}{\lambda _1^2} - \frac{23.17^2}{\alpha \lambda _1}\right] \nonumber \\&\sigma _{\beta _n}^2 \approx \sigma _z^2\left( \frac{\beta }{\lambda _1}\right) ^2\left[ \frac{29.88^2}{\beta ^2} + \frac{15.45^2}{\lambda _1^2} - \frac{27.41^2}{\beta \lambda _1}\right] \\&\sigma _{\gamma _n}^2 \approx \sigma _z^2\left( \frac{\gamma }{\lambda _1}\right) ^2\left[ \frac{27.14^2}{\gamma ^2} + \frac{15.45^2}{\lambda _1^2} + \frac{28.47^2}{\gamma \lambda _1}\right] \nonumber \\&\sigma _{\kappa _n}^2 \approx \sigma _z^2\left( \frac{\kappa }{\lambda _1}\right) ^2\left[ \frac{(0.66C)^2}{\kappa ^2} + \frac{15.45^2}{\lambda _1^2} - \frac{0.024\sigma _z}{\kappa \lambda _1}\right] \nonumber \end{aligned}$$

(60a)

$$\begin{aligned}&\mathrm{For}~\mathrm{worst}~\mathrm{case}~\mathrm{of}~\sigma _{\lambda _2} = 69.329\sigma _z, \nonumber \\&\sigma _{\alpha _n}^2 \approx \sigma _z^2\left( \frac{\alpha }{\lambda _2}\right) ^2\left[ \frac{29.88^2}{\alpha ^2} + \frac{69.33^2}{\lambda _2^2} - \frac{55.09^2}{\alpha \lambda _2}\right] \nonumber \\&\sigma _{\beta _n}^2 \approx \sigma _z^2\left( \frac{\beta }{\lambda _2}\right) ^2\left[ \frac{29.88^2}{\beta ^2} + \frac{69.33^2}{\lambda _2^2} - \frac{53.11^2}{\beta \lambda _2}\right] \\&\sigma _{\gamma _n}^2 \approx \sigma _z^2\left( \frac{\gamma }{\lambda _2}\right) ^2\left[ \frac{27.14^2}{\gamma ^2} + \frac{69.33^2}{\lambda _2^2} + \frac{61.3^2}{\gamma \lambda _2}\right] \nonumber \\&\sigma _{\kappa _n}^2 \approx \sigma _z^2\left( \frac{\kappa }{\lambda _2}\right) ^2\left[ \frac{(0.66C)^2}{\kappa ^2} + \frac{69.33^2}{\lambda _2^2} + \frac{0.024\sigma _z}{\kappa \lambda _2}\right] \nonumber \end{aligned}$$

(60b)