An efficient surface registration using smartphone

Abstract

Gathering 3D object information from the multiple spatial viewpoints typically brings up the problem of surface registration. More precisely, registration is used to fuse 3D data originally acquired from different viewpoints into a common coordinate system. This step often requires the use of relatively bulky and expensive robot arms (turntables) or presents a very challenging problem if constrained to software solutions. In this paper we present a novel surface registration method, motivated by an efficient and user-friendly implementation. Our system is inspired by the idea that three out of generally six registration parameters (degrees of freedom) can be provided in advance, at least to some degree of accuracy, by today’s smartphones. Experimental evaluations demonstrated the successful point cloud registrations of $\sim$10,000 points in a matter of seconds. The evaluation included comparison with state-of-the-art descriptor methods. The method’s robustness was also studied and the results using 3D data from a professional scanner showed the potential for real-world applications.

Appendix: Computing orientation from the accelerometer and magnetometer measurements

For the completeness of the proposed method, in this section we provide a general computational framework of how to compute orientation data directly from both the accelerometer and the magnetometer measurements. For convenience we have proposed our method using a smartphone. However, there may be researches working in the applications where accelerometer and magnetometer are already provided as stand-alone pieces of hardware and which may be then preferred over a smartphone. It is useful to note that the orientation sensor is basically a so-called logical sensor, what means that the output data delivered by the sensor are the result of data gathered by the physical sensor, e.g., accelerometer and magnetometer, including some processing and filtering. Some professional orientation sensors include additional physical sensors such as gyroscopes for data stability under dynamic conditions [35]. However, for our work, a simple accelerometer and magnetometer will suffice. Accelerometers are devices sensitive to the difference between the linear acceleration of the sensor and the local gravitational field, whereas magnetometers measure the components of earth’s magnetic field.

We define an orientation sensor (phone) coordinate system as depicted in Fig. 2 (for simplicity a smartphone is shown again, but it should be clear that generally it can be any device containing an accelerometer and a magnetometer). It is also assumed that the accelerometer and magnetometer, contained inside the physical sensor, are aligned with the sensor coordinate system. It is convenient to define a sensor reference position using gravitational and magnetic field vectors (Fig. 2), where the sensor in its reference position is positioned flat and facing northwards. Consequently, all sensor rotations are expressed with respect to that reference. With this assumption, a three-axis accelerometer in the earth’s gravitational field, and undergoing no linear acceleration, will have the output $\mathbf{G}_{\mathbf{p}}$ given by:

$$\begin{aligned} G_\mathrm{p}= \left[ \begin{array}{l} G_x\\ G_y\\ G_z\\ \end{array}\right] =R_\mathrm{S} \cdot \left[ \begin{array}{l} 0\\ 0\\ g\\ \end{array}\right] \end{aligned}$$

(6)

where rotation matrix $\mathbf{R}_{\mathbf{S}}$ describes a sensor orientation from the reference position and $g=9.8065$ ms$^{-2}$ is the gravitational constant. $\mathbf{R}_{\mathbf{S}}$ is composed as the product of three basic rotations $\mathbf{R}_{\mathbf{X}}$, $\mathbf{R}_{\mathbf{Y}}$ and $\mathbf{R}_{\mathbf{Z}}$ about $X_{\mathrm{S}}$, $Y_{\mathrm{S}}$ and $Z_{\mathrm{S}}$ axis, respectively; so that each rotation is defined by the corresponding rotation angle (Fig. 2):

$$\begin{aligned}&R_X =\left[ {{\begin{array}{c@{\quad }c@{\quad }c} 1 &{} 0 &{} 0 \\ 0 &{} {\cos \hbox {Pitch}} &{} {-\sin \hbox {Pitch}} \\ 0 &{} {\sin \hbox {Pitch}} &{} {\cos \hbox {Pitch}} \\ \end{array} }} \right] \end{aligned}$$

(7)

$$\begin{aligned}&R_Y =\left[ {{\begin{array}{c@{\quad }c@{\quad }c} {\cos \hbox {Roll}} &{} 0 &{} {\sin \hbox {Roll}} \\ 0 &{} 1 &{} 0 \\ {-\sin \hbox {Roll}} &{} 0 &{} {\cos \hbox {Roll}} \end{array} }} \right] \end{aligned}$$

(8)

$$\begin{aligned}&R_Z =\left[ {{\begin{array}{c@{\quad }c@{\quad }c} {\cos \hbox {Yaw}} &{} {-\sin \hbox {Yaw}} &{} 0 \\ {\sin \hbox {Yaw}} &{} {\cos \hbox {Yaw}} &{} 0 \\ 0 &{} 0 &{} 1 \end{array} }} \right] \end{aligned}$$

(9)

There are six possible ways of combining these three rotation matrices to compose the rotation matrix $\mathbf{R}_{\mathbf{S}}$, acknowledging that a different rotation ordering would yield to a different set of Yaw (azimuth), Pitch and Roll rotation angles. The question is: Can we choose any of this six possible rotation orderings and based on Eq. (6) be able to retrieve the rotation angles? The matter of fact is that a three-component accelerometer vector reading $\mathbf{G}_{\mathbf{P}}$ also needs (ideally) to satisfy the following constraint:

$$\begin{aligned} \sqrt{G_x^2 +G_y^2 +G_z^2 } =g=9.8060~\hbox {ms}^{-2} \end{aligned}$$

(10)

Hence, there are actually only two degrees of freedom, which would be insufficient to find unique values for all three Yaw, Pitch and Roll angles, based on $\mathbf{G}_{\mathbf{P}}$ only. Fortunately, two out of six rotation orderings (xyz and yxz rotation order) yield a functional form of Eq. (6) dependent on two rotation angles only: Roll and Pitch. More specifically, for xyz rotation order, i.e., $\mathbf{R}_{\mathbf{S}}=\mathbf{R}_{\mathbf{X}}\cdot \mathbf{R}_{\mathbf{Y}}\cdot \mathbf{R}_{\mathbf{Z}}$, we obtain:

$$\begin{aligned}&{G_\mathrm{p}} = \left[ {\begin{array}{c} {{G_x}} \\ {{G_y}} \\ {{G_z}} \\ \end{array} } \right] = \left[ {\begin{array}{c} {g \cdot \sin \hbox {Roll}} \\ { - g \cdot \cos \hbox {Roll} \cdot \sin \hbox {Pitch}} \\ {g \cdot \cos \hbox {Roll} \cdot \cos \hbox {Pitch}} \end{array} } \right] \end{aligned}$$

(11)

$$\begin{aligned}&\tan \hbox {Pitch} = \frac{{ - {G_y}}}{{{G_z}}}\end{aligned}$$

(12)

$$\begin{aligned}&\tan \hbox {Roll} = \frac{{{G_x}}}{{\sqrt{G_y^2 + G_z^2} }} \end{aligned}$$

(13)

Then, two rotation angles can be retrieved as:

$$\begin{aligned}&\hbox {Pitch} = a\tan 2( - {G_y},~{G_z})\end{aligned}$$

(14)

$$\begin{aligned}&\hbox {Roll} = a\tan 2\frac{{{G_x}}}{{\sqrt{G_y^2 + G_z^2} }} \end{aligned}$$

(15)

where Android convention is used restricting Pitch angle in the range between $-180^{\circ }$ and $180^{\circ }$ ($a\tan 2$ here represents the tangent inverse function in the range $-180^{\circ }$ and $180^{\circ }$), and Roll angle in the range between $-90^{\circ }$ and $90^{\circ }$. It is worth noting that in practice when $G_y\approx 0$ and $G_z\approx 0,$ computing the inverse tangent function of Eq. (12) becomes unstable. There is no perfect solution for this problem but one common approach is to use instead of (12) the following approximate expression to compute the Pitch angle:

$$\begin{aligned} \tan \hbox {Pitch} = \frac{{ - {G_y}}}{{\sqrt{\mu \cdot G_x^2 + G_z^2} }} \end{aligned}$$

(16)

where typically $\mu =0.01$ [34]. The preceding text demonstrated that accelerometer alone is sufficient to reveal two out of three orientation angles, needed for the proposed method. On those findings we build next how to recover the remaining orientation Yaw angle.

A three-axis magnetometer in the earth’s magnetic field will have output $\mathbf{B}_{\mathbf{p}}$ given by:

$$\begin{aligned} {B_\mathrm{p}} = \left[ {\begin{array}{c} {{B_x}} \\ {{B_y}} \\ {{B_z}} \\ \end{array} } \right] = {R_\mathrm{S}} \cdot \left[ {\begin{array}{c} 0 \\ {B \cdot \cos \delta } \\ { - B \cdot \sin \delta } \\ \end{array} } \right] \end{aligned}$$

(17)

where rotation matrix $\mathbf{R}_{\mathbf{S}}$ describes the sensor orientation from the reference position (Fig. 2), B is the geomagnetic field strength which varies over the earth’s surface, and $\delta$ is the angle of inclination of the geomagnetic field measured downwards from horizontal and varies over the earth’s surface, too. Fortunately, both $\delta$ and B cancel out during the subsequent computation. Assuming $\mathbf{R}_{\mathbf{S}}$ is defined through the three successive rotations $\mathbf{R}_{\mathbf{X}}$, $\mathbf{R}_{\mathbf{Y}}$, and $\mathbf{R}_{\mathbf{Z}}$ around system axes X, Y and Z, respectively, and using xyz rotation order, Eq. (17) can be re-written as:

$$\begin{aligned} R_Y^{ - 1} \cdot R_X^{ - 1} \cdot \left[ {\begin{array}{c} {{B_x}} \\ {{B_y}} \\ {{B_z}} \\ \end{array} } \right]= & {} {R_Z} \cdot \left[ {\begin{array}{c} 0 \\ {B \cdot \cos \delta } \\ { - B \cdot \sin \delta } \\ \end{array} } \right] \nonumber \\= & {} \left[ {\begin{array}{c} { - B \cdot \cos \delta \cdot \sin \hbox {Yaw}} \\ {B \cdot \cos \delta \cdot \cos \hbox {Yaw}} \\ { - B \cdot \sin \delta } \end{array} } \right] \end{aligned}$$

(18)

Since two rotation angles Pitch and Roll have already been found using accelerometers measurements, after some algebraic manipulation of (18), the Yaw angle is computed as:

$$\begin{aligned}&{B_2} = {B_x} \cdot \cos \hbox {Roll} - {B_z} \cdot \cos \hbox {Pitch} \cdot \sin \hbox {Roll}\nonumber \\&\qquad \quad + {B_y} \cdot \sin {\text {Pitch}} \cdot \sin \hbox {Roll}\nonumber \\&{B_1} = {B_y} \cdot \cos \hbox {Pitch} + {B_z} \cdot \sin \hbox {Pitch}\nonumber \\&\hbox {Yaw}= a\tan 2( -{B_2},{B_1}) \end{aligned}$$

(19)

Equation (19) provides Yaw readings in the range between $-180^{\circ }$ and $180^{\circ }$, though the Android version we have been experimenting with appears to use a convention that transforms Yaw readings into the range between $0^{\circ }$ and $360^{\circ }$. We conclude that all three rotation angles, required initially by the proposed registration method, can be neatly retrieved in general case using combined accelerometer and magnetometer measurements.