1 Introduction

In recent years, Optical 3D shape measurement techniques are widely used in various fields such as biomechanics, intelligent monitoring, robot navigation, industrial quality control, and human-computer interaction. Among plenty of state-of-the-art methods, fringe projection profilometry (FPP) [1,2,3,4,5,6,7,8], which is based on the principle of structured light and triangulation, has been proven to be one of the most promising techniques due to its inherent advantages of non-contactness, high accuracy, high efficiency, and low cost. In a conventional measurement system based on fringe projection profilometry (FPP) consisting of a camera and a projector, it cannot obtain the 360-degree overall 3D shape results of the object with complex surfaces due to the limited or occluded field of view, which leads to the limits on the application of FPP. Therefore, for such problems, it is necessary to carry out multiple measurements from different views to obtain the overall shape of the object.

In general, in order to achieve multiple measurements of the tested object from different views, it can be classified into three main categories: methods based on turn-table [9, 10], methods based on movable robot arm [11, 12], and measurement systems with plane mirrors [13,14,15]. In the first method, the tested object is placed on a turn-table, acquiring the whole 3D data by multiple rotations. Based on an idea contrary to the first method, the second method requires the measurement system to be mounted on a movable robot arm to perform multiple measurements around the tested object. Besides, the complicated post-processing operation for the scanned data must be performed using point cloud registration algorithms such as Iterated Closest Point (ICP). As a result, these fringe projection systems cannot be applied for the real-time overall acquisition of dynamic scenes, which requires multiple measurements and registration algorithms that are time-consuming and laborious. Different from the first two methods, measurement systems with plane mirrors can capture fringe images of the measured object from three different perspectives simultaneously including a real camera and two virtual cameras obtained by plane mirrors, making it possible to achieve panoramic 3D shape reconstruction only by single-shot measurement.

Several methods based on systems with plane mirrors has been presented in the survey articles and achieved remarkable success. Epstein et al. firstly introduced plane mirrors into FPP to create virtual cameras and projectors [13]. By tracking the relative positions of camera, projector, and mirrors, an interactive reconstruction system with structured light can provide 3D points to accurately estimate the pose of a mirror, while also reconstructing 3D points on the object. However, there are still some limitations in this system, which needs multiple measurements because the entire surface of the object cannot be illuminated at the same time. To solve this issue, Lanman et al. presented an orthographic projection system using a DLP projector and a Fresnel lens, which illuminated passive optical scatterers to create a volumetric display [14]. And then they designed an unambiguous Gray code sequence to facilitates the establishment of the correspondence of projectors and cameras, recovering a dense 3-D point cloud data of the entire object surface. In addition, some conventional calibration procedures are used to obtain accurate mirror calibration due to the lack of a suitable reflection model for the mirror, but it is complex and difficult to implement. Mariottini et al. systematically studied the catadioptric of the mirror to propose an ideal catadioptric model that helps convert the virtual surfaces reflected by the mirror into their true positions [16]. Following this idea, Chen et al. premade two speckle patterns on the mirrors to transform all surface portions into a common global coordinate thus allowing a straightforward full-surface 360-deg profile and deformation measurements [15]. However, this method requires the plane mirror with the front surface reflection, otherwise the thickness of the plane mirror should be considered [17]. In addition, it will inevitably introduce errors into the calibration of the mirror due to the nonuniform thickness of the print paper with speckles. On the other hand, since the paper is fixed on the plane mirror, it leads to limited 3D measurement volume for the system with plane mirrors.

In this work, a system calibration method for panoramic 3D shape measurement with plane mirrors is proposed. Firstly, the ideal reflection model for the plane mirror is review. Then, a flexible new calibration technique is proposed to easily calibrate the mirror. In the proposed technique, the calibration of the mirror is discussed mathematically to ensure the effectiveness and rationality of the calibration process, it only requires the camera to observe a set of feature point pairs (including real points and virtual points) to achieve the solution of the reflection matrix for plane mirrors. The entire calibration process is divided into two steps: the initial estimation of the reflection matrix and the precise calibration using the Levenberg-Marquardt algorithm with the bundle adjustment strategy. Finally, the estimated calibration information is used to convert 3D point cloud data obtained from real and virtual perspectives into a common world coordinate system, making it possible to obtain full-surface 3D data of the object. Benefitting from the robust and high-performance calibration method, experimental results verify that our method can achieve high-accuracy and panoramic 3D shape measurement.

2 Principle

2.1 The Reflection Model for the Plane Mirror

In this subsection, we will discuss in detail how to establish the ideal reflection model for the plane mirror. Firstly, let us consider the measurement system with a plane mirror as shown in Fig. 1, where an arbitrary 3D point of the tested object in the world coordinate system (X, Y, Z) is denoted by \({X^{o}(x^{o}, y^{o}, z^{o})}\), O is the original point of the world coordinate system, \({d^r_w}\) is the distance between O and the mirror, \({X^{r}(x^{r}, y^{r}, z^{r})}\) represents the corresponding virtual point of \({X^o}\) due to the reflection of the plane mirror, \({d^r_o}\) is the distance between \({X^o}\) and the mirror, and \({n^r}\) is the normal vector of the mirror. From Fig. 1, we represent the reflection of plane mirror in the vector way:

Fig. 1.
figure 1

The schematic diagram of the measurement system with a plane mirror.

Full size image
$$\begin{aligned} \overrightarrow{OX^r} = \overrightarrow{OX^o} + \overrightarrow{X^{o}X^{r}}. \end{aligned}$$
(1)

Due to \({\overrightarrow{X^oX^r} = 2 d^r_o \overrightarrow{n^r}}\), Eq. (1) can also be rewritten as:

$$\begin{aligned} \overrightarrow{OX^r} = \overrightarrow{OX^o} + 2 d^r_o \overrightarrow{n^r}. \end{aligned}$$
(2)

Besides, the relationship between \({d^{r}_{o}}\) and \(\overrightarrow{OX^o}\) can be calculated:

$$\begin{aligned} {d^r_o} = {d^r_w} - \overrightarrow{OX^o} \cdot \overrightarrow{n^r}, \end{aligned}$$
(3)

where \({\cdot }\) is the dot product of vectors, \({\overrightarrow{OX^o} \cdot \overrightarrow{n^r}}\) is a value instead of a vector, so combining Eqs. (2) and (3) yields:

$$\begin{aligned} \overrightarrow{OX^r} = \overrightarrow{OX^o} + 2 d^r_w \overrightarrow{n^r} - 2\overrightarrow{OX^o} \cdot \overrightarrow{n^r}\overrightarrow{n^r}. \end{aligned}$$
(4)

In this subsection, \({X^o}\), \({X^r}\), and \({n^r}\) are matrices with the size of \({3 \times 1}\). Due to \({\overrightarrow{OX^o} \cdot \overrightarrow{n^r} = {(X^o)}^{T}n^r = {(n^r)}^{T}X^o}\), Eq. (4) can also be rewritten in the matrix way:

$$\begin{aligned} X^r = (I - 2n^r{(n^r)}^{T})X^o + 2 d^r_w n^r, \end{aligned}$$
(5)

where I is the \({3 \times 3}\) identity matrix. So we have:

$$\begin{aligned} \left[ \begin{array}{cc} X^r\\ 1 \end{array} \right] = \left[ \begin{array}{cc} I - 2n^r{(n^r)}^{T} &{} 2 d^r_w n^r\\ 0 &{} 1 \end{array} \right] \left[ \begin{array}{cc} X^o\\ 1 \end{array} \right] . \end{aligned}$$
(6)

So the reflection matrix \({D^r}\) for plane mirrors is defined as the following formula:

$$\begin{aligned} {D^r} = \left[ \begin{array}{cc} I - 2n^r{(n^r)}^{T} &{} 2 d^r_w n^r\\ 0 &{} 1 \end{array} \right] . \end{aligned}$$
(7)

It is noted that \({D^r}\) is involutory (e.g., \({(D^r)^{-1} = {D^r}}\)) in [16]:

$$\begin{aligned} \left[ \begin{array}{cc} I - 2n^r{(n^r)}^{T} &{} 2 d^r_w n^r\\ 0 &{} 1 \end{array} \right] \left[ \begin{array}{cc} X^r\\ 1 \end{array} \right] = \left[ \begin{array}{cc} X^o\\ 1 \end{array} \right] . \end{aligned}$$
(8)

It can be found according to Eq. (8) that the reflection matrix \({D^r}\) for plane mirrors can be obtained immediately if \({n^r}\) and \({d^r_w}\) are known, and the 3D point cloud data obtained from virtual perspectives can be converted into the 3D point cloud data in the real world coordinate system. As a result, the core challenge for the panoramic 3D measurement is to calculate \({n^r}\) and \({d^r_w}\) quickly and accurately.

2.2 The Calibration Method for the Plane Mirror

At present, the conventional method requires artificially attaching a printing paper on the plane mirror to acquire the attitude information of the plane mirror for realizing the calibration of the plane mirror [15]. However, this method requires the plane mirror with the front surface reflection, otherwise, the thickness of the plane mirror should be considered [17]. In addition, it will inevitably introduce errors into the calibration of the mirror due to the nonuniform thickness of the print paper with speckles. On the other hand, since the paper is fixed on the plane mirror, it leads to limited 3D measurement volume for the system with plane mirrors.

In this subsection, the calibration of the mirror is discussed mathematically to ensure the effectiveness and rationality of the calibration process. The entire calibration process is divided into two steps: the initial estimation of the reflection matrix and the precise calibration using the Levenberg-Marquardt algorithm with the bundle adjustment strategy.

The Initial Estimation of the Reflection Matrix

Firstly, let \({n^r}\) is \({(a^r, b^r, c^r)}\), Eq. (8) can also be rewritten in detail:

$$\begin{aligned} \left[ \begin{array}{cccc} 1-2(a^r)^2 &{} -2 a^rb^r &{} -2 a^rc^r &{} 2a^rd^r_w\\ -2 a^rb^r &{} 1-2(b^r)^2 &{} -2 b^rc^r &{} 2b^rd^r_w\\ -2 a^rc^r &{} -2 b^rc^r &{} 1-2(c^r)^2 &{} 2c^rd^r_w\\ 0 &{} 0 &{} 0 &{} 1\\ \end{array} \right] \left[ \begin{array}{cccc} x^r\\ y^r\\ z^r\\ 1 \end{array} \right] = \left[ \begin{array}{cccc} x^o\\ y^o\\ z^o\\ 1 \end{array} \right] . \end{aligned}$$
(9)

From Eq. (9), it requires the camera to observe N feature 3D point pairs (including real points \({X^o}\) and virtual points \({X^r}\)) to achieve the solution of the reflection matrix for plane mirrors. Minimizing Eq. (9) is a nonlinear minimization problem, which requires the accurate initial guesses of \({n^r}\) and \({d^r_w}\) obtained using the technique described in the following section. In addition, we have from Eq. (9):

$$\begin{aligned}{}[1-2(a^r)^2]x^r - 2a^rb^ry^r - 2a^rc^rz^r + 2a^rd^r_w = x^o, \end{aligned}$$
(10)
$$\begin{aligned} -2 a^rb^rx^r + [1-2(b^r)^2]y^r - 2 b^rc^rz^r + 2b^rd^r_w = y^o, \end{aligned}$$
(11)
$$\begin{aligned} -2 a^rc^rx^r - 2 b^rc^ry^r + [1-2(c^r)^2]z^r + 2c^rd^r_w = z^o. \end{aligned}$$
(12)

Combining Eqs. (10) and (11) yields:

$$\begin{aligned} a^r(y^r-y^o) + b^r(x^o - x^r) = 0. \end{aligned}$$
(13)

Likewise, the other two formulas can be derived from Eqs. (10), (11), and (12):

$$\begin{aligned} a^r(z^r-z^o) + c^r(x^o - x^r) = 0. \end{aligned}$$
(14)
$$\begin{aligned} b^r(z^r-z^o) + c^r(y^o - y^r) = 0. \end{aligned}$$
(15)

So combining Eqs. (13), (14), and (15) yields:

$$\begin{aligned} \left[ \begin{array}{ccc} y^r-y^o &{} x^o - x^r &{} 0\\ z^r-z^o &{} 0 &{} x^o - x^r\\ 0 &{} z^r-z^o &{} y^o - y^r \end{array} \right] \left[ \begin{array}{ccc} a^r\\ b^r\\ c^r \end{array} \right] = 0. \end{aligned}$$
(16)

Solving Eq. (16) is a least-squares minimization problem for obtaining an initial guess of \({n^r (a^r, b^r, c^r)}\). In our method, SVD can be implemented to yield the exact solution for the least-squares minimization problems, and the last column vector of V obtained using SVD is the initial guess \({n_0^r (a_0^r, b_0^r, c_0^r)}\). And Eqs. (10), (11), and (12) can also be rewritten for estimating the initial guess of \({d^r_w}\):

$$\begin{aligned} r_1(d^r_w) = 2a_0^rd^r_w + [1-2(a_0^r)^2]x^r - 2a_0^rb_0^ry^r - 2a_0^rc_0^rz^r - x^o = 2a_0^rd^r_w + c_1, \end{aligned}$$
(17)
$$\begin{aligned} r_2(d^r_w) = 2b_0^rd^r_w - 2 a_0^rb_0^rx^r + [1-2(b_0^r)^2]y^r - 2 b_0^rc_0^rz^r - y^o = 2b_0^rd^r_w + c_2, \end{aligned}$$
(18)
$$\begin{aligned} r_3(d^r_w) = 2c_0^rd^r_w - 2 a_0^rc_0^rx^r - 2 b_0^rc_0^ry^r + [1-2(c_0^r)^2]z^r - z^o = 2c_0^rd^r_w + c_3, \end{aligned}$$
(19)
$$\begin{aligned} f(d^r_w) = \sum _{n=1}^N r_1^2(d^r_w) + r_2^2(d^r_w) + r_3^2(d^r_w), \end{aligned}$$
(20)

where \({c_1}\), \({c_2}\), and \({c_3}\) are the constant values now. Since there is only one variable \({d^r_w}\) in Eqs. (17), (18), and (19), minimizing Eq. (20) is a quadratic equation problem with one unknown, and the first-order derivative of \({f(d^r_w)}\):

$$\begin{aligned} f'(d^r_w) = \sum _{n=1}^N 8[(a_0^r)^2 + (b_0^r)^2 + (c_0^r)^2]d^r_w + 4(a_0^rc_1 + b_0^rc_2 + c_0^rc_3). \end{aligned}$$
(21)

Due to the normal vector \({n_0^r}\) (e.g., \({(a_0^r)^2 + (b_0^r)^2 + (c_0^r)^2 = 1}\)), so we have:

$$\begin{aligned} f'(d^r_w) = \sum _{n=1}^N 8d^r_w + 4(a_0^rc_1 + b_0^rc_2 + c_0^rc_3). \end{aligned}$$
(22)

Therefore, the initial guess of \({d^r_w}\) is:

$$\begin{aligned} d^r_w = \frac{\sum _{n=1}^N -1/2(a_0^rc_1 + b_0^rc_2 + c_0^rc_3)}{N}. \end{aligned}$$
(23)

There is the minimum value of \({f(d^r_w)}\) if Eq. (23) is established.

The Precise Calibration Using the Levenberg-Marquardt Algorithm with the Bundle Adjustment Strategy

Next, Eqs. (10), (11), and (12) can be rewritten again based on the Levenberg-Marquardt algorithm:

$$\begin{aligned} g_1(a^r, b^r, c^r, d^r_w) = [1-2(a^r)^2]x^r - 2a^rb^ry^r - 2a^rc^rz^r + 2a^rd^r_w - x^o, \end{aligned}$$
(24)
$$\begin{aligned} g_2(a^r, b^r, c^r, d^r_w) = -2 a^rb^rx^r + [1-2(b^r)^2]y^r - 2 b^rc^rz^r + 2b^rd^r_w - y^o, \end{aligned}$$
(25)
$$\begin{aligned} g_3(a^r, b^r, c^r, d^r_w) = -2 a^rc^rx^r - 2 b^rc^ry^r + [1-2(c^r)^2]z^r + 2c^rd^r_w - z^o, \end{aligned}$$
(26)
$$\begin{aligned} \sum _{n=1}^N g_1^2(a^r, b^r, c^r, d^r_w) + g_2^2(a^r, b^r, c^r, d^r_w) + g_3^2(a^r, b^r, c^r, d^r_w), \end{aligned}$$
(27)

where N is the total number of 3D point pairs. Minimizing Eq. (27) is a nonlinear minimization problem, which is solved with the Levenberg-Marquardt algorithm. It is worth noting that there are two key factors (\({X^r}\) and \({X^o}\)) affecting the accuracy of the final optimization. It is well known that FPP is capable of acquiring high-precision 3D data of actual feature points. In our system, the precision of 3D measurement obtained using traditional multi-frequency phase-shifting profilometry is about \(30\,{\upmu }\mathrm{m}\). Therefore, the influence of the second factor \({X^r}\) should be considered primarily. In above calibration, \({X_n^r}\) (\({n = 1, 2, 3, \cdots , N}\)) were always taken as known input data. However, the low-precision 3D measurement for the virtual points (caused by the imperfect flatness of the mirror or the uneven reflectivity of the mirror) will introduce systematic errors into the final calibration results with low reliability. By further enhancing the manufacturing quality of the plane mirror, this disadvantage can be overcome to some extent to improve the performance of the calibration, but it is expensive and time-consuming. Therefore, the bundle adjustment strategy should be introduced to try to avoid problems caused by the mirror with low quality [18]. So Eq. (27) can be rewritten according to the bundle adjustment strategy:

$$\begin{aligned} \sum _{n=1}^N g_1^2(a^r, b^r, c^r, d^r_w, X_n^r) + g_2^2(a^r, b^r, c^r, d^r_w, X_n^r) + g_3^2(a^r, b^r, c^r, d^r_w, X_n^r). \end{aligned}$$
(28)

Although the total number of variables has been increased from 4 to \({4+3N}\), minimizing Eq. (27) is still a nonlinear minimization problem that can be solved with the Levenberg-Marquardt method.

3 Experiments

In the experiment, a mirror-assisted FPP system is built to verify the actual performance of the proposed method as shown in Fig. 2. This system includes a monochrome camera (Basler acA2440-75 um with the resolution of \({2448 \times 2048}\)), a DLP projector (LightCrafter 4500Pro with the resolution of \({912 \times 1140}\)), two plane mirrors with the front surface reflection (the size of 30 cm \(\times \) 30 cm). In Fig. 2, since the camera is placed above the projector, a series of horizontal fringe patterns are projected by the projector and captured by the camera.

Fig. 2.
figure 2

The diagram of the mirror-assisted FPP system.

Full size image

In the calibration process of the mirror, our calibration method needs to capture multiple pose data (6 postures are used in this experiment) of the circular calibration board with high precision, each of which can provide 15 feature point pairs as shown in Fig. 3. Then, these feature point pairs are used to perform in sequence the initial estimation of the reflection matrix and the precise calibration using the Levenberg-Marquardt algorithm with the bundle adjustment strategy. In order to quantitatively analyze the robustness of the proposed calibration method, the calibration residual errors at different steps are calculated as shown in Table 1. From the comparison results in Table 1, it can be found that our method can provide a relatively accurate initial guess of \({n^r}\) and \({d^r_w}\) with the RMS of 0.0704 mm or 0.0611 mm. Based on these estimations, the calibration residual errors can be further decreased to 0.0578 mm or 0.0534 mm using the Levenberg-Marquardt algorithm, which confirms its effectiveness. However, the low-precision 3D measurement for the virtual points will introduce systematic errors into the process of the calibration, which leads to the calibration results with low reliability. Therefore, the bundle adjustment strategy should be introduced into the calibration method to obtain results with high precision in Table 1.

To further evaluate the accuracy of the proposed approach, a standard ceramic spheres with a diameter of 50.8 mm is measured using our system, and the single-view 3D measurement results and the full-surface 3D measurement results are presented in Figs. 4(a)–(c) and (g)–(i). Then, for the single-view 3D measurement results, we perform the sphere fitting to obtain separately the measured errors with the RMS of 27.577 \(\upmu \)m, 47.531 \(\upmu \)m, and 44.791 \(\upmu \)m shown in Figs. 4(d)–(e), and the accuracy of full-surface measurement result is 65.122 \(\upmu \)m shown in Figs. 4(j)–(l). This result verifies that the proposed method can realize high-accuracy and panoramic 3D shape measurement.

Fig. 3.
figure 3

The measurement result of the circular calibration board. (a) One pose data of the circular calibration board can provide 15 feature point pairs, (b) The 3D data of virtual points, (c) The 3D data of real points.

Full size image
Table 1. Comparison of calibration residual errors.
Full size table

Finally, a Voltaire model is measured and the corresponding full-surface 3D reconstruction results are shown in Fig. 5(a). And then, the corresponding results from three different views are presented to illustrate the reliability of our method which can achieve robust panoramic 3D shape measurement for objects with complex surfaces in Figs. 5(b)–(d).

Fig. 4.
figure 4

The 3D measurement results of a standard ceramic spheres. (a)–(c) The single-view 3D measurement results. (d)–(f) The corresponding distribution of the errors of (a)–(c). (g)–(i) The full-surface 3D measurement results. (j)–(l) The corresponding distribution of the errors of (g)–(i).

Full size image
Fig. 5.
figure 5

The measurement results of a Voltaire model. (a) The full-surface 3D reconstruction results of a Voltaire model. (b)–(d) The corresponding results of (a) from three different views.

Full size image

4 Conclusion

In conclusion, we proposed a system calibration method for panoramic 3D shape measurement with plane mirrors. By introducing plane mirrors into the traditional fringe projection profilometry (FPP), our system can capture fringe images of the measured object from three different perspectives simultaneously including a real camera and two virtual cameras obtained by plane mirrors, realizing panoramic 3D shape reconstruction only by single-shot measurement. Then, a flexible new calibration technique is proposed to easily calibrate the mirror. In this work, the calibration of the mirror is firstly discussed mathematically to ensure the effectiveness and rationality of the calibration process, it only requires the camera to observe a set of feature point pairs (including real points and virtual points) to achieve the solution of the reflection matrix for plane mirrors. By the initial estimation of the reflection matrix and the precise calibration using the Levenberg-Marquardt algorithm with the bundle adjustment strategy, the robust calibration information with high performance can be acquired to recovery full-surface 3D data of the object. Finally, experimental results verify that our method can achieve high-accuracy and panoramic 3D shape measurement with the precision of 65.122 \(\upmu \)m.