Edge-preserved fringe-order correction strategy for code-based fringe projection profilometry
Introduction
Fringe projection profilometry (FPP) is widely accepted as an effective method for non-contact measurement [1], [2], [3], [4], [5], and the phase value is often selected as the link between the captured photographs and the three-dimensional (3D) information of the objects [6], [7], [8]. The phase obtained directly from the FPP is wrapped between and rad and must be unwrapped to form the absolute phase map. Numerous unwrapping algorithms have been proposed over the past few decades [9], [10], [11], [12], and the temporal phase unwrapping (TPU) algorithm [13] has merits over other methods when the measured surface contains sharp edges. Because TPU is a multi-frame method, composite phase shifting schemes [14], [15], [16] (e.g., dual-frequency, bi-frequency, and 2+2 phase shifting methods) are targeted at addressing the speed limitation of TPU. However, to guarantee the reliability of phase unwrapping, the fringe frequency and modulation are limited, which provides comparatively low accuracy.
Code-based FPP (CBFPP) is used in numerous fields [12], [17] (e.g., ground-truth establishment, industrial monitoring) because of its simple structure and highly reliable unwrapping [18], [19], [20], [21], [22], [23]. Moreover, when implemented by the dithering technique [24] and high-speed hardware, the speed of 3D measurement can reach kilohertz levels [25], [26], [27], [28], [29]. Each CBFPP scheme contains two groups of fringe patterns, namely (i) the phase-shifted patterns to calculate the wrapped phase and (ii) the codeword patterns for phase unwrapping. The most significant feature of this method is that the codewords are aligned with the rad discontinuities of the wrapped phase. However, the phase-jump regions of the wrapped phase map have been shown to be unstable, and the fringe orders around these regions are easily altered [13], [30], [31]. In addition, in some CBFPP cases, the fringe orders not around the phase-jump regions tend to fail [32], [33]. Some of these noises even merge to form dense point artifacts (DPAs), a problem that current research finds difficult to solve [9].
Various filtering approaches have been proposed to tackle these fringe-order problems [34], [35], [36], [37], [38], [39], [40], [41]. Zhang [35] developed a method that filters the noise to facilitate phase monotonicity. This method removes nearly all the fringe-order noises except for those that change in a similar way as does the absolute phase. Zhang [36] then improved the previous algorithm by using the discontinuities of the wrapped phase. The algorithm is effective at removing all the fringe-order noises around the phase-jump regions. However, that method belongs to the class of proprietary algorithms and has difficulty correcting all the fringe-order errors of phase-based CBFPP [22], [42], [43]. Deng et al. [37] devised a simple threshold filtering approach to suppress the fringe-order noise. That method performs well with high speed but cannot remove the noises when the camera lens is too defocused. Zhang et al. [38] added one complementary Gray-code pattern to stagger the fringe orders, thereby ensuring the correctness of the phase unwrapping but limiting the measurement speed. Wang et al. [42] separated the noise-suspicious regions into several subregions to correct the fringe orders, a method that tackles part of the problem of DPAs.
A median filter is a nonlinear filter that replaces each entry with the median value of the neighboring entries. It is a filter that removes fringe-order noises fast and accurately [39]. Nevertheless, a median filter is also classified as a neighbor filter, unavoidably polluting those data that are surrounded by regions of abrupt change [42], [44]. Zheng et al. [40] designed an adaptive median filter to weaken the problem of neighboring pollution. However, although the pollution effect is weakened greatly, the framework requires different median filters to be iterated numerous times. Consequently, the process restricts the ability of CBFPP to process in real time [41].
To ensure correctness and processing speed simultaneously, the present paper proposes a framework to correct fringe-order noises, and the overall process is a two-fold one:
- 1.
Coarse correction. The coarse correction step prevents the problem of neighboring pollution and removes most of the fringe-order noises with low computational cost. The sinusoidal fringe patterns serve the dual purpose of wrapped-phase retrieval and sharp-edge protection. A simple modified phase comparison algorithm is applied for further denoising. The corresponding theories are provided in Section 3.1.
- 2.
Fine correction. The fine correction process is aimed at eliminating the remaining noises that contain the DPAs that are difficult to remove with traditional methods. After separating and identifying the regions (Section 3.2.1), the unprocessed fringe-order noises can be removed by eight-neighbor filtering (ENF) (Section 3.2.2).
To suppress the fringe-order errors in a dynamic scene, Zuo et al. [45] developed a state-of-the-art method called reliability-guided compensation (RGC). Guided by the minimum projection distance map (one of the reliability functions), this method corrects the fringe-order errors from the high-reliability regions toward the low-reliability regions. Therefore, in Zuo’s method, a reliability function (e.g., the minimum projection distance map, fringe modulation, or phase error) and the processing path must be predefined before the correction work. However, some factors limit the application of RGC. Because different algorithms are needed in different scenarios, different reliability functions must be designed to map these algorithms, which brings inconvenience in practical processing. Moreover, the formation of the fringe-order errors is usually caused by several known or unknown factors (e.g., fringe modulation, numerically unstable points). Consequently, it is difficult to set denoising assessment criteria that can guarantee the reliability of denoising in different measurement scenarios. By contrast, the processing path and the assessment criteria of the proposed method are determined by the noise regions only. For this reason, the proposed method has the following advantages when compared with previous filtering algorithms:
- 1.
The correction algorithm is a general algorithm that can deal with the fringe-order problem of all types of CBFPP.
- 2.
The process is noninvasive and can remove the fringe-order noises around regions of abrupt change.
- 3.
The proposed method is effective for solving the problem of DPAs.
- 4.
The processing is sufficiently fast to allow filtering to be done in a short time.
The rest of this paper is organized as follows. Section 2 introduces CBFPP and explains the theoretical basis for the formation mechanisms of different types of fringe-order noise ( and ). Section 3 describes the principles of the proposed correction framework. Sections 4 and 5 present the simulation and experimental results, respectively. Sections 6 and 7 discuss and summarize the paper.
Section snippets
Fringe projection profilometry
As presented in Fig. 1, the 3D reconstruction principle of classical FPP is based on the triangulation relationship between the projector, the measured object, and the camera. Therefore, it is important to match the features across the camera and projector with high reliability.
Among all the matching techniques, phase-shifting profilometry (PSP) is dominant because of its high accuracy, spatial resolution, and data density. More details about PSP can be found elsewhere [46]. Generally, the
Fringe-order correction procedures
To prevent the problem of neighboring pollution, in this paper the edges of the regions of abrupt change in the measured object act as segmentation lines to separate all the valid regions. However, because of the texture of the object and the problem of DPAs, the edges of the object cannot be extracted directly from the scene or the absolute phase map.
Fig. 4 shows the flow chart of the proposed correction processing. The major steps of the process are coarse correction and fine correction.
Simulations
The phase-coding method with 14 pattern periods is selected for simulation. The smooth data generated by the patterns without noise, shown in Fig. 9(a) and (b), are selected as the ground truth. The data contain one step height region and an “O” shaped hole. Gaussian noise (with a signal-to-noise ratio of 25) is added to the phase-coding patterns to generate the noisy data. As shown in Fig. 9(c) and (d), sparsely distributed noise, periodic noise, and DPAs largely influence the reconstruction
Experiments
The experimental setup shown in Fig. 11 comprises objects, a camera (BFS-U3-13Y3M-C), and a projector (LightCrafter 4500). The resolution of the projector is 1140 912, and the camera resolution is 1280 1024 with a 16-mm focal lens. The 16-period grating groups are generated by computer, projected onto the objects by the projector, and captured by the camera.
Discussion
We discuss the proposed method regarding the following aspects.
- 1.
Processing speed. The average computation time for each dynamic frame using MATLAB 2016b in a computer assembly with an Intelë Core i5 CPU @ 3.00 GHz and 8.0 GB RAM is 1.132 s with the proposed method. Considering related work [33], [35], [40], the present processing speed is higher than that of most existing algorithms (e.g., adaptive median filter, valid point detection). In addition, the processing could be accelerated even
Conclusions
In this paper, we explained and verified the formation mechanisms of different types of fringe-order noise and proposed a correction framework to solve the problem of fringe-order noise. There are two key points in this correction framework. For coarse correction, an edge-determined segmentation process is proposed and applied to preserve the edge of the measured object. Such a protection process would not protect any fringe order noises. For fine correction, the processing path and the
CRediT authorship contribution statement
Ji Deng: Conceptualization, Methodology, Software, Validation, Writing - original draft. Jian Li: Investigation. Hao Feng: Visualization, Writing - review & editing. Shumeng Ding: Investigation. Yu Xiao: Investigation. Wenzhong Han: Investigation. Zhoumo Zeng: Supervision, Project administration.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgment
This work was supported by the Major Scientific Instrument and Equipment Development Project in National Key Research and Development Program of China (grant no. 2016YFF0101802) and National Natural Science Foundation of China (grant no. 61873183).
References (54)
- et al.
A state of the art in structured light patterns for surface profilometry
Pattern Recognit.
(2010) - et al.
Fringe projection decamouflaging
Opt. Lasers Eng.
(2020) - et al.
Review of phase measuring deflectometry
Opt. Lasers Eng.
(2018) - et al.
Fourier transform profilometry: a review
Opt. Lasers Eng.
(2001) - et al.
Reliability-guided phase unwrapping algorithm: a review
Opt. Lasers Eng.
(2004) - et al.
Fringe projection techniques: whither we are?
Opt. Lasers Eng.
(2010) - et al.
Direction-determined phase unwrapping using geometric constraint of the structured light system: the establishment of minimum phase map
Opt. Commun.
(2017) - et al.
Temporal phase unwrapping algorithms for fringe projection profilometry: a comparative review
Opt. Lasers Eng.
(2016) - et al.
Dual-frequency pattern scheme for high-speed 3-d shape measurement
Opt. Express.
(2010) - et al.
High-speed three-dimensional profilometry for multiple objects with complex shapes
Opt. Express.
(2012)
Ternary gray code-based phase unwrapping for 3d measurement using binary patterns with projector defocusing
Appl. Opt.
Modified gray-level coding method for absolute phase retrieval
Sensors
Genetic method to optimize binary dithering technique for high-quality fringe generation
Opt. Lett.
Improved intensity-optimized dithering technique for 3d shape measurement
Opt. Lasers Eng.
Pixel-by-pixel absolute phase retrieval using three phase-shifted fringe patterns without markers
Opt. Lasers Eng.
Fringe order correction for the absolute phase recovered by two selected spatial frequency fringe projections in fringe projection profilometry.
Rev. Sci. Instrum.
Error correcting coding-theory for structured light illumination systems
Opt. Lasers Eng.
Valid point detection in fringe projection profilometry
Optics Express
3D information detection with novel five composite fringe patterns
Mod. Phys. Lett. B
High-resolution, real-time 3d imaging with fringe analysis
Journal of Real-Time Image Processing
Micro fourier transform profilometry (ftp): 3d shape measurement at 10,000 frames per second
Opt. Lasers Eng.
Flexible depth segmentation method using phase-shifted wrapped phase sequences
Opt. Lasers Eng.
Omnidirectional depth segmentation using orthogonal fringe patterns and multi-scale enhancement
Opt. Lasers Eng.
A modified fast parallel algorithm for thinning digital patterns
Pattern Recognit. Lett.
Gpu-assisted high-resolution, real-time 3-d shape measurement
Opt. Express.
Automatic identification and removal of outliers for high-speed fringe projection profilometry
Opt. Eng.
Structured-light 3d surface imaging: a tutorial
Adv. Opt. Photonics
Cited by (12)
Full-period fringe order correction method based on dual-threshold cellular automaton
2024, Optics and Laser TechnologyHigh-speed 3-D reconstruction based on phase shift coding and interleaved projection
2023, Expert Systems with ApplicationsAn efficient high dynamic range 3D shape reconstruction method based on double phase-shifting profilometry
2024, Measurement Science and TechnologyDEPTH-FROM-DEFOCUS-BASED CONVENIENT COAXIAL PROJECTION PROFILOMETRY FOR LARGE MEASUREMENT RANGE
2024, Mechatronic Systems and Control