Elsevier

Signal Processing

Volume 182, May 2021, 107959
Signal Processing

Edge-preserved fringe-order correction strategy for code-based fringe projection profilometry

https://doi.org/10.1016/j.sigpro.2020.107959Get rights and content

Highlights

  • This paper proposes and verifies the formation mechanisms of different types of fringe-order noise.

  • The proposed filter can solve the problem of dense point artifacts.

  • The proposed algorithm is noninvasive and suitable for all types of code-based fringe projection profilometry.

  • The processing is sufficiently fast for filtering to be done in a short time.

  • A novel noise edge-extraction algorithm is proposed to separate regions of abrupt change.

Abstract

Code-based fringe projection profilometry (CBFPP) is widely used because of its simple structure, robustness, and potential high speed. One of the most important steps in this technique is to obtain the fringe orders correctly. However, fringe orders tend to fail easily, especially around the phase-jump regions. First, this paper analyzes the formation mechanisms of different types of fringe-order noise. Then, an edge-preserved fringe-order correction strategy has been proposed to tackle the fringe order noise problem. Unlike the proprietary denoising algorithms, the correction approach is suitable for all types of CBFPP. Furthermore, the present algorithm is fast and can process in a short time. Moreover, the entire filtering procedure is noninvasive and can deal with dense point artifacts. The simulation and experimental results are presented to verify the feasibility and effectiveness of the proposed method.

Introduction

Fringe projection profilometry (FPP) is widely accepted as an effective method for non-contact measurement [1], [2], [3], [4], [5], and the phase value is often selected as the link between the captured photographs and the three-dimensional (3D) information of the objects [6], [7], [8]. The phase obtained directly from the FPP is wrapped between π and π rad and must be unwrapped to form the absolute phase map. Numerous unwrapping algorithms have been proposed over the past few decades [9], [10], [11], [12], and the temporal phase unwrapping (TPU) algorithm [13] has merits over other methods when the measured surface contains sharp edges. Because TPU is a multi-frame method, composite phase shifting schemes [14], [15], [16] (e.g., dual-frequency, bi-frequency, and 2+2 phase shifting methods) are targeted at addressing the speed limitation of TPU. However, to guarantee the reliability of phase unwrapping, the fringe frequency and modulation are limited, which provides comparatively low accuracy.

Code-based FPP (CBFPP) is used in numerous fields [12], [17] (e.g., ground-truth establishment, industrial monitoring) because of its simple structure and highly reliable unwrapping [18], [19], [20], [21], [22], [23]. Moreover, when implemented by the dithering technique [24] and high-speed hardware, the speed of 3D measurement can reach kilohertz levels [25], [26], [27], [28], [29]. Each CBFPP scheme contains two groups of fringe patterns, namely (i) the phase-shifted patterns to calculate the wrapped phase and (ii) the codeword patterns for phase unwrapping. The most significant feature of this method is that the codewords are aligned with the 2π rad discontinuities of the wrapped phase. However, the phase-jump regions of the wrapped phase map have been shown to be unstable, and the fringe orders around these regions are easily altered [13], [30], [31]. In addition, in some CBFPP cases, the fringe orders not around the phase-jump regions tend to fail [32], [33]. Some of these noises even merge to form dense point artifacts (DPAs), a problem that current research finds difficult to solve [9].

Various filtering approaches have been proposed to tackle these fringe-order problems [34], [35], [36], [37], [38], [39], [40], [41]. Zhang [35] developed a method that filters the noise to facilitate phase monotonicity. This method removes nearly all the fringe-order noises except for those that change in a similar way as does the absolute phase. Zhang [36] then improved the previous algorithm by using the discontinuities of the wrapped phase. The algorithm is effective at removing all the fringe-order noises around the phase-jump regions. However, that method belongs to the class of proprietary algorithms and has difficulty correcting all the fringe-order errors of phase-based CBFPP [22], [42], [43]. Deng et al. [37] devised a simple threshold filtering approach to suppress the fringe-order noise. That method performs well with high speed but cannot remove the noises when the camera lens is too defocused. Zhang et al. [38] added one complementary Gray-code pattern to stagger the fringe orders, thereby ensuring the correctness of the phase unwrapping but limiting the measurement speed. Wang et al. [42] separated the noise-suspicious regions into several subregions to correct the fringe orders, a method that tackles part of the problem of DPAs.

A median filter is a nonlinear filter that replaces each entry with the median value of the neighboring entries. It is a filter that removes fringe-order noises fast and accurately [39]. Nevertheless, a median filter is also classified as a neighbor filter, unavoidably polluting those data that are surrounded by regions of abrupt change [42], [44]. Zheng et al. [40] designed an adaptive median filter to weaken the problem of neighboring pollution. However, although the pollution effect is weakened greatly, the framework requires different median filters to be iterated numerous times. Consequently, the process restricts the ability of CBFPP to process in real time [41].

To ensure correctness and processing speed simultaneously, the present paper proposes a framework to correct fringe-order noises, and the overall process is a two-fold one:

  • 1.

    Coarse correction. The coarse correction step prevents the problem of neighboring pollution and removes most of the fringe-order noises with low computational cost. The sinusoidal fringe patterns serve the dual purpose of wrapped-phase retrieval and sharp-edge protection. A simple modified phase comparison algorithm is applied for further denoising. The corresponding theories are provided in Section 3.1.

  • 2.

    Fine correction. The fine correction process is aimed at eliminating the remaining noises that contain the DPAs that are difficult to remove with traditional methods. After separating and identifying the regions (Section 3.2.1), the unprocessed fringe-order noises can be removed by eight-neighbor filtering (ENF) (Section 3.2.2).

To suppress the fringe-order errors in a dynamic scene, Zuo et al. [45] developed a state-of-the-art method called reliability-guided compensation (RGC). Guided by the minimum projection distance map (one of the reliability functions), this method corrects the fringe-order errors from the high-reliability regions toward the low-reliability regions. Therefore, in Zuo’s method, a reliability function (e.g., the minimum projection distance map, fringe modulation, or phase error) and the processing path must be predefined before the correction work. However, some factors limit the application of RGC. Because different algorithms are needed in different scenarios, different reliability functions must be designed to map these algorithms, which brings inconvenience in practical processing. Moreover, the formation of the fringe-order errors is usually caused by several known or unknown factors (e.g., fringe modulation, numerically unstable points). Consequently, it is difficult to set denoising assessment criteria that can guarantee the reliability of denoising in different measurement scenarios. By contrast, the processing path and the assessment criteria of the proposed method are determined by the noise regions only. For this reason, the proposed method has the following advantages when compared with previous filtering algorithms:

  • 1.

    The correction algorithm is a general algorithm that can deal with the fringe-order problem of all types of CBFPP.

  • 2.

    The process is noninvasive and can remove the fringe-order noises around regions of abrupt change.

  • 3.

    The proposed method is effective for solving the problem of DPAs.

  • 4.

    The processing is sufficiently fast to allow filtering to be done in a short time.

The rest of this paper is organized as follows. Section 2 introduces CBFPP and explains the theoretical basis for the formation mechanisms of different types of fringe-order noise (NoisePJ, NoiseCP and NoiseNT). Section 3 describes the principles of the proposed correction framework. Sections 4 and 5 present the simulation and experimental results, respectively. Sections 6 and 7 discuss and summarize the paper.

Section snippets

Fringe projection profilometry

As presented in Fig. 1, the 3D reconstruction principle of classical FPP is based on the triangulation relationship between the projector, the measured object, and the camera. Therefore, it is important to match the features across the camera and projector with high reliability.

Among all the matching techniques, phase-shifting profilometry (PSP) is dominant because of its high accuracy, spatial resolution, and data density. More details about PSP can be found elsewhere [46]. Generally, the

Fringe-order correction procedures

To prevent the problem of neighboring pollution, in this paper the edges of the regions of abrupt change in the measured object act as segmentation lines to separate all the valid regions. However, because of the texture of the object and the problem of DPAs, the edges of the object cannot be extracted directly from the scene or the absolute phase map.

Fig. 4 shows the flow chart of the proposed correction processing. The major steps of the process are coarse correction and fine correction.

Simulations

The phase-coding method with 14 pattern periods is selected for simulation. The smooth data generated by the patterns without noise, shown in Fig. 9(a) and (b), are selected as the ground truth. The data contain one step height region and an “O” shaped hole. Gaussian noise (with a signal-to-noise ratio of 25) is added to the phase-coding patterns to generate the noisy data. As shown in Fig. 9(c) and (d), sparsely distributed noise, periodic noise, and DPAs largely influence the reconstruction

Experiments

The experimental setup shown in Fig. 11 comprises objects, a camera (BFS-U3-13Y3M-C), and a projector (LightCrafter 4500). The resolution of the projector is 1140 × 912, and the camera resolution is 1280 × 1024 with a 16-mm focal lens. The 16-period grating groups are generated by computer, projected onto the objects by the projector, and captured by the camera.

Discussion

We discuss the proposed method regarding the following aspects.

  • 1.

    Processing speed. The average computation time for each dynamic frame using MATLAB 2016b in a computer assembly with an Intelë Core i5 CPU @ 3.00 GHz and 8.0 GB RAM is 1.132 s with the proposed method. Considering related work [33], [35], [40], the present processing speed is higher than that of most existing algorithms (e.g., adaptive median filter, valid point detection). In addition, the processing could be accelerated even

Conclusions

In this paper, we explained and verified the formation mechanisms of different types of fringe-order noise and proposed a correction framework to solve the problem of fringe-order noise. There are two key points in this correction framework. For coarse correction, an edge-determined segmentation process is proposed and applied to preserve the edge of the measured object. Such a protection process would not protect any fringe order noises. For fine correction, the processing path and the

CRediT authorship contribution statement

Ji Deng: Conceptualization, Methodology, Software, Validation, Writing - original draft. Jian Li: Investigation. Hao Feng: Visualization, Writing - review & editing. Shumeng Ding: Investigation. Yu Xiao: Investigation. Wenzhong Han: Investigation. Zhoumo Zeng: Supervision, Project administration.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

This work was supported by the Major Scientific Instrument and Equipment Development Project in National Key Research and Development Program of China (grant no. 2016YFF0101802) and National Natural Science Foundation of China (grant no. 61873183).

References (54)

  • D. Zheng et al.

    Ternary gray code-based phase unwrapping for 3d measurement using binary patterns with projector defocusing

    Appl. Opt.

    (2017)
  • X. Chen et al.

    Modified gray-level coding method for absolute phase retrieval

    Sensors

    (2017)
  • L. William et al.

    Genetic method to optimize binary dithering technique for high-quality fringe generation

    Opt. Lett.

    (2013)
  • J. Sun et al.

    Improved intensity-optimized dithering technique for 3d shape measurement

    Opt. Lasers Eng.

    (2015)
  • C. Jiang et al.

    Pixel-by-pixel absolute phase retrieval using three phase-shifted fringe patterns without markers

    Opt. Lasers Eng.

    (2017)
  • Y. Ding et al.

    Fringe order correction for the absolute phase recovered by two selected spatial frequency fringe projections in fringe projection profilometry.

    Rev. Sci. Instrum.

    (2017)
  • R. Porras-Aguilar et al.

    Error correcting coding-theory for structured light illumination systems

    Opt. Lasers Eng.

    (2017)
  • H. Wang et al.

    Valid point detection in fringe projection profilometry

    Optics Express

    (2015)
  • H. Deng et al.

    3D information detection with novel five composite fringe patterns

    Mod. Phys. Lett. B

    (2017)
  • N. Karpinsky et al.

    High-resolution, real-time 3d imaging with fringe analysis

    Journal of Real-Time Image Processing

    (2012)
  • C. Zuo et al.

    Micro fourier transform profilometry (μftp): 3d shape measurement at 10,000 frames per second

    Opt. Lasers Eng.

    (2018)
  • J. Deng et al.

    Flexible depth segmentation method using phase-shifted wrapped phase sequences

    Opt. Lasers Eng.

    (2019)
  • J. Deng et al.

    Omnidirectional depth segmentation using orthogonal fringe patterns and multi-scale enhancement

    Opt. Lasers Eng.

    (2020)
  • Y. Chen et al.

    A modified fast parallel algorithm for thinning digital patterns

    Pattern Recognit. Lett.

    (1988)
  • S. Zhang et al.

    Gpu-assisted high-resolution, real-time 3-d shape measurement

    Opt. Express.

    (2006)
  • S. Feng et al.

    Automatic identification and removal of outliers for high-speed fringe projection profilometry

    Opt. Eng.

    (2013)
  • J. Geng

    Structured-light 3d surface imaging: a tutorial

    Adv. Opt. Photonics

    (2011)
  • Cited by (12)

    View all citing articles on Scopus
    View full text