1. Introduction

Three-dimensional shape measurement has been widely used in the manufacturing, entertainment, security and medical industry, where fringe projection technique is one of the most popular methods owing to its accuracy, speed, flexibility and versatility [1, 2]. However, for the objects with deep holes, grooves and steps, traditional stereo-based methods usually fail to completely measure the regions with self-occlusions due to the perspective difference between camera and projector caused by the system baseline [3]. Vertical scanning setup is developed to overcome such barriers, where the optical axis of projector and camera commonly placed parallelly or coaxially. The depth is encoded in the projector parameter retrieved from the fringe patterns, and recovered using the parameter-to-depth mapping established after calibration.

In terms of 3D reconstruction principle, the vertical scanning setups can be categorized into two types: shape-from-defocus and projector-triangulation. Shape-from-defocus is first proposed in the computer vision community [4]. The depth is recovered by analyzing the blur kernel of the defocused image. For fringe projection technique, the depth information is encoded in the projector lens defocus, which results in the degradation of the projected fringe patterns. This degradation can be estimated quantitatively using fringe contrast [5], modulation (called modulation measuring profilometry, MMP) [6–9], or phase error [10, 11], and then the depth mapping is established from these parameters directly. Another attempt is to recover the optical transfer function (OTF) of the defocus using spatial [12] or temporal [13, 14] system identification. Zhang and Nayar [13] established the depth mapping using the harmonics ratio, however it cannot separate the defocus blur from other types, such as translucency blur. Zhen et al. [14] modelled OTF as an ideal Gaussian function and established the depth mapping from its standard deviation σ, however this approximation is only valid in small defocus [15]. In general, the sensitivity of the aforementioned shape-from-defocus method is highly dependent on the variation of defocus degree, which requires the projector defocus changes dramatically within certain depth range. This requirement limits its sensitivity, especially when using telecentric camera. In addition, the degradation also results in lower SNR when retrieved the parameter from defocused projection.

Unlike stereo-triangulation using camera-projector constraint, the projector-triangulation methods utilize the triangulation formed between the projector optical axis and its emitted light ray. For a fixed camera pixel, the ray it observed varies in depth. The depth is recovered using the pinhole model with the localization of the emitted light ray estimated either in projector digital micromirror device (DMD) coordinates or camera image coordinates. This localization commonly achieves through phase-based methods. Liu et al. [16] estimated the displacement using speckle and fringe patterns. Zhao et al. [17, 18] proposed using the phase retrieved from circular fringe patterns to calculate the depth. However, the phase-based method requires sophisticated calibration to detect the zero-phase coordinate [18].

Parallel single-pixel imaging (PSI) technique can capture the light transport coefficients (LTCs) that describe the distribution of the received light for each camera pixel in projector coordinates. It has been proved in our previous studies that the LTCs captured by PSI can be used in stereo-triangulation setup, which enables high signal-to-noise ratio (SNR), high quality 3D shape measurement even under strong global illumination [19]. In this paper, we introduced the PSI technique to the vertical scanning setup with projector-triangulation principle called paraxial 3D shape measurement. The projection of the LTCs captured by PSI is utilized for the localization of the corresponding projector pixel to improve the robustness, and the depth is encoded in its radial distance to the optical center. To simplify the problem of optical center detection, third-order polynomial fitting is used for depth mapping and calibration. Experiments verified that the proposed method achieves high accuracy measurement, and high robustness to the detection error of projector optical center.

The reminder of this paper is given as follows. Section 2 explains the principle of the proposed method, including the review of paraxial 3D shape measurement and two-stage localization using PSI technique. Section 3 demonstrates the experiment setup and results. Section 4 summarizes the paper.

2. Principle

Unlike stereo-based 3D shape measurement using disparity, the depth information in paraxial system is encoded in the coordinates of the observed projector pixel by camera and then recovered from the calibration beforehand using projector-triangulation. In the proposed method, the depths are encoded in the radial distance to the optical center in projector DMD coordinates, which is determined using two-stage localization with PSI technique by projecting horizontal and vertical fringe patterns.

In this section, we first review the basic principle of paraxial 3D shape measurement. Then, the two-stage localization method with PSI is introduced.

2.1 Paraxial 3D shape measurement

The paraxial 3D shape measurement system consists of one telecentric camera and one projector, where both optical axes are set parallelly. A typical setup of paraxial 3D shape measurement system is illustrated in Fig. 1, where the telecentric camera observes the projected patterns reflected by the object and transmitting out a beam splitter.

 

Fig. 1. Schematic diagram of paraxial 3D shape measurement system.

Download Full Size | PDF

Regarding projector as a pinhole model, which is shown in Fig. 2, the Z-axis of world coordinates is parallel to the projector optical axis, whereas the X-Y plane is perpendicular to the optical axis and aligned with the image plane. Considering a camera pixel observes a 3D point P(X, Y, Z), owing to the telecentricity, the XY coordinates can be calculated as

 

Fig. 2. The geometric model of projector

Download Full Size | PDF

Given that telecentric lens has no perspective effect and low distortion, the depth information can be encoded in the projector image coordinates observed by each camera pixel. To be specific, assuming the radial distance of P in projector pixel coordinates ($\overline {p{O_p}}$) is r_p, then the depth Z can be estimated as

One thing should be mentioned is that this kind of reconstruction principle is only valid when P is distant from the projector’s optical center O_p. Otherwise the gradient will drop dramatically which decreases the measurement sensitivity. That is the reason why the projector and camera cannot be perfectly uniaxial.

2.2 Two-stage localization using PSI

It can be seen that projector pixel localization is the key step to estimate r_p. Traditional fringe projection methods use phase information to determine correspondences. However, defocus will degrade the fringe quality and the encoded phase suffers from interreflection, translucent and other global illumination conditions. Here, we propose using LTCs projection captured by PSI technique to establish the pixelwise mapping between camera pixel coordinates and projector pixel coordinates, which improves the robustness of paraxial 3D shape measurement. The aim of two-stage localization is to determine the corresponding projector pixel and calculate radial distance r_p.

2.2.1 Light transport coefficients for localization

LTCs describe the radiance distribution received by a camera pixel from the projector DMD coordinates. Mathematically, the radiance captured by a camera pixel (x_c, y_c) can be expressed as

For the localization problem in paraxial 3D shape measurement, because only the coordinates of LTC center are needed, Fourier slice theory [20] can be introduced to simplify the acquisition process. Unlike stereo-matching that requires the 2D distribution of LTCs, the horizontal and vertical projections containing the peak information of the LTCs are sufficient for localization only. In practice, the localization can be separated as two stages: the rough localization and fine localization.

2.2.2 Rough localization

Adaptive regional SI technique [21] is used for rough localization. According to the Fourier slice theorem, the 1D projection of an image can be recovered from its 1D scan in the frequency domain using inverse Fourier transformation (IFT).

The schematic diagram is shown in the Fig. 3. In this step, only horizontal and vertical fringe patterns with different frequencies are projected to recover the LTC’s 1D projection. The patterns can be expressed as

For each fringe pattern, the camera response is

After collecting all the Fourier series, the 1D projection of LTC can be recovered by applying IFT. Hence,

For a given camera pixel (x_c0, y_c0), the rough location of its corresponding projector pixel $({\tilde{x}_p},{\tilde{y}_p})$ can be determined through the peak of these 1D signals, hence

 

Fig. 3. The schematic diagram of rough localization. Vertical and horizontal fringes are projected on to the object. The camera captures the patterns and calculates the center of the visible region using Adaptive Regional SI technique [21].

Download Full Size | PDF

2.2.3 Fine localization

For the 3D measurement tasks with low accuracy, the rough localization is acceptable. However, for precision reconstruction, fine localization in subpixel level is necessary. Given that the high frequencies signals provide detailed information about the peak of LTC signal, additional fringe patterns are needed in this stage. Here, we use the periodical extension patterns in PSI technique to sharpen the LTC signal. Given that it’s a modification of PSI technique, it is recommended to refer to [19] for detailed theoretical background.

"The schematic diagram is illustrated in the Fig. 4. The projected patterns are the periodical extension of the base fringe patterns with the size that only covers the visible region of the camera pixel [19]. Given that the task is localization instead of 2D matching, Fourier slice theorem is also introduced to further reduce the data acquisition. Similar to the rough localization stage, only vertical and horizontal periodical extension patterns are utilized to obtained the 1D projections of the LTC signal. These patterns can be expressed as

In practice, the localization contains the following steps:

3. Experiment

The experiment setup is shown in Fig. 5. The DMD-based projector is PDC03 with the resolution of 1280×800, whereas the telecentric camera is Basler camera with telecentric lens MML044-SR130VI-18C. The resolution of camera is 1800×1500, while the magnification factor for the telecentric lens is 0.44. The optical axis of projector and camera are set parallelly, where only half of the projected region is recorded. The measurement volume is 28×23.5×4 mm³. We use 32 frequencies to localize the visible region in rough localization stage, and the size of visible region is set as 20×20 projector pixels. Consequently, given that only horizontal and vertical fringes are used, the total number of projected patterns is 208 considering the conjugate symmetry of Fourier transform. The detailed calculation method of the required pattern number can be found in Ref. [19].

 

Fig. 4. The schematic diagram for fine localization. (a) The visible region (orange box) of a camera pixel (x_c0, y_c0) is located using Adaptive SI technique in the rough localization stage, where the 1D projection of the LTC patch (h_s) can be placed back in the projector pixel coordinates. (b) after the coordinates transformation of h_S, the subpixel coordinate of the direct illumination light (x*_p, y*_p) can be obtained and the radial distance r_p is calculated.

Download Full Size | PDF

 

Fig. 5. Experiment setup.

Download Full Size | PDF

In this section, we first introduce the calibration procedure. Then, we test the performance of paraxial 3D reconstruction on 5 objects with different materials and surface textures. After that, the comparison experiments between traditional and proposed methods are conducted. And finally, standards are used to evaluate system accuracy.

3.1 Calibration

As shown in Eq. (3), the parameters calibrated for depth reconstruction are the coefficients of third-order polynomial. Owing to the nature of PSI technique, pixelwise calibration for each camera pixel can be easily achieved. A standard plate with translation stage is utilized to sample the LTC projections at different depths, where the digital indicator displays the depth digits. Before calibration, the translation stage is adjusted to align with the optical axis of telecentric camera. To determine the accurate optical center, a projector calibration is conducted, which takes the projector as an inverse camera [23]. The optical center is (738.6, 323.4).

The schematic diagram of calibration is illustrated in Fig. 6. At each depth Z_i, we projected the horizontal and vertical PSI fringe patterns generated using Equation (5-6) and (14-15). The projected fringe patterns were recorded by the telecentric camera, and then the two-stage localization was used to calculate the radial distance r_p for each camera pixel, where the optical center is set as the center of DMD coordinates. After collecting all the images at different depths, the radial distances were fed to the third-order polynomial fitting algorithm to determine the coefficients in Eq. (3). In this experiment, we sampled 34 depths equally for calibration. The total calibration scope was 4 mm in depth.

 

Fig. 6. The schematic diagram of calibration. The projector is fixed, whereas the standard plate is placed at different depths using translation stage, which is represented as the image plane. By projecting the PSI patterns and using the two-stage localization, the radial distance r_p in DMD coordinates can be calculated. According to the pinhole model, the r_p decreases as the depth increases.

Download Full Size | PDF

3.2 Paraxial 3D shape measurement

To test the robustness of the proposed method, five objects with different materials and surface textures are measured, including painted metal plate, frosted metal, coins and translucent ceramics. The objects are placed within the calibration scope determined in the previous section. For each object, 208 vertical and horizontal fringe patterns are projected onto the object and recorded by the telecentric camera. Two-stage localization is used to calculate the radial distance. The 3D coordinate is recovered using Eq. (1) and (3) after calibration. The reconstruction results are shown in Fig. 7, where the first column displays the objects, the second displays the image captured by the telecentric camera, and the third displays the reconstructed results.

 

Fig. 7. Experiments on objects with different textures and materials. The first column are the measured objects, where (a) is a painted metal plate with V-figure countersink, (b) a black metal plate with T-figure countersink and frosted texture, (c) and (d) Chinese coins, (e) a ceramic gauge block with translucent property. The second column (f-j) are the images captures by the telecentric camera, whereas the third column (k-o) are the corresponding reconstructed 3D data. The pseudo color illustrates the depth range.

Download Full Size | PDF

The first object (Fig. 7(a)) is a metal plate with through-hole and V-figure countersink, which is painted with white powders to make the surface diffuse. As can be seen in Fig. 7(k), the slope of countersink is clearly measured. There are some missing points at the upper side because the plate has a tilt angle and the points beyond the calibration scope are eliminated. The second object (Fig. 7(b)) is a black metal plate with its original frosted surface texture, where the holes are also precisely reconstructed. The third and fourth (Fig. 7(c-d)) object are coins with scratched surface, where the letters and grooves can still be seen in the results. The fifth object (Fig. 7(e)) is a ceramic block with a little translucent property. It can be seen that objects with surface textures other than the calibration plate are still measured with high complicity, which means that the reconstruction is insensitive to the surface texture and the proposed method can achieve robust measurement.

3.3 Comparison between traditional methods

To further investigate the performance of the proposed method, traditional PSI localization [19] and phase-shifting localization are used as the control group to compare the performance between the traditional methods and the proposed one.

We first compare the proposed method with the traditional PSI technique reported in [19] to compare the measurement quality. The original PSI was combined with the stereo triangulation setup. In this experiment, we transplant the traditional PSI algorithm to the paraxial system setup to make it applicable. The localization procedure is same as the determination of direct illumination point in the stereo-based PSI but without epipolar constraint. Accordingly, the traditional PSI method requires 1132 fringe patterns to capture all the necessary frequencies for two-dimensional inverse Fourier transform [19]. And once the corresponding projector pixels are determined, the reconstruction procedure described in Section 2.1 are applied. The experiment results in the form of depth map are displayed in Fig. 8. From the absolute deviation calculated from the difference between (b) and (c), it can be seen that the reconstruction quality of the proposed method is almost same with the traditional one, except that the traditional PSI method reconstructed more edge points. However, owing to the further use of Fourier slice theory, the proposed method only requires 208 fringe patterns, which significantly improves the measurement speed and computational efficiency.

 

Fig. 8. The comparison between traditional PSI method and the proposed method with same paraxial setup. (a) is the V-figure countersink as the object. (b) is the depth map reconstructed using traditional PSI with full-direction frequencies and 2D inverse Fourier transform. (c) is the depth map reconstructed using the proposed method. (d) is the absolute deviation between (b) and (c), which is calculated from the difference between the two depth maps.

Download Full Size | PDF

In the second experiment, we replace the two-stage localization technique in the paraxial system with traditional 4-step phase-shifting as the control group. The system is recalibrated. At each depth, horizontal and vertical phase-shifting patterns are projected and the corresponding projector pixel coordinate is determined using the unwrapped absolute phase. In this experiment, the fringe periods are chosen as 22, 24 and 26, and the coins with scratched surface are measured. Reconstruction results are demonstrated in Fig. 9. Each column displays the same reconstruction, where the first row is the data shown in depth map to illustrate the missing data, whereas the second row in the form of point clouds to highlight the outliers. (a), (b), (e) and (f) are the data measured using phase-based technique, whereas (c), (d), (g) and (h) are the data using proposed method. The missing data and outliers of the control group is significantly noticeable than the proposed method, which means the complex surface texture degrades the SNR of phase-based method. In contrast, the proposed method can achieve more complete measurements with higher SNR even under this complex texture condition. This is because the PSI-based method recorded more images, which provides more information for the 3D reconstruction.

 

Fig. 9. Comparison between phase-shifting localization technique (Phase-based method) and two-stage localization technique (The proposed method). (a-d) are the depth maps of the measured coins, which illustrates the missing parts. (e-h) are the same reconstructed coins above but in the form of point clouds with different angles, where the yellow boxes highlight the outliers. (a), (b), (e) and (f) are the data measured using phase-shifting technique where the missing part and outliers can be clearly seen. (c), (d), (g) and (h) are the data measured using two-stage localization with PSI technique, which results in better quality.

Download Full Size | PDF

3.4 Accuracy evaluation

To further investigate the performance of the proposed method, gauge blocks and standard plate are measured. All the objects are placed within the calibration scope and the projected patterns are same as the previous experiments. We first test the accuracy, and then the influence of optical center detection error is investigated to test the robustness.

Four diffuse gauge blocks (1.2 mm, 1.4 mm, 1.6 mm, 2.0 mm) are utilized to test the depth accuracy. The gauge blocks were assembled as standards, where the heights are the thickness of the blocks. The bottom gauge is placed at the same distance, where the distance of optical path from projector to object is about 23 cm.The measured height is calculated using plane fitting. The experiments results are given in Table 1 and Fig. 10. As can be seen in Table 1, the max error in depth is within 20 μm. The error increases as the top gauge block close to the projector. One explanation of this thickness error is the residual error of polynomial fitting. The r_p increases as the object close to the projector, which results in the increasing residual error of high order terms. Besides, the misalignment error in the calibration can also influence this depth mapping. However, more elaborate calibration will improve the accuracy.

 

Fig. 10. Experiment results on gauge blocks. Two gauges are assembled as T-shape, where the height is the thickness of the upper gauge. The green part illustrates the surface of the upper gauge whereas the blue part the surface of the bottom gauge. (a-d) are the reconstructed results of gauge blocks where the height for each block are 1.2 mm, 1.4 mm, 1.6 mm and 2.0 mm respectively.

Download Full Size | PDF

Table 1. Experiment data on gauge blocks

View Table | View all tables in this article

A standard plane is also measured, where the RMSE after plane fitting is 43 μm. The reconstruction results are shown in Fig. 11(a), whereas the error distribution is shown in Fig. 11(b). It is noticed that the error on the right side is slightly larger than that of the left side. This is probably because the right side is near the optical center, where the gradient of the change of the radial distance r_p is smaller than the far side which results in larger noises. As illustrated in Fig. 12, the gradient of the far side pixel (150,70) almost doubles that of the pixel (1700,70). In practice, this region can be eliminated by only using the far side of the projected plane to further improve the accuracy. Besides, the random error induced by the projector, such as quantitative error and intensity error, can be further reduced when using high power projector with high resolution and high-quality lens.

 

Fig. 11. Experiment results on standard plane. (a) is the 3D reconstruction of standard plane. (b) is the error distribution after plane fitting, where the RMSE of plane fitting is 43 μm.

Download Full Size | PDF

 

Fig. 12. The calibration curve of camera pixel (150,70) and (1700,70). (a) is the standard plane captured by the telecentric camera, where the locations of the two sample pixels in the image coordinates are illustrated. The blue spot is (150,70) whereas the orange spot is (1700,70). (b) demonstrates the calibration curve of each camera pixel after removing the offset (the initial distance).

Download Full Size | PDF

We also investigate the robustness on optical center selection by adjusting the optical center coordinate and comparing the plane fitting error. Experiment results are demonstrated in Table 2 and Fig. 13. It can be seen that the maximum error change is 3 μm when the detection error of the optical center is about 100 pixels, which proves that the third-order polynomial fitting is robust to the selection of optical center. As can be seen in Fig. 13, although the error increases in the near side corners, the error distribution in the far side is nearly invariant. Therefore, it can be inferred that the optical center can be approximated as the center of DMD as long as the effective measurement region is far from the optical center. In practice, the near side region can be eliminated by moving the projector optical axis further outside the camera’s field of view.

 

Fig. 13. The error distribution after plane fitting. (a) is reconstructed with (738.6, 323.4), which is the calibration results; (b) is with (618, 323); (c) is with (618, 394); (d) is with (640, 400), which is the center of the DMD plane; (e) is with (650, 410);

Download Full Size | PDF

Table 2. Experiment results of the robustness on optical center detection

View Table | View all tables in this article

4. Conclusion

In this research, we proposed a paraxial 3D measurement method with the PSI theory. The depth information is encoded in the radial distance to the optical center in projector pixel coordinates, and the camera-projector coordinates mapping is obtained using two-stage localization using PSI with Fourier slice theorem. Five objects and standards are used to test the performance of the proposed method. Experiments verified the feasibility and showed that this method can achieve robust measurements with high accuracy. The depth accuracy achieves up to 20 μm while the plane-fitting error 43 μm.

From the perspective of 3D measurement, the paraxial setup can alleviate the problem of occlusion and shadow, which reduces the missing parts and improves the complicity. Compared with phase-based method, the proposed method enables high SNR measurement, which is more robust to the noises and insensitive to complex surface textures owing to the merits of PSI technique. In addition, the third order polynomial fitting simplifies the optical center detection and overcomes the zero-phase problem. In the context of PSI, it proves that paraxial setup also can be used for high accuracy 3D reconstruction besides stereo triangulation. Using the Fourier slice theorem that only requires horizontal and vertical fringes, the number of fringe patterns utilized in traditional PSI is greatly reduced. However, just as other uniaxial or off-axial 3D measurement system, the measurement range is limited because of the use of telecentric camera. Future research can be focused on simple but accurate calibration, error analysis, efficient LTC reconstruction, and multi-sensor fusion with traditional FPP.