Abstract
Using 1-bit binary patterns for three-dimensional (3D) shape measurement has been demonstrated as being advantageous over using 8-bit sinusoidal patterns in terms of achievable speeds. However, the phase quality generated by binary pattern(s) typically are not high if only a small number of phase-shifted patterns are used. This paper proposes a method to improve the phase quality by representing each pattern with the difference of two binary patterns: the first binary pattern is generated by triangular pulse width modulation (TPWM) technique, and the second being π shifted from the first pattern that is also generated by TPWM technique. The phase is retrieved by applying a three-step phase-shifting algorithm to the difference patterns. Through optimizing the modulation frequency of the triangular carrier signal, we demonstrate that a high-quality phase can be generated for a wide range of fringe periods (e.g., from 18 to 1140 pixels) with only six binary patterns. Since only 1-bit binary patterns are required for 3D shape measurement, this paper will present a real-time 3D shape measurement system that can achieve 30 Hz.
© 2017 Optical Society of America
1. Introduction
Three-dimensional (3D) shape measurement plays a significant role in many areas ranging from manufacturing to medicine. Numerous techniques have been developed including Moiré, holography, and fringe projection. Among these methods, the digital fringe projection technique has been exhaustively studied and widely applied due to its simple setup, automatic data processing, high-speed and high-resolution measurement capabilities [1–3]. However, simultaneously achieving high accuracy and high speed remains challenging.
For high-speed 3D shape measurement, it has been demonstrated that using 1-bit binary patterns is advantageous over 8-bit sinusoidal phase-shifted fringe patterns especially on the digital-light-processing (DLP) projection platform [4]. By properly defocusing the projector, square binary patterns becomes pseudo sinusoidal patterns [5]. Researchers have achieved speed breakthroughs with the binary defocusing technique on the DLP platform [4], yet such a method requires careful adjustment of the projector’s lens to within a small out-of-focus range, limiting its depth measurement capability.
To improve phase quality, Ayubi et al. [6] developed the sinusoidal pulse width modulation (SPWM) technique, Wang and Zhang [7] developed the optimal pulse width modulation (OPWM) technique, and later Zuo et al. [8] developed the optimization strategies for the SPWM technique. These modulation and optimization techniques could improve phase quality when the fringe period is relative small, however, if the fringe period is very large or too small, they fail to produce high-quality phase. To improve the phase quality for large fringe periods, Xian and Su developed the area modulation technique [9] to generate high-quality sinusoidal patterns using the high-accuracy fabrication, and Wang and Zhang [10] employed the dithering/halftoning technique developed in the printing field to generate high-quality fringe patterns through defocusing. However, the area modulation and dithering technique only works well for large fringe periods. In summary, the attempts to optimize the binary pattern itself have substantially improved measurement quality, yet they only work well for a limited range of fringe period variations especially when a small number of phase-shifted fringe patterns are used.
In lieu of relying solely on those patterns, researchers also attempted to improve phase quality by taking advantage of the temporally acquired information. Ayubi et al. [11] directly took 8 binary bits of the sinusoidal pattern, projected and captured each bit pattern separately sequentially, and finally combined them after image acquisition. The concept of this idea is good, yet, due to non-perfect hardware system and sampling, such a method does not work well in practice. Furthermore, it requires the acquisition of 8 images for each sinusoidal fringe pattern, which is not desirable for high-speed applications. Zhu et al. [12] developed a method to improve phase quality by optimizing four binary dithered patterns. Yet, this method still requires the 4 patterns to be used to represent one sinusoidal pattern. Silva et al. [13] proposed the colored PWM technique to reduce the number of images to be three. Unlike the previous temporal structured pattern modulation methods, this technique requires the blend of color for pattern generation. Even though such a method could improve the phase quality, they ignored the fact that the color spectrum response of the projector and that of the camera, making such a technique very difficult for practical hardware systems to achieve similar level of quality as the simulation demonstrates. Zuo et al. [14] then employed triangular pulse width modulation (TPWM) technique, and represented each desired patterns with an average of 2 binary patterns for better quality phase generation. This method works reasonably well for middle range fringe periods, but may be challenging to be directly employed for the small or large fringe periods.
This paper proposes a method to generate high-quality fringe pattern by taking the difference of two 1-bit binary patterns: the first pattern is generated by the TPWM technique, and the second being π shifted from the first pattern is also generated by TPWM technique. A three-step phase-shifting algorithm is then applied to the difference of modulated patterns for phase retrieval. Through optimizing the modulation frequency of the triangular carrier signal, we will demonstrate that high-quality phase can be generated for a wide range of fringe periods (e.g., from 18 to 1140 pixels) with only six binary patterns. This paper will present the theoretical foundation of the proposed technique, and the optimization framework for “best” carrier frequency selection, and both simulation and experimental results will be carried out for validation. Finally, the optimized patterns are implemented on the DLP development kit (e.g., LightCrafter 4500) to achieve 30 Hz high-quality 3D shape measurement.
Section 2 explains the principle of the proposed method. Section 3 shows simulation and experimental results to verify the performance of the proposed method; and Sec. 4 summarizes the paper.
2. Principle
2.1. Three-step phase-shifting algorithm
Phase-shifting methods are widely used in optical metrology because of their speed and accuracy [15]. In this paper, we adopt a three-step phase-shifting algorithm with a phase shift of 2π/3 for high-speed application; and the fringe patterns can be mathematically described as,
where I′ (x, y) is the average intensity, I″ (x, y) the intensity modulation, and ϕ(x, y) the phase to be solved for. The phase can be calculated from Eqs. (1)–(3) as The arctangent function yields a phase value ranging [−π, +π) with 2π discontinuities. In order to remove the 2π discontinuities and obtain the absolute phase map, a temporal phase unwrapping algorithm (e.g., two- and multi-frequency phase-shifting algorithm) is usually required. The unwrapped phase can be converted to (x, y, z) coordinates for each point once the system is calibrated.2.2. Double-pattern triangular pulse width modulation (TPWM) technique
Figure 1 illustrates the basic principle of the TPWM technique. By comparing a reference sinusoidal signal with a carrier triangular signal, the TPWM wave can be generated with switched binary-state pulses. For a naturally sampled TPWM signal, the spectrum can be modeled by a double Fourier integral analysis method [16]. And if the carrier triangular signal is symmetric about zero with an amplitude of Cm (= 1 in our case), and the reference sinusoidal signal with an amplitude of R1 (= 1 in our case) without DC offset, the modulated signal switches from −1/2 to +1/2 (binary states), as shown in the bottom row images of the Fig. 1. This special case of TPWM technique is also called bipolar double-edge modulation (BDEM) [16]. Mathematically, the modulated signal can be described as
where M = 2R1/Cm is the modulation index, θ0 is the initial phase of the reference sinusoidal signal, θc is the initial phase of the carrier signal, and is the Bessel functions of the first kind. Here j denotes the imaginary symbol.Assume θc = θ0 = 0 in Eq. (5), then
Figure 1(a) illustrates an example of generating PWM signal with initial phase of θ0 = 0.If the reference sinusoidal wave is shifted by π when θc = 0, as shown in Fig. 1(b), another TPWM wave can be obtained, whose mathematical representation can be written as
Taking the the difference of these two TPWM signals shown in Eqs. (7) and (8) leads to
where m = 2m′ and n = 2n′ − 1.Figure 2 illustrates the difference signal by taking the difference between those two TPWM signals.
By examining Eqs. (7) and (8), the difference of these two equations, as shown in Eq. (9), has the following properties
- all the harmonics at frequency introduced by the carrier signal are cancelled. That is, no signal at frequencies f = mfc present in the difference signal;
- when m + n are even numbers, all the sideband harmonics are cancelled. That is, no signals at frequencies f = mfc + nf0 present in the difference signal if m + n are even numbers;
- when m are odd and n are even numbers, all the sideband harmonics are cancelled. That is, no signals at frequencies f = mfc + nf0 present in the difference signal if m are odd and n are even numbers; and
- there is no DC signal.
2.3. Computation framework for optimal modulation frequency selection
For any frequency (f0) fringe pattern (i.e., the fringe period T = 1/f0 pixels), we generated the resultant three phase-shifted modulated patterns and three π shifted modulated patterns with modulation frequencies ranging from 3 f0 to (T/2 − 1) f0. And for each modulation frequency, we applied n × n Gaussian filters with standard deviation of n/3, here n = 3, 5, 7, …, 15, 17, 19. These filter sizes represent various amounts of defocusing levels. The phase was retrieved from the difference fringe patterns by employing the three-step phase-shifting algorithm. We computed the phase root-mean-square (rms) error for each filter size, and plotted the phase rms error as a function of filter size for each modulated pattern. These error plots were then inspected to determine the “best” candidate for a given frequency f0 fringe patterns.
Figure 3 shows an example of selecting the best modulation frequency for the fringe frequency of f0 = 1/60 pixels−1, or fringe period of T = 60 pixels. The modulation frequencies we used range from fc = 3 f0 to fc = 29 f0, and thus 27 different candidates were created. Four representative candidates and the associated phase rms error plots are shown in Fig. 3. From all phase rms error plot, one can clearly see that the modulation frequency fc = 25 f0 should not be considered further because the overall phase rms error (shown in Fig. 3(h)) is larger than other three especially when filter sizes are small. Comparing to the phase rms plots using modulation frequencies fc = 14 f0 and fc = 22 f0, the resultant phase rms error obtained from the modulation frequency of fc = 11 f0 is significantly larger when the filter size n = 3, therefore, modulation frequency fc = 11 f0 should not be considered further. For the remaining two candidates, although when the filter size is n = 3 the phase error is slightly smaller when modulation frequency is fc = 14 f0, its phase error drops less when filter size increases to n = 5, therefore, the modulation frequency fc = 22 f0 was chosen as the “best” candidate.
Figure 4 shows the frequency spectrum analysis of the “best” modulation pattern chosen from the previous step. Figure 4(a)–4(c) respectively shows the PWM pattern, the π phase-shifted pwm pattern, and the difference pattern between the regular and the π phase-shifted pattern. We then took one horizontal cross section of each pattern, and analyzed its frequency spectrum. Figure 4(d) and 4(e) respectively shows the frequency spectrum of the regular PWM pattern and that of the π phase-shifted pattern. On can clearly see that the DC component is not zero, and the dominant 2nd – 6th harmonics are not zero, resulting in phase error and thus measurement error for 3D shape measurement. Figure 4(f) shows the spectrum of the difference pattern. It clearly shows no DC component, and the 2nd, 4th–6th order harmonics are completely eliminated. The only low-order harmonic component remained is the 3rd order, which can be effectively eliminated by a three-step phase-shifting algorithm [17]. Therefore, only a small amount of defocusing is necessary to effectively suppress the 7th and higher order harmonics for high-quality phase-based 3D shape measurement.
It is important to note that when we chose the “best” candidate, the chosen phase-shifting algorithm plays a critical role. Although the pattern shown in Fig. 3(a) may appear the best sinusoidal structures, the phase error is clearly much larger than the apparently less sinusoidal pattern shown in Fig. 3(c). This fact was mistakenly understood by Ayubi et al. [18], leading to their incorrect comments on the optimal pulse width modulation paper written by Zhang and Wang [7]. It should also be noted that if a different phase-shifting algorithm is chosen, the “best” candidate may be different. Therefore, our selection is closely tied to the phase-shifting algorithm used for high-speed applications.
3. Experiments
We first employed the optimal modulation frequency selection process discussed in Subsec. 2.3 to determine the “best” modulation frequencies for fringe periods ranging from T = 18 to T = 1140 pixels (i.e., fringe frequency f0 = 1/T ranging from 1/1140 to 1/18). Table 1 summarizes the fringe periods and the corresponding “best” modulation frequency for each fringe period. Note that on the table K denotes the modulation frequency of K × f0.
Figure 6 shows the phase rms error for all different fringe periods with the best modulation frequency summarized in Table 1. We then applied four different filter sizes to those best modulated patterns and compute phase rms errors. Figure 5(a)–5(d) respectively shows the resultant filtered pattern with filter size n = 3, 9, 15, 21 pixels for the pattern shown in Fig. 3(g) where T = 60 pixels.
Figure 6(a) shows phase rms error for all these patterns under different amounts of defocusing (e.g., applying different sizes of Gaussian filters). This figure shows that the phase rms error is very small overall (around 0.1 rad) when fringe patterns are nearly focused (i.e., the filter size is n = 3), even for very wide fringe patterns (e.g., T = 1140 pixels). This simulation result indicates that the proposed method can work well for a wide range of fringe patterns even under a small amount of defocusing (or the projector is nearly in focus). One may also notice when the filter size increases (i.e., the amount of defocusing increases), the phase error decreases, as expected. However, if the amount of defocusing is too much (e.g., the filter size n = 19 pixels), the phase error for the narrow fringe patterns (e.g., T = 18 pixels) actually increases due to the reduced fringe contrast.
We then evaluated the quality of the modulated binary patterns through experiments. We developed a 3D shape measurement system that includes a complementary metal-oxide-semiconductor (CMOS) camera (model: PointGrey GS3-U3-23S6M-C) that is attached with a 12 mm focal length lens (model: Computar M1214-MP2). The camera resolution was set as 640 × 480 and a DLP developmental kit (model: Texas Instruments Lightcrafter 4500) with a 912 ×1140 native resolution. The Lightcrafter projector was synchronized with the camera by using an external microprocessor (model: Arduino UNO) that simultaneously sends two trigger signals: one to the camera and one to the projector. The system was calibrated using the out-of-focus structured light system calibration method developed by Li et al. [19].
A flat white paper was measured to characterize the performance. We captured six binary patterns for each fringe period. We compared the measured phase quality by taking the difference between the measured phase Φ and a reference phase Φr. The reference map was generated by taking 20 averages of 18-step phase-shifted fringe patterns with a fringe period of T = 18 pixels, and then smoothed out by a Gaussian filter with a size of 21 × 21 pixels. To determine phase error, we took the difference between the measured phase map Φ with a fringe period of T pixels and the reference phase map Φr with a fringe period of 18 pixels as
Seven different amounts of defocusing was used and Figure 6(b) shows phase difference rms error for all fringe periods. They overall conform with the simulation data fairly well: the phase errors are consistently very small even when the projector is nearly focused.Since high-quality phase can be generated with different fringe periods, it is straightforward to realize multi-frequency phase-shifting algorithm with these modulated patterns for absolute 3D shape measurement. Using the same 3D shape measurement hardware system, we measured a complex statue with fringe periods of T = 36, 120, 360 and 1140 pixels. Figures 7 and 8 show the results. Figure 7(a) shows the photograph of the statue measured. Figure 7(b)–7(e) respectively shows one of the phase-shifted fringe patterns for fringe period 36, 120, 360, and 1140 pixels. These patterns clearly demonstrated that the projector is nearly focused since the binary structures are quite obvious on long period fringe patterns. We then computed the difference fringe patterns, and Figure 7(f) shows one difference pattern with a fringe period of T = 36 pixels. The wrapped phase can be computed from these difference fringe patterns for each fringe periods. Figures 7(g)–7(j) show the corresponding wrapped phase maps.
The temporal phase unwrapping algorithm can be then applied to unwrap all high-frequency phase maps, and the unwrapped high-frequency phase map can then be used for 3D shape reconstruction. Figure 8 shows the results. Since when the fringe period T = 1140 pixels, one single fringe covers the entire projection span, no phase unwrapping is necessary. In other words, the wrapped phase, ϕ1140, is identical to unwrapped phase Φ1140 (i.e., Φ1140 = ϕ1140). There are multiple methods to unwrap the highest frequency phase (i.e., ϕ36 at T = 36 pixels) with these four frequency wrapped phase maps. We first directly used Φ1140 to unwrap the high-frequency phase ϕ36 for 3D shape recovery, as shown in Fig. 8(a). We then combined one of the middle range frequency phase ϕ360 or ϕ120 with Φ1140 to unwrap the high frequency phase for 3D reconstruction. Basically, we used Φ1140 to unwrap the middle range frequency phase, and used the unwrapped middle range frequency phase to unwrap the high frequency phase. Figure 8(b) and 8(c) respectively shows the 3D result by adding the wrapped phase of fringe period 360 pixels, ϕ360 and that of fringe period of 120 pixels, ϕ120. Finally, we combined all these frequency phase maps together to unwrap the highest frequency phase and reconstructed 3D shape, as shown in Fig. 8(d).
Comparing with all these different combinations, one can visually see that directly unwrapping the highest frequency phase with the lowest frequency phase provides overall high-quality result, although there some unwrapping artifacts near the boundary areas: the spiky noise points as shown in Fig. 8(e) due the largest difference between the low fringe period and the high fringe period; adding a middle range phase can reduce the boundary unwrapping artifacts but may not be able to completely eliminate them all; and using all four frequency phase provides the cleanest result, as expected. These experimental results demonstrated that 1) the phase quality generated by the proposed method are indeed high even for the very wide fringe pattern (i.e., T = 1140 pixels) since it can be used to unwrap the high frequency phase directly, and 2) adding other frequency components can increase measurement quality but requiring the acquisition of additional fringe patterns, which may not be desirable for high speed applications.
We also carried out real-time 3D shape measurement using the same hardware setup. For these experiments, the projector refreshes binary patterns at 360 Hz, and the camera captures the projected fringe patterns at the same frame rate. After data acquisition, we employed the two-frequency phase-shifting algorithm, and chose the low frequency fringe period as 1140 pixels and the high-frequency fringe period as 36 pixels. Since 12 patterns are needed to recover one 3D geometry, the actual 3D shape measurement speed is 30 Hz. Figure 9 shows some 3D frames and the associated Visualization 1 and Visualization 2 contain the entire captured sequences. Again, the phase quality from such a low frequency fringe patterns is sufficient to unwrap such a high-frequency phase map, albeit there are some phase unwrapping artifacts near the boundary areas. It is important to note that we employed a small (i.e., 5 × 5) filter to reduce phase unwrapping artifacts.
Furthermore, we measured dynamic hand gesture motions and facial expressions with the enhanced two-frequency phase-shifting algorithm [20]. Here we chose the fringe period of 360 pixels for the low-frequency fringe patterns and 36 pixels for the high-frequency fringe patterns. The entire setting was the same as those data shown in Fig. 9, and the hands and facial motions were ensured to be as close as possible. Figure 10 shows some typical frames of the measurement results; and the associated Visualization 3 and Visualization 4 show the entire captured sequences. Comparing with 3D data reconstructed from the standard two-frequency phase-shifting algorithm, the enhanced two-frequency phase-shifting algorithm further improved the quality of reconstruction by reducing phase unwrapping artifacts near the boundary areas, albeit the overall depth range is limited.
All these simulation and experimental results demonstrated that our proposed a double-pattern TPWM technique could generate high-quality phase for a wide range of fringe periods with only six binary patterns. With this capability, the multi-frequency phase-shifting algorithm was successfully realized for absolute phase retrieval.
4. Summary
This paper has presented a method to generate high-quality phase by representing each pattern with the difference of two binary patterns: the first binary pattern is generated by triangular pulse width modulation (TPWM) technique, and the second being π shifted from the first pattern is also generated by TPWM technique. This paper also presented a framework to select the optimal modulation frequency of the triangular carrier signal to generate the highest possible phase quality when a phase-shifting algorithm is applied. Both simulation and experiment demonstrated that high-quality phase can be generated for a wide range of fringe periods (e.g., from 18 to 1140 pixels) with only six binary patterns. This paper further developed a 3D shape measurement system that achieved 30 Hz using binary patterns generated by the TWPM technique.
Funding
Directorate for Engineering (ENG), National Science Foundation (NSF), (CMMI-1531048); National Natural Science Foundation of China, (61603360).
References and links
1. S. Gorthi and P. Rastogi, “Fringe projection techniques: Whither we are?” Opt. Laser. Eng. 48, 133–140 (2010). [CrossRef]
2. S. Zhang, “Recent progresses on real-time 3-d shape measurement using digital fringe projection techniques,” Opt. Laser Eng. 48, 149–158 (2010). [CrossRef]
3. X. Su and Q. Zhang, “Dynamic 3-d shape measurement method: A review,” Opt. Laser. Eng 48, 191–204 (2010). [CrossRef]
4. B. Li, Y. Wang, J. Dai, W. Lohry, and S. Zhang, “Some recent advances on superfast 3d shape measurement with digital binary defocusing techniques,” Opt. Laser Eng. 54, 236–246 (2014). [CrossRef]
5. X. Y. Su, W. S. Zhou, G. Von Bally, and D. Vukicevic, “Automated phase-measuring profilometry using defocused projection of a Ronchi grating,” Opt. Commun. 94, 561–573 (1992). [CrossRef]
6. G. A. Ayubi, J. A. Ayubi, J. M. D. Martino, and J. A. Ferrari, “Pulse-width modulation in defocused 3-d fringe projection,” Opt. Lett. 35, 3682–3684 (2010). [CrossRef] [PubMed]
7. Y. Wang and S. Zhang, “Optimal pulse width modulation for sinusoidal fringe generation with projector defocusing,” Opt. Lett. 35, 4121–4123 (2010). [CrossRef] [PubMed]
8. C. Zuo, Q. Chen, S. Feng, F. Feng, G. Gu, and X. Sui, “Optimized pulse width modulation pattern strategy for three-dimensional profilometry with projector defocusing,” Appl. Opt. 51, 4477–4490 (2012). [CrossRef] [PubMed]
9. T. Xian and X. Su, “Area modulation grating for sinusoidal structure illumination on phase-measuring profilometry,” Appl. Opt. 40, 1201–1206 (2001). [CrossRef]
10. Y. Wang and S. Zhang, “Three-dimensional shape measurement with binary dithered patterns,” Appl. Opt. 51, 6631–6636 (2012). [CrossRef] [PubMed]
11. G. A. Ajubi, J. A. Ayubi, J. M. D. Martino, and J. A. Ferrari, “Three-dimensional profiling with binary fringes using phase-shifting interferometry algorithms,” Appl. Opt. 50, 147–154 (2011). [CrossRef]
12. J. Zhu, P. Zhou, X. Su, and Z. You, “Accurate and fast 3d surface measurement with temporal-spatial binary encoding structured illumination,” Opt. Express 24, 28549–28560 (2016). [CrossRef] [PubMed]
13. A. Silva, J. L. Flores, A. Munoz, G. A. Ayubi, and J. A. Ferrari, “Three-dimensional shape profiling by out-of-focus projection of colored pulse width modulation fringe patterns,” Appl. Opt. 56, 5198–5203 (2017). [CrossRef] [PubMed]
14. C. Zuo, Q. Chen, G. Feng, S. Feng, F. Feng, R. Li, and G. Shen, “High-speed three-dimensional shape measurement for dynamic scenes using bi-frequency tripolar pulse-width-modulation fringe projection,” Opt. Laser Eng. 51, 953–960 (2013). [CrossRef]
15. D. Malacara, ed., Optical Shop Testing (John Wiley and Sons, NY, 2007), 3rd ed. [CrossRef]
16. J. Sun, Pulse Width Modulation (Springer, NY, 2012), chap. 2, pp. 25–61.
17. L. Ekstrand and S. Zhang, “Three-dimensional profilometry with nearly focused binary phase-shifting algorithms,” Opt. Lett. 36, 4518–4520 (2011). [CrossRef] [PubMed]
18. G. A. Ayubi and J. A. Ferrari, “Optimal pulse width modulation for sinusoidal fringe generation with projector defocusing: comment,” Opt. Lett. 36, 808 (2011). [CrossRef]
19. B. Li, N. Karpinsky, and S. Zhang, “Novel calibration method for structured light system with an out-of-focus projector,” Appl. Opt. 53, 3415–3426 (2014). [CrossRef] [PubMed]
20. J.-S. Hyun and S. Zhang, “Enhanced two-frequency phase-shifting method,” Appl. Opt. 55, 4395–4401 (2016). [CrossRef] [PubMed]