In this paper, a temporal shift unwrapping technique is presented for solving the problem of shift wrapping associated with spatial shift estimation (SSE)-based fringe pattern profilometry (FPP). Based on this technique, a novel 3D shape measurement method is proposed, where triangular patterns of two different spatial frequencies are projected. The patterns of the higher frequency are used to implement the FPP, and the one with lower frequency is utilized to achieve shift unwrapping. The proposed method is able to solve the shift unwrapping problem associated with the existing multi-step triangular pattern FPP by projection of an additional fringe pattern. The effectiveness of the proposed method is verified by experimental results, where the same accuracy as existing multi-step triangular pattern FPP can be achieved, but enabling the measurement of objects with complex surface shape and high steps.
© 2014 Optical Society of America
As an enabling technology for noncontact three-dimension (3D) profile measurement, fringe projection profilometry (FPP)  has attracted increasing research efforts due to many potential applications. Figure 1 shows the structure of a FPP system, consisting of a digital video projector, a CCD camera and the reference plane. The project generates a set of images with a particular fringe pattern, which are casted respectively onto the reference plane and the surface of the object, and the reflections are captured by the CCD camera. Due to the variance of the height of the object surface, the image patterns reflected by the object surface are a deformed version of the ones from the reference plan. These deformed patterns carry the information of surface shape, and hence can be used to retrieve the profile of the object.
Many approaches have been developed during the past decades for FPP, e.g., Fourier transform profilometry (FTP) , phase shifting profilometry (PSP) [3–5], modulation measurement profilometry (MMP) , spatial phase detection (SPD) [7, 8], phase lock loop (PLL) profilometry , Moiré technique (MT) , laser triangulation measurement , color-coded fringe projection [12–14] and other methods [15, 16]. Among these approaches, the most widely used are those based on analysis of the phase maps of the fringe patterns, such as FTP and PSP. With these phase based approaches, fringe patterns projected are sinusoidal or periodic, and the deformed fringe patterns are considered as the result of phase modulation of the original patterns projected. Detection of phase maps from original and deformed fringe patterns enables the retrieval of the 3D shape.
Although phase based approaches have many advantages, they also suffers from a number of weaknesses. A major problem is the influence of nonlinear distortions inherent to digital video projection [17, 18], which makes it difficult for the original fringe patterns to be either sinusoidal or even periodic, which are required by phase based approaches. As an effort to solve the problem, an approach was proposed by Hu, et al. [19–21], referred to as spatial shift estimation (SSE) profilometry. Rather than relying on the analysis of the two phase maps, the technique utilizes the spatial shift of the corresponding pixels on the two fringe patterns, thus leading to a great advantage. The projected fringe patterns are no longer required to be sinusoidal or even periodic, resulting in an increased flexibility for the selection of the fringe patterns. Recently, Wu, et al.  introduced a multiple-step triangular-pattern spatial shift estimation algorithm by combining inverse function based shift estimation (IFSE)  with the multiple-step triangular-pattern phase shifting algorithm . Compared to the IFSE approach in , the measurement accuracy is greatly improved due to the use of multiple patterns. Also compared to the existing intensity ratio based techniques , the proposed in  does not suffer from the influence of non-linear distortion.
While the multiple-step triangular-pattern spatial shift method is characterized by the above advantages, its performance relies on the result of shift unwrapping. This problem arises as the result of fringe reuse (that is, use of fringes with periodic light intensity variance). In the SSE based approaches, spatial shift between corresponding pixels on the two fringe patterns can only be detected within the range of [0, λ], where λ is the wavelength, or the width of the individual fringe, i.e., number of pixels per fringe stripe. Obviously, shift unwrapping is also required in order to correctly restore the 3D shape of the object surface , which is a challenging task particularly in the cases of noisy image patterns and discontinuities of the object surface. To solve the shift unwrapping problem, a multiple-wavelength unwrapping algorithm was proposed in , where, in order to unwrap the shift of a pair of fringe patterns with frequency f (i.e., the number of fringes on the image), a set of fringe patterns with their frequencies between 1 and f are also projected. This method is not suitable for fast 3D measurement, as the number of image patterns can be rather large.
The shift unwrapping problem is similar to the phase unwrapping problem in phase based FPP. Phase unwrapping problem arises because the phase can only be detected within the principle value range of [-π, π], but the true phase can exceed the range. In order to retrieve the actual surface shape of the object, phase unwrapping must be carried out to obtain the actual phase maps. Ding, et al. [26, 27] developed an approach to recover the absolute phase maps of two image patterns with selected frequencies. Inspired by the method , we introduce a new approach to unwrap the shift maps using two image patterns with selected frequencies. Based on this approach, we propose a novel 3D shape measurement method based on projection of a set of triangular patterns with a higher frequency, and a triangular pattern with a lower frequency. The former are utilized to yield a high frequency shift map by the multi-step triangular pattern profilometry method in , and the latter is used to obtain a shift map of lower frequency using the single-step triangular-pattern spatial shift estimation algorithm . With the method to be proposed, the corresponding absolute shift maps can be retrieved, enabling the measurement of the profile of object surface. Compared with the multiple frequency shift estimation algorithm in , the number of image patterns is greatly reduced since only one additional image is used.
This paper is organized as follows. In Section 2 we firstly give a brief introduction on the conventional phase based fringe pattern profilometry and the spatial shift estimation based techniques. Then in Section 3 we indicate that the unwrapping problem in spatial shift estimation approach is similar to the phase unwrapping problem in phase based fringe pattern profilometry. The paper then introduces a method for unwrapping spatial shift maps of two fringe patterns with different frequencies. Based on this method, a novel 3D shape measurement approach is proposed. Finally in Section 4 a set of experimental results are presented to demonstrate performance of the proposed method. Section 5 concludes the paper.
2. Principle of fringe pattern profilometry
2.1 Phase based approaches for FPP
FPP is based on the triangulation principle described as follows. Without loss of generality and for the simplicity in expression, we assume that the projector has a fringe structure, where the light intensity varies periodically alone x direction, while keeping constant along y direction, as shown in Fig. 1. We use, and to denote the variance of light intensity of the fringe pattern on the reference plane, the object surface and the height distribution along x coordinate respectively. We also assume that the reference plane and the object surface have the same reflective characteristics.
The phase based FPP utilize fringe patterns that are periodic and can be expressed as [29, 30]:
2.2 SSE based approaches for FPP
The phase based FPP methods suffer from some limitations. In particular, the fringe patterns used to project must be sinusoidal or periodic in order that the phase maps of and exist and can be detected. However, due to many undesired factors inherent to digital projection, such as geometrical distortion and nonlinear intensity distortion, purely sinusoidal fringe patterns are hard to produce. In order to solve these problems, Hu, et al.  introduced a method which is based on the SSE, where the 3D shape can be calculated by the following:
The spatial shift based approach has a particular advantage. The projected fringe patterns are no longer required to be sinusoidal, thus leading to significant flexibility in the design of the fringe patterns. A number of approaches were proposed to retrieve the [19–21]. Among these approaches, the one referred to as IFSE  is particularly interesting. Base on this approach, a single-step triangular-pattern spatial shift estimation algorithm is proposed in . Wu, et al.  also introduced a multiple-step triangular-pattern spatial shift estimation algorithm by combining IFSE with the multiple-step triangular-pattern phase shifting algorithm , which greatly improved the accuracy of measurement.
3. The unwrapping problem
With most phase based approaches, can only be identified within the range of [-π, π]. In other words, is obtained by modulo 2π operation (called phase wrapping), resulting in discontinuities in its values. In order to recover the absolute phase , we have:26].
The wrapping problem also exists in SSE approaches. As has a fringe structure with a periodic fringe of width λ, can only be detected within the main value of , where λ is the width of an individual fringe referred to as the spatial wavelength of the fringe pattern. In other words, is wrapped into. Thus the true shift function, denoted as , should be continues and recovered from as follows:
Similar to the phase unwrapping associated with PDE, we also must restore from the wrapped , and the process is referred to as shift unwrapping .
3.1 Shift unwrapping of two fringe patterns with different frequencies
In order to unwrap the spatial shift, a simple method was proposed in  based on the relationship between the adjacent pixels. However, the method does not work if the intensity difference of neighborhood pixels is greater than λ/2, and hence the method does not work for the objects with high steps on the surface. Recently, a new a multiple-wavelength unwrapping algorithm for spatial shift estimation approach is proposed . In this method, a series of fringe patterns with their frequencies increasing by a constant factor are projected. The first image has only one fringe covering the whole measurement area (i.e., its frequency is 1), and the remaining ones are characterized by their frequency increasing by a constant factor (or their wavelengths decreasing by a constant factor). For example, for unwrapping an image pattern with 16 fringes, at least 5 image patterns are required. The method is not suitable for fast 3D measurement, as the number of image patterns can be rather large. Therefore, the reduction of the number of image patterns while maintaining the accuracy of shift unwrapping is still a challenging problem.
To address this problem, inspired by the method proposed for phase unwrapping in , we introduce a method for unwrap the shift based on projection of two fringe patterns with different spatial frequency. We assume that the measurement area has a resolution of W × H, and that the normalized spatial frequencies of the two patterns are and respectively, which are positive integers representing the total number of fringes on the respective patterns. The corresponding wavelengths are and . Combining this with Eq. (7) yields the following:26], we are able to derive a unique mapping from to and, as follows:26] we can show that each segment corresponds to different values of as well as different combinations of and. Hence a table can be generated forming a unique mapping from to and.
As , we can uniquely determine andfrom the value of . Let us choose and , the mapping relationship is shown in Table 1.
From the above results we can reconstruct the absolute shift maps of two fringe patterns by following steps:
- 1. Select two frequencies and construct a mapping table, making sure the table provides a unique mapping from to and. It should be noticed that the principle of frequency pair selection is similar to the principle mentioned in , where the selection of frequency pair has the restriction that two selected frequencies must be coprime, otherwise the mapping will not be unique.
- 2. Project two fringe patterns onto the object and acquire the two shift distance maps and by a spatial estimation algorithm.
- 3. Calculate by rounding its value to the closest integer, denoted as M. Using the lookup table derived in Step 1, find the row (or entry) whose value of is the closest to M. Record the corresponding andin the same row.
- 4. Using andobtained in Step 3, reconstruct the absolute shift maps using Eq. (8).
Since the unwrapped spatial shift map is retrieved point by point, compared to algorithms which use neighborhood pixels to do the unwrapping , the method we propose here can measure the surface profile with larger steps or multiple separate objects.
3.2 3D shape measurement based on projection of triangular patterns of two selected frequencies
In this section, a novel 3D shape measurement method based on projection of triangular patterns of two selected frequencies is proposed. As both andcan be used to reconstruct the absolute shift maps, we can select the one with lower measurement noise. Usually, the higher the spatial frequency used, the smaller the noise induced. Hence, we will project a set of triangular patterns with a higher frequency, from which a shift map can be obtained by the multiple-step triangular-pattern spatial shift estimation algorithm . Then, a triangular pattern with a lower frequency is projected, and the shift map can be retrieved by the single-step triangular-pattern spatial shift estimation algorithm . The absolute shift map of the higher frequency is recovered by the method described in 3.1, which is then used to determine the 3D shape. The procedure of the 3D shape measurement can be described using the flow chart in Fig. 2.
From the flow chart in Fig. 2, we can find that two different methods are used for the retrieving of the wrapped shift map. For the image pattern with higher frequency, the multiple-step triangular-pattern spatial-shifting algorithm is used. Here N is the number of images captured, which depends on the steps of the algorithm. As mentioned in method , at least two steps are required. However, the measurement accuracy can be improved by increasing the number of the spatial-shift steps. On the other side, the single-step triangular-pattern spatial-shifting algorithm is applied for the image pattern with the lower frequency, requiring only a single image pattern.
It can be seen from the flow chart, compared the proposed method to the existing single frequency multiple-step triangular-pattern spatial-shifting algorithm, only one additional image pattern is projected, which is much more efficient in terms of the number of fringe patterns required in contrast to the existing multiple-frequency shift unwrapping method .
4. Experiments and results
In order to test the performance of the approach proposed in Section 3, experiments were carried out in our laboratory. The triangular fringe patterns  are projected by a HITACHI CP-X260, and a Duncan Tech MS3100 3CCD digital camera is used to capture the fringe patterns. The digital camera is placed on top of the projector with a distance of 350.3 mm. The distance between the camera lens and the reference plane is 1295.1 mm. The resolution of the CCD camera is 1392 × 1039 pixels, and the field of vision for CCD camera is 250mm × 187mm. Hence, the equivalent spatial resolution is 0.1796 mm/pixel. The fringe projection system is calibrated using method in .
To verify that the proposed method can measure the surface profile with arbitrary step height, we first chose a flat box of 84mm high as the measured object. Figure 3(a) shows the photograph of the object. Figure 3(b) shows the captured fringe image of object with spatial frequency f = 5. Figures 3(c)–3(f) show the captured 4-step fringe images of object with spatial frequency f = 8.
We firstly use single frequency multiple-step triangular-pattern spatial-shifting algorithm to reconstruct this object. The 4-step fringe images with normalized spatial frequency f = 8 are used, where the wrapped is retrieved using method in , and it is unwrapped using method in . In contrast, the proposed method is also used. An additional fringe images with normalized spatial frequency f = 5 is captured, and the wrapped is retrieved using method in . Figure 4 shows the cross section of the object and Fig. 5 shows the 3D reconstruct results.
Figure 4(a) and Fig. 5(a) show that, if we only use patterns of the higher frequency, the object step height is measured as about 24 mm, which is obviously incorrect in contrast to the true value of 84mm. However, with the proposed method, the step height can be measured successfully, with the result very close to 84mm. This experiment demonstrated that the proposed approach can successfully perform the measurement even when the surface profile has a large step.
As mentioned in Section 3, since the measurement is performed on point-by-point basis, this technique can also be used to measure multiple separate objects or islands. To verify this, we measured two separate objects, a mask model and a plaster hand model. Both objects have complex surface shape. The proposed method is used with two spatial frequencies and respectively. We performed two experiments by fixing the higher frequency, but using different number of steps (3 and 6-steps) for retrieving the wrapped . Figure 6(a) shows the two objects and Fig. 6(b) shows the captured fringe images of the objects with frequency . Figure 7 shows the captured images of the objects with 3-step triangular patterns with frequency , and Fig. 8 depicts the 6-step fringe images of objects with frequency .
Figure 9 and Fig. 10 show the reconstructed 3D surface shape of the object using the proposed unwrapping method, where Fig. 9 shows the result using 3-step fringe images and Fig. 10 gives the result using 6-step fringe images. It can be seen from these figures that two separated objects is reconstructed successfully in both measurements. The height information for those places with high step drops (edge of mask and hand, nose part) is retrieved correctly. Some details, such as mouth and eyes part, are also recovered. This result demonstrated that the proposed method can successfully measure those complex objects with arbitrary step height, even if two objects are completely separate. Besides, it is noticed that the quality of the reconstructed model is also improved with the increase of the number of steps for the higher frequency image.
In this paper, we studied the shift wrapping problem associated with SSE based FPP. Compared to phase based approaches, SSE techniques are advantageous in that non-sinusoidal fringe patterns can also be employed, and that they do not suffer from the nonlinear distortion associated with the digital fringe projection. However, similar to the phase unwrapping problem encountered in phase based approaches, a shift unwrapping problem also exists in SSE based approaches. Hence we presented a temporal shift unwrapping technique to overcome this limitation. Based on this technique, we introduced a novel 3D shape measurement method. This method is an improvement of the existing multiple-step triangular-pattern spatial shift estimation algorithm , by the projection of only one additional image with a lower frequency. In order to test the performance of our proposed method, we carried out experiments on three different objects with significant steps and complex surface shapes: a flat box, a mask, and a plaster hand-model. The results show that the proposed method works very well, where complex objects surface with significant step height or multiple separate objects can be measured successfully with the same accuracy as the method in . Compare to the existing multiple-wavelength unwrapping algorithm for spatial shift estimation approach , the number of image pattern required is greatly reduced.
It should be pointed out that similar to the method in , the selection of frequency pair has the restriction that two selected frequencies must be coprime. In order to meet such a requirement, the total number of pixels perpendicular to the fringe must be an integer multiple of the number of pixels within a fringe. However, such a selection may not be possible in some cases. Taking an example where the measurement area is 800 × 600 pixels, if the selected frequencies are and, the numbers of pixels per fringe period will be about 75 and 46.1538, respectively, which are not integers and thus are not implementable. As a future work we will investigate the possibilities that the selected frequencies does not need to be an integer, thus yielding more flexibility for the design of the patterns.
References and links
1. S. Gorthi and P. Rastogi, “Fringe projection techniques: whither we are?” Opt. Lasers Eng. 48(2), 133–140 (2010). [CrossRef]
2. X. Su and W. Chen, “Fourier transform profilometry: a review,” Opt. Lasers Eng. 35(5), 263–284 (2001). [CrossRef]
3. H. Zhang, M. J. Lalor, and D. R. Burton, “Spatiotemporal phase unwrapping for the measurement of discontinuous objects in dynamic fringe-projection phase-shifting profilometry,” Appl. Opt. 38(16), 3534–3541 (1999). [CrossRef] [PubMed]
4. M. Halioua and H. C. Liu, “Optical three-dimensional sensing by phase measuring profilometry,” Opt. Lasers Eng. 11(3), 185–215 (1989). [CrossRef]
6. X. Su, L. Su, W. Li, and L. Xiang, “New 3D profilometry based on modulation measurement,” Proc. SPIE 3853, 1–7 (1998). [CrossRef]
7. S. Toyooka and M. Tominga, “Spatial fringe scanning for optical phase measurement,” Opt. Commun. 51(2), 68–70 (1984). [CrossRef]
9. R. Rodríguez-Vera and M. Servin, “Phase locked loop profilometry,” Opt. Laser Technol. 26(6), 393–398 (1994). [CrossRef]
12. P. Huang, Q. Ho, F. Jin, and F. Chiang, “Colour-enhanced digital fringe projection technique for high-speed 3-D surface contouring,” Opt. Eng. 38(6), 1065–1071 (1999). [CrossRef]
13. Z. Zhang, C. E. Towers, and D. P. Towers, “Robust color and shape measurement of full color artifacts by RGB fringe projection,” Opt. Eng. 51(2), 021109 (2012). [CrossRef]
14. J. Guo, X. Peng, J. Yu, X. Liu, A. Li, and M. Wang, “Real-time 3D imaging by using color structured light based on Hilbert transform,” Proc. SPIE 8856, 885624 (2013). [CrossRef]
15. A. J. Moore and F. Mendoza-Santoyo, “Phase demodulation in the space domain without a fringe carrier,” Opt. Lasers Eng. 23(5), 319–330 (1995). [CrossRef]
16. J. Villa, M. Servin, and L. Castillo, “Profilometry for the measurement of 3-D object shapes based on regularized filters,” Opt. Commun. 161(1–3), 13–18 (1999). [CrossRef]
20. Y. Hu, J. Xi, J. Chicharo, E. Li, and Z. Yang, “Discrete cosine transform-based shift estimation for fringe pattern profilometry using a generalized analysis model,” Appl. Opt. 45(25), 6560–6567 (2006). [CrossRef] [PubMed]
21. Y. Hu, J. Xi, J. Chicharo, W. Cheng, and Z. Yang, “Inverse function analysis method for fringe pattern Profilometry,” IEEE Trans. Instrum. Meas. 58(9), 3305–3314 (2009). [CrossRef]
22. K. Wu, J. Xi, Y. Yu, and Z. Yang, “3D profile measurement based on estimation of spatial shifts between intensity ratios from multiple-step triangular patterns,” Opt. Lasers Eng. 51(4), 440–445 (2013). [CrossRef]
23. P. Jia, J. Kofman, and C. English, “Multiple-step triangular-pattern phase shifting and the influence of number of steps and pitch on measurement accuracy,” Appl. Opt. 46(16), 3253–3262 (2007). [CrossRef] [PubMed]
24. P. Cao, J. Xi, J. Chicharo, and Y. Yu, “A fringe period unwrapping technique for digital fringe profilometry based on spatial shift estimation,” Proc. SPIE 7432, 743208 (2009). [CrossRef]
25. P. Cao, J. Xi, Y. Yu, and Q. Guo, “Spatial shift unwrapping for digital fringe profilometry based on spatial shift estimation,” J. Electron. Imaging 23(4), 043002 (2014). [CrossRef]
27. Y. Ding, J. Xi, Y. Yu, W. Q. Cheng, S. Wang, and J. F. Chicharo, “Frequency selection in absolute phase maps recovery with two frequency projection fringes,” Opt. Express 20(12), 13238–13251 (2012). [CrossRef] [PubMed]
28. P. Cao, J. Xi, Y. Yu, and Q. Guo, “Digital fringe profilometry based on triangular fringe patterns and spatial shift estimation,” Proc. SPIE 9110, 91100C (2014). [CrossRef]
29. M. Takeda, H. Ina, and S. Kobayashi, “Fourier-transform method of fringe-pattern analysis for computer-based topography and interferometry,” J. Opt. Soc. Am. 72(1), 156–160 (1982). [CrossRef]
31. Q. Hu, P. S. Huang, Q. Fu, and F. Chiang, “Calibration of a three-dimensional shape measurement system,” Opt. Eng. 42(2), 482–493 (2003). [CrossRef]