## Abstract

The out-of-plane shape determination in a generalized fringe projection profilometry is presented. The proposed technique corrects the problems in existing approaches, and it can cope well with the divergent illumination encountered in the generalized profilometry. In addition, the technique can automatically detect the geometric parameters of the experimental setup, which makes the generalized fringe projection profilometry simple and practical. The concept was verified by both computer simulations and actual experiments. The technique can be easily employed for out-of-plane shape measurements with high accuracies.

©2006 Optical Society of America

## 1. Introduction

Fringe projection profilometry has been well established for out-of-plane shape and warpage measurements [1]–[16]. In the conventional fringe projection profilometry [1], a collimated light beam is employed to project straight and equally spaced fringes on a specimen, and the images of the specimen and the projected fringes are captured from a different direction other than the illumination direction. The departure of a viewed fringe on the specimen from a straight line contains information of the departure of the specimen surface from the reference surface, i.e., the out-of-plane height (shape or warpage). Generally, the out-of-plane shape or warpage is obtained through extracting the relevant phase distributions from the projected fringes.

In recent years, digital LCD or DLP projectors, due to their multiple advantages such as convenience and low cost, have been widely used in fringe projection profilometry for out-of-plane shape or warpage measurements. In practice, although the projector-based technique is relatively easy to implement, the measurement accuracy is subjected to uncertainties in the system. These uncertainties can further cause substantial errors in shape or warpage determinations. Specifically, the primary uncertainty originates from the divergent or uncollimated illumination which not only induces nonlinearities to the reference carriers but also makes an accurate determination of the out-of-plane height difficult. In addition, the techniques based on divergent projection normally require specific and precise experimental setups, but the geometric parameters such as the projection angle and the location of the focal point of projector lens are quite difficult to determine precisely. In existing techniques [3]–[13], these uncertainties are generally more or less neglected, thus the corresponding measurement accuracies are inevitably diminished.

To cope with the nonlinear carrier problem, a number of approaches have been proposed. A typical scheme is capturing the images of reference plane and test object separately to obtain the full-field fringe phase maps [5] [13]. However, this handling is inconvenient in actual application; moreover, because the distributions of carrier fringe pitches remain unknown, the out-of-plane height determination is only an approximation. Recently, Chen et al. [14]–[16] described a least-squares approach to determine the nonlinear carrier phases with a polynomial function, and their algorithm is very effective for the reference carrier detection under nonparallel illumination. In spite of the advances in the carrier phase detection, it is noted that a corresponding technique to accurately determine the out-of-plane height is lacking. The existing algorithms either rely on a precise experimental setup which is impractical to achieve or simply use a phase-subtraction scheme. It is important to point out that the algorithms based on a direct subtraction of carrier phases from the projection fringe phases, which is originally employed for parallel or collimated illumination case, have been extended and applied to projector-based profilometry in the literatures [9]–[12]. However, the algorithms based on phase-subtraction are not theoretically rigorous for the case of divergent illumination. Due to above reasons, the existing techniques normally provide the out-of-plane shape approximately instead of accurately.

Other notable works include a self-calibrating system developed by Schreiber et al. [7] and a multiple-parameters technique introduced by Salas et al. [8] for carrier subtraction and height detection. These two techniques are very complicated and subjected to errors due to manual intervention, which makes the techniques impractical.

In this paper, a mathematical derivation of the carrier phase function for fringe projection profilometry with divergent illumination is presented, and a rigorous algorithm to determine the out-of-plane shape for a generalized fringe projection profilometry is proposed. The proposed technique allows using a conventional projector (i.e., the illumination is divergent) in the experiment; furthermore, it neither requires a specific and precise experimental setup nor requires the geometric parameters of experimental setup to be measured. The technique is capable of quickly providing out-of-plane shape and warpage measurements with high accuracies for the generalized fringe projection profilometry with divergent illuminations.

## 2. Principle

#### 2.1. Experimental setup

A typical setup of the generalized fringe projection profilometry with a digital projector is illustrated in Fig. 1. The experimental system mainly comprises a LCD or DLP digital projector and a CCD camera. A continuously shifting pattern with straight and equally spaced fringes is projected through the projector onto the specimen surface and the reference plane, which are then captured by the CCD camera in the normal view direction. It is noteworthy that the proposed technique does not require the geometric parameters of the experimental setup to be precisely determined as long as they are reasonably located so that the projection shadows can be avoided in the regions of interest. In addition, the CCD camera may be set in any appropriate view other than the normal one although the normal view is desired for easy image mapping purpose.

#### 2.2. Projection fringe pitch and carrier phase function on the reference plane

Figure 2 illustrates the schematic geometry of the fringe projection system. Point *P* is the assumed point light source of the projector and it has a distance *d*
_{1} from the reference plane; the dot-dashed line indicates the projection direction at an angle of *α*. Point *O* is a starting point (can be arbitrarily set) on the reference plane and line *OQ* is perpendicular to the projection direction. In practice, it is convenient to choose point *O* at the left edge of the digital fringe image. An arbitrary light beam *PR* meets the reference plane at point *R* at an angle *β*. Assuming the uniform fringe pitch or spacing along line *OQ* is *p*_{OQ}
and the distance from point *O* to point *R* is *x*, then the fringe phase function on the reference plane, i.e., carrier phase function, can be obtained according to the truth that the phases at points *T* and *R* are identical:

where the superscript “r” denotes the reference plane where carrier phase is determined, *ϕ*
_{0} is the fringe phase at point O on the reference plane, and *l*_{OT}
is the length of line segment OT. From the geometry of the triangle *OTR* in Fig. 2, *l*_{OT}
can be determined as

where *H*=*d*
_{1}/sin*α*, *A*=*d*
_{1}/tan*α*+*d*
_{0}, *d*
_{0} and *d*
_{1} are the geometric distances as shown in Fig. 2. Substituting Eq. (2) into Eq. (1) yields

where *B*=*ϕ*
_{0}
*A* and
$C={\varphi}_{0}+\frac{2\pi}{{p}_{\mathit{OQ}}}H$
. Accordingly, the fringe pitch *p*^{r}
(*x*) on the reference plane can be obtained from Eq. (3) as

It is evident from Eq. (4) that the projected fringes on the reference plane are nonuniform and the fringe pitch is a parabolic function of distance *x*.

The parameters *A*(> 0), *B*, and *C* must be determined in order to obtain the out-of-plane height in following image data analysis and processing. Instead of being calculated from the geometric parameters which are impractical to measure precisely, the parameters *A*, *B*, and *C* in Eq. (3) should be determined by an inverse process because the carrier phase *ϕ*^{r}
(*x*) at numerous points on the reference plane can be experimentally obtained. A simple way to determine the parameters is to utilize the explicit solutions as following

$$C=\frac{\left[{\varphi}_{e}^{r}\left({x}_{1}\right){x}_{1}-{\varphi}_{e}^{r}\left({x}_{2}\right){x}_{2}\right]\left[{\varphi}_{e}^{r}\left({x}_{3}\right)-{\varphi}_{e}^{r}\left({x}_{2}\right)\right]-\left[{\varphi}_{e}^{r}\left({x}_{2}\right){x}_{2}-{\varphi}_{e}^{r}\left({x}_{3}\right){x}_{3}\right]\left[{\varphi}_{e}^{r}\left({x}_{2}\right)-{\varphi}_{e}^{r}\left({x}_{1}\right)\right]}{\left[{\varphi}_{e}^{r}\left({x}_{3}\right)-{\varphi}_{e}^{r}\left({x}_{2}\right)\right]\left({x}_{1}-{x}_{2}\right)-\left[{\varphi}_{e}^{r}\left({x}_{2}\right)-{\varphi}_{e}^{r}\left({x}_{1}\right)\right]\left({x}_{2}-{x}_{3}\right)}$$

$$B={\varphi}_{e}^{r}\left({x}_{1}\right)\left(A+{x}_{1}\right)-C{x}_{1}$$

where the subscript “e” denotes experimental results, and *x*
_{1}, *x*
_{2}, and *x*
_{3} are three arbitrary and separate points on the reference plane. Although this direct scheme is very easy to implement, the results obtained may contain substantial errors due to the noises in the experimental images. To reduce calculation errors, a least-squares approach, which uses a large amount of data points, can be employed. The least-squares algorithm finds the coefficients that minimize the following expression

In the equation, *N*(*N*≫3) is the number of data points where the carrier phases ${\varphi}_{e}^{r}$
can be experimentally obtained. The parameters *A*, *B*, and *C* can be obtained from the following equations

#### 2.3. Out-of-plane height calculation

Figure 3 illustrates the schematic geometry of the specimen surface and the reference plane. Points *P* and *C* represent projector point source and camera point source, respectively. Their projection and view directions are indicated with dot-dashed lines. A light beam meets the specimen surface at point *I* at an incidence angle of *θ*; the beam would hit the reference plane at point *J* if the specimen is removed. The camera-recorded coordinate of point *I* is the same as that of point *K* on the reference plane, i.e., *x*. From Fig. 3, the phase function on the specimen surface can be obtained as

Here the superscript “s” denotes the specimen surface.

In Fig. 3, letting *h*(*x*) denote the height of point *I* relative to the reference plane, then it is easy to get the the following relation

where
${D}_{0}=\left(\frac{{d}_{0}}{{d}_{1}}+\frac{{d}_{0}^{\prime}}{{d}_{2}}\right)$
,
${D}_{1}=\left(\frac{1}{{d}_{1}}-\frac{1}{{d}_{2}}\right)$
,
${D}_{2}=\frac{1}{{d}_{1}}$
, *d*
_{0}′ and *d*
_{2} are the geometric distances as shown in Fig. 3. From Eq. (9), the height of point *I* can be obtained as

where Δ*x* can be calculated from Eq. (8) with the phase function on the specimen surface determined experimentally, that is,

Substituting Eq. (11) into Eq. (10) yields

To provide accurate out-of-plane shape or warpage measurements, it is evident from Eq. (12) that *D*
_{0}, *D*
_{1} and *D*
_{2} (therefore *d*
_{1}, *d*
_{2}, *d*
_{0} and *d*
_{0}′) must be acquired in advance. As previously described, however, it is impractical to directly measure the corresponding geometric parameters with high accuracies. In practice, a gage object whose out-of-plane shape is precisely known can be used in the experiment to determine the parameters *D*
_{0}, *D*
_{1} and *D*
_{2} through an inverse calculation. Again, a least-squares approach can be employed to accurately calculate the parameters through using a large amount of data points. The linear least-squares function can be expressed as

$$\phantom{\rule{.9em}{0ex}}\phantom{\rule{.9em}{0ex}}\phantom{\rule{.9em}{0ex}}\phantom{\rule{.9em}{0ex}}\phantom{\rule{.9em}{0ex}}\phantom{\rule{.9em}{0ex}}\phantom{\rule{.9em}{0ex}}\phantom{\rule{.9em}{0ex}}\phantom{\rule{.9em}{0ex}}\phantom{\rule{.9em}{0ex}}\phantom{\rule{.2em}{0ex}}\phantom{\rule{.2em}{0ex}}-{D}_{2}{\left[B+C{x}_{i}-\left(A+{x}_{i}\right){\varphi}_{e}^{s}\left({x}_{i}\right)\right]{h}^{g}\left({x}_{i}\right)\}}^{2}$$

and *D*
_{0}, *D*
_{1} and *D*
_{2} can obtained from

where the superscript “g” denotes the gage object, and *M* is the number of data points on the gage where the heights are precisely known and the projection fringe phases can be experimentally obtained. It should be noted from Eq. (10) that if the gage object has a constant height *h*(*x*)=*h*
_{0} (e.g., the object is a cuboid block), an incorrect solution set of *D*
_{0}=*D*
_{1}=0 and
${D}_{2}=\frac{1}{{h}_{0}}$
would be derived. In this case, an additional different gage object must be employed in the experiment to get the correct parameters.

After finalizing *D*
_{0}, *D*
_{1} and *D*
_{2} from Eq. (14), the heights of points on the specimen surface can be readily determined with Eq. (12). It should be noted from Fig. 3 that the detected height *h*(*x*) is associated with the corresponding physical point *I* on the specimen surface, so there is no need to link *h*(*x*) with the reference point *R*. The discrepancy between *I* and *R* originates from the inherent characteristic of the camera imaging, which may be negligible in practice (the discrepancy is exaggerated in Fig. 3 for description purpose).

## 3. Simulation

A computer simulation was implemented to verify the validity of the proposed algorithm. The projection fringe image used in the simulation is shown in Fig. 4, which represents a partial sphere of peak height 10.72*mm* and curvature of 0.0125*mm*
^{-1}, a cuboid gage block of height 10*mm*, and another cuboid gage block of height 20*mm* on a flat surface. The fringe patterns in Fig. 4 were generated according to the geometric relations of a divergent illumination setup, and the conventional four-frame phase shifting conceptwas adopted to obtain four phase-shifted images in the simulation. The non-uniformity of the projection fringes due to divergent illumination is evident in the figure.

As a comparison, two existing techniques, in addition to the proposed one, were incorporated in the simulation:

- A conventional technique which was originally developed for parallel or collimated illumination case. Since the algorithm has been used in divergent illumination application, it was investigated in the simulation.

For both techniques, the most widely used phase-subtraction algorithm [4], [9]–[12], [14] is employed for the determination of the out-of-plane heights. Table 1 summarizes the techniques incorporated.

The out-of-plane shapes obtained from the three different techniques are shown in Fig. 5. In each figure, the results have been normalized to demonstrate the relative heights. It is easy to see from the figure that the proposed technique can detect the out-of-plane shape more accurately than the existing techniques. Figure 6 shows a comparison of the absolute height values (with the flat surface as the reference plane) along the horizontal centerlines in the images. It is noted that the coefficients *k* in the relevant governing equations of the conventional and polynomial algorithms were intentionally set to specific values so that the detected average height of the second cuboid block is identical with the theoretical height of 20*mm*. From Fig. 6, it is evident that the results obtained from the proposed technique exactly match the theoretical heights, while the results from the other two techniques show apparent discrepancies. The simulation clearly proves the validity of the proposed technique.

## 4. Application considerations

#### 4.1. About phase shifting, fringe order, and phase unwrapping

In fringe projection profilometry, the captured experimental fringe patterns should include certain specific phase-shifting amounts in order to accurately obtain phase distributions with a typical phase shifting algorithm. This requires a synchronization between shifting the projection fringes and capturing the fringe images, which is not easy to implement, especially for real-time measurements. This requirement restricts the practical application of the fringe projection profilometry. To cope with this problem, the proposed approach employs the advanced iterative algorithm (AIA) for phase determinations [17]. The AIA can accurately extract phase distributions from any arbitrarily phase-shifted fringe patterns while keep the processing realtime. Consequently, the projection fringes may shift continuously at any speed and the camera can capture the phase-shifted images at any time (e.g., in a real-time manner) during the experiment. This approach significantly reduces the requirements on the hardware and software implementations and makes the application of the already-convenient generalized fringe projection profilometry more practical.

A typical experimental image of the proposed fringe projection profilometry normally contains the images of the specimen, the reference plane and the calibration gage(s). Due to the existing of the multiple objects and projection shadows, the fringe orders in the captured digital images are usually discontinuous among different objects. It is worth noting that this issue was not clearly addressed in previous work, thus the test object must have some surface points contact the reference plane to make the fringe orders continuous across the whole field [3]–[14]. Such a restriction highly limits the applicability of the fringe projection profilometry. In this paper, this practically important issue is easily resolved by adding one or more markers in each valid and isolated region to indicate the relative fringe orders. An example of projecting fringe-order markers will be demonstrated in the following section.

Another problem associated with the isolated regions is that the phase unwrapping algorithm should be able to simultaneously unwrap the phases and assign correct offsets to the phases in multiple isolated areas. There are already several advanced phase unwrapping algorithms (e.g., PCG algorithm) available for such a task. An overview of the algorithms can be found in Ref. [18].

#### 4.2. About the calibration gage and the reference plane

The calibration gage(s) is useful for the detection of the governing parameters *D*
_{0}, *D*
_{1}, and *D*
_{2} required for the out-of-plane height determination without measuring the geometric parameters *d*
_{0}, *d*′_{0}, *d*
_{1}, and *d*
_{2} of the experimental system. It is necessary to ensure that the gage height is accurate enough to eliminate the influence on the measurement uncertainty. Once the governing parameters are finalized, the calibration gage(s) does not have to be included in the experiments unless the corresponding geometric parameters have been changed.

The fringe projection profilometry provides the out-of-plane height measurements relative to the reference plane. The reference plane is not necessarily the background plane and is not necessarily located behind the objects of interest to be measured. Similar to the calibration gage, the reference plane may be removed from the experimental setup once the parameters *A*, *B*, and *C* have been detected and unchanged.

## 5. Experiments

Two experiments have been conducted to demonstrate the applicability of the proposed technique. One experiment intends to verify the measurement accuracy of the technique, and another one aims to show its practicability. The experimental procedure comprises the following major steps:

- Setup the reference plane, the calibration and test objects, the projection and camera systems. It is preferable to set the projection at an inclined incidence angle and the camera at a normal view with respect to the reference plane. There is no need to measure the geometric parameters.
- Shift the projection fringes continuously and capture a series of images. One image with fringe order markers, which are generated and controlled by the software, should be captured as well.
- Process the images to obtain the full-field out-of-plane shape. This step involves extracting wrapped and unwrapped phases, offsetting fringe orders, detecting governing parameters, and determining height distributions.

#### 5.1. Experiment one: accuracy examination

A cuboid block of nominal height 33.0*mm* was tested in the experiment for accuracy examination. Figure 7 shows six experimental images with randomly phase-shifted projection fringes. In the images, the cuboid specimen can be found to the left of two gage blocks where the one in dark gray is 50.8*mm* high and the white one is 36.5*mm* high. All the blocks are located against the background reference plane. From Fig. 7, it is clear that the illumination from the projector to the objects and reference plane is uncollimated, and the fringe pitches have relatively large variations across the field.

Figure 8(a) shows the wrapped phase map in the regions of interest, including the reference plane, the test object, and two gage blocks. Figure 8(b) is the corresponding unwrapped phase map. Figs. 8(c) and 8(d) are the 2-D and 3-D color plots of the out-of-plane shape (height) detected by the proposed technique. The maximum and minimum heights on the specimen surface relative to the reference plane were detected to be 33.3*mm* and 32.7*mm* respectively, and the average height of the whole surface is 33.0*mm*. This indicates a difference of ±0.3*mm* or ±0.9% from the nominal height and a perfect match for the average height. As a comparison, Fig. 8(e) shows the result obtained from the polynomial technique, which yielded a maximum height of 39.2*mm* and a minimum one of 36.4*mm*. The result verifies that the proposed technique is capable of measuring full-field out-of-plane shape or warpage with high accuracies.

#### 5.2. Experiment two: practicability examination

The practicability of the proposed technique has been examined as well. The experiment is similar to the previous one except that the cuboid block was replaced with a tool box which has a more complex shape, as shown in Fig. 9(a). The maximum height was carefully measured to be 43.26*mm* with a caliper gage. Figure 9(b) is the projection fringe pattern with fringe order markers. As previously stated, the markers are necessary for correctly assigning the whole-field fringe orders because they are discontinuous among different objects.

Figure 10 shows the experimental results obtained by the propose technique. Figure 10(a) is the wrapped phase map, and Fig. 10(b) is the unwrapped one. The 2-D height map is shown in Fig. 10(c), and Fig. 10(d) shows the 3-D out-of-plane height distributions in two different views. The maximum height was determined to be 43.4*mm*, which indicates a discrepancy of 0.14*mm* or 0.3% from the value measured by the caliper gage. Unlike the caliper gage, however, the fringe projection profilometry yields the full-field information of the out-of-plane shape.

## 6. Further consideration

The proposed technique requires that the projected fringes must be vertical so that the carrier phase distribution is actually a 1-D problem, i.e., it is a function of *x* only. For the tilted projection fringes, the effect of *y* component must be incorporated. It is easy to understand that the carrier phase function can be expressed, based on the 1-D case, as

which requires a nonlinear least-squares method to solve for the parameters. Due to the introduction of more parameters, Eq. (15) may be capable of considering the effect of a tilting camera as well. However, a rigorous description of the out-of-plane height determination associated with Eq. (15) is very difficult. The relevant work will be conducted in the future.

## 7. Conclusion

The out-of-plane shape determination in generalized fringe projection profilometry with divergent illumination is presented. The proposed technique corrects the problems in existing approaches, and it can cope well with the divergent illumination encountered in the generalized profilometry. Because the technique can automatically detect the geometric parameters of the experimental setup, it makes the generalized fringe projection profilometry quite simple and convenient. The concept was verified by computer simulations. For the practical application of the generalized fringe projection profilometry, a scheme which uses the proposed algorithm and AIA phase shifting algorithm has been proposed and successfully applied to actual experimental measurements. The simulation and experiments shows that the proposed technique can be easily employed for out-of-plane shape and warpage measurements with high accuracies.

## References and links

**1. **K. Creath and J. Wyant, “Moiré and fringe projection techniques,” in. Optical Shop Testing, 2nd Ed., Ed. by D. Malacara, New York: Wiley, 653–685 (1992).

**2. **F. Chen, G. Brown, and M. Song, “Overview of 3-D shape measurement using optical methods,” Opt. Eng. **39**, 10–22 (2000). [CrossRef]

**3. **M. Takeda and K. Mutoh, “Fourier transform profilometry for the automatic measurement of 3-D object shapes,” Appl. Opt. **22**, 3977–3982 (1983). [CrossRef] [PubMed]

**4. **V. Srinivasan, H. Liu, and M. Halioua, “Automated phase-measuring profilometry of 3-D diffuse objects,” Appl. Opt. **23**, 3105–3108 (1984). [CrossRef] [PubMed]

**5. **V. Srinivasan, H. Liu, and M. Halioua, “Automated phase-measuring profilometry: a phase mapping approach,” Appl. Opt. **24**, 185–188 (1985). [CrossRef] [PubMed]

**6. **P. Huang, Q. Hu, F. Jin, and F. Chiang, “Color-encoded digital fringe projection technique for high-speed threedimensional surface contouring,” Opt. Eng. **38**, 1065–1071 (1999). [CrossRef]

**7. **W. Schreiber and G. Notni, “Theory and arrangements of self-calibrating whole-body three-dimensional measurement systems using fringe projection technique,” Opt. Eng. **39**, 159–169 (2000). [CrossRef]

**8. **L. Salas, E. Luna, J. Salinas, V. Garcia, and M. Servin, “Profilometry by fringe projection,” Opt. Eng. **42**, 3307–3314 (2003). [CrossRef]

**9. **C. Quan, C. Tay, X. Kang, X. He, and H. Shang, “Shape measurement by use of liquid-crystal display fringe projection with two-step phase shifting,” Appl. Opt. **42**, 2329–2335 (2003). [CrossRef] [PubMed]

**10. **C. Tay, C. Quan, H. Shang, T. Wu, and S. Wang, “New method for measuring dynamic response of small components by fringe projection,” Opt. Eng. **42**, 1715–1720 (2003). [CrossRef]

**11. **C. Tay, C. Quan, T. Wu, and Y. Huang, “Integrated method for 3-D rigid-body displacement measurement using fringe projection,” Opt. Eng. **43**, 1152–1159 (2004). [CrossRef]

**12. **J. Pan, P. Huang, and F. Chiang, “Color-coded binary fringe projection technique for 3-D shape measurement,” Opt. Eng.44,, 023606 (2005). [CrossRef]

**13. **H. Guo, H. He, Y. Yu, and M. Chen, “Least-squares calibration method for fringe projection profilometry,” Opt. Eng. **44**, 03360–03363, (2005). [CrossRef]

**14. **L. Chen and C. Quan, “Fringe projection profilometry with nonparallel illumination : a least-squares approach,” Opt. Lett. **30**, 2101–2103 (2005). [CrossRef] [PubMed]

**15. **Z. Wang and H. Bi, “Comment on “Fringe projection profilometry with nonparallel illumination: a least-squares approach”,” Opt. Lett. **31**, 1972–1973 (2006). [CrossRef] [PubMed]

**16. **L. Chen and C. Quan, “Reply to Comment on “Fringe projection profilometry with nonparallel illumination: a least-squares approach”,” Opt. Lett. **31**, 1974–1975 (2006). [CrossRef]

**17. **Z. Wang and B. Han, “Advanced iterative algorithm for phase extraction of randomly phase-shifted interferograms,” Opt. Lett. **29**, 1671–1673 (2004). [CrossRef] [PubMed]

**18. **Z. Wang, “Development and application of computer-aided fringe analysis,” Ph.D. Dissertation, University of Maryland at College Park, 2003. http: //hdl.handle.net/1903/31