For three-dimensional shape measurement, phase-shifting techniques are widely used to recover the objective phase containing height information from images of projected fringes. Although such techniques can provide an accurate result in theory, there might be considerable error in practice. One main cause of such an error is distortion of fringes due to nonlinear responses of a measurement system. In this paper, a postprocessing method for compensating distortion is proposed. Compared to other compensation methods, the proposed method is flexible in two senses: (1) no specific model of nonlinearity (such as the gamma model) is needed, and (2) no special calibration data are needed (only the observed image of the fringe is required). Experiments using simulated and real data confirmed that the proposed method can compensate multiple types of nonlinearity without being concerned about the model.
© 2016 Optical Society of America
Measuring three-dimensional (3D) shapes is a quite fundamental and important task for many industrial and scientific situations. The fringe projection technique is a 3D shape measurement method that has been developed by many researchers [1–4]. It acquires 3D information from two-dimensional images by analyzing sinusoidal patterns, or fringes, projected on an object to be measured. The phase-shifting method is a popular branch of the fringe projection methods and utilizes several fringe patterns with shifted phase. It can accurately reconstruct the objective phase, which contains 3D information, and has been successfully applied to many practical measurement problems.
Nowadays, it has become more common to use commercially available projectors and cameras for fringe projection methods, because their costs have been greatly reduced. However, to improve the visual quality for humans, most commercial devices have a nonlinear intensity response, which distorts the projected fringes. Since phase-shifting algorithms are designed based on purely sinusoidal fringes, distorted fringes provide inaccurate results. Unfortunately, measurement error caused by this nonlinear effect cannot be reduced by a denoising method designed for random noise , because the error appears as a periodic pattern. Therefore, any nonlinearity between input signal and measured intensity must be compensated before or within a reconstruction step of the objective phase. This problem has been addressed by many researchers, and many methods for reducing the error have been proposed [6–11].
In the case of phase-shifting techniques, measurement error caused by the nonlinear effect can be decreased by increasing the shifting steps . Therefore, accurate results are obtained by taking the time to acquire many images of fringes if the measuring object does not move. On the other hand, there are some situations in which acquiring many images is not possible: for example, in real-time phase-shifting methods [13,14]. Then the error must be compensated by software acting on either the input signal or the acquired images. In any case, it is better to have a signal-processing method that reduces error and improves accuracy, and thus it is worthwhile to investigate it further.
One popular approach for the compensation is the gamma-model-based methods [12,15–23]. They assume that nonlinearity can be approximated well by the gamma model: for a parameter , nonlinearity is written as the th power of a function of the input signal. The main advantage of the gamma model is its simplicity, which allows detailed analysis of phase error through series expansion and its coefficients. Although the effectiveness of the gamma model has been proved, it cannot perfectly compensate nonlinearity, which results from not only the gamma effect but also some other causes. This modeling error has been reduced by generalizing the gamma model through including the effect of defocusing and ambient light [17–20]. Nevertheless, some situations where the generalized gamma models cannot perfectly represent the nonlinearity exist in practice.
A lookup-table-based method is another popular approach [13,24–27]. It consists of constructing a table that represents nonlinear error from reference data measured in advance, and the effect of the nonlinearity is directly compensated by the table. Since unknown nonlinear effects of a system are revealed by the table obtained from real data, no specific model of nonlinearity has to be assumed. Therefore, it can be applied to any nonlinear measurement system, including the gamma model. Some research utilized an artificial neural network, which also does not require any specific model, for obtaining the nonlinear relation of the input pattern and observed illumination . Such neural-network-based methods may have the possibility of improving their performance using some recently developed learning techniques . However, many of these methods require input data, which may not be available for some situations, as well as observed images. Moreover, a table or network must be constructed each time the setting of a measurement system is changed, and such construction may consume time.
In this paper, as an alternative method, we propose a general framework for approximating and compensating the nonlinear distortion of a fringe. The proposed method tries to estimate an inverse map of the nonlinearity from only observed images. Thus, it can be applied to any measurement system easily, because no information about the system is required. In addition, any model, such as polynomials, can be used for approximating nonlinearity. Our contributions include (1) proposing a simple objective function measuring the degree of nonlinearity, (2) modeling nonlinearity in a flexible form, and (3) obtaining an analytic solution of the model parameters when a linear combination model is applied. The aim of this paper, however, is not to compete with the current state-of-the-art methods, which may require many reference images, but rather to show how our framework can treat multiple types of nonlinearity in the same manner. The proposed method should be able to be integrated with the state-of-the-art methods without much effort because of its flexibility.
This paper is organized as follows. In Section 2, a mathematical model of distorted fringes is reviewed briefly. Then, our method of modeling and compensating the nonlinear effect is introduced in Section 3. The proposed method is formulated as an optimization problem minimizing the energy of harmonic components caused by distortion. Therefore, the objective function is introduced first, and then the modeling of a nonlinear map and solution to the optimization problem are described. In Section 4, the proposed method is evaluated by simulations and real data, and finally the paper is concluded in Section 5.
2. FRINGE DISTORTION
Let be an obtained image of a fringe pattern, and denotes the position vector. In addition, the nonlinearity of the observation process is represented by a pixel-wise nonlinear map as
Distortion of is often analyzed by its Fourier series,
Although the concept of minimizing higher harmonic components is straightforward, separating them from the observed images is not an easy task because of the modulation caused by . One strategy for avoiding this difficulty is to utilize a simple model, including the gamma model, which has only a few parameters to determine. On the other hand, we will not simplify the model of nonlinearity so much but simplify the objective function to be minimized. That is, we propose to approximate the coefficients of the harmonic components by the Fourier transform rather than the Fourier series in Eq. (2).
3. BLIND COMPENSATION OF A DISTORTED FRINGE
In this section, a general framework for compensating unknown nonlinearity is proposed. It is formulated as an optimization problem minimizing a criterion of fringe distortion. Here, the proposed criterion is introduced first, and then the optimization method is described.
A. Fourier-Transform-based Criterion of Distortion
The two-dimensional Fourier transform is defined by
Since the harmonic components caused by distortion are lying in the high-frequency region of the spectrum, the total power within this region,
A simple modification to overcome this dependency on the magnitude in Eq. (4) is to calculate the ratio of the power of the low- and high-frequency components,
B. Gamma-Model-based Compensation
Although the objective function, Eq. (5), can be calculated numerically without effort, minimizing it is not easy because the variable of Eq. (6) is , which is a nonlinear continuous function lying on an infinite-dimensional space. Therefore, it must be expressed by a finite set of parameters. As an illustrative example, the well-known gamma model of the simplest form is presented here, because it has been used for compensation in many articles.
Compared to the multivariate counterpart, a single-variable optimization problem is extremely easy. Therefore, if a simple gamma model,6), the optimization problem becomes tractable, because the variable to be optimized is a single parameter: finding 8). Since can be calculated by a single fast Fourier transform, solving Eq. (8) by a line search method, which requires the calculation of only several times, is reasonably fast.
C. Compensation by Inverse Map Estimation
In the previous subsection, it is shown by example that a simple gamma model can be incorporated with the optimization problem. Any other single parameter model can also be used for Eq. (6) in the same manner. However, utilizing a model with multiple parameters results in a quite difficult problem, because it becomes a nonconvex optimization problem, which has a lot of local minima. Here, we show that a linear combination model can overcome this difficulty and provide a general framework for solving Eq. (6) globally.
A linear combination of a known set of functions ,6) as a finite-dimensional optimization problem: finding
In the above model, can be computed only from the observed image . Therefore, by constructing a matrix as9) can be expressed by a matrix–vector multiplication 10) can be written as a minimization of with respect to : finding
This simplification of allows us to further simplify the original problem as follows:17) is an eigenvector corresponding to the smallest generalized eigenvalue . There are many eigenvalue solvers that can numerically solve Eq. (18) without much effort. After obtaining a set of optimal parameters , Eq. (9) provides an estimated nonlinear map compensating the distortion effect.
Although the main portion of the proposed method has been described, one slight modification of the above formulation is needed to obtain a meaningful result. As the matrices and are constructed through the Fourier transform, their elements are complex numbers in general. Then, directly solving Eq. (18) yields a complex solution that cannot be applied to calibrate distortion. Thus, the above formulation must be modified so that all solutions contain only real numbers.
For a complex scaler , is simply18) with and , we obtain the real counterpart of the generalized eigenvalue problem,
D. Summary of the Proposed Algorithm Solving Eq. (10)
The proposed algorithm for compensating the nonlinear effect by solving Eq. (10), described in the previous subsection, can be summarized as the following procedure:
- (1) Input a fringe image . Some preprocessing such as noise reduction might be applied for better estimation.
- (2) Transform the image by a set of predefined approximating functions .
- (3) Calculate the two-dimensional Fourier transform of each piece of transformed data as .
- (4) Vectorize and concatenate them horizontally to construct a matrix .
- (5) Split into two matrices and by selecting its rows corresponding to the high- and low-frequency regions.
- (6) Separate the real and imaginary parts of and , and concatenate them vertically as and .
- (7) Solve the generalized eigenvalue problem for the matrix pencil .
- (8) Set the eigenvector corresponding to the smallest generalized eigenvalue as the optimal parameter .
- (9) Compensate the distortion effect by calculating the linear combination in Eq. (9) with the obtained solution .
In the above construction of matrix , multiple images (or small patches of images whose size may differ from each other) can be taken into account simultaneously by stacking corresponding matrices vertically: . Moreover, window functions can be utilized for calculating the Fourier transform in order to separate the fundamental and harmonic components better.
Note that the matrices and are both square and their size is same as the number of approximating functions in Eq. (9). Therefore, the computational cost of solving the above eigenvalue problem does not depend on the image size. This should be an advantage when working on a huge fringe image. Although image size has an impact on the construction of the matrices, the fast Fourier transform allows an efficient implementation. The main computational cost of the proposed algorithm may be the calculation of the matrix–matrix multiplications, and , which can be executed in parallel by multicore processors.
E. Choice of Approximating Functions
In the above framework, there are many options for the choice of approximating functions in the linear combination model. If one knows some information about the inverse map , that knowledge can be used in choosing suitable functions. However, in many cases, there is no such prior information. Therefore, the set of functions has to be chosen so that any unknown map can be approximated reasonably well.
One popular choice for such approximation is the sequence of polynomials , or monomial, because the set of polynomials can approximate all continuous functions arbitrarily well. Its close relation to the Taylor series should also be a reason for its popularity. However, it is well known in the approximation theory that a monomial is exponentially ill conditioned, i.e., numerically unstable.
In this paper, for numerical stability, the normalized Legendre polynomial is chosen for ,30].
In order to confirm the flexibility and effectiveness of the proposed framework, it was applied to four simulated data and three measured data. The simple gamma model in Eq. (7) and the linear combination model in Eq. (9) with the Legendre polynomials [Eq. (23)] were compared in each experiment by solving Eq. (8) and Eq. (10). For calculating the Fourier transform in this section, a tapered Hann window (smooth trapezoidal window) was used (at the edge of the data, about a half-period of fringes was multiplied by the half-part of the Hann window).
As illustrative examples, four types of nonlinearity were simulated. For simplicity, one-dimensional fringes for the three-step phase-shifting method were considered.
Nonlinearly transformed fringes were obtained from
Case (a): simple gamma,
The parameters in the above maps were selected so that each reconstructed phase,
Without any additional information, the distorted fringes were compensated by the proposed method. For comparison, the simple gamma model [Eq. (7)] and polynomial model [Eq. (9) with Eq. (23)] were applied to all four cases. For the polynomial model, fringes were scaled as so that their values are in , because the Legendre polynomials are defined in . The maximum degree of the polynomial was chosen as 15. The boundary of low-frequency region and high-frequency region was set to 1.5 times the fundamental frequency (as described in Section 3.A, the DC component is not included in ).
Figure 1 shows the compensation results, where only the first quarter, , is depicted. Figure 1(a) shows that the gamma model accurately compensated the gamma nonlinearity. This result confirms that the proposed criterion [Eq. (5)] can properly measure the degree of distortion and can be used for compensating fringe distortion. On the other hand, as in Figs. 1(b)–1(d), the gamma model cannot compensate the error caused by other types of nonlinearity because of model mismatch. The risk of such model mismatching is one drawback of a model-based compensation method.
In contrast to the gamma model, the polynomial model was able to compensate all four types of nonlinearity to some extent. Although the polynomial itself is also a model, it can approximate any continuous function arbitrarily well as the degree of the polynomial increases. Therefore, by using a polynomial of sufficiently high degree, the proposed method can compensate distortion caused by any type of (continuous) nonlinearity without knowledge of it. Table 1 shows the root-mean-square (RMS) error of each result. From the table, it is obvious that the gamma model can compensate only gamma distortion, while a polynomial model can compensate any nonlinear distortion.
B. Measured Flat Surface
To confirm performance in real situations, the proposed method was applied to real data. Measured data were obtained by an EPSON EB-1735W in projecting fringe patterns and a SIGMA dp2 Quattro in capturing monochromatic images.
In order to compare with the simulations in Fig. 1, a flat surface was measured by projecting two different three-step fringe patterns:
Case (a): sinusoidal patterns without defocus,
Case (b): binary patterns with defocus using the projector.
Defocusing acts as a low-pass filter on the binary fringe, which results in a sinusoidal-like waveform . Parameters for compensation were chosen to be the same as for the simulations: the degree of the polynomial was 15, and the boundary of and was 1.5 times the fundamental frequency.
Figure 2 shows single lines of the compensated results for the measured data, for which the RMS error values are shown in Table 2. Since the case in Fig. 2(a) is an ordinary situation in which the gamma model can be applied, both the gamma model and the polynomial model reduced similar amounts of error. On the other hand, the compensation result for the gamma model contains more error than the polynomial model for the case in Fig. 2(b). This should be caused by model mismatch of the gamma model, as illustrated in Fig. 1.
Although the error was reduced by the proposed method for both cases, the amount of reduction was not as convincing as in the previous simulation. Possible reasons for these results are the following. First, the proposed method only account for phase reconstruction error related to the nonlinear distortion of fringes. However, there are other sources of error including vibration, illuminance variation, and random noise. Every nonideal situation not only contributes to error but also reduces the accuracy of estimation for the compensation algorithm. This could be a factor that is limiting the practical performance of the proposed method. Nevertheless, the proposed method can be combined with other methods that compensate error not caused by distortion to improve the result. Second, the illumination and nonlinearity of the observed images depend on its position in general. Therefore, a nonlinear map compensating distortion should also be position dependent. However, in this paper, the proposed method only considers a single nonlinear map uniformly applied to each pixel value. This global nature could be another factor limiting the performance. Although it must be possible to extend the proposed method to adapt to such nonuniformity, some effort is needed for the modification.
C. Application to Real Objects
In order to demonstrate the performance of the proposed method for a practical situation, with an actual application in mind, the ear pinna of a dummy head was measured. An ear pinna is an important part of the ear that supplies information regarding the position of a sound source to the human. Measuring the shape of an actual ear pinna enables the simulation of localized sound, which can be used for personalizing a 3D audio system [31,32].
The ear pinna of a dummy head (NEUMANN KU100) was measured by the four-step phase-shifting method. Figure 3 shows one of the four fringe images, and Fig. 4 shows the corresponding measured result with and without compensation. Since the illumination of the images varies according to the position, the top 20 rows of the images (nearly flat part of the images) were used for estimating optimal parameters. Also, the nonlinear map obtained for compensating reconstruction error was scaled at each position according to the illumination of the images. For better comparison, cross sections of the results are also depicted in Fig. 5.
It can be seen in Figs. 4 and 5 that the result obtained without compensation contains error as a periodic pattern, which could be problematic for later application. On the other hand, the result compensated by the proposed method contains less error. These figures indicate that the polynomial model suppressed more error than the gamma model, as in Fig. 2. Therefore, we conclude that the proposed method can compensate measurement error caused by nonlinearity that is not only related to the gamma model but also related to other models.
In this paper, a methodology of compensating nonlinear distortion of sinusoidal fringes was proposed. The proposed criterion of distortion allows us to formulate a compensation method as an optimization problem. Moreover, the linear combination model allows an analytic solution of the optimization problem, which can be obtained by a simple procedure. Through simulations and real measurements, it was confirmed that the proposed method can reduce error caused by multiple types of nonlinearity from observed data only.
As mentioned in Section 4.B, at least two points should be improved to obtain more accurate measurement results. Since there is not only nonlinearity but also several other sources of error, combining the proposed method with other compensation methods would be preferable. Moreover, position-adaptive modification of the proposed method should be investigated in the future to deal with nonlocal behavior of illumination and nonlinear distortion of fringes.
Japan Society for the Promotion of Science (JSPS) (15J08043, 16J06772).
1. M. Takeda and K. Mutoh, “Fourier transform profilometry for the automatic measurement of 3-D object shapes,” Appl. Opt. 22, 3977–3982 (1983). [CrossRef]
2. P. S. Huang, F. Jin, and F. P. Chiang, “Quantitative evaluation of corrosion by a digital fringe projection technique,” Opt. Lasers Eng. 31, 371–380 (1999). [CrossRef]
3. S. Zhang, “Recent progresses on real-time 3D shape measurement using digital fringe projection techniques,” Opt. Lasers Eng. 48, 149–158 (2010). [CrossRef]
4. S. S. Gorthi and P. Rastogi, “Fringe projection techniques: whither we are?” Opt. Lasers Eng. 48, 133–140 (2010). [CrossRef]
5. K. Yatabe and Y. Oikawa, “Convex optimization based windowed Fourier filtering with multiple windows for wrapped phase denoising,” Appl. Opt. 55, 4632–4641 (2016). [CrossRef]
6. X. Y. Su, W. S. Zhou, G. von Bally, and D. Vukicevic, “Automated phase-measuring profilometry using defocused projection of a Ronchi grating,” Opt. Commun. 94, 561–573 (1992). [CrossRef]
7. K. Hibino, B. F. Oreb, D. I. Farrant, and K. G. Larkin, “Phase shifting for nonsinusoidal waveforms with phase-shift errors,” J. Opt. Soc. Am. A 12, 761–768 (1995). [CrossRef]
8. Y. Surrel, “Design of algorithms for phase measurements by the use of phase stepping,” Appl. Opt. 35, 51–60 (1996). [CrossRef]
9. Y. Hu, J. Xi, E. Li, J. Chicharo, and Z. Yang, “Three-dimensional profilometry based on shift estimation of projected fringe patterns,” Appl. Opt. 45, 678–687 (2006). [CrossRef]
10. M. J. Baker, J. Xi, and J. F. Chicharo, “Elimination of γ non-linear luminance effects for digital video projection phase measuring profilometers,” in Proceedings of IEEE International Symposium on Electronic Design, Test and Applications (DELTA) (IEEE, 2008), pp. 496–501.
11. B. Pan, Q. Kemao, L. Huang, and A. Asundi, “Phase error analysis and compensation for nonsinusoidal waveforms in phase-shifting digital fringe projection profilometry,” Opt. Lett. 34, 416–418 (2009). [CrossRef]
12. K. Liu, Y. Wang, D. L. Lau, Q. Hao, and L. G. Hassebrook, “Gamma model and its analysis for phase measuring profilometry,” J. Opt. Soc. Am. A 27, 553–562 (2010). [CrossRef]
13. P. S. Huang, C. Zhang, and F.-P. Chiang, “High-speed 3-D shape measurement based on digital fringe projection,” Opt. Eng. 42, 163–168 (2003). [CrossRef]
14. K. Ishikawa, K. Yatabe, N. Chitanont, Y. Ikeda, Y. Oikawa, T. Onuma, H. Niwa, and M. Yoshii, “High-speed imaging of sound using parallel phase-shifting interferometry,” Opt. Express 24, 12922–12932 (2016). [CrossRef]
15. H. Guo, H. He, and M. Chen, “Gamma correction for digital fringe projection profilometry,” Appl. Opt. 43, 2906–2914 (2004). [CrossRef]
16. T. Hoang, B. Pan, D. Nguyen, and Z. Wang, “Generic gamma correction for accuracy enhancement in fringe-projection profilometry,” Opt. Lett. 35, 1992–1994 (2010). [CrossRef]
17. Z. Li and Y. Li, “Gamma-distorted fringe image modeling and accurate gamma correction for fast phase measuring profilometry,” Opt. Lett. 36, 154–156 (2011). [CrossRef]
18. X. Zhang, L. Zhu, Y. Li, and D. Tu, “Generic nonsinusoidal fringe model and gamma calibration in phase measuring profilometry,” J. Opt. Soc. Am. A 29, 1047–1058 (2012). [CrossRef]
19. P. Zhou, X. Liu, Y. He, and T. Zhu, “Phase error analysis and compensation considering ambient light for phase measuring profilometry,” Opt. Lasers Eng. 55, 99–104 (2014). [CrossRef]
20. J. Yao, C. Xiong, Y. Zhou, H. Miao, and J. Chen, “Phase error elimination considering gamma nonlinearity, system vibration, and noise for fringe projection profilometry,” Opt. Eng. 53, 094102 (2014). [CrossRef]
21. S. Ma, C. Quan, R. Zhu, L. Chen, B. Li, and C. J. Tay, “A fast and accurate gamma correction based on Fourier spectrum analysis for digital fringe projection profilometry,” Opt. Commun. 285, 533–538 (2012). [CrossRef]
22. S. Zhang, “Comparative study on passive and active projector nonlinear gamma calibration,” Appl. Opt. 54, 3834–3841 (2015). [CrossRef]
23. Z. Cai, X. Liu, X. Peng, Z. Zhang, H. Jiang, Y. Yin, and S. Huang, “Phase error compensation methods for high-accuracy profile measurement,” Meas. Sci. Technol. 27, 045201 (2016). [CrossRef]
24. S. Zhang and S. Yau, “Generic nonsinusoidal phase error correction for three-dimensional shape measurement using a digital video projector,” Appl. Opt. 46, 36–43 (2007). [CrossRef]
25. Y. Xu, L. Ekstrand, J. Dai, and S. Zhang, “Phase error compensation for three-dimensional shape measurement with projector defocusing,” Appl. Opt. 50, 2572–2581 (2011). [CrossRef]
26. K. Liu, S. Wang, D. L. Lau, K. E. Barner, and F. Kiamilev, “Nonlinearity calibrating algorithm for structured light illumination,” Opt. Eng. 53, 050501 (2014). [CrossRef]
27. C. Zhang, H. Zhao, L. Zhang, and X. Wang, “Full-field phase error detection and compensation method for digital phase-shifting fringe projection profilometry,” Meas. Sci. Technol. 26, 035201 (2015). [CrossRef]
28. M. J. Baker, J. Xi, and J. F. Chicharo, “Neural network digital fringe calibration technique for structured light profilometers,” Appl. Opt. 46, 1233–1243 (2007). [CrossRef]
29. S. Sonoda and N. Murata, “Sampling hidden parameters from oracle distribution,” in Proceedings of International Conference on Artificial Neural Networks (ICANN) (Springer, 2014), pp. 539–546.
30. L. N. Trefethen, “Householder triangularization of a quasimatrix,” IMA J. Numer. Anal. 30, 887–897 (2010). [CrossRef]
31. M. Otani and S. Ise, “Fast calculation system specialized for head-related transfer function based on boundary element method,” J. Acoust. Soc. Am. 119, 2589–2598 (2006). [CrossRef]
32. M. Otani, T. Hirahara, and S. Ise, “Numerical study on source-distance dependency of head-related transfer functions,” J. Acoust. Soc. Am. 125, 3253–3261 (2009). [CrossRef]