3D shape measurement of discontinuous specular objects based on advanced PMD with bi-telecentric lens

Zhenqi Niu; Nan Gao; Zonghua Zhang; Feng Gao; Xiangqian Jiang

doi:10.1364/OE.26.001615

1. Introduction

Specular free-form surfaces are recently widely used in many fields, such as optics, car industry, aerospace, and micro-electro mechanical systems (MEMS). The research of three dimensional (3D) shape measurement for these objects is still in the early stage. The traditional fringe projection techniques [1–4] can only measure diffused surfaces. In order to measure specular objects by using fringe projection, the surface characteristics need to be changed in advance, for example, coating the surface by powder. This operation will slow the measurement speed and reduce the measurement accuracy.

Phase measuring deflectometry (PMD) [5–8], which is aimed at measuring specular free-form surfaces, has been widely studied in recent years, because of its advantages of large dynamic range, non-contact operation, full-field measurement, fast acquisition, high precision and automatic data processing. PMD has been applied to measure aspheric mirror [9] and dynamic specular surface [10]. Phase information of the deformed fringe patterns is demodulated to obtain the slope of the measured specular surface and then 3D shape of the measured specular surface can be reconstructed by integrating the gradients. Each point of a reconstructed shape depends on the gradients of surrounding points because of the integration procedure, and the regularity of the surface must be carefully considered [11–13]. Because the slope data calculation and the integration procedure are sensitive to the system errors, the obtained 3D shape data are inaccurate. The other drawback is that discontinuous specular objects cannot be measured because of the integration procedure. In order to measure discontinuous specular objects, some researchers have proposed several improved PMD methods. Guo et al [14] presented a measurement method with one camera and one LCD plane that is moved into two or more positions to complete specular measurement by special geometrical structure. However, the accuracy of measurement depends on the precisions of the system parameters and the speed is limited due to the horizontal and vertical fringe patterns projected in the LCD screen. It is difficult to accurately calibrate the measurement system and establish the system structure [15]. Knauer et al [16] proposed a stereo deflectometry to obtain the height and gradient information simultaneously. The system consists of two cameras and one LCD screen. However, the measurement accuracy depends on the process of an iterated algorithm and the precision of the geometric relationship between the cameras and LCD screen.

There are two problems in the existing PMD methods. One is a limited depth of field (DOF) of the camera lens because the specular surface under test and the virtual image of the screen have different distances to the camera. The captured fringe pattern image is blurred because of the limited DOF [14, 17], so that it is difficult to accurately calibrate the system parameters and measure specular object with a large depth. The other problem is system calibration of PMD. It is an important and basic step to obtain the accurate measurement results, especially to calibrate the relative orientation between the camera and LCD screen [14, 18–20]. However, this is a difficult and challenging problem because the camera cannot directly see the LCD screen [21–23], as illustrated in Fig. 1. Generally, there are two kinds of geometric calibration method. One is to use a plane mirror with markers on the surface to calibrate the relative orientation between the camera and LCD screen. The known separations between markers determine the position of the mirror [19, 20]. Obviously, it is troublesome and inconvenient to accurately determine the position of these markers. The other is to utilize a plane mirror without any markers. The mirror reflects the pattern displayed on the LCD screen several times to determine the relative relationship between the camera and the LCD screen [21–23]. All these calibration methods aim at the pinhole camera, without considering the lens distortion. According to the principle of photogrammetry, lens distortion has great effect on high-precision 3D measurement. A bi-telecentric lens has negligible distortion, so that it can be used to improve the measurement system. However, there is not an available method to calibrate the relative orientation between the LCD screen and camera with a bi-telecentric imaging lens. .

Fig. 1 Schematic diagram of the relationship between a pinhole camera and an LCD screen.

Download Full Size | PDF

By using a bi-telecentric lens, this paper proposes a new advanced PMD method having unlimited imaging DOF. A novel mathematical model is established for measuring 3D shape of discontinuous specular objects by directly establishing the relationship between absolute phase and depth. The camera captures the deformed fringe patterns reflected by the measured specular surface from the same viewpoint of the displayed LCD screens. An auxiliary plane mirror is used to calibrate the measuring system by reflecting the pattern displayed on the LCD screen three times. Three specular objects have been measured to verify the accuracy and evaluate the performance of the proposed 3D measuring method.

The rest of the paper is organized as follows. In Section 2, the principle of measurement and calibration are demonstrated. Section 3 shows the system calibration and the 3D measurement results of specular objects by using simulation data. In Section 4, the actual experiments on several specular objects are carried out and a quantitative evaluation of the measurement system is given. Finally, Section 5 concludes the paper.

2. Principle

The proposed measurement method displays sinusoidal fringe patterns onto an LCD screen, which is moved to two different positions along normal direction of a reference plane. A camera captures the deformed fringe patterns reflected by the measured specular surface from the same viewpoint of the screens through a flat beam splitter. A four-step phase shift algorithm and an optimum multiple-fringe numbers selection method [24, 25] are used to calculate the wrapped and unwrapped phase data pixel by pixel from the captured sinusoidal fringe patterns. According to the established mathematical model, 3D shape of discontinuous specular surface can be directly reconstructed from the calculated absolute phase map.

2.1. Measurement method

Figure 2 shows the measurement principle used to obtain 3D shape of discontinuous specular objects. It consists of an LCD screen, a camera with a bi-telecentric lens, a flat beam splitter, and a reference specular plane. The LCD screen is used as the diffusive structured light source, which can be moved by a translating stage. In order to directly obtain 3D shape of specular objects from the deformed fringe patterns, the sinusoidal fringe patterns need to be displayed on the LCD screen located at two known positions of P1 and P2. The beam splitter is used to change the light path of the camera, so that the optical axis of lens is vertical to the LCD screen. Because of the characteristics of bi-telecentric lens, only the light rays that are approximately parallel to the optical axis of the lens can form the image, as illustrated in Fig. 2. Two light rays are displayed from LCD screens and reflected into the camera by the measured specular object surface and the reference specular plane.

Fig. 2 Schematic diagram of the measurement principle.

Download Full Size | PDF

The LCD screen remains parallel to the reference specular plane that is vertical to the optical axis of camera. ∆d is the translating distance of the screen by the stage and d is the distance between the reference specular plane and the LCD screen located at P1. The two incident rays correspond to the same reflection light. The phases of the two incident rays are $φ_{r 1}$ and $φ_{r 2}$ on the reference plane and $φ_{m 1}$ and $φ_{m 2}$ on the measured free-form surface. Because normal vector of the reference plane is parallel to the optical axis of camera, the incident ray coincides with the reflected ray through the reference plane. $α$ is the angle between the other incident ray and the reflected ray through the measured object surface. The period of the displayed fringe pattern is q. Parameter h stands for height of the measured specular surface with respect to the reference plane.

According to the geometric relationship in Fig. 2, the following equations can be deduced.

\frac{φ_{m 1} - φ_{r 1}}{2 π} q = (d - h) \tan α

\frac{φ_{m 2} - φ_{r 2}}{2 π} q = (Δ d + d - h) \tan α

From Eqs. (1) and (2), height of the measured specular surface is

h = d - \frac{Δ d (φ_{m 1} - φ_{r 1})}{(φ_{m 2} - φ_{r 2}) - (φ_{m 1} - φ_{r 1})}

Because d is a constant, it can be moved to the left part. Depth information can be defined as H = d-h, which is the distance between the measured specular surface and the LCD screen located at P₁. H can be directly calculated from phase data. It clearly shows that depth H can be directly calculated from the captured fringe patterns only if the parameter ∆d and phase information on the reference plane mirror are determined beforehand. Because the optimum three-fringe numbers selection method will be used to independently calculate the absolute phase pixel by pixel, discontinuous specular objects surfaces can be measured by the proposed method. However, when normal of the measured specular point is vertical to the LCD screen, the reference phase equals to the phase of the corresponding point on the measured specular surface. In this case, denominator of Eq. (3) is zero, so that the points whose normal is vertical to the LCD screen on specular objects cannot be measured effectively.

2.2. Camera model with bi-telecentric lens

A bi-telecentric lens has the property of purely orthographic projections of scene points and maintains a constant magnification over a specific range of object distances [26–28]. In a bi-telecentric system, only the light rays that are approximately parallel to the optical axis of the lens pass the aperture stop and form the image. The bi-telecentric lens accurately reproduces dimensional relationships within its telecentric depth, and it is robust to small differences in the distance between the lens and the camera’s sensor. The bi-telecentric lens shows a very low distortion and offers a large DOF. A pinhole model of wide angle cameras is not applicable for the bi-telecentric imaging system. The camera model with a bi-telecentric lens is demonstrated in Fig. 3 [29]. The bi-telecentric lens simply performs a magnification in both X and Y directions, while it is not sensitive to the depth in Z direction.

Fig. 3 Model of bi-telecentric camera imaging.

Download Full Size | PDF

Assuming a 3D point and the corresponding 2D point are denoted $P {[\begin{matrix} X_{W} & Y_{W} & Z_{W} \end{matrix}]}^{T}$ and $p = {[\begin{matrix} u & v \end{matrix}]}^{T}$ , respectively. By carrying out ray transfer matrix analysis for such an optical system, the projection of an arbitrary point P to computer image coordinate in pixels is expressed as Eq. (4) [30].

[\begin{matrix} u \\ v \\ 1 \end{matrix}] = [\begin{matrix} m / d_{u} & 0 & 0 \\ 0 & m / d_{v} & 0 \\ 0 & 0 & 1 \end{matrix}] [\begin{matrix} r_{11} & r_{12} & r_{13} & t_{x} \\ r_{21} & r_{22} & r_{23} & t_{y} \\ 0 & 0 & 0 & 1 \end{matrix}] [\begin{matrix} X_{w} \\ Y_{W} \\ Z_{W} \\ 1 \end{matrix}]

where m is the effective magnification of the bi-telecentric lens;

r_{i j}

is the element of rotation matrix R and

t_{x}, t_{y}

are elements of the translation matrix T;

d_{u}

and

d_{v}

are the sizes of a pixel in the x and y directions, respectively. It is obvious that there is not a principal point

u_{0} v_{0}

for the telecentric lens from Eq. (4), because the bi-telecentric lens performs parallel projection and there is not a projection center [27, 28].

\begin{array}{l} {\begin{matrix} δ_{x} = k_{1} x_{u} (x_{u}^{2} + y_{u}^{2}) + h_{1} (3 x_{u}^{2} + y_{u}^{2}) + 2 h_{2} x_{u} y_{u} + s_{1} (x_{u}^{2} + y_{u}^{2}) \\ δ_{y} = k_{1} y_{u} (x_{u}^{2} + y_{u}^{2}) + 2 h_{1} x_{u} y_{u} + h_{2} (x_{u}^{2} + 3 y_{u}^{2}) + s_{2} (x_{u}^{2} + y_{u}^{2}) \end{matrix} \\ [\begin{matrix} x_{d} \\ y_{d} \end{matrix}] = [\begin{matrix} x_{u} \\ y_{u} \end{matrix}] + [\begin{matrix} δ_{x} \\ δ_{y} \end{matrix}] \end{array}

where

(x_{u}, y_{u})

are the image coordinates if a perfect orthographic projection model is used;

(x_{d}, y_{d})

are the actual image coordinates on the image plane;

(k_{1}, h_{1}, h_{2}, s_{1}, s_{2})

are the distortion coefficients.

2.3. Geometric calibration

2.3.1. Bi-telecentric imaging model by a plane mirror

Through a plane mirror, the imaging geometric relationship between the camera and mirrored LCD screen is illustrated in Fig. 4.

Fig. 4 Schematic diagram of bi-telecentric imaging model by a plane mirror.

Download Full Size | PDF

Markers with known separation in between are generated by software and displayed on the LCD screen. The screen is reflected by the plane mirror placed at three different positions $π_{j} (j = 1, 2, 3)$ [23] and the virtual image of the markers is captured by the CCD camera. Assuming ${}^{X}p^{i} (i = 0, 1, 2, \dots, k)$ denotes the position of the marker points in the world coordinate system {X}. These positions are modeled as located at

{}^{C}P {}^{i}= R {}^{X}P^{i} + T

in the camera coordinate system {C} with a rotation matrix R and a translation vector T. According to the law of reflection, the reference point

^{C} p^{i}

is mirrored by

π_{j}

as

^{C} p_{m j}^{i}

in {C}. Each mirror

π_{j}

is defined by its normal vector

^{C} n_{j}

and its distance

d_{j}

from C.

The relationship between $^{C} p^{i}$ and its reflection $^{C} p_{m j}^{i}$ by mirror $π_{j}$ is known as Householder transformation. However, the bi-telecentric lens is not sensitive to depth in Z direction. Therefore, the third element $t_{z}$ of the translation vector T cannot be calibrated. The real 3D reflection points cannot be estimated. Moreover, the ideal imaging model cannot be used to calibrate posture of the camera with the bi-telecentric lens. In order to address the problem, a new measurement model is described in the following subsection.

2.3.2. Measurement model

Assuming $T = [\begin{matrix} 0 & 0 & 0 \end{matrix}]$ , a new imaging model can be obtained where original point $^{C} p^{0}$ of the LCD screen and the corresponding original point $^{C} p_{j}^{0}$ are located in the original point of the camera, as illustrated in Fig. 5. In the new model, the law of reflection is still effective. The LCD screen is reflected by the plane mirror located at three different positions $π_{j} (j = 1, 2, 3)$ and the markers on the screen are captured. The reflection of the reference point $^{C} p^{i}$ mirrored by $π_{j}$ appears as $^{C} p_{j}^{i}$ in {C}.

Fig. 5 Measurement model.

Download Full Size | PDF

Distance $t_{j}^{i}$ from the mirror $π_{j}$ to $^{C} p_{j}^{i}$ is equal to the distance from the mirror $π_{j}$ to $^{C} p^{i}$ according to the law of reflection. The relationship is expressed by

{}^{C}P^{i} = 2 t_{j}^{i} \cdot {}^{C}n_{j} + {}^{C}P_{j}^{i}

where

t_{j}^{i}

can be represented as

t_{j}^{i} = {}^{C}n_{j}^{T} \cdot ({}^{C}P_{j}^{0} - {}^{C}P_{j}^{i})

Removing $t_{j}^{i}$ from the two equations can obtain

{}^{C}P^{i} = 2 ({}^{C}n_{j}^{T} \cdot ({}^{C}P_{j}^{0} - {}^{C}P_{j}^{i})) \cdot {}^{C}n_{j} + {}^{C}P_{j}^{i}

Considering the reference 3D point $^{C} p^{i}$ is unique, its two mirrored 3D points $^{C} p_{j}^{i}$ and $^{C} p_{j'}^{i} (j \neq j')$ reflected by two different mirror positions $π_{j}$ and $π_{j'}$ , respectively, are all applicable to the Eq. (9).

From Eq. (9), it concludes that not only can 3D position of the ith reference point $^{C} p^{i}$ be estimated from each of three mirrors $π_{j}$ and the corresponding mirrored position $^{C} p_{j}^{i}$ , but the estimated 3D positions from different mirrors should be equal to each other, which can be represented

\begin{array}{l} {}^{C}P^{i} = 2 \cdot ({}^{C}n_{j}^{T} \cdot ({}^{C}P_{j}^{0} - {}^{C}P_{j}^{i})) \cdot {}^{C}n_{j} + {}^{C}P_{j}^{i} \\ = 2 \cdot ({}^{C}n_{j'}^{T} \cdot ({}^{C}P_{j'}^{0} - {}^{C}P_{j'}^{i})) \cdot {}^{C}n_{j'} + {}^{C}P_{j'}^{i} \end{array}

2.3.3. Solution of rotation matrix R

Assuming $m_{j j'} = n_{j} \times n_{j'}$ denotes the axis vector which lies along the intersection of the two mirror planes, the axis vector $m_{j j'}$ is perpendicular to both mirror plane normal vector $n_{j}$ and $n_{j'}$ .

{\begin{matrix} {}^{C}n_{j}^{T} \cdot m_{j j'} = 0 \\ {}^{C}n_{j'}^{T} \cdot m_{j j'} = 0 \end{matrix}

Multiplying $m_{j j'}$ on the right side of Eq. (10) obtains

\begin{array}{l} {}^{C}P {_{j}^{i}}^{T} \cdot m_{j j'} = {}^{C}P {_{j'}^{i}}^{T} \cdot m_{j j'} \\ \Leftrightarrow {({}^{C}P_{j}^{i} - {}^{C}P_{j'}^{i})}^{T} \cdot m_{j j'} \end{array}

By applying the orthogonality constraint for the mirrored positions of reference points $^{C} p^{i} (i = 0, 1, 2, \dots, k)$ , the following equation holds

[\begin{matrix} {({}^{C}P_{j}^{1} - {}^{C}P_{j'}^{1})}^{T} \\ {({}^{C}P_{j}^{2} - {}^{C}P_{j'}^{2})}^{T} \\ ⋮ \\ {({}^{C}P_{j}^{i} - {}^{C}P_{j'}^{i})}^{T} \end{matrix}] \cdot m_{j j'} = Q_{j j'} \cdot m_{j j'} = 0

Multiplying $Q_{j j'}^{T}$ on the left side of Eq. (13),

Q_{j j'}^{T} Q_{j j'} m_{j j'} = M_{j j'}^{T} \cdot m_{j j'} = 0.

Since $M_{j j'}$ is a 3 × 3 positive semidefinite matrix, $M_{j j'}$ can be calculated as the eigenvector corresponding to the smallest eigenvalue of $M_{j j'}$ .

The axis vectors $m_{12}$ , $m_{23}$ and $m_{31}$ are obtained as the eigenvectors corresponding to the smallest eigenvalues of $M_{12}$ , $M_{23}$ and $M_{31}$ , respectively. Using these axis vectors can compute the normal vector of the mirrors by the following equation.

{\begin{matrix} {}^{C}n_{1} = \frac{(m_{11} \times m_{13})}{‖ m_{11} \times m_{13} ‖} \\ {}^{C}n_{2} = \frac{(m_{23} \times m_{12})}{‖ m_{23} \times m_{12} ‖} \\ {}^{C}n_{3} = \frac{(m_{31} \times m_{23})}{‖ m_{31} \times m_{23} ‖} \end{matrix}

After removing $^{C} p^{i}$ from Eqs. (6) and (9), a large system of linear equations can be derived

A Z = B

where

\begin{array}{l} A = {[\begin{matrix} x_{1} I_{3} & ... & x_{k} I_{3} & x_{1} I_{3} & ... & x_{k} I_{3} & x_{1} I_{3} & ... & x_{k} I_{3} \\ y_{1} I_{3} & ... & y_{k} I_{3} & y_{1} I_{3} & ... & y_{k} I_{3} & y_{1} I_{3} & ... & y_{k} I_{3} \end{matrix}]}^{T} \\ Z = {[\begin{matrix} r_{1}^{T} & r_{2}^{T} \end{matrix}]}^{T} \\ B = {[\begin{matrix} b_{1}^{1} & ... & b_{1}^{k} & b_{1}^{1} & ... & b_{1}^{k} & b_{1}^{1} & ... & b_{1}^{k} \end{matrix}]}^{T} \\ b_{j}^{i} = {(2 ({}^{C}n_{j}^{T} \cdot ({}^{C}P_{j}^{0} - {}^{C}P_{j}^{i})) \cdot {}^{C}n_{j} + {}^{C}P_{j}^{i})}^{T} \end{array}

Without loss of generality, $^{X} p^{i} = {(x_{i}, y_{i}, 0)}^{T} (1 \leq i \leq k)$ is used in these equations. $r_{1}$ and $r_{2}$ are the first and second column vectors of R, that is, $R = (\begin{matrix} r_{1} & r_{2} & r_{3} \end{matrix})$ .

The least-squares solution for Z can be computed by $Z = A^{*} \cdot B$ , where $A^{*}$ is the Moore-Penrose pseudo-inverse of A. The third column vector of rotation matrix $r_{3}$ can be computed as follows [30].

r_{3} = \frac{(r_{1} \times r_{2})}{‖ r_{1} ‖ \cdot ‖ r_{2} ‖}

Through the above procedure, the rotation matrix R between LCD screen and camera can be calculated. Therefore, the parallel relationship between them can be guaranteed by adjusting the LCD screen according to the calculated rotation matrix R.

3. Simulation

The simulated experiments were carried out to evaluate the proposed measurement method. To generate the simulated data, the following default values were used. The original distance from the camera to mirrors was 340 mm. The normal vectors $^{C} n_{j} (j = 1, 2, 3)$ of mirror $π_{j}$ were set to (sinθ_z∙sinθ_x + cosθ_x∙cosθ_z∙sinθ_y, sinθ_x∙cosθ_z + cosθ_x∙sinθ_z∙sinθ_y, cosθ_x∙cosθ_y), where $θ_{k} (k = x, y, z)$ was the angle with respect to each axis, and drawn randomly within the ranges of (−20≤θ_x≤20, −20≤θ_y≤20, −20≤θ_z≤20). The rotation matrix and translation vector from the world coordinate system of LCD screen to the camera coordinate system were set to $I_{3}$ and ${[300, 100, 160]}^{T}$ , respectively. The world coordinates of reference 3D points were defined as [0, f, g]^T, (f = 0, 20, …, 60, g = 0, 20, …, 60). Gaussian noise with zero-mean and standard deviation σ = 0, 0.2, 0.4, …, 3 was added to the synthesized reference 3D points. Using the simulated data, the system was calibrated and then a simulated specular plane and a simulated specular sphere were measured.

3.1. Geometric calibration

The normal vectors $^{C} n_{j}$ and rotation matrix R could be estimated by Eqs. (15)-(17). By using the preset value and the estimated results, the errors of R and $^{C} n_{j}$ were quantitatively evaluated. E_r is defined as the error by using Riemannian distance [31] between the estimated R and the ground truth value $R_{g}$

E_{r} = \frac{1}{\sqrt{2}} {‖ \log (R^{T} \cdot R_{g}) ‖}_{F}

\log R^{'} = {\begin{matrix} 0 θ = 0 \\ \frac{θ}{2 \sin θ} (R^{'} - R^{' T}) θ \neq 0 \end{matrix}

where

θ = \cos^{- 1} \frac{t r R' - 1}{2}

.

The physical meaning of Riemannian distance reflects appropriately the distance between two poses in a dynamic sense [32]. The Riemannian distance E_r defines the minimal absolute value of the angle by which the coordinate system R must be rotated around an arbitrary axis in order to align it with the coordinate system $R_{g}$ [33].

Error of $^{C} n_{j}$ is defined as the estimated angle against the ground truth $n_{g, j}$

E_{n} = \frac{1}{3} \sum_{j = 1}^{3} \cos^{- 1} ({}^{C}n_{j}^{T} \cdot n_{g, j})

Figure 6(a) shows the error against different noise. The mirror normal vectors are ${[0.493, 0.057, 0.868]}^{T}, {[0.485, 0.585, - 0.651]}^{T}, {[0.396, - 0.330, 0.857]}^{T}$ . Figure 6(b) shows the error against different mirror angles. The mirror postures were changed up to ± 20 degrees. When the angle is greater than 15 degrees, the errors of En and Er are not sufficiently obvious. The simulated data were added Gaussian noise with zero-mean and standard deviation σ = 2. These results quantitatively prove that the proposed method is insensitive to a certain noise level and the estimation accuracy increases with increasing the mirror rotation angle.

Fig. 6 Calibration errors. (a) Different noise, and (b) different mirror postures.

Download Full Size | PDF

3.2. Measurement results

Distance $Δ d$ between the two LCD screens is 30mm and distance d between LCD screen at P1 and the reference specular plane is 100mm. The LCD screen has a resolution of 2048 $\times$ 1536 pixels and a pixel size of 15 um $\times$ 15 um. The effective magnification of the bi-telecentric lens is 0.057. The camera has a pixel size of 3.45 um $\times$ 3.45 um. Twelve vertical fringe patterns having the optimum fringe numbers of 64, 63, and 56 were generated with 5% random noise by software, as shown in Fig. 7. A specular plane with size of 100 mm $\times$ 100 mm and a specular sphere with radium of 67.434 mm were simulated. 3D shape data of the two simulated specular objects were measured by using the proposed method, as shown in Fig. 8(a) and Fig. 9(a).

Fig. 7 Three sets of original fringe patterns having the optimum fringe numbers. (a) 64, (b) 63, and (c) 56.

Download Full Size | PDF

Fig. 8 Simulated results of a specular plane. (a) Simulated specular plane, and (b) absolute error.

Download Full Size | PDF

Fig. 9 Simulated results of a specular sphere. (a) Simulated specular sphere, and (b) absolute error.

Download Full Size | PDF

To verify the measurement accuracy, the mean squared errors is defined as

M S E = \sqrt{\frac{\sum_{i = 0}^{m} ({(H_{c} - H_{r})}^{2})}{m}}

where

H_{c}

is the calculated depth value,

H_{r}

is the preset value, and m is number of the measured points on the specular surface.

Figure 8(b) and 9(b) illustrate absolute error of the reconstructed plane and the reconstructed sphere between the ground truth and the calculated height. Their mean squared errors are 0.011mm and 0.015mm. Figure 9(b) clearly shows that the absolute error is increasing at the region of smooth surface. The main reason is that the measurement method is limited to measure the specular surface that is unparalleled to the LCD screen, as stated in Section 2.1. Only vertical fringe patterns (as shown in Fig. 7) on the LCD screen were used to calculate the absolute phase on the specular surface, so that the phase along the middle column had the same value as that on the reference plane. Therefore, depth data along the middle column was calculated inaccurately. In actual measurement, the problem of inaccurate data can be improved by changing the specular surface with respect to the reference plane, and then the measured data from different positions are merged together.

3.3. Influence of parallelism

When the reference plane is not parallel to the LCD screen, the angle a between the reference plane and the LCD screen is varied from 0 degrees to 1.5 degrees in increments of 0.5 degrees. Using the simulated measurement system, the relationship between angle a and the mean squared error of the above reconstructed plane is obtained as shown in Fig. 10.

Fig. 10 Relation between a and mean squared error.

Download Full Size | PDF

Figure 10 shows that with angle a increasing, the mean squared error of the measurement results increases gradually. The main reason for this trend is that with the angle a increasing, the deviations of the measured parameter d and the reference phase increase along x direction, which causing the error of measurement. Therefore, the deviation of angle a should be guaranteed in actual measurements.

4. Experiments and results

A full-field 3D shape measurement system for specular objects has been setup to test the proposed method. Three separate plat mirrors, a concave mirror and an artificial specular step have been measured to show the feasibility of directly measuring depth information from the calculated absolute phase data.

4.1. Hardware system

The developed hardware system consists of a computer, a camera with a bi-telecentric lens, an LCD screen, a flat beam splitter and a translating stage, as illustrated in Fig. 11. The camera is a SVS-Vistek camera with the model number of ceo655CVGE from Germany. It had a CS-mount lens interface, a resolution of 2448 $\times$ 2050 pixels, a frame rate of 10 fps at full resolution, and a GigE interface. The bi-telecentric lens has the model number of GCO230105 with magnification 0.057, which has a C-mount interface. Because of CS-mount of the camera, a 5-mm tube is added between the lens and the camera. The LCD screen is model LP097QX2 from LG (Seoul, Korea) and has a resolution of 2048 $\times$ 1536 pixels. The linear stage has a position accuracy of 1μm. After the system has been calibrated, parameter $Δ d$ is determined as 50 mm by the stage. Depth information of the reflected object surface could be measured by the developed system.

Fig. 11 Experimental system.

Download Full Size | PDF

4.2. Process of measurement

The measurement process mainly includes the following four steps, as shown in the following flow chart in Fig. 12.

Fig. 12 The flow chart of measuring process.

Download Full Size | PDF

Step 1: Calibrate the intrinsic parameters of camera with the bi-telecentric lens to decrease the error of system calibration and 3D reconstruction.

Step 2: Establish the measurement system. First, the position of the mirrored object reflected three times by a plane mirror is obtained in the coordination system of camera. Second, the axis vector $m_{j j'}$ is calculated by Eq. (14). Third, each normal vector of mirrors located in three positions is estimated by Eq. (15). Fourth, the rotation matrix of the LCD screen is calculated by using Eqs. (16) and (17). The above four steps are repeated until the LCD screen is vertical to the optical axis of camera. Finally, the reference plane mirror is adjusted and the first three steps are repeated until the reference plane mirror is vertical to the optical axis of camera.

Step 3: Obtain reference fringe patterns reflected by the reference plane mirror. In the whole measurement process, this step is performed once to obtain the reference absolute phase which is compared with the deformed absolute phase on the measured specular object.

Step 4: Calculate the absolute phase from the captured deformed fringe patterns reflected by the measured specular object.

Step 5: Calculate the 3D measurement data. Based on Eq. (3), the depth data of the measured object can be calculated using the phase difference and the parameter $Δ d$ that is determined by the translating stage.

The steps 3-5 are detailed in the following chart, as shown in Fig. 13.

Fig. 13 Process of calculating the 3D measurement data.

Download Full Size | PDF

4.3. Measurement results

After calibrating the geometric parameters, three reflected objects were measured by the developed 3D system. They are three separate plat mirrors, a concave mirror from Micro-nano Optical Corporation (Beijing, China) which has a radius of 400 mm, and an artificial specular step.

Twelve fringe patterns having the optimum fringe numbers of 64, 63, and 56 were generated by software and displayed sequentially on the LCD screens. The reflected fringe patterns by the specular surface are deformed and captured by the camera. Figure 14 shows one of the captured deformed fringe patterns on the specular objects. Because the optimum three-fringe number selection method was used to unwrap the wrapped phase [24], the absolute phase of each pixel is independently determined, as shown in Fig. 15. Figure 16 illustrates the obtained depth information. The results show that the proposed method can directly measure discontinuous specular object surfaces.

Fig. 14 Captured deformed fringe patterns on the specular objects. (a) Three flat mirrors, (b) concave mirror, and (c) artificial specular step.

Download Full Size | PDF

Fig. 15 Absolute phase map of the three measured objects. (a) three flat mirrors, (b) concave mirror, and (c) artificial specular step.

Download Full Size | PDF

Fig. 16 Depth information of the three measured objects. (a) Three plat mirrors, (b) concave mirror, and (c) artificial specular step.

Download Full Size | PDF

4.4. Performance analysis

In order to quantitatively evaluate the accuracy of the proposed method, the radius of the measured concave mirror and the distance between neighboring steps of the artificial step were estimated. The radius of the concave mirror was 400.0318 mm calculated from the measured 3D shape data, so that the error was 31.8µm. In order to further verify the accuracy, the error of radius between the actual value and the estimated value was calculated as 38.7 um. The distance between neighboring steps of the artificial step was measured by a coordinate measuring machine (CMM), as the first column shown in Table 1. All the points of one measured step surface were fitted onto a plane to calculate the distance between neighboring steps. The measured distance between neighboring steps was calculated using the average of the distance from all the points obtained on the other step surface to the fitted plane, as the second column shown in Table 1. The absolute error (absolute difference between the average measured distance and the actual distance) and the mean squared error are listed in the third and fourth columns in Table 1. The absolute error in the distance between neighboring steps and the mean squared error were below 25.3 um and 28.5 um, respectively. The experimental results demonstrate that the proposed method can directly reconstruct the 3D shape of specular objects with high precision and reliability.

Table 1. Evaluation results for the artificial specular step (unit: mm)

View Table

5. Conclusion

This paper presents a novel full-field 3D shape measurement method based on a bi-telecentric system for measuring discontinuous specular surfaces by building the direct relationship between absolute phase and depth data. An LCD screen is moved to two different positions by an accurate translating stage during the procedure of measurement. Fringe pattern sets are generated by software and displayed on the LCD screen. The reflected fringe patterns are deformed with respect to the shape and slope of the specular surfaces and captured by a CCD camera. The bi-telecentric lens gives a very low distortion and offers a large DOF. Therefore, the system can measure the specular surface with large depth change. In addition, a new posture calibration method of measurement system has been proposed. An auxiliary plane mirror without markers is introduced and implemented in the calibration procedure. Based on the law of reflection, the relationship between object and the corresponding mirrored object can be directly established. After the parameter of the system has been calibrated, depth information can be directly obtained from the obtained absolute phase. Because depth directly relates to absolute phase without gradient integration, the proposed method can measure discontinuous specular objects. The experimental results show that the system effectively obtains full-field 3D shape information of discontinuous specular objects.

The proposed method has the following advantages. 1). Simple: the system calibration can be completed by using a plane mirror without markers. 2). Accurate and flexible: the measurement system can be calibrated accurately and flexibly due to a very low distortion and a large DOF of the bi-telecentric lens. 3). Discontinuous: discontinuous specular surfaces can be measured because of usage of the optimum three-fringe number selection method and without gradient integration.

Funding

National Key R&D Program of China (2017YFF0106404); National Natural Science Foundation of China (51675160); Key Basic Research Project of Applied Basic Research Programs supported by Hebei Province (15961701D); Research Project for High-level Talents in Hebei University (GCC2014049); Talents Project Training Funds in Hebei Province (A201500503); European Horizon 2020 through the Marie Sklodowska-Curie Fellowship Scheme (707466-3DRM); UK’s Engineering and Physical Sciences Research Council (EPSRC) funding of Future Advanced Metrology Hub (EP/P006930/1).

References and links

1. Z. Cai, X. Liu, X. Peng, Y. Yin, A. Li, J. Wu, and B. Z. Gao, “Structured light field 3D imaging,” Opt. Express 24(18), 20324–20334 (2016). [CrossRef] [PubMed]

2. C. Zuo, Q. Chen, G. Gu, S. Feng, and F. Feng, “High-speed three-dimensional profilometry for multiple objects with complex shapes,” Opt. Express 20(17), 19493–19510 (2012). [CrossRef] [PubMed]

3. J. Xu, S. Liu, A. Wan, B. Gao, Q. Yi, D. Zhao, R. Luo, and K. Chen, “An absolute phase technique for 3D profile measurement using four-step structured light pattern,” Opt. Lasers Eng. 50(9), 1274–1280 (2012). [CrossRef]

4. C. Zuo, L. Huang, M. Zhang, Q. Chen, and A. Asundi, “Temporal phase unwrapping algorithms for fringe projection profilometry: a comparative review,” Opt. Lasers Eng. 85, 84–103 (2016). [CrossRef]

5. M. Petz and R. Tutsch, “Reflection grating photogrammetry: a technique for absolute shape measurement of specular free-form surfaces,” Proc. SPIE 5869, 58691D (2005). [CrossRef]

6. T. Zhou, K. Chen, H. Wei, and Y. Li, “Improved system calibration for specular surface measurement by using reflections from a plane mirror,” Appl. Opt. 55(25), 7018–7028 (2016). [CrossRef] [PubMed]

7. C. Guo, X. Lin, A. Hu, and J. Zou, “Improved phase-measuring deflectometry for aspheric surfaces test,” Appl. Opt. 55(8), 2059–2064 (2016). [CrossRef] [PubMed]

8. J. Xu, N. Xi, C. Zhang, Q. Shi, and J. Gregory, “A Robot-assisted Back-imaging Measurement System for Transparent Glass,” IEEE-ASME T. MECH. 17(4), 779–788 (2012). [CrossRef]

9. Y. Tang, X. Su, F. Wu, and Y. Liu, “A novel phase measuring deflectometry for aspheric mirror test,” Opt. Express 17(22), 19778–19784 (2009). [CrossRef] [PubMed]

10. L. Huang, C. S. Ng, and A. K. Asundi, “Dynamic three-dimensional sensing for specular surface with monoscopic fringe reflectometry,” Opt. Express 19(13), 12809–12814 (2011). [CrossRef] [PubMed]

11. L. Huang, M. Idir, C. Zuo, K. Kaznatcheev, L. Zhou, and A. Asundi, “Shape reconstruction from gradient data in an arbitrarily-shaped aperture by iterative discrete cosine transforms in South well configuration,” Opt. Lasers Eng. 67, 176–181 (2015). [CrossRef]

12. L. Huang, M. Idir, C. Zuo, K. Kaznatcheev, L. Zhou, and A. Asundi, “Comparison of two-dimensional integration methods for shape reconstruction from gradient data,” Opt. Lasers Eng. 64, 1–11 (2015). [CrossRef]

13. A. Agrawal, R. Raskar, and R. Chellappa, “What is the range of surface reconstructions from a gradient field,” in Computer Vision–ECCV 2006 (Springer, 2006), pp. 578–591.

14. H. Guo, P. Feng, and T. Tao, “Specular surface measurement by using least squares light tracking technique,” Opt. Lasers Eng. 48(2), 166–171 (2010). [CrossRef]

15. L. Song, H. Yue, H. Kim, Y. Wu, Y. Liu, and Y. Liu, “A study on carrier phase distortion in phase measuring deflectometry with non-telecentric imaging,” Opt. Express 20(22), 24505–24515 (2012). [CrossRef] [PubMed]

16. M. Knauer, J. Kaminski, and G. Hausler, “Phase measuring deflectometry: a new approach to measure specular free-form surfaces,” Proc. SPIE 5457, 366–376 (2004). [CrossRef]

17. Y. L. Xiao, X. Su, W. Chen, and Y. Liu, “Three-dimensional shape measurement of aspheric mirrors with fringe reflection photogrammetry,” Appl. Opt. 51(4), 457–464 (2012). [CrossRef] [PubMed]

18. Y. Xiao, X. Su, and Z. You, “Pose transfer geometrical calibration for fringe-reflection optical three-dimensional measurement,” Opt. Commun. 305, 143–146 (2013). [CrossRef]

19. Y. Tang, X. Su, Y. Liu, and H. Jing, “3D shape measurement of the aspheric mirror by advanced phase measuring deflectometry,” Opt. Express 16(19), 15090–15096 (2008). [CrossRef] [PubMed]

20. Y. Liu, S. Huang, Z. Zhang, N. Gao, F. Gao, and X. Jiang, “Full-field 3D shape measurement of discontinuous specular objects by direct phase measuring deflectometry,” Sci. Rep. 7(1), 10293 (2017). [CrossRef] [PubMed]

21. H. Ren, F. Gao, and X. Jiang, “Iterative optimization calibration method for stereo deflectometry,” Opt. Express 23(17), 22060–22068 (2015). [CrossRef] [PubMed]

22. Y. L. Xiao, X. Su, and W. Chen, “Flexible geometrical calibration for fringe-reflection 3D measurement,” Opt. Lett. 37(4), 620–622 (2012). [CrossRef] [PubMed]

23. K. Takahashi, S. Nobuhara, and T. Matsuyama, “Mirror-based camera pose estimation using an orthogonality constraint,” IPSJ Trans. Comput. Vision Appl. 8(0), 11–19 (2016). [CrossRef]

24. Z. Zhang, C. E. Towers, and D. P. Towers, “Time efficient color fringe projection system for 3D shape and color using optimum 3-frequency Selection,” Opt. Express 14(14), 6444–6455 (2006). [CrossRef] [PubMed]

25. C. E. Towers, D. P. Towers, and J. D. Jones, “Optimum frequency selection in multifrequency interferometry,” Opt. Lett. 28(11), 887–889 (2003). [CrossRef] [PubMed]

26. J. S. Kim and T. Kanade, “Multiaperture telecentric lens for 3D reconstruction,” Opt. Lett. 36(7), 1050–1052 (2011). [CrossRef] [PubMed]

27. F. Zhu, W. Liu, H. Shi, and X. He, “Accurate 3D measurement system and calibration for speckle projection method,” Opt. Lasers Eng. 48(11), 1132–1139 (2010). [CrossRef]

28. D. Li, C. Liu, and J. Tian, “Telecentric 3D profilometry based on phase-shifting fringe projection,” Opt. Express 22(26), 31826–31835 (2014). [CrossRef] [PubMed]

29. D. Li and J. Tian, “An accurate calibration method for a camera sensor with telecentric lenses,” Opt. Lasers Eng. 51(5), 538–541 (2013). [CrossRef]

30. Z. Zhang, “A flexible new technique for camera calibration,” IEEE T. Pattern Anal. 22(11), 1330–1334 (2000). [CrossRef]

31. M. Moakher, “Means and averaging in the group of rotations,” SIAM J. Matrix Anal. Appl. 24(1), 1–16 (2002). [CrossRef]

32. M. Sekimoto, S. Arimoto, S. Kawamura, and J. Bae, “Skilled-motion plannings of multi-body systems based upon Riemannian distance,” in Proceedings of IEEE International Conference on Robotics and Automation (IEEE, 2008), pp. 1233–1238.

33. I. Kruzhilov, “Small-angle rotation method for star tracker orientation,” J. Appl. Remote Sens. 7(1), 073479 (2013). [CrossRef]

Actual distance	Measured distance	Absolute error	Mean square error
3.9868	3.9615	0.0253	0.0285
7.0248	7.0481	0.0233	0.0272
5.0062	4.9850	0.0212	0.0239
6.0986	6.1212	0.0226	0.0260

3D shape measurement of discontinuous specular objects based on advanced PMD with bi-telecentric lens

Abstract

1. Introduction

2. Principle

2.1. Measurement method

2.2. Camera model with bi-telecentric lens

2.3. Geometric calibration

2.3.1. Bi-telecentric imaging model by a plane mirror

2.3.2. Measurement model

2.3.3. Solution of rotation matrix R

3. Simulation

3.1. Geometric calibration

3.2. Measurement results

3.3. Influence of parallelism

4. Experiments and results

4.1. Hardware system

4.2. Process of measurement

4.3. Measurement results

4.4. Performance analysis

5. Conclusion

Funding

References and links

Cited By

Figures (16)

Tables (1)

Equations (22)

Optics Express