1. Introduction

With their high flexibility and reasonably high accuracy, calibration methods based on the pin-hole model have contributed immensely to advances in machine vision applications such as three-dimensional measurements [1,2], shape reconstruction [3,4], quality inspection [5] and lane detection for autonomous cars [6,7]. The requirement to further enhance accuracy is driven by industrial needs, such as those from the 3C (computer, communications, consumer electronics) and advanced manufacture industries. With escalating labor costs and an acute shortage of skilled workers, these industries are facing immense pressure to automate their manufacturing processes. Accurate 3D measurement is a basic building block of an automated inspection and production system in advanced manufacture.

The intuitive idea to improve the accuracy of an optical system is by increasing its magnification to fully utilize the resolution of the imaging sensor. However, high magnification results in limited DOF. Many technologies have been developed for highly accurate measurement with extended DOF, such as scanning electron microscope [8], focus stacking [9], computer tomography [10], shape-from-silhouette [11], telecentric stereo vision systems [12–15] and common stereo vision systems [16–19]. The scanning electron microscope and focus stacking methods can achieve very high accuracy while they take long time to measure small objects with relatively large DOF, which cannot fulfill production requirements in advanced manufacture. Above other image-based methods conduct a fast measurement, while the measurement accuracy is restricted. Among them, the highest measurement accuracy is around 5 $\mu m$ with two-megapixel camera at a 0.3× magnification [16]. Since there is a trade-off between the accuracy of 3D measurement and its magnification, is it possible to keep increasing the magnification of the vision system to achieve an even higher 3D measurement accuracy? In fact, the magnification can be increased physically, but the 3D measurement accuracy is limited by camera calibration accuracy at high magnification. Small DOF induced by high magnification shown in the black dotted line in Fig. 1(a) causes large errors in calibration as reported by the pin-hole calibration methods [20–27]. The reason of existing large calibration errors with small DOF is illustrated in Fig. 1(b). Once the image of the absolute conic is determined, camera parameters can be estimated. However, when the DOF is small, the obtained feature points (red crosses) on the absolute conic concentrate within the red region of Fig. 1(b). This makes estimation of the absolute conic inaccurate or unsolvable. Therefore, to measure accurately, it is necessary to work out how to precisely calibrate the camera parameters at high magnification when the DOF is small (but much larger than zero).

 

Fig. 1. (a) Properties against the working distance (WD) of a camera system. Red solid curve: magnification [28]; violet dashed curve: error of one pixel (reciprocal of magnification [29]); black dotted curve: depth of field [30];(b) reason of low calibration accuracy with small depth of field.

Download Full Size | PDF

To overcome this difficulty, the thin-lens model [31] is utilized instead of the pin-hole model. The thin-lens model can be regarded as a set of numerous pin-hole models with different camera settings (different focal lengths or principal distances) as shown in Fig. 2. Since a single pin-hole model at high magnification cannot provide enough information to precisely determine camera parameters, the pin-hole models at different magnifications are combined together, which then forms the thin-lens model. Once the relationship among camera parameters at different camera settings can be established, calibration information at low magnification can be utilized for the calibration at high magnification. Therefore, extra information at low magnification may make the calibration at high magnification stable and with low calibration errors. Meanwhile, the focal length or principal distance of a vision system is varied in the thin-lens model to ensure captured images in focus, which extends the DOF of the vision system. Therefore, both high accuracy and extended DOF can be achieved. To mechanically realize the interaction between different pin-hole models, a commercially available camera system that can change its principal distance by moving the imaging sensor is adopted [32] because it holds reliable and accurate mechanical properties compared to other thin-lens camera systems such as zoom lenses, liquid lenses, etc. [33,34].

 

Fig. 2. Schematic diagram of the thin-lens model. The focal length or principal distance are variables in the thin-lens model, while parameters in the pin-hole model.

Download Full Size | PDF

Since this thin-lens camera system is similar to other zoom camera systems, should it be calibrated using existing calibration methods for zoom cameras? Existing zoom camera calibration methods [33–41] mostly share the same strategy: individual monofocal calibrations at selected camera settings and interpolation of camera parameters followed by local or global optimizations. The individual monofocal calibration (intrinsic and extrinsic parameters) is mainly performed using the methods based on the pin-hole model [20–27]. These methods are reported to have large errors in calibrated camera parameters at high magnification due to small DOF. Furthermore, the calibration workload is massive due to the interpolation procedure. Normally at least 100 images are required for the calibration which is highly impractical. Self-calibrations require fewer images (around 30 images) but the calibration accuracy is lower [23,33,41]. In addition, calibration results from the interpolation procedure are not promising due to the lack of physical constraints. This makes calibration susceptible to errors due to noise in data. The situation becomes even worse with higher magnification. To solve these problems and realize accurate measurement with extended DOF, an efficient calibration method based on the thin-lens model for the camera with focus-related intrinsic parameters is proposed, including the camera model of the proposed thin-lens camera, estimation of initial camera parameters and optimization strategies. The paper also provides a new calibration strategy for the thin-lens camera systems which is building the relationship between camera parameters at different camera settings first (this relationship could be the polynomial used in the conventional methods [33–41] or the physical constraint) followed by solving all coefficients of camera parameters together. This strategy can improve both the accuracy and practicality of thin-lens camera calibration compared to the conventional strategy (monofocal calibration, interpolation and optimization). With the proposed calibration method based on the thin-lens model, accurate measurement with extended DOF for small objects can be realized.

2. Camera model

The camera model describes the mathematical relationship between objects in the 3D world coordinates and its projective points in the frame buffer coordinates. The pin-hole model only functions well within small DOF at high magnification. This also highly reduces camera calibration accuracy at high magnification. Instead, the thin-lens model varies its focal length or principal distance to extend its DOF, which makes accurate calibration and large DOF possible. Both the pin-hole and thin-lens model should fulfill the Gaussian optical formula in Eq. (1) [42] to ensure objects in focus. However, the mathematical difference is that the focal length or principal distance in the Gaussian optical formula are parameters for the pin-hole model, but variables for the thin-lens model.

In this paper, the thin-lens camera that varies its principal distance by moving the imaging sensor is adopted. The thin-lens camera model in this paper is elaborated as the following four steps that transfers the points of objects in the world coordinates to the points in the frame buffer coordinates.

i) Points in the world coordinates ${\textbf {P}_\textbf {W}}\textrm { = }{\left [ {{X_W},{Y_W},{Z_W}} \right ]^T}$ to points in the camera coordinates ${\textbf {P}_\textbf {C}}\textrm { = }{\left [ {{X_C},{Y_C},{Z_C}} \right ]^T}$

The world coordinate is defined by the user, which is normally built based on the calibration target during calibration process. The origin of the camera coordinate is at the optical center of the lens. Its $Z_C$ axis coincides with the optical axis and its $X_C$ and $Y_C$ axes are parallel to the cross section of the lens.

Points in the world coordinates $\mathbf {P_W}$ and points in the camera coordinates $\mathbf {P_C}$ are related by a rotation matrix $\mathbf {R}$ and a translation matrix $\mathbf {T}$ based on Eq. (2).

Since the camera coordinate is established on the lens and the lens does not move when the imaging sensor is moved in the thin-lens camera, the rotation and translation matrices $\mathbf {R},\mathbf {T}$ remain constant during the movement of the imaging sensor in this paper as Eq. (3).

ii) Points in the camera coordinates ${\textbf {P}_\textbf {C}}\textrm { = }{\left [ {{X_C},{Y_C},{Z_C}} \right ]^T}$ to points in the ideal image coordinates ${\textbf {P}_\textbf {u}}\textrm { = }{\left [ {{X_u},{Y_u}} \right ]^T}$

The points in the ideal image coordinates ${\textbf {P}_\textbf {u}}$ are transferred from the points in the camera coordinates ${\textbf {P}_\textbf {C}}$ through the projective projection of a lens. The ideal image coordinate is with the unit of length. In this step, the projective process is only related to the principal distance of the lens. The relationship between ${\textbf {P}_\textbf {u}}$ and ${\textbf {P}_\textbf {C}}$ is expressed in Eq. (4) based on triangular geometry.

Due to the movable imaging sensor, the intrinsic matrix $\mathbf {A_j}$ that contains the information of the principal distances and image center is no longer constant but a function of the imaging sensor’s displacement $d_j$. The imaging sensor is moved by a step motor and its displacement $d_j$ is read from the encoder. Since the captured image is the sharpest when the thin-lens model is satisfied, the displacement of the imaging sensor $d_j$ is determined by an image sharpness assessment function like the cumulative probability of blur detection (CPBD) [43]. The imaging sensor is assumed to move straightly constrained by mechanical and physical properties of linear motors, which can be verified by the experimental results with real data in the section 4.2. However, the imaging sensor cannot moves totally along the optical axis of the lens. Therefore, the displacement of the imaging sensor $d_j$ induces a change in both the principal distance and the image center simultaneously. This focus-related intrinsic matrix $\mathbf {A_j}$ can be expressed as Eq. (5).

iii) Points in the ideal image coordinates ${\textbf {P}_\textbf {u}}\textrm { = }{\left [ {{X_u},{Y_u}} \right ]^T}$ to Points in the distorted image coordinates ${\textbf {P}_\textbf {d}}\textrm { = }{\left [ {{X_d},{Y_d}} \right ]^T}$

Distortion is an important factor affecting the calibration accuracy. It has been explained in [22,44] that the distortion of a lens can be divided into three categories: radial distortion, decentering distortion and thin prism distortion. Thin prism distortion is induced by the tilt of the lens which can be compensated by the difference in principal distances along the $u$ and $v$ axes. Therefore, radial distortion and decentering distortion are considered in this paper.

The points in the distorted image coordinates $(X_d,Y_d)$ are expressed by the distortion model [22,44] in Eq. (6).

The above distortion model has been well adopted and verified in the pin-hole model case, where the magnification does not vary much. However, Seidel’s aberration theory [45] suggests that distortion is exacerbated at higher magnification. To measure accurately at high magnification, the coefficients ${k_1},{k_2},{p_1},{p_2}$ can no longer be constants. In this paper, a quadratic polynomial is utilized to describe the relationship between the coefficients and the principal distance based on previous studies [34,38–40] in Eq. (8).

iv) Points in the distorted image coordinates ${\textbf {P}_\textbf {d}}\textrm { = }{\left [ {{X_d},{Y_d}} \right ]^T}$ to Points in the frame buffer coordinates ${\textbf {P}_\textbf {f}}\textrm { = }{\left [ {{u},{v}} \right ]^T}$

The imaging sensor discretizes the image through lens, transferring the point in the distorted image coordinates $\mathbf {P_d}$ to the points in the frame buffer coordinates $\mathbf {P_f}$.

This relationship is presented in Eq. (9).

3. Camera calibration

In the camera model introduced in the section 2, there are in total twenty intrinsic parameters ${\alpha ,k_{\alpha },\beta ,k_{\beta },u_0,k_{u_0},v_0,}$ $k_{v_0},{a_i}\textrm { }\left ( {i = 1,2,3\cdots 12} \right )$ and twelve extrinsic parameters $r_i(i=1,2,3\cdots 9)$ $,t_x,t_y,t_z$ to be calibrated for the proposed thin-lens camera. The number of extrinsic parameters is reduced to six when the Rodrigues form of the rotation matrix is applied to ensure its orthogonality. The initial estimation of camera parameters and bundle adjustment optimization for the thin-lens camera are presented as follows.

3.1 Estimation of camera parameters

i) Estimating ${\alpha ,\beta ,u_0,v_0}$ in the initial state

The initial state is set by users as the reference position of the imaging sensor, which is normally at low magnification. Therefore, large DOF is available in the initial state of the thin-lens camera. Meanwhile, ${d_j}$ is set to zero so that the intrinsic matrix $\mathbf {A_j}$ in the initial state becomes the commonly-used pin-hole intrinsic matrix. The four parameters ${\alpha ,\beta ,u_0,v_0}$ in the intrinsic matrix $\mathbf {A_j}$ of the initial state can be calibrated using many different methods based on the pin-hole model [20–27].

ii) Estimating ${r_i(i=1,2,3\cdots 9),t_x,t_y,k_{u_0},k_{v_0}}$ using the iterative radial alignment constraint (IRAC)

In projective geometry, the direction of the vector from the origin $O_u$ to the point $(X_u,Y_u)$ in the ideal image coordinates is radially aligned with the direction of the vector from the optical center of the lens $O_C$ to the object points ${(X_C,Y_C,Z_C)}$ in the camera coordinates. This is called the radial alignment constraint (RAC) expressed in Eq. (10).

For each $i^{th}$ calibration point at the $j^{th}$ position of the imaging sensor with $(X_{u,i,j},Y_{u,i,j})$ in the ideal image coordinate, Eq. (11) is derived from Eq. (10).

With more than seven of Eq. (11), an over-determined system of linear equations can be established. ${r_i(i=1,2,3\cdots 9),t_x,t_y}$ can be calculated as the least square solution to this over-determined system with additional conditions from orthogonality of $\mathbf {R}$. Details can refer to Tsai’s method [21].

Now, only ${k_{u_0},k_{v_0}}$ remain unknown. Next, how to iteratively calculate ${k_{u_0},k_{v_0}}$ will be presented. The linear representation of the RAC in Eq. (10) is presented in Eq. (12).

If the image center $(C_{X,j},C_{Y,j})$ is not estimated correctly, Eq. (12) will not be equal to zero. Let $(\Delta {C_{X,j}},\Delta {C_{Y,j}})$ be the estimated error of the image center. Since $({u_0},{v_0})$ is already obtained in the first step, the estimated error of the image center is only related to the estimated error of ${k_{u_0},k_{v_0}}$. The biased image center $({C_{X,j}}',{C_{Y,j}}')$ is calculated as Eq. (13).

Equation (14) follows from Eq. (12) with deviated image center.

In the Lenz’s work [46], the deviation of the image center $(\Delta C_X,\Delta C_Y$) is constant for all the calibration points. However, the deviation of the image center is different for the calibration points at different positions of the imaging sensor in this paper. They are expressed as follows.

Since the expression of the biased rotation and translation parameters is not used in the estimation of the camera parameters, the average approximation in Eq. (16) is reasonable to show how the deviation of the image center biases the rotation and translation parameters.

Once $({k_{{u_0}}},{k_{{v_0}}})$ is correctly estimated, the value of Eq. (14) is zero. Therefore, the problem becomes minimizing the cost function for $m$ calibration points each at $n$ positions of the imaging sensor as follows.

However, a singularity problem occurs as shown in Fig. 3(a) when ${t_y} = 0,{t_x} \ne 0$, because $t_y$ is regarded as a denominator in Eq. (11). Although it can simply be set to nonzero by experimental setups [21], the deviation of $t_y$ during iteration caused by the incorrect image center will force ${t_y}'$ to be zero as shown in Eq. (15), which will make optimization unstable. To solve this singularity problem, the original denominator $t_y$ in Eq. (11) is replaced by $t_x$ when ${t_y} = 0,{t_x} \ne 0$,leading to Eq. (18). ${r_i(i=1,2,3\cdots 9),t_x,t_y}$ is calculated based on Eq. (18) when ${t_y} = 0,{t_x} \ne 0$.

 

Fig. 3. Singularity and solution of the IRAC. (a.1) Errors of the RAC towards $k_{u_0},k_{v_0}$ with singularity. (a.2) Top view of a.1. (b.1) Errors of the RAC towards $k_{u_0},k_{v_0}$ without singularity. (b.2) Top view of b.1.

Download Full Size | PDF

The graphical representation of the cost function in Eq. (17) without singularity after modification is shown in Fig. 3(b). The idea of minimizing the error of the RAC to obtain the true image center was first proposed by Lenz [46]. However, this paper shows how to apply the IRAC to obtain the variation of the image center in a thin-lens camera. Besides, the singularity problem of the RAC is proposed and solved. The strict convexity of the IRAC has also been proved in Appendix A.

iii) Estimating ${k_\alpha },{k_\beta },{t_z}$

With the prior estimated parameters, ${k_\alpha },{k_\beta },{t_z}$ can be calculated as the least square solution based on Eq. (19).

For accurate measurement, a lens with little distortion is commonly adopted [24–27]. Therefore, the initial distortion coefficients are set to zero in this case. However, if the distortion is too serious, the distortion coefficients in the initial state are regarded as the initial values. The final distortion coefficients are obtained by applying bundle adjustment.

3.2 Bundle adjustment

Since the positions of image points are affected by acquisition noise, the initial estimated intrinsic and extrinsic parameters are biased. Therefore, bundle adjustment (BA) is utilized to jointly optimize the camera parameters and camera poses by refining the errors between observed points in the frame buffer coordinates and projected points from the world coordinates modeled [47]. The refined intrinsic and extrinsic parameters are determined based the procedures described below.

Figure 1(a) shows that the nonlinearity of the change in magnification at different WDs is small when the WD is above three focal lengths. This will cause four pairs of camera parameters $\alpha ,{k_\alpha },\beta ,{k_\beta },{u_0},{k_{{u_0}}},{v_0},{k_{{v_0}}}$ to be strongly correlated with each other, which makes the Jacobian matrix for optimization semi-definite. The Pearson product-moment correlation coefficient (PPCC) is defined as $r(X,Y) = \frac {{{\textrm {cov}} (X,Y)}}{{{\sigma _X} \cdot {\sigma _Y}}}$ [48]. $r(\frac {{\partial {u_{i,j}}}}{{\partial \alpha }},\frac {{\partial {u_{i,j}}}}{{\partial {k_\alpha }}}) = 0.9379$ when the WD is larger than five focal lengths. It suggests a linear relationship between $\frac {{\partial {u_{i,j}}}}{{\partial \alpha }},\frac {{\partial {u_{i,j}}}}{{\partial {k_\alpha }}}$ at a large WD (low magnification). But this correlation coefficient becomes small as the magnification increases. To ensure the robustness of the optimization, an optimization strategy is proposed as shown in Fig. 4. In the BA process, the PPCC is evaluated first. One-step BA for all camera parameters is conducted with a small PPCC lower than 0.8 (this threshold is experimentally obtained), and two-step BA is iteratively conducted first on M then on N with a large PPCC. After the optimization process, the refined camera parameters can be obtained. The optimization method used in the paper is Levenberg-Marquardt algorithm [49].

 

Fig. 4. Optimization Strategy. ${p_{f,i,j}} = {(u _{i,j},{v_{i,j}})^T}$ is the observed feature points in the frame buffer coordinates; $\widehat {{p_{f,i,j}}}({P_{W,i,j}},\textbf {M},\textbf {N},{\textbf {d}_\textbf {j}})$ is the projected feature points in the frame buffer coordinates with the camera parameters $\textbf {M},\textbf {N}$ and the displacement of the imaging sensor ${\textbf {d}_\textbf {j}}$ obtained by step motor; $m$ is the number of feature points at each $j^{th}$ position of the imaging sensor; $n$ is the number of positions of the imaging sensor; $\textbf {M} = ({k_\alpha },{k_\beta },{k_{{u_0}}},{k_{{v_0}}},o{m_i}(i = 1,2,3),{t_x},{t_y},{t_z},{a_i}(i = 1,2,3\cdots 12))$; $o{m_i}(i = 1,2,3)$ is the Rodrigues form of the rotation matrix ${r_i}(i = 1,2,3\cdots 9)$ to ensure its orthogonality; and $\textbf {N} = (\alpha ,\beta ,{u_0},{v_0})$.

Download Full Size | PDF

3.3 Summary of the proposed calibration method

The recommended calibration procedures are given below:

i) Calibrate the camera intrinsic parameters in the initial state using pin-hole calibration methods [20–27] to obtain $\alpha ,\beta ,u_0,v_0$.

ii) Create 3D calibration space at different magnifications of the thin-lens camera to prepare calibration data. The imaging sensor moves automatically to stay all feature points in the 3D calibration space in focus based on an image sharpness assessment function like the CPBD [43].

iii) Compute ${r_i(i=1,2,3\cdots 9),t_x,t_y,k_{u_0},k_{v_0}}$ using the Iterative Radial Alignment Constraint (IRAC).

iv) Estimate ${k_\alpha },{k_\beta },{t_z}$ from the least square solution to Eq. (19).

v) Perform the BA optimization presented in the section 3.2 to obtain the refined intrinsic and extrinsic parameters.

4. Numerical simulations and experiments

4.1 Simulations

Simulations are conducted to investigate calibration performance affected by four different effects: the noise level in calibration data, the accuracy of the motor moving the imaging sensor, the number of images for calibration, and variation in magnification required for calibration. All the simulations are conducted under the following default setups unless otherwise stated.

Default setups. The simulated thin-lens camera that moves its imaging sensor has the following parameters: $\alpha =4480, \beta =4480, k_\alpha =4,k_\beta =4, u_0=940, v_0=515, k_{u_0}=1, k_{v_0}=1, k_1=0.001,k_2=p_1=p_2=0$. These parameters are selected close to the real situation. The resolution of the imaging sensor is 1920 x 1080 pixels and its pixel size is 2.9 $\mu m$.

The first step of the proposed calibration method is to estimate ${\alpha ,\beta ,u_0,v_0}$ in the initial state at low magnification. In this study, this step is conducted using Zhang’s method [20]. The uncertainty is about 0.1% for $\alpha , \beta$ and 0.5% for $u_0,v_0$ in reality. Since this step has been well evaluated, the calibration results of other camera parameters will be focused on to evaluate the proposed calibration method. In simulation, 0.1% and 0.5% uncertainties are applied to $\alpha , \beta$ and $u_0,v_0$ after the first step of the proposed calibration method.

The calibration target for the calibration at high magnification is a checkerboard with 6 x 4 1 mm squares (7 x 5 corner points) moved by a perpendicular displacement platform whose magnification gradually decreases from 1× to 0.25×, constructing 3D calibration space. The total number of corner points for calibration is 35 x 15 (15 is the number of images with different magnifications) = 525. The moving interval between the positions of the calibration target is 2 mm for the first five positions and 3 mm for the rest. The interval is enlarged for the rest to achieve enough variations in the magnification. The simulated rotation and translation matrix between this 3D calibration field and camera coordinates is $om=[0.2,0.2,0.2]^T$ (Rodrigues form), $T=[2,2,25]^T$(unit in millimeters). In reality, the standard deviation (SD) of noise added onto the image increases with higher magnification. Therefore, the Gaussian noise with 0 mean and $\sigma = magnitude\; \rm {x}\; magnification$ SD is added to the projected image points in the frame buffer coordinates. The default magnitude of SD in the Gaussian noise is 0.25 pixels. The accuracy of the step motor that moves the imaging sensor is 10 $\mu m$ in simulation.

The influence of the noise level in calibration data. Different magnitudes of the Gaussian noise ranging from 0 to 2.5 pixels are applied to the calibration data. For each noise level, 50 trials with 525 corner points each are conducted. Therefore, the average relative errors of calibrated parameters ${k_\alpha ,k_\beta ,k_{u_0},k_{v_0},\textbf {om},\textbf {T}}$ with regard to the noise level are shown in Fig. 5(a). As the results shown, the relative errors of calibrated parameters keep increasing linearly with the magnitude of SD in the Gaussian noise. The average relative errors of $k_{\alpha }, k_{\beta }$ increase more rapidly than those of other parameters. That is because calibrating $k_{\alpha }, k_{\beta }$ is the last step in estimating camera parameters. Therefore, the estimation errors of other parameters are added to the estimation of $k_{\alpha }, k_{\beta }$ which causes them to have larger errors than other parameters. For the magnitude of SD = 1, which fits most of the noise in reality, all the relative errors of calibrated parameters are lower than 2.5%.

 

Fig. 5. Performance of the proposed calibration method affected by different factors. (a) Magnitude of SD in the Gaussian noise (pixels). (b) Step accuracy of motors ($\mu m$). (c) Number of images. (d) The minimum magnification.

Download Full Size | PDF

The influence of the motor that moves the imaging sensor. Since a thin-lens camera with a movable imaging sensor is adopted in this paper, concern arises about how the accuracy of the motor affects the calibration robustness and measurement accuracy. To evaluate calibration robustness, the Gaussian noise with 0 mean and SD ranging from 0 $\mu m$ to 100 $\mu m$ is imposed on the motor data, since the uncertainty of a displacement platform for an accurate measurement task is normally within 100$\mu m$ in reality. For each level of SD, 50 trials with 525 corner points each are conducted and the average relative errors of calibrated parameters ${k_\alpha ,k_\beta ,k_{u_0},k_{v_0},\textbf {om},\textbf {T}}$ with regard to motor accuracy are shown in Fig. 5(b). It can be observed that the motor accuracy has little influence on the estimation of rotation parameters. Other parameters ${k_\alpha ,k_\beta ,k_{u_0},k_{v_0},\textbf {T}}$ increase linearly with the loss of motor accuracy. For motor displacement with 100 $\mu m$ uncertainty, the average relative errors of all calibrated parameters remain below 3.5%. Therefore, the parameter estimation is robust with regard to motor accuracy.

To evaluate the effect of motor accuracy on measurement accuracy, the partial derivatives of points in the frame buffer coordinates towards the motor displacement are derived in Eq. (20), regardless of the slight influence of distortion. With the real data of the calibration process in the section 4.2, the maximum error in the frame buffer coordinates induced by the motor displacement error occurs at the boundary of the image, which is about 0.3 pixels (1.32 $\mu m$ at a 0.66× magnification). This will be further verified by the measurement accuracy with real data in the section 4.2 below.

It can be observed that the average relative errors are around 1 % when the number of images is over two. There is a sudden fall in the value of these parameters when the number of images used for calibration is changed from two to three. This is caused by a lack of independent equations required for solution. It is recommended that four images should be adopted for calibration to ensure a robust calibration.

The influence of the variation in magnification. This experiment aims to investigate the calibration performance with regard to the variation in magnification of calibration data. In default setups, the magnification of calibration data is from 1× to 0.25×. The maximum magnification of calibration data is fixed at 1×, while the minimum magnification is modified from 0.1× to 0.8× to evaluate calibration performance. The magnification in this simulation is evenly varied. For each minimum magnification, 50 trials with 35 x 15 corner points each are conducted and the average relative errors of calibrated parameters ${k_\alpha ,k_\beta ,k_{u_0},k_{v_0},\textbf {om},\textbf {T}}$ with regard to the minimum magnification were shown in Fig. 5(d).

It can be seen that the average relative errors of rotation parameters $\textbf {om}$ increase slightly as the minimum magnification increases, but the errors of other parameters increase to a greater extent. However, the average relative errors of all parameters are lower than 5% when the minimum magnification is smaller than 0.75$\rm {x}$. The average relative errors when the minimum magnification is 0.1 are slightly large, likely because of large noise in the calibration data at low magnification.

4.2 Experiments

In this section, variations in calibrated intrinsic and extrinsic parameters with different numbers of images and the same number of images with different sets are investigated to evaluate the effectiveness and robustness of the proposed calibration method. Meanwhile, 3D measurement results with camera parameters obtained using the proposed method and prior methods [33,50] are compared to evaluate the accuracy and generality.

Experimental setups. The setup is depicted in Fig. 6. The calibration target (checkerboard) is perpendicularly fixed to a Parker displacement platform. A 3D calibration world space is constructed by moving this displacement platform. The imaging sensor of the thin-lens camera is moved by a step motor to ensure a clear image always. The exact specifications of the above devices are given in the caption of Fig. 6. To calibrate a thin-lens camera, a total of eight images are taken at different magnifications ranging approximately from 0.3538× to 0.6728× as shown in Fig. 7. The images’ size of the calibration board in Fig. 7 gradually decreases with the reduction of the magnification. If the whole field of view of the camera will be utilized for measurement, a larger calibration board with some known markers should be utilized to make the image of the calibration board take around two-thirds of the field of view. However, the reason that the images of the calibration board become smaller in Fig. 7 in this paper is that the stereo vision system with the thin-lens cameras holds a greatly varied coincident field of view, which is different from the one based on the pin-hole model. The coincident field of view of two cameras are labeled with blue frames in Fig. 7. It gradually increases to a certain degree and drops when the magnification gradually decreases. It could also be observed that the captured calibration target almost takes over half of the coincident field of view. And the calibration accuracy can be guaranteed within its effective field of view.

 

Fig. 6. Experimental setup. Checkerboard: a pattern of 5 x 4 squares each measuring 0.5 mm x 0.5 mm, produced in Nanosystem Fabrication Facility at HKUST by photolithography with $0.25\mu m$ accuracy; displacement platform: $1\mu m$ accuracy; thin-lens camera: AF301 from AFScope company [32], micro-step accuracy is 0.625 $\mu m$; and lens: 25mm, Edmund Optics #85357

Download Full Size | PDF

 

Fig. 7. A total of eight images used for calibration. Magnification from 0.3538× to 0.6728×. The blue frames indicate the coincident field of view of the calibrated camera in a stereo vision system with two thin-lens cameras.

Download Full Size | PDF

Since the image is the sharpest when the thin-lens model is satisfied, we utilize an image sharpness evaluation function - the cumulative probability of blur detection (CPBD) [43] - to find the proper principal distance at which the sharpest image is acquired. The principal distance in this paper is changed by the step motor whose displacement $d_j$ is read from the encoder. The micro-step accuracy of the step motor is 0.625 $\mu m$ [32]. One step in the step motor is 40 $\mu m$ and subdivided into 64 micro steps by the controller AN41908A. The corner points of the checkerboard are extracted by Matlab calibration toolbox [51].

Calibration performance with different numbers of images for calibration. Table 1 gives the calibrated parameters and root mean square error (RMSE) in pixels obtained respectively from the initial estimation (initial) and bundle adjustment (BA) after three iterations (3 iters) using different numbers of images. The calibrated results from different numbers of images are consistent with each other. The BA converges very fast, normally within two to three iterations. The RMSE between the corner points in the frame buffer coordinates and their projected image points with calibrated parameters is 0.7158 pixels for initial estimation and 0.4739 pixels after BA in average. At least five images are adopted in the calibration, although it is claimed in the simulation that four images are enough for robust parameters’ estimation. The combined noises in real data causes the calibration results with four images in the experiment to be biased.

Table 1. Calibrated intrinsic parameters, extrinsic parameters and RMSE in pixels obtained respectively from initial estimation (initial) and bundle adjustment after 3 iterations (3 iters) with different numbers of images.

View Table | View all tables in this article

Calibration performance with different sets of images. To further investigate the calibration performance, Table 2 gives the calibrated parameters, RMSE in pixels, their averages and SD after BA (three iterations) with different image sets containing five images each. The image set {12345} means that the first, second, third, fourth and fifth images in Fig. 7 are adopted for calibration. The small SD of all calibrated parameters indicates the effectiveness of the proposed calibration method. The RMSE is 0.4479 pixels on average. Considering the magnification of the vision system, the RMSE in the 3D world coordinates ranges from 1.93 $\mu m$ to 3.67 $\mu m$.

Table 2. Calibrated intrinsic parameters, extrinsic parameters and RMSE in pixels with different sets of images.

View Table | View all tables in this article

Measurement accuracy evaluation. Three 3D test fields to evaluate measurement accuracy are shown in Fig. 8. They are constructed by three checkerboards with 1.5mm, 1.0mm and 0.5mm squares (0.25$\mu m$ accuracy) which are moved by a perpendicular displacement platform (1$\mu m$ accuracy). The magnifications of these three 3D test fields range from 0.1218×, 0.2059×, and 0.4814× to 0.3915×, 0.5655×, and 0.6554×, respectively.

 

Fig. 8. 3D test fields for evaluation of measurement accuracy.

Download Full Size | PDF

The above three uncalibrated 3D test fields are reconstructed from the camera parameters which are respectively calibrated using the proposed method and prior methods [33,50]. For the proposed method, 15 images are used to estimate $\alpha ,\beta ,u_0,v_0$ by Zhang’s method [20] as the first step. Eight images shown in Fig. 7 are adopted to estimate the remaining parameters. A total of 23 images are utilized for calibration which can be further decreased if necessary. For Chen’s method [50], monofocal calibrations are conducted at 10 different principal distances with 15 images each where the magnification ranges from 0.030× to 0.696×. The monofocal calibration is conducted using Zhang’s method [20]. The monofocal camera parameters are interpolated by a cubic polynomial regarding the change of principal distances $d_j$. Quadratic or quartic polynomials are also tested, but a cubic polynomial performs better. In all, 15 x 10 =150 images are used for calibration with this method. Zheng [33] obtained focal length values from the EXIT file which is unavailable for a prime lens or most industrial zoom lenses. Therefore, the focal length and image center at each principal distance are still obtained using Zhang’s method which are in fact even more accurate than the EXIT file. The monofocal calibration and interpolation process are the same with ones in the above method, while BA is applied afterwards.

Since higher accuracy at high magnification is pursued in this paper, the same data are used for all the above BAs with the magnification ranging from 0.3538× to 0.6728×. Therefore, the 3D test fields with high magnification (above 0.3538×) can be utilized to analyze the measurement accuracy of different calibration methods. The 3D test fields with low magnification (below 0.3538×) can be utilized to analyze the generality of different calibration methods. The calibrated parameters should be closer to the ground truth with the greater generality, otherwise over-fitting problems may exist.

The comparison of measurement accuracy is shown in Fig. 9. Each point in this figure represents the average RMSE of about 50 corner points on one checkerboard at a certain principal distance.

 

Fig. 9. Comparison of the 3D measurement accuracy of the proposed methods and prior methods. (a) Monofocal calibration + Interpolation [50]. (b) Monofocal calibration + BA [33]. (c) The proposed method. (d) Monofocal calibration results.

Download Full Size | PDF

The proposed method has greater measurement accuracy than the other two methods at any magnification. It is interesting to see a linear model performing much better than a cubic polynomial. This is due to our earlier claim that the monofocal calibration results become inaccurate at high magnification. When the magnification is 0.696×, the calibration uncertainties of the principal distance and image center using Zhang’s method are around 7.5% and 12.6% respectively. Two examples of the calibrated principal distance and image center $\alpha , u_0$ regarding $d_j$ by monofocal camera calibrations are given in Fig. 9(d). Physically, the principal distance and image center should change linearly with $d_j$. The principal distance closely follows this trend at low magnification but fails to do so at high magnification. However, the calibrated image center always fluctuates which cannot be accurately estimated by the monofocal calibration. The reason might be the following fact. The majority function of moving the imaging sensor is to change the principal distance to ensure the captured image stays in focus. However, the changes of the image center are a side effect due to the manufacturing or assembly process. The small variation of the image center cannot be easily estimated. Meanwhile, estimation errors of the image center are five times worse than those of principal distance [20]. Therefore, the inaccurate monofocal calibration causes interpolation and bundle adjustment (BA) to be biased. The other reason is that the step motor moving the imaging sensor applies a linear physical constraint to the camera model. If a high-order polynomial is used to interpolate camera parameters, an over-fitting problem can easily occur due to the inaccurate estimation of camera parameters, especially for the image center. The method involving monofocal calibration and BA performs better at high magnification but far worse at low magnification than the one involving monofocal calibration and interpolation because the data for BA is at high magnification.

It can also be observed from Figs. 9(a)–(c) that the measurement accuracy increases with higher magnification. This conforms to the fact that the accuracy of the vision system can be increased by fully utilizing the resolution of the imaging sensor. The RMSE of points in the 3D world coordinates with the proposed method decreases from 22.02 $\mu m$ to 1.66 $\mu m$ as the magnification increases. Although the magnification of the data for BA ranges from 0.3538× to 0.6728×, camera parameters obtained using the proposed method yield the equivalent measurement performance for the scenario with magnification higher than 0.1218×. This result indirectly verifies the generality of the proposed method. Meanwhile, the RMSE of the method involving monofocal calibration and BA suddenly soars at low magnification because the BA process causes inaccurate parameters to over-fit the scenario at high magnification. This greatly reduces the generality.

In addition, existing calibration methods require more than 100 images for calibration as mentioned before [34,37,38,50]. By contrast, our proposed method needs at most 23 images which greatly reduces the calibration workload.

As our experimental results have demonstrated, the proposed camera calibration method based on the thin-lens model can effectively and accurately calibrate a camera with focus-related intrinsic parameters to fully utilize the resolution at high magnification.

The accuracy and generality of the proposed method exceed those of prior methods at any magnification. Meanwhile, the number of images required and the calibration workload are one-fifth of those associated with existing methods. This makes the proposed method much more practical and easy to use.

5. Conclusion

An efficient camera calibration method based on the thin-lens model is developed to calibrate a thin-lens camera that changes its principal distance by moving the imaging sensor, so that small objects can be accurately measured with extended DOF at high magnification. The camera model of the thin-lens camera is studied and established. The IRAC is utilized to estimate initial camera parameters. The singularity problem in the IRAC is presented and solved and its strict convexity is proved. Meanwhile, the distortion model for the scenario with varied magnifications and the bundle adjustment strategy are proposed to effectively obtain refined camera parameters. A new calibration strategy for the thin-lens camera system is also proposed in this paper, which is building the relationship between different camera parameters at different camera settings first followed by solving all the parameters together. It may improve both the practicality and accuracy of thin-lens camera calibration.

The simulation and experimental results confirm the superiority of the proposed method over existing calibration methods in terms of effectiveness, accuracy, generality and practicality. The RMSE of points in the 3D world coordinates with the proposed method decreases from 22.02 $\mu m$ to 1.66 $\mu m$ when the magnification of the vision system increases from 0.12× to 0.66×. The proposed method is at least 2.5 times more accurate than calibration methods based on the pin-hole model [33,50]. It is also worth noting that the optimization data only covers the magnification ranging from 0.3538× to 0.6728×. The small RMSE of reconstruction out of the optimization range indicates the strong generality of the proposed method which means that the calibrated parameters are close to the ground truth. Furthermore, the proposed calibration method requires 10 to 23 images only which greatly reduces the calibration workload, as opposed to over 100 images for calibration methods based on the pin-hole model [33–41].

In the near future, we will develop a stereo vision system based on the thin-lens model with this calibration method. The possibility of extending the proposed method to other zoom camera systems with variable fields of view will also be examined.