The calibration of computer vision systems that contain the camera and the projector usually utilizes markers of the well-designed patterns to calculate the system parameters. Undesirably, the noise and radial distortion exist universally, which decreases the calibration accuracy and consequently decreases the measurement accuracy of the related technology. In this paper, a method is proposed to remove the noise and radial distortion by registering the captured pattern with an ideal pattern. After the optimal modeled pattern is obtained by registration, the degree of freedom of the total calibration markers is reduced to one and both the noise and radial distortion are removed successfully. The accuracy improvement in a structured light scanning system is over 1024 order of magnitude in the sense of mean square errors. Most importantly, the proposed method can be readily adopted by the computer vision techniques that use projectors or cameras.
© 2015 Optical Society of America
Noise removal or reduction [1–10] is a hot topic in optics and it usually determines the application prospect of the related technology because the noise induced error during the calibration stage may affect the measurement accuracy greatly and limits their applications. This paper mainly deals with the noise of projector calibration and camera calibration in the structured light scanning system [11–19]. The structured light scanning methods [11–19] project some designed patterns onto the surface of the object to be measured. From the distorted pattern, the surface's profile could be calculated based on the system parameters calculated from a set of calibrated markers. Thus, the calibration accuracy is critical for the measurement accuracy of these methods. Traditionally, these methods use as many calibration markers as they can until the calibration accuracy improvement was close to zero growth. Similarly, the camera calibration  makes use of multiple views of the well-designed pattern to calculate intrinsic and extrinsic parameters and it uses as many views of the designed pattern as it can until the calibration accuracy could not be improved significantly any more. Unfortunately, none of them could remove the noise completely in both calibration stage and measurement stage even if they might reduce the noise to a low level by averaging more markers or views of the designed pattern. In addition, multiple views could not reduce or remove the radial distortions, which must be rectified for robust measurement.
In this paper, we propose a method to eliminate both the noise and radial distortion during calibration for a structured light system  which uses a Pico laser projector to generate the structured light pattern that suffers both radial distortion and noise. Consequently, the measurement accuracy is very poor. In , a SNF laser is used instead of the projector. The radial distortion is avoided and the contrast of the image is also increased because of the higher power of SNF laser compared to the Pico laser projector. Hence, the measurement accuracy is increased over 10 order of magnitude compared to that of . To increase the measurement accuracy of the structured light system with a projector , we model the markers in the perfectly designed pattern with an arbitrarily assigned center as rays. We then search for an optimal plane that intercepts the rays and produces an optimal pattern that is closest to the actually captured pattern. When the actually captured pattern is replaced by the optimal pattern, the noise and radial distortion are removed with the degree of freedom of all the calibration markers reduced to one.
2. The structured light scanning system
The structured light scanning system  is illustrated in Figs. 1(a) and 1(b). Figure 1 shows the principle of the structured light method and the established system where a Pico laser projector is used to produce the structured light pattern. The optical center of the projector is denoted as C and its symmetry point relative to the plane p1 is C'. C' and the horizontal plane form the virtual camera that is used to compute the equations of the planes p2 and p3 with camera c2 and camera c3 respectively. Plane p1 is defined as the reference plane, z = 0 and it originates at O. The laser ray is projected onto and reflected by plane p1 onto a beam splitter which splits the ray into two parts that intercept planes p2 and p3 respectively. During calibration, the poses of the three cameras are estimated. Then the equation of the diffusive plane p2 is computed by camera c2 and the virtual camera. In the same way, the equation of the diffusive plane p3 is computed by camera c3 and the virtual camera. With the equation of the diffusive plane, the Homography between the camera and the diffusive plane and the camera coordinates of the interception points, the 3D world coordinate of the interception point can be computed. When the points on p3 are computed, they are mapped to p4. Then, two points intercepting one ray are obtained and the ray can be determined uniquely with closed form solution. With the incident rays determined by camera c1, the 3D coordinates of the points on the specular surface are computed as the closed form solutions of the intersections of the incident rays and the reflected rays.
We make use of the pattern designed in this structured light system as an instance to describe the proposed method and illustrate its effectiveness. During calibration, a set of bright dots (markers) are used to calculate the system parameters. The two dimensional coordinates of these points in the camera view need to be computed, where the noises are introduced when they are computed as the mean of all the corner or bright pixels. Figure 2(a) shows the designed pattern and it is projected by a Pico Laser Projector onto a horizontal diffusive plane. The brightest point in the center denotes the center marker. A dragonfly camera captures the projected pattern as shown in Fig. 2(b). To demonstrate the noise, we select 44 points around the center marker and compute their x coordinates and y coordinates as the mean of the bright pixels after segmentation by the proposed method in . Then we calculate the difference of the adjacent dots for the x coordinate and y coordinate respectively. Figures 2(c) and 2(d) show the calculated x coordinate differences and y coordinate differences respectively which reflect the noise that changes randomly. The differences should change regularly without noise according to the designed pattern while noise adds the random variations. The purpose of the proposed method is to remove these random variations (noise) completely.
For the structured light system, we define the noise as the random variations caused during image capturing that is affected by different influencing light sources and automatic image processing that is affected by the unevenly distributed gray-scales of the markers. The most commonly encountered lens distortions are radial distortions  that are addressed in   [23,24] with different methods. In , the radial distortion of the projector is corrected independently by adjusting the coefficients of the projected pattern with inverse distortion. In , the radial distortion was modeled and incorporated into the camera calibration process. In , inverse distortion patterns different from  are used to correct the radial distortion. In , Hough transformation was used to correct the radial distortions. Unfortunately, none the above mentioned methods could remove the radial distortions completely. In addition, these correction methods are also easy to be affected by the noise, which was evaluated with Gaussian noise in . In this paper, a new method is proposed to remove the noise and radial distortion as a whole by registering the captured pattern with an ideal pattern.
3. The proposed method
In most cases [11–20], the designed patterns keep the distances between adjacent markers the same for convenience. In this section, we propose a 3D pattern modeling method to eliminate the noise based on the distances between the center marker and the other markers. The proposed method is more general because it does not require the distances between adjacent markers equal, which might benefit more computer vision applications in the future. The proposed method contains the following steps:
- Step 1: Model the rays with designed markers and an arbitrarily assigned projection center The unit of the markers can be chosen as convenient as pixel or as mm depending on the convenience of the application. The modeled rays are formulated as:
where is the th marker in the designed pattern.
The intercepted points (markers) are computed in a virtual coordinate system instead of the world coordinate system. A registration is thus needed between the original points and the intercepted points to convert the coordinates correctly. We register the two set of points based on the least square errors by finding the transformation matrix that makes the sum of square errors, minimum.
The proposed method can be summarized as follows. The rays are modeled with an arbitrarily assigned center outside of the plane that the markers lie in. It relies on the plane that intercepts the modeled rays to generate the modeled pattern. Then a transformation is used to match the modeled pattern with the original computed pattern. Without noise and radial distortion, the modeled pattern and the original computed pattern should match completely. Because the projection center was arbitrarily assigned for ray modeling, we need to make sure that the transformation matrix exists before we could calculate it by least squares estimation. Hence, the proposed method relies on the following two lemmas.
Lemma 1: Two different planes and intercept the modeled rays, with central projection and produce two different patterns and . denotes the number of markers in the pattern. and can be transformed to each other by a transformation matrix .
From the property of central projection and Homography definition, there is Homography between two sets of planar markers. The following two formulations hold.
Equation (11) can be rewritten in the format of and as:
Let and , then Eq. (16) can be rewritten as:
From the above equation, we obtain the value of and respectively.
By definition, the following formulation holds.
The transformation matrix exists as:
The lemma is proved.
Lemma 2: For a given pattern , two set of modeled rays, and are obtained with two different projection centers and. Suppose a plane intercepts the modeled rays with a pattern and a plane intercept the modeled rays with a pattern. and can be transformed to each other by a transformation matrix .
According to Lemma 1, there is a transformation matrix between and that is produced by plane intercepting the modeled rays.
There is also a transformation matrix between and that is produced by plane intercepting the modeled rays.
As can be seen, the transformation matrix exists:
The lemma is proved.
For the practical implementation, the searching range to find the optimal parameters is limited since it will be intractable to search all the possible values thoroughly. The searching ranges is chosen as and respectively during experiments. The complexity of the searching is 106 and it takes less than one minute in MATLAB.
Since the searching range is fixed, the center used to model the rays will affect the equations of the modeled rays, which might in turn affect the final pattern modeling accuracy significantly. Thus, an additional search around the arbitrarily assigned center is performed to find a center that could yield more accurate registration results by the following two error criteria.
Where denotes the th modeled point and denotes the th original point.
As can be seen, the proposed 3D pattern modeling method is operated in the three dimensions for better accuracy. Based on Eq. (10), the proposed method could be simplified into the 2D pattern modeling method as follows:
where is a constant and the transformation matrix is defined as:
In , the authors use Homography estimation to remove the perspective distortion instead of radial distortion and we did not find any literature that utilizes the Homography transformation to remove the radial distortion. Hence, we claim that both the proposed 3D pattern modeling method and the proposed 2D pattern modeling method are original. The 2D pattern modeling method is a simplified version of the 3D pattern modeling method.
4. Experimental results
Firstly, the exemplary pattern shown in Fig. 2 is modeled by the 3D pattern modeling method and Figs. 3(a) and 3(b) show the modeled coordinates (in red) versus the original original coordinates (in blue). To see the noise removal effect of the proposed method, the differences of the and coordinate for these 45 points after modeling are plotted in Figs. 3(c) and 3(d) respectively to compare with those plotted in Figs. 2(c) and 2(d). It is seen that the proposed method works well and the noise (random variation) is eliminated successfully. Please note that the original points refer to the points computed directly from the captured image and the modeled points refer to the points on the registered ideal pattern. The registered ideal pattern is computed from the designed ideal pattern by Eqs. (1)-(9) or Eqs. (34)-(39).
For the modeled results shown in Fig. 3, the computed mean squared errors (MSE), and are 2.8061 and 2.0916 respectively. We search a new projection center in a small range in three dimensions and find the center that yields the minimum MSE. The MSEs, and are reduced to 2.2204 and 1.6357 respectively and the results are shown in Fig. 4. Since the improvement of MSEs is not significant, the visual difference is not very obvious. In fact, the modeled coordinates with the new center match better than the modeled coordinates with the original center, which indicates that finding the optimal parameters is a challenging engineering problem that needs great effort.
Secondly, two sets of detected corners of a camera calibration pattern  are modeled by the 3D pattern modeling method to compare the accuracy of without center searching and with center searching. The results of modeling the first set are shown in Fig. 5 and Fig. 6. There are obvious mismatches between the modeled coordinates and the original coordinates in Fig. 5. The computed and are 1.8155 and 2.8142 respectively. After searching around the projection center within the range in three dimensions, the MSEs are reduced to = 1.074 and = 1.6262 respectively. The modeling results with the new center are shown in Fig. 6. It is seen that the modeled points and original ones match significantly better than those in Fig. 5. The modeling results for the other set of corner points are shown in Fig. 7 and Fig. 8. The computed MSEs for the modeling results without center searching in Fig. 7 are = 2.0002 and = 3.2223 respectively. With a new searched center, the MSEs are reduced to = 1.8155 and = 2.8142 respectively.
Thirdly, we use a different camera calibration pattern to compare the performance of the 3D pattern modeling method and 2D pattern modeling method and the results are shown in Fig. 9 and Fig. 10. In Fig. 9(b), there are obvious mismatches between the modeled coordinates and the original coordinates. In Fig. 9(c), the mismatches between the modeled points and the original points are also significant. On the contrary, both the coordinates and points modeled by the 3D pattern modeling method match very well with the original ones as shown in Fig. 10. Figure 11 shows more results of the proposed 3D pattern modeling method on different captured camera calibration patterns with different orientations and resolutions. As can be seen, the modeled points match the corners well.
Fourthly, we show the modeling results of the pattern of a SNF laser during measuring the 3D weld pool shape. The pattern contains 11 rows and 11 columns of laser points. The distances between adjacent points in each row or column are equal. Different from the camera and the projector, the SNF laser does not have the problem of radial distortion. Hence, the function of the proposed pattern modeling method in this specific application is to remove the noise only. Figure 12 shows the results of modeling the laser pattern. As can be seen, both the modeled coordinates and points match the original ones well.
Fifthly, we compare the accuracy and computation time of the proposed methods in Table 1. The 3D method1 represents the 3D method without center searching and the 3D method2 represents the 3D method with center searching. It is seen that the proposed 3D pattern modeling method is superior to the 2D pattern modeling method in accuracy while the 2D method is significantly more efficient than the 3D method. Hence, the 2D method might be used in cases that do not require extremely high accuracy.
Sixthly, the reconstuction accuracy of the structured light system is used to demonstrate the effectiveness of the proposed 3D pattern modeling method. In , the following equation is used to compute the reconstruction accuracy.
where denotes the error in coordinate, denotes the error in coordinate and denotes the error in coordinate. denotes the th reconstructed point and denotes the th original point. Without 3D pattern modeling, the measurement accuracy of the developed system is 50.8 μm2 in coordinate, 39.1 μm2 in coordinate and 6.8 μm2 in coordinate. Then, we compute the errors of reconstructing the flat mirror with 3D pattern modeling of the camera coordinates in c2 and c3 in addition to the world coordinates in p1, p2 and p3. The reconstruction measurement accuracy is 1.2 × 10−24 μm2 in coordinate, 4.46 × 10−24 μm2 in coordinate and 6.08 × 10−25 μm2 in coordinate. The accuracy improvement is over 1024 order of magnitude.
In this paper, a pattern modeling method is proposed to remove the noise and radial lens distortion during computer vision calibration that uses projector or camera by registering the captured pattern with an ideal pattern. When the pattern is modeled as a whole, the degree of the freedom of the all the calibration markers (points) is reduced to one. After the optimal modeled pattern is obtained by registration, the noise and radial distortion can be removed effectively.
The major contributions of this paper include:
- (2), compared to the state of literatures   [23,24] that deal with the radial lens distortion, the proposed pattern modeling method is capable of removing the radial distortion with noise as a whole while the state of art methods have to consider the effect of the noise during correcting the radial distortions.
- (3), the proposed method could conducts the pattern modeling in 3D for better accuracy and in 2D for better efficiency, which is very flexible to meet the requirements of different computer vision applications.
- (4), two lemmas are proposed and proved to validate the correctness of modeling the pattern in 3D with an arbitrarily assigned projection center.
References and links
1. A. Wong, A. Mishra, K. Bizheva, and D. A. Clausi, “General Bayesian estimation for speckle noise reduction in optical coherence tomography retinal imagery,” Opt. Express 18(8), 8338–8352 (2010). [CrossRef] [PubMed]
3. F. Pan, W. Xiao, S. Liu, F. Wang, L. Rong, and R. Li, “Coherent noise reduction in digital holographic phase contrast microscopy by slightly shifting object,” Opt. Express 19(5), 3862–3869 (2011). [CrossRef] [PubMed]
6. M. Szkulmowski, I. Gorczynska, D. Szlag, M. Sylwestrzak, A. Kowalczyk, and M. Wojtkowski, “Efficient reduction of speckle noise in optical coherence tomography,” Opt. Express 20(2), 1337–1359 (2012). [CrossRef] [PubMed]
7. Y. Wang, P. Meng, D. Wang, L. Rong, and S. Panezai, “Speckle noise suppression in digital holography by angular diversity with phase-only spatial light modulator,” Opt. Express 21(17), 19568–19578 (2013). [CrossRef] [PubMed]
8. S. M. Jung, S. M. Yang, K. H. Mun, and S. K. Han, “Optical beat interference noise reduction by using out-of-band RF clipping tone signal in remotely fed OFDMA-PON link,” Opt. Express 22(15), 18246–18253 (2014). [CrossRef] [PubMed]
10. P. Memmolo, V. Bianco, M. Paturzo, B. Javidi, P. A. Netti, and P. Ferraro, “Encoding multiple holograms for speckle-noise reduction in optical display,” Opt. Express 22(21), 25768–25775 (2014). [CrossRef] [PubMed]
12. Z. Z. Wang, X. Y. Huang, R. G. Yang, and Y. M. Zhang, “Measurement of mirror surfaces using specular reflection and analytical computation,” Mach Vision Appl. 24(2), 289–304 (2013). [CrossRef]
13. Z. Z. Wang, “Robust measurement of the diffuse surface by phase shift profilometry,” J. Opt. 16(10), 105407 (2014). [CrossRef]
15. C. Guan, L. G. Hassebrook, D. L. Lau, and V. G. Yalla, “Improved composite-pattern structured light profilometry by means of postprocessing,” Opt. Eng. 47(9), 097203 (2008).
16. C. Je, S. W. Lee, and R. H. Park, “Colour-stripe permutation pattern for rapid structured-light range imaging,” Opt. Commun. 285(9), 2320–2331 (2012). [CrossRef]
17. C. Je, K. H. Lee, and S. W. Lee, “Multi-projector color structured-light vision,” Signal Process. Image Commun. 28(9), 1046–1058 (2013). [CrossRef]
19. W. Jang, C. Je, Y. Seo, and S. W. Lee, “Stuctured-light stereo: Comparative analysis and integration of structured-light and active stereo for measuring dynamic shape,” Opt. Lasers Eng. 51(11), 1255–1264 (2013). [CrossRef]
20. Z. Y. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. on PAMI 22(11), 1330–1334 (2000). [CrossRef]
21. Z. Z. Wang, “Monitoring of GMAW weld pool from the reflected laser lines for real time control,” IEEE Trans, on Ind, Inform 10(4), 2073–2083 (2014).
22. D. C. Brown, “Decentering distortion of lenses,” Photogramm. Eng. 32(3), 444–462 (1966).
23. R. Cucchiara, C. Grana, A. Pratzi, and R. Vezzani, “A hough transform-based method for radial lens distortion correction,” ICIAP 1, 182–187 (2003).
24. J. P. Villiers, F. W. Leuschner, and R. Geldenhuys, “Centi-pixel accurate real-time inverse distortion correction,” Proc. SPIE 7266, 726611 (2008). [CrossRef]
25. A. K. Geetha and S. Murali, “Automatic rectification of perspective distortion from a single image using plane homography,” IJCSA 3(5), 47–58 (2013). [CrossRef]