Abstract
Previous stereo-vision based methods for measuring bent tube axis usually approximate the centerlines of image contours as the axis projection, inevitably resulting in reconstruction errors. This phenomenon is more significant as the tube diameter increases. In this paper, a perspective projection model for any cross section of bent tube was established. Based on this model, a way to locate the precise projected position on image planes of points laying on axis was proposed and 3D coordinates of axis points are reconstructed by binocular stereo vision. We measured three bent tubes with different diameters. Compared with classical approaches, this method effectively reduces the reconstruction errors with measurement accuracy practically independent of diameter.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Pipelines are the key parts of electro-mechanical products in aeronautics and astronautics field. It commonly consists of complex metal bent tubes produced by tube benders. Although every tube is made according to its CAD model, guaranteeing its manufacture precision is a great challenge due to spring-back and many disturbances during the fabricating progress. For successful and non-stress assembly, processed tubes are required to be measured and compared with their CAD models to evaluate whether their assembly demands are satisfied. Therefore fast and precise 3D bent tube measurement is quite important.
Contact measuring instruments such as coordinate measuring machine [1] (CMM) and mechanical gauges [2] are the main approaches for tube measurement in early stage. CMM has high precision but its operation is quite complex. Besides, the contact between the probe and tube surface may lead to partial deformation. Modeling method needs to fabricate molds for tubes with their CAD models in advance causing high cost and poor universality. Metronor light pen [3] is a new 3D reconstruction equipment combing stereo vision and tactile measurement, but moving the probe to scan the whole surface leads to a complicated operation.
In the past few years, stereo vision has been widely studied [4] and applied in the field of non-contact measurement [5] owing to its multiple advantages such as high accuracy, high efficiency, non-contact and automation. Therefore, many methods of measuring bent tube based on stereo vision have emerged. These automatic propositions can mainly be divided into two parts, bent tube surface measurement and space axis reconstruction. GOM ATOS Core 3D scanner [6] is a typical representative of measuring tube surface based on binocular stereo vision with the accuracy of 0.02 mm. However, it has the problem of poor efficiency because of 3D data registration. Bosemann [7] mentioned that AICON [8] provided a monocular vision method using photogrammetry. Though this method has high accuracy, it requires a scale benchmarking to provide constraint condition and the automation of the measurement is weak.
Compared with surface measurement, tube axis reconstruction has the superiority of full automation. So it has become an important direction that the scholars and companies paid great attention to. AICON developed TubeInspect [9] pipeline measuring system with the accuracy of 0.1mm. T. Zhang and P. Jin [10–13] used the center lines of image contours together with stereo vision to reconstruct 3D axis within 0.2s and the accuracy is nearly equal to AICON. Liu [14] proposed an accurate way to measure the position of pipe end faces that is a supplement for T. Zhang. Nevertheless, the above methods regarded the center lines of tube contours on image plane as the projected position of axis. This assumption is just a kind of approximate calculation with a great systemic error, especially for tubes with large diameter. Figure 1 describes that there exists two rays passing through the optical center and intersecting with an arbitrary cross-section of bent tube at two points of tangency and . These two rays intersects the image plane at the points and , which are the projection points of and . In fact, the image plane is usually not perpendicular to the cross section, so the projection point of cross section center is not the midpoint of line on image plane. Considering that the contour axis is the midpoint of each pair of projection points and , thus the projection position of tube axis is deviated from the image contour axis.
Accurately modeling the projecting process of bent tube axis is an effective way to improve the reconstruction precision. Ascending to 1990s, plenty of researchers have studied the perspective projection model of bent tubes, while most results focused on straight homogeneous circular cylinders or surface of revolutions [15–17]. There were only a few researches dealing with bent tubes: Boseman [18] divided the tube axis into several portions, and treated each segment as straight lines. It caused a considerable margin of error because the major parameters are measured via junctions of straight lines. Lee [19,20] mapped the surface points to axis through local geometric properties around points and fitted axis, solving the problem of straight-line approximation. But it needs a point cloud from pip surface. Caglioti [21] came up with a way using the geometric properties of tube surface to reconstruct a full 3D reconstruction of the shape from a single perspective image or even a fully-uncalibrated image under exact perspective. It required the radius of tube to be known. Although the above-mentioned means provided some perspective models for straight or bent tubes, there exists some matters like large errors or demanding more constraint conditions.
Addressing the benefits and disadvantages of existing measurement methods, this work utilizes binocular stereo vision to reconstruct bent tube axis. An accurate model for projective process of space axis and a way for calculating the precise projected position of axis on image plane are proposed. The method can further improve the precision of axis measurement on the basis of keeping user-friendly control. Besides, the radius of tube is not needed in advance.
2. Proposed approach
The bent tube surface can be treated as a family of circular cross-section with constant radius R, and each cross-section center lies on the axis of tube. Therefore, constructing the perspective projection model of any cross section and calculating the projected position of every cross-section center can obtain the whole projected position of axis.
2.1 perspective projection of an arbitrary cross section
Figure 2 depicts the perspective projection of an arbitrary cross-section on a bent tube. is a circle located on a supporting plane P. is the normal vector of plane P, intersecting at the cross-section center . Planes and are the tangent planes to the tube surface at point and separately, passing through the optical center O. Point is the center of the chord . is the intersection line of planes and .Plane bisects the two planes and , passing through the line and the point . The points and are the projections of the points and respectively. Likewise, the same correlation applies to the point and the point , as well as the pointand the point. The planes and separately intersects the image plane at the lines and . is the intersection point of the lines and , that can also be regarded as the intersection point of the line and the image plane.
The plane P is given by , where is and d is the distance from the plane P to O. Each pointlying on the cross-section satisfies the equation . As shown in Fig. 2, since the planes and are the tangent planes to tube surface, they are perpendicular to the plane P and the line is perpendicular to the plane P. So the line and the vector have the same direction in space. According to the definition of vanishing point in ([21,22]), the intersecting point for the lines and is the vanishing point for vector on the image plane. The point is given by , and the vector can be written as , where .
On the basis of camera imaging model, the points and satisfy the relationship shown in Eq. (1),
where stands for a scale factor, is the camera matrix, and .Let the inverse matrix of be written as Eq. (2). Then Eq. (1) is written as Eq. (3).Because the point lies on the plane P, it also satisfies . Therefore the coordinates of takes the form as Eq. (4).The point can be expressed as due to symmetry. The above derivation process introduces the perspective projection model for the points and . In fact, each cross-section can only be projected to two points on image plane except for end faces. Considering there are some geometric constraints between the points and , the projected position on image plane of the point could be obtained by the points . So when we compute the projected position of an axis, the above model can represent the projective process of every cross-section, including two end faces.
2.2 Projective position of cross section center
The planes and can be regarded as the back projection planes of the lines and (Fig. 3) respectively, so the normal vectors of these two planes can be expressed as and . Then since the plane bisects the planes and , a Lagrange function is given by with and . We define the normal vector of plane as , and solve the appropriate value of that satisfies the equation . Then the expression of the vector can be given as .
Similarly to Eq. (1), the point could be written as Eq. (5):
Located on the planes P and , the point satisfies and , where. Moreover, since the point is the center of the cross-section , there exists the condition . Combining Eq. (1)~(5), a new formula is obtained as Eq. (6),where the definition of some parameters in Eq. (6) is shown in Eq. (7). Other definition of parameters are defined in Eq. (4) and its dual equation.To further simplify Eq. (6), the variables of the last row in Eq. (6) is written as Eq. (8).Finally a succinct equation for solving the coordinates of the point is given by Eq. (9) with .
2.3 Image properties of arbitrary cross section
Caglioti [19] has provided the coupling way for contour points, making it possible to determine which pairs of imaging points are actually coupled for a cross-section. The points and are coupled only if the angle coincides with the angle , so it could be enforced as Eq. (10),
where is the absolute conic of the image, with . Then a loss function to estimate the difference between the angles and is constructed as Eq. (11).At first, applying the contour extracting method presented in literature [21], we can get both the sub-pixel contours of tube image and the normal vectors for each point on contours. After that we subdivid the edge points into two chains located on the facing contours and found an initial pair of coupled points. Starting from this pair, other pairs are found incrementally by searching along the points nearby the last pair and choosing the one minimizing Eq. (11).
2.4 3D reconstruction of bent tube axis
Figure 4 shows the construction process using binocular stereo. Firstly we calculate the projected positions for tube axis on two image planes. Then the epipolar geometry is used to search the corresponding image points in two views. Finally for each pair of correlative points, we reconstruct the point in space that is projected to these two image points and got the whole axis.
3. Experiments and analysis
The measurement system is shown in Fig. 5, consisting of binocular stereo visual sensor, a background light source and a computer. The binocular stereo visual sensor comprises two industrial GC1380H cameras with resolution of pixels and two Schneider Cinegon 1.4/12mm lenses. In order to evaluate our method, every testing tube is measured by GOM ATOS core 300 scanner, with a precision of 0.02 mm. Each group of experiment results is compared with the data from the scanner. The calibration results of binocular stereo visual sensors are as Table 1.
3.1 Result using method with axis projection model
Three tubes with different diameters but with two bending angles are measured for several times. To quantify our measuring results, we first reconstructed their axes by the approach described in Section 2, and then separated each axis into straight line sections and arc sections and fitted these sections separately. The angle between any two adjacent lines is regarded as the bending angle of tube, and the radius of arc is treated as the bending radius of tube. Based on this, bending angle and radius are the parameters to be measured in the following experiments.
3.1.1 Results for tube with diameter about 6 mm
In this section we measured a bent tube with diameter about 6 mm, and the tube is shown in Fig. 6(a). The metrical data of GOM scanner on two bending angles are 112.25° and 99.79°, and the results of two radius are 25.17 mm and 24.82 mm.
The tube was repeatedly measured 10 times in the same surroundings but with different visual angles. The reconstruction result is shown in Fig. 6(b), and the detail data is listed in Table 2. From the second to the last line in Table 2, the root mean square error (RMSE) of two bending angles are 0.020° and 0.020°, and the RMSE of two radius are 0.022 mm and 0.021 mm. The above results vividly indicated the high measurement repeatability accuracy and stability of the system. Compared with the results given by the GOM scanner, the root mean square deviation (RMSD) of bending angles are 0.030° and 0.031°, and the RMSD of radius are 0.029 mm and 0.029 mm.
3.1.2 Result for tube with diameter about 15 mm
Next we measured a thicker tube with diameter about 15 mm displayed in Fig. 7(a).The GOM measuring data are 100.64° and 122.45° for bending angles and 63.23 mm and 65.69 mm for radius. Figure 7(b) and Table 3 show the reconstruction results together. Compared with the results of the GOM scanner, the RMSD of bending angles are 0.031° and 0.029°, and the RMSD of radius are 0.028 mm and 0.029 mm.
3.1.3 Result for tube with diameter about 24 mm
Finally we measured the thickest tube with diameter about 24 mm shown in Fig. 8(a). The GOM results for bending angles are 87.95° and 108.90°, and radius are 72.32 mm and 72.67 mm. The specific measurements are shown in Fig. 8(b) and Table 4. The RMSD of bending angles are 0.027° and 0.031°, and the RMSD of radius are 0.026 mm and 0.029 mm.
The results above dedicate that for bent tubes with different diameters, our method is not affected by the factor of tube diameter. Among these results, the RMSD of bending angles is within 0.031°, and the RMSD of radius is within 0.031 mm. So the measuring performance of our method is excellent.
3.2 Result using method without axis projection model
We also reconstructed those tubes by the method using contour center lines as the projected position of tube axis and made a comparison between the results gained from the traditional method and our method. According to Jin’s method [9] we got the contour center lines in images and reconstructed 3D axes of three tubes. A serious of results are presented as follows.
3.2.1 Result for tube with the diameter about 6 mm
Table 5 reveals the results of the first tube measured by the method without perspective projection model. Contrast with the data measured from the GOM scanner, the RMSD of two bending angles are 0.035° and 0.034°, and the RMSD of radius are 0.035 mm and 0.032 mm.
It’s obvious that for this method the measurement precision are lower than our approach, while the repeatability accuracy is similar to our method. Table 6 lists the comparison results of two methods on the same tube, where the method 1 represents for our method and the method 2 stands for the traditional method. The last line in the table provides the percentage of precision improvement that indicates the measurement accuracy of the first tube can be raised by 14.4% in average after adding a perspective projection model.
3.2.2 Result for tube with diameter about 15 mm
Then we did the same experiment as subsection 3.2.1 on the second tube and gained the results in Table 7. Two bending angles have the RMSD of 0.037° and 0.038°, and the RMSD of radius are 0.042 mm and 0.038 mm.
Similar to the conclusion from the last experiment, our method has a higher measurement precision than the traditional method while maintaining the same repeatability accuracy. Table 8 implies that the measuring accuracy can be improved by 30.4% on average.
3.2.3 Result for tube with diameter about 24 mm
Finally the identical content is repeated on the third tube. From Table 9 the RMSD of bending angles are 0.040° and 0.044°, and the RMSD of radius are 0.041 mm and 0.041 mm.
The similar conclusion is gained as the foregoing discussion. From Table 10 the measuring accuracy is increased by 47.3% on average.
The above details turns out that some system error can be corrected by adding the perspective projection model. Contrast with the traditional method, our method is able to decrease the root mean square deviation of the measurements and ensures the high repeatability accuracy. Furthermore, the reconstruction precision decreases as the diameter increases in traditional method, but our method is not sensitive to the changing of diameter. So when the diameter is larger, the proposed method behaves better in improving the measuring precision.
3.3 The influence of image noise on reconstruction results
To the test images, we add several Gaussian noises with mean 0 and varying standard deviations (SD) in same pictures. Using these pictures, a series of new 3d reconstruction results can be obtained and the biases between GOM results and new results are used to evaluate the influence of noises. For each noise level, we repeated the experiment for ten times. The final average measurement results of four key parameters are shown in Fig. 9. As can be seen, the RMSD increases slightly as the SD increases when the SD is lesser than 20. But after 20, the RMSD increases significantly. This is mainly because the greater image noise can flush image details. Under the high noise level, the loss of small regions feature is great. So it is hard to detect the accurate contour, and the fuzzy contour will result in large deviation of the final measured value.
4. Conclusion
This paper proposes a high precision bent tube axis reconstruction method based on the accurate calculation of the projected image position of bent tube space axis. The method can reduce the error caused by approximately estimating the projected position of tube axis. Meanwhile, binocular stereo vision requires no external constraint conditions and has high precision compared with monocular reconstruction. The proposed method is able to calculate the 3D coordinates of points located on tube axis without knowing tube radius in advance.
In contrast experiments, three tubes with different diameters but two corners were measured. The experimental results show that the root mean square deviation to the GOM scanner is reduced after appending the projection model while keeping a high repeatability accuracy. The average measurement accuracy of the tube with the diameter about 6 mm could be increased by 14.4%. The similar results occurred on rest tubes: the tube with about 15 mm diameter was raised by 30.4% and the one with about 24 mm diameter was improved by 47.3%. A better conclusion comes about that the measurement precision of our method is practically independent of tube diameter. The proposed method will heighten the correction effect of reconstruction error as the diameter increases.
The method can also be applied to multi-camera vision systems based on reconstructing tube axis. This cannot only improve the veracity of bent tube reconstruction but also solve the problem that the information of tube can’t be obtained completely at some angles.
Funding
Aeronautical Science Foundation of China 2017ZE51062.
References
1. X. Ji and G. Gong, “Study of Realization Technologies for 3D Reconstruction of Superchargers′ Impellers,” J. Mech. Sci. Technol. 25(3), 322–325 (2006).
2. Y. Wei and A. Thornton, “Tube production and assembly systems: the impact of compliance and variability on yield,” presented at Design Engineering Technical Conferences and Computer and Information in Engineering Conference, Baltimore, Maryland, 10–13 Sept. (2000).
3. Extreme Networks white paper, “Virtual metropolitan area networks,” (Extreme Networks, 2001), http://www.extremenetworks.com/technology/whitepapers/vMAN.asp.
4. F. Gu, H. Zhao, X. Zhou, J. Li, P. Bu, and Z. Zhao, “Photometric invariant stereo matching method,” Opt. Express 23(25), 31779–31792 (2015). [CrossRef] [PubMed]
5. Z. Liu, X. Li, F. Li, and G. Zhang, “Flexible dynamic measurement method of three-dimensional surface profilometry based on multiple vision sensors,” Opt. Express 23(1), 384–400 (2015). [CrossRef] [PubMed]
6. A. S. Metronor, Homepage, “Metronor One,” (Metronor, 2011–2018), http://www.metronor.com.
7. Gesellschaft für optische Messtechnik mbH Homepage, “ATOS Core – Optical 3D Scanner,” (GOM, 2011–2018), https://www.gom.com/metrology-systems/atos/atos-core.html.
8. W. Bösemann, “Industrial photogrammetry-accepted METROLOGY tool or exotic niche,” presented at the International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, Prague, Czech Republic,12–19 July. (2016).
9. AICON 3D Systems GmbH Homepage, “Aicon-scanner,” (AICON, 2011–2018), https://www.aicon3d.com/de-DE/products/aicon-scanner.
10. AICON 3D Systems GmbH Homepage, “Tubeinspect,”(AICON, 2011–2018), https://www.aicon3d.com/de-DE/products/tubeinspect.
11. P. Jin, J. Liu, S. Liu, and X. Wang, “Automatic multi-stereo-vision reconstruction method of complicated tubes for industrial assembly,” Assem. Autom. 36(4), 362–375 (2016). [CrossRef]
12. S. Liu, J. Liu, P. Jin, and X. Wang, “Tube measurement based on stereo-vision: a review,” Int. J. Adv. Manuf. Technol. 92(5-8), 2017–2032 (2017). [CrossRef]
13. T. Zhang, J. Liu, S. Liu, C. Tang, and P. Jin, “A 3D reconstruction method for pipeline inspection based on multi-vision,” Measurement 98, 35–48 (2017). [CrossRef]
14. T. Zhang, J. Liu, C. Tang, and S. Liu, “3D reconstruction method of bent tube with NURBS curve,” in 2nd International Conference on Mechatronics and Applied Mechanics (ICMAM2012), C.K. Wang and J.Guo, ed. (Applied Mechanics and Materials, 2013), pp. 112–118. [CrossRef]
15. S. Liu, P. Jin, J. Liu, X. Wang, and P. Sun, “Accurate measurement method for tube’s endpoints based on machine vision,” Chin. J. Mech. Eng. 30(1), 152–163 (2017). [CrossRef]
16. C. Doignon and M. de Mathelin, “A degenerate conic-based method for a direct fitting and 3-d pose of cylinders with a single perspective view,” in Proceedings of IEEE International Conference on Robotics and Automation, (IEEE, 2007), pp. 4220–4225. [CrossRef]
17. C. Liu and W. Hu, “Relative pose estimation for cylinder-shaped spacecrafts using single image,” in IEEE Transactions on Aerospace and Electronic Systems, (IEEE, 2014), pp. 3036–3056.
18. K. Wong, P. Mendonça, and R. Cipolla, “Reconstruction of surfaces of revolution from single uncalibrated views,” Image Vis. Comput. 22(10), 829–836 (2004). [CrossRef]
19. W. Bösemann, “The optical tube measurement system OLM photogrammetric methods used for industrial automation and process control,” presented at International Archives of Photogrammetry and Remote Sensing, Part B5, Vienna (1996).
20. I.-K. Lee, “Curve reconstruction from unorganized points,” Comput. Aided Geom. Des. 17(2), 161–177 (2000). [CrossRef]
21. I.-K. Lee and K.-J. Kim, “Shrinking: Another method for surface reconstruction,” in Geometric Modeling and Processing, 2004. Proceedings, (IEEE, 2004), pp. 259–266.
22. Caglioti and A. Giusti, “Reconstruction of canal surfaces from single images under exact perspective,” in European Conference on Computer Vision, A. Prinz, ed. (Springer, 2006), pp. 289–300.