Open Access
Add to BibSonomy
Abstract
A prototype of a 3D shape measurement device with two cameras was developed, and the prototype was used to confirm the effectiveness of the 3D shape measurement method that does not require camera parameters. For 3D shape measurement using a fringe projection method, generally the pixel coordinate in the image and phase information of the projected fringe pattern are used; however, 3D coordinates can be obtained from only three fringe phase values. Recently, authors proposed this method as a feature quantity type whole-space tabulation method. There were no camera parameters required because pixel coordinates were not used, and thus a camera calibration-free 3D shape measurement can be realized. Moreover, the experimental evaluation was performed using the prototype having two cameras. Although these cameras were located at different positions and had lenses with different focal lengths, their measured shapes of an object were almost identical. An experiment of 3D shape measurement using an uncalibrated camera was performed. The effectiveness of the proposed method was quantitatively validated from experimental result obtained using the developed prototype.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
For measuring 3D shapes, the fringe projection method is a useful technique [1,2]. Moreover, for 3D measurements, 3D coordinates can be obtained from at least three independent values such as pixel coordinates and phase information of the projected fringe pattern. For many conventional methods, the 3D coordinates of a point on an object are computed from the pixel coordinates and phase information of the projected fringe pattern; however, these methods are not robust to the vibrations of measurement instruments, e.g., the positions of imaging sensors and lenses are easily adjusted by vibration, which may result in systematic errors.
Authors proposed a method known as a whole-space tabulation method (WSTM) using linear LED devices [3,4] in the previous study. In WSTM, the relation of an unwrapping phase of the fringe, including the aberration of lenses and the spatial coordinates at each pixel of a camera is stored in a table. This table is built as a calibration process; however, it is still difficult to minimize systematic errors because of the dynamic position of a lens owing to vibration. Although multiple methods, which do not require camera calibration, have been proposed for 3D shape measurement [5,6], it is difficult to adapt these methods to the Fringe projection method; moreover, a self-calibrating shape-measuring system was proposed [7,8]. In this method, called “Phasogrammetry,” every point on an object is obtained by at least four phase values of fringe patterns projected from at least two projectors [9,10]. The projector projects both horizontal and vertical fringe patterns to specify each line projected from the projector. The method of photogrammetric calculation is then used to obtain 3D coordinates of a point on the object from the four-phase values. Note that this method is robust to vibrations of the measurement instrument because it does not require camera calibration. Similarly, a pre-calibration-free 3D shape measurement method, which uses both horizontal and vertical fringe pattern, was proposed [11]. In the abovementioned method, the phase information of the projected 2D fringe patterns is used for matching images obtained by two cameras.
However, authors had suggested that in a 3D measurement, 3D coordinates can be obtained from at least three independent values as feature quantities such as the pixel coordinates and phase information of the projected fringe pattern. Furthermore, a set of three feature quantities corresponds one by one to a set of 3D coordinates, i.e., a table of correspondence between the feature quantities and 3D coordinates can be produced. Authors proposed this method as a feature quantity type whole-space tabulation method (F-WSTM) [12]. Moreover, a camera calibration-free 3D shape measurement can be realized using F-WSTM because camera parameters are not required. The 3D coordinates at a target point on an object are obtained only from the three phases at that point.
In this study, to confirm the effectiveness of the F-WSTM, a prototype of a 3D shape measurement device with two cameras is developed. As a preliminary to explain the prototype, the principles of F-WSTM and the calibration method to obtain a table of correspondence between three feature quantities and 3D coordinates are explained. An experiment of 3D shape measurement using an uncalibrated camera is performed.
2. Principles of a feature quantity type whole-space tabulation method
Figure 1 shows the principles of F-WSTM, which requires three stable projectors fixed in a 3D shape measurement device. In Fig. 1, projector PA projects fringe patterns onto an object. The fringes are then imaged by a camera for measurement. The phase map of the fringe image can then be obtained using a phase analysis method such as a phase-shifting method [13] or Fourier transform method [14]. Note that phase ϕA is the unwrapped phase obtained by certain phase unwrapping methods [15–17]. Therefore, the projectors PA, PB, and PC may be considered to be projecting unwrapped phases ϕA, ϕB, and ϕC at point P on the object, respectively. The camera then considers the set of unwrapped phases ϕA, ϕB, and ϕC as feature quantities.
Here, an arbitrary set of unwrapped phases ϕA, ϕB, and ϕC has a one-to-one correspondence with a set of 3D coordinates (x, y, z) if it is uniquely defined in the measurement space. This relationship can be built as a table of feature quantities and 3D coordinates (right plot in Fig. 1). In this table, each (ϕA, ϕB, ϕC) element is assigned its corresponding 3D coordinates (x, y, z), and this table is built in advance as a calibration process. The 3D coordinates of point P are quickly retrieved from this table using a set of unwrapped phases ϕA, ϕB, and ϕC.
3. Fringe projector using a linear LED device
In this section, the construction of a fringe projector using a linear LED device is explained. Figure 2 shows a schematic of a projected fringe pattern based on a light-source-stepping method (LSSM) [3,4] with a linear LED device and a grating glass such as a Ronchi ruling. The grating glass is placed between the light source and the object. A shadow of the grating glass is then projected onto the object.
Three light sources S1, S2, and S3 are located at the same z position and spaced at regular intervals. As shown in Fig. 2, when the active light source is changed from S1 to S2, the position of the projected fringe pattern will be changed from the continuous line to the larger dashed line. Similarly, when the active light source changes from S2 to S3, the position of the projected fringe pattern changes from the larger-dashed line to smaller-dashed line. Consequently, the projected fringe pattern can be shifted by changing the active light source.
In cases that light source comprises lined LEDs, the projected fringe pattern becomes brighter when the direction of the LED is aligned with the grating direction on the grating glass. The LED enables the active light source to be quickly changed, i.e., the phase of the projected fringe pattern can be quickly shifted. The phase of the projected fringe pattern can be obtained using a phase-shifting method, i.e., the phase-shifted fringe pattern yields wrapped phase distribution in the measurement space. A wrapped phase at an arbitrary point in the measurement space can be obtained using the fringe projector.
4. Proposed calibration method
In this method, a reference plane is used for the calibration process. A 2D fringe pattern with a known regular pitch is fixed on the surface of the reference plane. Figure 3 shows the configuration of the camera for calibration and reference planes, and the reference plane can be shifted for the normal direction of the plane. The z-direction is defined as the normal direction, and the x- and y-directions are then defined as the directions for the 2D fringe fixed on the reference planes. The reference plane is placed on a linear stage to step-by-step move to the z-direction. The surface of the reference plane at each z position is considered as a reference point.
A pixel of the camera obtains an image along the ray line L. The pixel coordinate is (i, j) in the camera, and the pixel images point Pn on reference plane Rn, where n (1 ≤ n ≤ N) is a reference plane number. The z-coordinate zn is then obtained as the z position of the reference plane at each point, and the x- and y-coordinates of point Pn are obtained from the 2D grating fixed on reference planes. The x and y coordinates can be accurately obtained using a phase analysis method such as the Fourier transform method and phase shifting method. This indicates that a set of {x, y, z} is obtained at each pixel on each reference plane, corresponding to the pixel coordinate (i, j) and the reference plane number n.
The calibration process of F-WSTM using the reference plane is explained below using Fig. 4. The process is demonstrated on 2D coordinates for simplicity. The 3D shape measurement device shown in Fig. 4 comprises two projectors A and B. The distributions of unwrapped phases ϕA and ϕB are projected onto reference planes from projectors A and B, respectively. The unwrapped phases ϕA and ϕB make a unique combination of {ϕA, ϕB} appearing at each point. A camera for calibration is located close to the 3D shape measurement device. As shown in Fig. 4, a reference plane is placed in front of the camera and 3D shape measurement device.
The reference planes located at each z position are imaged by the camera. At each pixel of the image, the x coordinate is obtained from the phase of the grating image. The z coordinate is the z position of the reference plane. The unwrapped phases ϕA and ϕB at each pixel are obtained by analyzing the phases of the projected fringe patterns produced by the two projectors. In other words, a set of {ϕA, ϕB, x, z} can be obtained at each pixel as a reference point shown as a closed red circle in Fig. 4. The unwrapped phases ϕA and ϕB can then be deemed feature quantities (ϕA, ϕB).
Figure 5 shows the table of coordinates corresponding to the feature quantities. Each grid point displayed in Fig. 5(a) and (b) shows a table element. Its contents are the coordinates (x, z) corresponding to the feature quantities, and this table is called “feature quantities - coordinates table (FQ-table)”.
Figure 5(a) shows a basic example of the FQ-table. Each reference point (a closed red circle) shown in Fig. 4(a) and (b) are located at a corresponding point on the feature quantities (ϕA, ϕB) space, as shown in Fig. 5(a). These points are distributed all over the measurement area. The contents of the table elements are calculated by interpolating the values at several reference points adjacent to the elements using a linear interpolation. The elements within the measurement area are shown as the closed blue circles in Fig. 5(a).
However, the elements outside the measurement area have no contents. In general, the measurement area becomes circumscribed by the specific concentrated area, which depends on the arrangement of projectors and camera, in the feature quantities (ϕA, ϕB) space. Therefore, the number of effective table elements is reduced. As a solution to this problem, i.e., the reduction problem, a coordinate transform from (ϕA, ϕB) to (FA, FB) is applied. Here, as shown in Fig. 5(b), the FA and FB axes are determined as shown in Fig. 5(a) in such an approach that the reference points are extensively spread in the transformed space of (FA, FB). The transformed feature quantities (FA, FB) can be obtained using a coordinate transform function such as an affine transform. Consequently, the number of effective table elements is increased.
The measurement process of F-WSTM is explained below using Fig. 6. A camera for measurement is located at the different position of the camera for calibration. The camera takes phase ϕA,P at a pixel imaging point P on an object when the distributions of unwrapped phases ϕA is projected from projector A as shown in Fig. 6(a). The camera takes phase ϕB,P at the pixel when the distributions of unwrapped phases ϕB is projected from projector B in the same way as shown in Fig. 6(b). The transformed feature quantities (FA,P, FB,P) are obtained from the feature quantities (ϕA,P, ϕB,P). A coordinates (xP, zP) at point P is obtained from the transformed feature quantities (FA,P, FB,P) using the FQ-table shown in Fig. 5(b). Here, the accuracy depends on the accuracy of movement of the reference plane and the accuracy of interpolation to obtain the table elements. The precision of the measured coordinates mainly depends on the density of the table elements and the precision of obtained phase.
In a practical sense, the number of feature quantities is three such as (ϕA, ϕB, ϕC) and (FA, FB, FC,) and the number of coordinates contained in an element of the FQ-table is three such as (x, y, z), i.e., a set of {ϕA, ϕB, ϕC, x, y, z} is obtained at each pixel as a reference point. The transformed feature quantities (FA, FB, FC,) are calculated from (ϕA, ϕB, ϕC). The coordinate (x, y, z) corresponding to the feature quantities (FA, FB, FC,) for each element within the measurement area is contained in the element.
5. Prototype of a 3D shape measurement device
A prototype of a 3D shape measurement device is developed for validating the principles of the F-WSTM. The prototype is constructed as shown in Fig. 7. Figure 8 shows an image of the prototype. The size is 320 mm × 150 mm × 150 mm. The prototype comprises three sets of fringe projectors: LED drivers, two cameras, and a micro-computer.
Each projector comprises a linear LED device and the abovementioned grating glass. Projectors A and B have vertical grating glasses and linear LED devices. They project vertical phase-shifted fringe patterns. Projector C has a horizontal grating grass and a linear LED device and projects horizontal phase-shifted fringe patterns. A set of unwrapped phases ϕA and ϕB obtained from projector A and B, respectively, gives information for the x and z coordinates. Unwrapped phase ϕC obtained from projector C gives information for the y coordinate. In other words, the unwrapped phases ϕA, ϕB and ϕC make a unique combination of {ϕA, ϕB, ϕC} appearing at each point inside the measurement volume.
The cameras are placed under projector C. Camera 1 has a wide-angle lens (f = 8 mm) and Camera 2 has a narrow-angle lens (f = 16 mm). A PC is connected to the 3D shape measurement device through a USB interface.
The PC then sends a control command to the micro-computer and it receives fringe images from the cameras. The micro-computer produces timing signals and the cameras synchronously trigger with each other after receiving the control command. The LED drivers turn on the specified lined LEDs mounted on each projector. The cameras consider images of the projected fringe patterns onto an object synchronously by lighting the LEDs. The PC gets the fringe images obtained from either Camera 1 or Camera 2. Camera 1 is then used for the calibration process. Both Camera 1 and Camera 2 are used for 3D shape measurement of an object.
6. Experiment of the 3D shape measurement
An experiment of the 3D shape measurement using the prototype was performed to confirm the effectiveness of the proposed method.
6.1 Experimental setup
Figures 9(a) and (b) show the image and arrangement of an experimental setup and a specimen, respectively. An LCD monitor is used as a reference plane. A diffuser sheet is attached on the surface of the LCD monitor such that the surface functions as a reference plane. A 2D grating displayed on the LCD monitor appears on the surface of the diffuser sheet. The LCD monitor can display grating patterns with phase-shifting to obtain accurate phase maps for both the x direction as well as the y direction. Moreover, projected fringe patterns are projected onto the same diffuser sheet surface.
The LCD monitor is fixed in the basement. A prototype of the 3D shape measurement device is located on a linear stage fixed in the basement. The linear stage moves in the z-direction defined as the normal direction of the surface of the LCD monitor. The movement of the 3D shape measurement device causes relative movement of the LCD monitor, i.e., it can be recognized that the xyz coordinate system is fixed to the 3D shape measurement device.
As a calibration process, an FQ-table is produced using reference planes. The initial distance between the 3D shape measurement device and the reference plane is 300 mm. The reference plane moved from z = 0 to z = 100 mm at 1.0 mm intervals. The movement range is defined as a measurement range. The FQ-table can be generated within the measurement range. At each position, the images of the grating patterns displayed on the LCD monitor with phase shifting and the images projected from three projectors incorporated into the 3D shape measurement device are captured by Camera 1, which has a wide-angle lens. The size of images captured by Camera 1 is 480 × 480 pixels. The FQ-table is produced from the images using the abovementioned calibration method.
A specimen is located with the measurement range. As shown in Fig. 10, the specimen, composed of aluminum and the surface is covered with matte white lacquer, has a four-sided truncated pyramid with a height of 10.07 mm. The vertical distance of upper area is 20.55 mm. The height of the specimen and the vertical distance of upper area are accurately measured by a depth-measuring microscope (HISOMET II, Union Optical Co., LTD.).
6.2 Experiment and evaluation
3D shape measurement was performed using Camera 2, which has a narrow-angle lens. Camera 2 is different from the camera (Camera 1) used for the calibration process. Figure 11 shows phase-shifted fringe images projected by linear LED device in projector A; the image size is 480 × 480 pixels. As shown in Fig. 12, (a) wrapped phase map was obtained from these four phase-shifted fringe images with the calculation for a four-step phase-shifting method. As shown in Fig. 13(a), the unwrapped phase map has been obtained from this wrapped phase maps with a conventional spatial phase unwrapping method. Figures 13(b) and (c) show unwrapped phase maps obtained using projectors B and C in the same manner, respectively.
A 3D coordinate corresponding to each pixel was obtained from these three unwrapped phase maps using the FQ-table. A combination of translational and rotational motions was applied to the 3D coordinates distribution in such a way that the lower part becomes the plane where z = 0. A transform matrix was generated using a least-square method. Figure 14(a) shows the height distribution after this translation. Figure 14(b) exemplifies a horizontal cross-section at the center of the object shown as a broken line in Fig. 14(a). The average and standard deviations of measured z-coordinates of the lower area (Area 2L) and the upper area (Area 2U) of the object are shown in Table 1, respectively. The size of Area 2L and Area 2U is 80 × 80 pixels (∼8 × 8 mm), and the difference between the two areas is 10.07 mm. This measured value (10.07 mm) is the same as the object’s height, which is 10.07 mm.
Table 1. Height Distribution
Figure 14(c) exemplifies a vertical cross-section at the center of the object shown as a broken line in Fig. 14(a). Linear regression lines L1, L2, and L3 of the right slope, the upper flat area, and the left slope are calculated as shown in Fig. 14(c). The intersection points C1 and C2 are obtained as the intersection of L1 and L2, and the intersection of L2 and L3, respectively. The distance between C1 and C2 was 20.38 mm. The difference from the actual distance measured by a function of horizontal displacement measurement of the depth-measuring microscope was 0.17 mm.
The measurement precision of this method mainly depends on the density of the table elements and the precision of obtained phase as mentioned above. In the case of this experiment, we produced a FQ-table having 512 × 512 × 512 elements for 100 × 100 × 100 mm measurement area approximately. It means that the resolution of the FQ-table is 0.2 mm approximately. This value is almost the same as the standard deviation of 0.19 mm shown in Table 1. So, in this case, it is surmised that the primary reason of this noise is the density of the table elements.
6.3 Experiment of vibration test
An experiment to confirm the robustness of the of the measurement device with the proposed method was performed. Figure 15 shows scenes with vibrating the measurement device using a robot arm. The vibration frequency was 1.6 Hz. Figure 16 shows height distributions measured before and after vibration along vertical cross-section horizontal cross-section at the center of the object Table 2 shows the comparison of measurement results before and after vibration. Average and standard deviation of height of upper area and lower area, respectively, are compared. The results show that no significant difference and standard deviation are occurred.
Fig. 16. Height distributions measured before and after vibration along vertical cross-section horizontal cross-section at the center of the object
Table 2. Comparison of measurement results before and after vibration
6.4 Measurement example of a general object using an uncalibrated camera
A measurement example of a general object using uncalibrated cameras was performed. Figure 17 shows the sequence of the measurement both an inner camera with wide-angle lens, which was used for making FQ-table in the calibration process, and an outer camera with narrow-angle lens. The outer camera was attached on the top panel of the 3D shape measurement device with loose hook and loop fastener. The target object was a white sculpture of Beethoven. The inner camera took the whole of the sculpture from the front. The outer camera took the face from the upper left direction. This measurement example shows that the uncalibrated camera is surely available to 3D shape measurement using the F-WSTM.
7. Summary
Authors proposed an F-WSTM as a novel method for 3D shape measurement. A camera-calibration-free 3D shape measurement can be realized using the F-WSTM. A prototype of a 3D shape measurement device with two cameras using the F-WSTM was developed in this study. Three fringe projectors using linear LED devices and two cameras were incorporated into the prototype. The experimental evaluation was performed using a specimen that has a four-sided truncated pyramid. A robust 3D shape measurement could then be performed using a camera, which was not calibrated. To summarize, the effectiveness of the proposed method was quantitatively validated based on the experimental result obtained with the developed prototype. The comparison to the traditional fringe projection systems against vibrations will be performed using the prototype as the further study.
Funding
Japan Science and Technology Agency (JST) (AS2915038S).
Disclosures
The authors declare no conflicts of interest.
References
1. F. Chen, G. M. Brown, and M. Song, “Overview of three-dimensional shape measurement using optical methods,” Opt. Eng. 39(1), 8–22 (2000). [CrossRef]
2. S. S. Gorthi and P. Rastogi, “Fringe projection techniques: whither we are?” Opt. Laser Eng. 48(2), 133–140 (2010). [CrossRef]
3. M. Fujigaki, Y. Oura, D. Asai, and Y. Murata, “High-speed height measurement by a light-source-stepping method using a linear LED array,” Opt. Express 21(20), 23169–23180 (2013). [CrossRef]
4. M. Fujigaki, T. Sakaguchi, and Y. Murata, “Development of a compact 3D shape measurement unit using the light-source-stepping method,” Opt. Laser Eng. 85, 9–17 (2016). [CrossRef]
5. R. Mohr and E. Arbogast, “It can be done without camera calibration,” Pattern Recognit. Lett. 12(1), 39–43 (1991). [CrossRef]
6. K. Achour and M. Benkhelif, “A new approach to 3D reconstruction without camera calibration,” Pattern Recognit. 34(12), 2467–2476 (2001). [CrossRef]
7. V. Kirschner, W. Schreiber, R. M. Kowarschik, and G. Notni, “Self-calibrating shape-measuring system based on fringe projection,” Proc. SPIE 3102, 5–13 (1997). [CrossRef]
8. W. Schreiber and G. Notni, “Theory and arrangements of self-calibrating whole-body 3-D-measurement systems using fringe projection technique,” Opt. Eng. 39(1), 159–169 (2000). [CrossRef]
9. P. Kuhmstedt, M. Heinze, M. Himmelreich, G. Notni, C. Brauer-Burchardt, and G. Notni, “Phasogrammetric optical 3D sensor for the measurement of large objects,” Proc. SPIE 5457, 56–64 (2004). [CrossRef]
10. A. Babaei and M. Saadatseresht, “Optimal selection of distortion model parameters for projection lenses using phasogrammetric self-calibration,” Earth Observ. Geomat. Eng. 3(2), 39–50 (2019). [CrossRef]
11. K. Zhong, Z. W. Li, R. F. Li, Y. S. Shi, and C. J. Wang, “Pre-calibration free 3D shape measurement method based on fringe projection,” Opt. Express 24(13), 14196 (2016). [CrossRef]
12. M. Fujigaki, Y. Kusunoki, and H. Tanaka, “Development of 3D shape measurement device using feature quantity type whole-space tabulation method,” in Advancements in Optical Methods & Digital Image Correlation in Experimental Mechanics3, 117–120 (2020).
13. V. Srinivasan, H. C. Liu, and M. Halioua, “Automated phase-measuring profilometry of 3-D diffuse objects,” Appl. Opt. 23(18), 3105–3108 (1984). [CrossRef]
14. M. Takeda and K. Mutoh, “Fourier transform profilometry for the automatic measurement of 3-D object shapes,” Appl. Opt. 22(24), 3977–3982 (1983). [CrossRef]
15. S. Zhang, “Absolute phase retrieval methods for digital fringe projection profilometry: A review,” Opt. Lasers in Eng. 107, 28–37 (2018). [CrossRef]
16. L. Song, Y. Chang, J. Xi, Q. Guo, X. Zhu, and X. Li, “Phase unwrapping method based on multiple fringe patterns without use of equivalent wavelengths,” Opt. Commun. 355, 213–224 (2015). [CrossRef]
17. M. Servin, M. Padilla, G. Garnica, and A. Gonzalez, “Profilometry of three-dimensional discontinuous solids by combining two-steps temporal phase unwrapping, co-phased profilometry and phase-shifting interferometry,” Opt. Laser Eng. 87(24), 75–82 (2016). [CrossRef]
