1. Introduction

Fringe patterns are essential for many precision measurements including laser interferometry [1], Moiré techniques [2] and holography [3]. Fringe patterns contain phase information that corresponds to height information [4], stress [5], strain [6], or other measurands [7]. To extract information a phase map is typically calculated first from the fringe pattern and then the required measurands and parameters are calculated [8].

For measuring large objects, different measurements are taken at different locations on the object and stitched together to form one image, which typically requires large overlap regions among the measurements. Information such as height, stress, strain, etc. can be obtained from this large stitched image without discontinuity. In cases where regions of interest (ROI) are small and discrete areas of the object, different strategies might be chosen. One strategy is based on scanning the whole object with the highest resolution followed by processing and stitching the acquired data. In this case, processed data outside the ROIs are discarded, wasting measurement time and effort. For example, suppose the part to be measured is an array of solder bumps deposited on a chip pad, and the solder bumps have to be carefully inspected. If there is a defect on one of the solder bumps, the location and details of the defect need to be identified. With this strategy, the profile of all solder bumps has to be acquired with the highest resolution, part by part, and then profiles stitched together from different parts to get one continuous profile. This approach is very time consuming and not efficient.

The other strategy is to take the high resolution measurements only of the ROIs. The difficulty here is that ROIs, have to be identified first and second, the relative location of the ROIs, with respect to a reference point on object, are unknown in general (if there is no overlap between ROIs).

To address these issues, a combination of multiscale measurement and stitching is proposed, namely a measurement with low resolution is taken, ROIs identified, and then measurements with high resolution are taken of the ROIs. This requires a strategy for inter-relating locations and magnification scales at different levels of zoom. Therefore the proposed technique offers measurements at different levels of zoom while stitches data at all zoom levels. Stitching means that the exact coordinates of all measurements at different magnifications are established relative to a common reference point.

To stitch data from different measurements, we use the fringe patterns from each measurement, which avoids error contributions that may originate from the phase map calculations. Also this technique speeds up the stitching process by avoiding phase map calculations prior to stitching. Hence, with this technique we can get detailed information (high resolution measurements) about the ROIs and their relative locations while saving time and calculation cost.

We have applied this technique to simulated moiré fringes [9]. In this paper, we report the application of this technique for fringe patterns from a laser interferometer and a projection moiré system, and show that this technique works for both fringe patterns and is capable of locating different ROIs over a range of magnifications.

2. Background

Laser interferometry is one of the techniques used for precision metrology [10]. It is used to measure a wide range of objects and surfaces [2]. There are different configurations for interferometers including the Michelson interferometer, the Mach-Zehnder interferometer, the Fizeau interferometer, etc. In all interferometers two or more waves interfere and the resultant interference pattern is used to calculate a wavefront change that results from transmission through or reflection from a test part of interest. The wavefront change, in turn, corresponds to the parameters and/or features of interest concerning the object under measurement. With laser interferometry, very precise measurements can be achieved down to sub-nanometer levels [11, 12].

Moiré technique is another optical method for form and surface measurements [2] [13–15]. It can be used to measure in-plane deformation and strain [16], as well as out-of-plane deformation and profilometry [17,18]. In-plane measurements are usually performed by attaching a grating to the object and then measuring the deformation of the grating. Out-of-plane measurements are based on illuminating the object with the structured light, usually a straight line grating, and measuring the deformation of reflected light from the object. In both cases the reflected light can be summed or multiplied by another grating to form a Moiré fringe pattern or the deformation can be directly measured. Moiré techniques are most common for form measurement and they can be used to realize measurements with uncertainties down to the micrometer level [19, 20].

Raw or intermediate output data from Moiré systems or laser interferometers is a fringe pattern that is processed to extract the desired measurand. For large objects these measurement systems usually have variable magnification capabilities that are typically accomplished with a zoom lens or different objective lenses. When reconfiguring the instrument for a measurement at a new magnification, the field of view does not necessarily stay centered and the value of the magnification (the new field of view) is only known approximately from the instrument settings. To quantitatively relate fine and coarse measurements, it is necessary to find the relative coordinates and value of the magnification with respect to a reference image that can be conveniently taken to be the measurement at the lowest magnification.. This can be accomplished using image registration.

Image registration is a technique to align two images from one scene [21]. These images can be taken with two different instruments, from two different angles, with two different levels of zoom, or at different times. One of the images is referred to as the reference image and the other one is the target image [21]. Image registration aligns the target image with the reference image. Mathematically speaking, the reference image (R) and the target image (T) belong to sets of d-dimensional images with d = 2; i.e. $R,T\in Img\left(d=2\right):=\left\{b:{ℝ}^2\to ℝ |b is 2-dimensional image\right\}$. Image registration minimizes the difference (or maximize the similarity) between the target and reference images. To quantify the difference (or similarity) of two images, a distance measure (similarity measure) has to be defined. The similarity measure used in this work is pattern intensity [22]. The pattern intensity measure for two images R and T is defined as

The pattern intensity operates on the difference between images. The average intensity of the captured fringe patterns at different zoom levels are different, and this is compensated for by multiplying one fringe pattern with parameter $\lambda$ and making the average intensity of both images equal. Parameter r determines the radius inside of which the difference comparison (calculation) is made. The parameter $\sigma$ determines the sharpness of the similarity measure. Smaller $\sigma$ means more sensitivity to the pixel differences and therefore a sharper similarity curve versus the difference between pixels. In our calculations values of $\sigma =10$ and r = 5 provide the best results for image registration of fringe patterns. The function $\phi$ is the transformation of the coordinates of the image. There are different transformations used to implement image registration [23-25]. If the image transformation includes only translation, scale, rotation and shear, it is called an affine transformation. Otherwise the transformation is called a non-rigid transformation. In the case of changing the zoom level, the transformation is dominated by translation and scaling only, so an affine transformation is sufficient to register the images.

The affine transformation used here is defined by $\phi \left(x\right)=Ax+b, A\in {ℝ}^{d\times d}$ with $\det \left(A\right)>0, b\in {ℝ}^d$ [26] where $\det \left(A\right)$ is determinant of matrix A. For the case of 2-d image transformation that includes translation and scaling, the transformation can be defined as

To summarize, the problem of 2-d image registration can be stated as follows [26]: Given a distance measure $\mathcal{D}:Img{(d=2)}^2\to ℝ$ and two images $R,T\in img(d)$, find a mapping $\phi :{ℝ}^d\to {ℝ}^d$ and a mapping $g:ℝ\to ℝ$ such that $\mathcal{D}\left(R,g\left(T\left(\phi \left(x,y\right)\right)\right)\right)$ is minimum.

In this work, the distance measure (similarity measure) is pattern intensity, transformation is an affine transformation with 2-d images of fringe patterns, and $g\left(x\right)=x$.

Now we present the simulation of the proposed technique, using image registration with the affine transformation and the pattern intensity distance measure.

Projection Moiré and laser interferometer are two systems simulated to evaluate the proposed technique. Figure 1 shows the schematics for the simulated systems. The projection Moiré (Fig. 1.a) consists of a light source and a grating to generate and project a series of parallel lines with specific spatial wavelength (fringe pattern), a lens system to form an image of the parallel fringe pattern on the object and a second lens system to image the distorted from the object interferes with an undistorted reference wavefront. The interference pattern is captured by a camera. The simulation was done in MATLAB. Parameters for projection Moiré are as follow: the angle between the camera and the normal to the object is 20°, and the angle between light source and normal to object is also 20°. The camera is 512 by 512 pixels. The pitch of the grating is 0.1 mm and the number of fringes at the lowest zoom level is 51.2.

 

Fig. 1 Schematics of simulated (a) projection Moiré and (b) laser interferometer systems.

Download Full Size | PDF

The electronic noise is considered as additive white Gaussian noise with an SNR of 20dB. The optics for projection and capturing the reflected light are telecentric lenses, i.e. illuminated and reflected rays are parallel.

The laser interferometer is simulated with the same camera as mentioned before. In addition, averaged exponential multiplicative speckle noise is added to the signal with a mean value that equals half the signal power. The laser wavelength is taken to be 633nm and additive white Gaussian noise with SNR of 20dB is added to simulate electronic noise.

The simulated object is taken to be a superposition of a rotationally symmetric cosine function with a period of 2 mm and an amplitude of 0.1mm (representing the large-scale form of the object), and a higher frequency additional structure is added that has a peak and valley shape with a period of 0.01mm and amplitude of 0.001 mm (representing ROI surface features). Figure 2 shows the object at zoom levels of 4 and 128. Note that at the lowest magnification (zoom level of 4), only the cosine shape of the object is recognizable. When the magnification is increased (zoom level of 128), the details surface features are clearly seen while the low spatial frequency cosine form is not.

 

Fig. 2 Simulated object at zoom levels of (a) 4 and (b) 128.

Download Full Size | PDF

With this defined object, simulations have been carried out for both the projection Moiré and the laser interferometer at different levels of zoom. Figure 3 shows the simulated fringe patterns from Moiré system at zoom levels of 1, 2, 8 and 16 and interferograms from laser interferometer at zoom levels of 64, 128, 256 and 512.

 

Fig. 3 Simulated fringe patterns and results of image registration. Images (a1), (a2), (b1) and (b2) are simulated fringe patterns from the projection moiré system at zoom levels of 1, 2, 8 and 16. Image (a3) is the scaled and translated version of (a2) (a2) by our proposed technique. It is located with respect to the coordinate system of image (a1). Image (b3) is the scaled and translated version of (b2), located with respect to the coordinate system of image (b1). Images (c1), (c2), (d1) and (d2) are simulated fringe patterns from the laser interferometer system at zoom levels of 64, 128, 256 and 512. Following the description above, image (c3) is the scaled and translated version of (c2), located with respect to the coordinate system of (c1). Similarly, image (d3) is the scaled and translated version of image (d2), located with respect to the coordinate system of (d1).

Download Full Size | PDF

For all simulated fringe patterns, we have assumed no translation error between fringe patterns at different zoom levels, without loss of generality, i.e. all fringe patterns have the same center in both X and Y directions for each zoom level (there is no drift in the system’s optical axis relative to the object). The goal is to find the amount of zoom (scale) and relative coordinate (translation) in both X and Y directions. Then, the translation should nominally be zero.

Figure 3 shows the results of image registration for the Moiré system and the laser interferometer, respectively. Simulation results are summarized in Table 1 . The error observed in the final results is dominated by the noise present in the fringe patterns, either electronic noise (projection moiré) or both electronic and speckle noise (laser interferometer).

Table 1. Image Registration Parameters for Simulated Fringe Patterns from Projection Moiré and Laser Interferometer

View Table | View all tables in this article

Since the camera has 512 pixels in each direction, 0.1% error in scale equals approximately half a pixel while 0.2% error corresponds to approximately one pixel error. Therefore, for fringes from the laser interferometer, the error is within one pixel and for the Moiré system the error is about one pixel. Hence, based on simulation results the proposed technique can find the scale and translation of different fringe patterns within one pixel.

3. Experiment

Figure 4 shows the experimental setup for the measurement system. The system is a combined projection Moiré and laser interferometer and is the combination of schematics of Fig. 1.a and 1.b. The zoom level is determined by the combination of two zoom lenses. For the projection Moiré, a projector is used as a combined light source and grating. The laser interferometer is a Michelson interferometer with a HeNe laser source.

 

Fig. 4 Experimental Setup. The setup is a combination of Moiré projection and laser interferometer (the schematic of the system is shown in Fig. 1). The projector relays an image of a parallel fringe pattern onto the object and the distorted fringe pattern is captured by the camera (SMX-M73 3 megapixel CMOS camera from SUMIX). One laser beam (laser beam on the top) illuminates the object and reflects back from the object, interfering with the reference laser beam (laser beam on the bottom). This interference pattern is captured by the camera. The lens systems consist of three zoom lenses (One 12.5X Precision Zoom Lens and two VZM 450 Zoom Imaging Lens all from Edmund Optics) and a 1X compact telecentric lens from Edmund Optics.

Download Full Size | PDF

Projection Moiré is used for the coarse (form) measurement, i.e. low levels of zoom, while the system switches to the laser interferometer for fine measurements; i.e. high level of zoom.

The object is coated array of solder bumps on a substrate. As mentioned before, projection Moiré is used to measure the form of the object, and the laser interferometer is used for measuring the fine surface features of the object. The fine feature measurement is for ROIs and not the whole surface of the object. In this case, the ROIs would be selected individual solder bumps. Therefore, the measurement procedure is as follows: first, the form of theobject is acquired, second the form measurement is used to identify ROIs, and finally the relative location and features in the ROIs are measured with high resolution. The details of converting the fringe patterns into phase maps and object profiles are not discussed here as it is not the focus of this work. These procedures are well documented [2,4,8]. Our focus is to show that the proposed technique can be used to find the relative location and magnification of the fringe patterns at different levels of zoom.

Figure 5 shows the fringe patterns acquired from the projection Moiré and the laser interferometer at different levels of zoom. Our image registration technique has been applied to these fringe patterns to calculate the parameters required finding the relative scale and the relative coordinates at each zoom. Figure 6 shows the results of image registration. The results show that the technique can relate the location of the fringe pattern at the highest level of zoom to the location of fringe pattern at the lowest level of zoom within fringe patterns of projection Moiré and laser interferometer, and spanning the range from the Moiré to the laserinterferometer measurements (Spanning from projection Moiré to the laser interferometer is explained later in this section). Since this can be done for all fringe patterns at the highest level of zoom, the relative location of the fringe pattern at the highest level of zoom can be determined with respect to a reference point at the lowest level of zoom. Table 2 shows the calculated parameters from the proposed technique.

 

Fig. 5 Fringe patterns from the solder bump array sample. Images (a1)-(a4) are from the projection Moiré system at approximate zoom levels of 1, 1.5, 2.5 and 3.5. Images (b1)-(b4) are from the laser interferometer system at approximate zoom levels of 8, 16, 24, and 48.

Download Full Size | PDF

 

Fig. 6 Fringe patterns and results of image registration. Images (a1), (a2), (b1), (b2), (c1) and (c2) are fringe patterns from the projection Moiré at zoom levels of 1, 1.5, 1.5, 2.5, 2.5 and 3.5. Image (a3) is scaled and translated version of (a2) by our proposed technique. It is located with respect to the coordinate system of (a1). Image (b3) is scaled and translated version of image (b2) located with respect to the coordinate system of (b1). Image (c3) is scaled and translated version of (c2) by our proposed technique. Image (c3) is located with respect to the coordinate system of (c1). Images (d1), (d2), (e1), (e2), (f1) and (f2) are fringe patterns from laser interferometer at zoom levels of 8, 16, 16, 24, 24 and 48. Analogous to above description, image (d3) is scaled and translated version of (d2), located with respect to the coordinate system of (d1). Likewise, image (e3) is scaled and translated version of (d2), located with respect to coordinate system of (e1). Image (f3), as well, is scaled and translated form of (f2), located with respect to the coordinate system of (f1).

Download Full Size | PDF

Table 2. Image Registration Parameters for Fringe Patterns from Projection Moiré and Laser Interferometer

View Table | View all tables in this article

Since the change of zoom level is not automated, the values experimentally known are only approximate values. Our technique determines the magnifications exactly, as presented in Table 2. The shift in X and Y between the different fringe patterns is also determined by the proposed technique. The nonzero shifts in both X and Y directions are dominated by errors in the optical setup and alignment of the optical elements and the object. The relationship between the fringe pattern location from the projection Moiré and the interferogram location from the laser interferometer can be determined if first a single camera is used for both projection Moiré and the laser interferometer, and second when we switch from projection Moiré to the laser interferometer, the zoom level does not change, i.e. the camera captures the same part of the object. Otherwise, the relative location can be found by first converting the fringe patterns into height profiles and then finding the relative location with the calculated height profile from the projection Moiré and the laser interferometer.

Figure 7 shows the overall process for a solder bump array when there is an overlap between projection Moiré and laser interferometer. Since there is overlap between fringe pattern acquired from projection Moiré and laser interferometer at zoom level 8, those two sets of registered fringe pattern can be combined to result in a continuous image from zoom level 1 to 48 (Fig. 7(c)).

 

Fig. 7 Fringe patterns and results of image registration where there is an overlap between fringes acquired from projection Moiré and laser interferometer. Image (a) is four registered fringe pattern acquired from laser interferometer at zoom levels of 8, 16, 32 and 48. Image (b) is four registered fringe pattern acquired from projection Moiré at zoom levels 1, 2, 4 and 8. Image (c) shows combined registered images from projection Moiré and laser interferometer.

Download Full Size | PDF

As mentioned before, applying image registration to fringe patterns, rather than the phase map or profile of object, results in better and more precise scale and location finding. The primary reason for this is that the registration step based on the fringe patterns does not include errors that might arise from the phase map or object profile calculation.

4. Conclusion

We have proposed a technique to generate a continuous image from different fringe patterns at different levels of magnification (zoom) and within same level of zoom. The technique is a combination of multiscale measurement and stitching, namely measuring data at different levels of zoom for regions of interest (RIOs) and then stitching the data onto a common surface coordinate system based on image registration techniques. Simulation results show that the proposed technique is capable of stitching and registering data to within one pixel. Experiments have been conducted to demonstrate the applicability for fringe patterns obtained from Moiré systems and laser interferometers. The proposed technique can be applied to other multi-scale measurement instruments. Calculations for error analysis of the proposed technique have been presented.

Results can be improved by using noise reduction techniques before applying the proposed technique and using adaptive image registration appropriate for a specific measurement instrument. The authors are investigating different transformations for fast image registration for different fringe patterns and instruments.