Subpixel digital image correlation has been applied to microscope images to analyze surface deformation. Nonintegral pixel shifting and successive approximation are used to calculate the subpixel element of the sample displacement without introducing systematic interpolation errors. Although in-plane displacement precision of better than 2% of a pixel, or < 15 nm at x10 magnification, is shown to be achievable, the use of even moderate numerical aperture microscope objectives render the technique sensitive to errors or variations in sample focusing. The magnitude of this effect is determined experimentally and a focus compensation method is described and demonstrated.
© 2002 Optical Society of America
Digital image correlation is a proven technique for measuring surface displacement and strain [1–7]. Many variants of correlation based techniques exist in the literature, for example electronic speckle photography [2,3], digital image/speckle correlation  and microscopic deformation analysis using correlation (MicroDAC or MDAC) [6,7]. All these techniques estimate surface displacement by calculating the position of maximum cross-correlation between undeformed and deformed image regions (subimages). The potential advantages of correlation based MDAC (the term MDAC is used here to denote the complete operation of image acquisition and processing) over competing techniques such as speckle, moiré interferometry  and stereo microscopy are that image acquisition can be performed by a conventional brightfield microscope over a wide range of optical magnifications using eye-safe, incoherent sample illumination, and sample preparation is straightforward. Our requirement was for instrumentation capable of analyzing thermally induced stress in microscopic sections through packaged microelectronic devices in a manufacturing environment. Although microscopic moiré has been used for this purpose , MDAC uses simple, stable optics and requires a less skilled operator.
Thermal deformation of microelectronic devices at temperatures of interest results in surface displacements in the nanometer to micrometer range [5–8]. Measurement requires microscopic image acquisition and an algorithm capable of subpixel resolution over a wide range of sample textures and contrast levels. In addition, the limited depth of focus of microscope optics has to be considered. Although many subpixel correlation algorithms are described in the literature, most are based on intensity interpolation [e.g. 1,5] or curve fitting to the discrete correlation peak . These approaches have been shown to introduce significant displacement and sample dependent systematic error [3–5]. In  it is demonstrated that a careful choice of interpolator can render these errors negligible for a particular sample, but not across a wide range of samples.
Section 2 of this paper describes an optical setup used for microscopic image acquisition, and an accurate subpixel displacement algorithm which iteratively applies the Fourier shift theorem to avoid the use of intensity interpolation or curve fitting. Experimental results from flat, well focused samples are presented. In Sect. 3 it is shown that small variations in sample focusing introduce significant error to the displacement measurement at typical magnifications. A simple algorithm to calculate a focus corrected in-plane displacement without optical hardware modification is described and applied to a tilted sample.
2. Microscopic deformation analysis using correlation
Images are acquired with the set-up illustrated in Fig. 1. Illumination source S is imaged into the back focal plane of microscope objective MO by relay lenses L2 and L3. The source is a fiber coupled illuminator, bandpass filtered at 500 nm with FWHM of 40 nm. Tube lens L1 together with the microscope objective [either Zeiss Epiplan, numerical aperture 0.13 (nominally x5) and NA 0.25 (x10)] form a magnified image onto the CCD camera (8-bit monochrome, 768 × 576 pixels of 11 × 11 μm). The object sampling corresponds to approximately 1.2 μm at x5 and 600 nm at x10.
Displacement measurement begins by acquiring at least two images, one of the undeformed sample (I1) and one after deformation (I2). Samples typically require refocusing after deformation because of the limited depth of focus of the objective lens. A motorized stage under computer control is used to ensure accurate and consistent auto-focus (Sect. 3).
As stated in Sect. 1, MDAC requires a precise image correlation algorithm. Very high image magnifications can be used to relax the precision requirement, but the reduced field of view, depth of focus and working distance of high NA microscope objectives limits this approach. The algorithm described here iteratively applies the Fourier shift theorem to calculate unbiased subimage cross-correlation coefficients at many subpixel displacements. This approach, while computationally intensive, avoids image interpolation or curve fitting to the correlation peak and can be applied to arbitrary sample textures and image contrasts. Subpixel subimage shifting allows accurate cross-correlation coefficients to be calculated for fully overlapping subimages. Sjödahl has shown that subpixel subimage shifting combined with curve fitting to the correlation peak could be used to reduce measurement error when compared to curve fitting alone . Here, we iteratively apply subpixel shifting to maximze the accuracy of direct cross-correlation calculations.
The Fourier shift theorem states that if F(s) is the Fourier transform of f(x), then shifted f(x - a) has the Fourier transform e -i2πas F(s) . Applying the discrete 2D shift theorem allows displacement of a subimage by an arbitrary subpixel amount. Subimages are transformed with an optimized fast Fourier transform , multiplied by a linear phase gradient in the frequency domain, then inverse transformed to generate a shifted version. The cyclic nature of discrete Fourier transforms causes image data which is shifted out from one side of the subimage to ‘wrap’ around to the opposite side . To avoid this effect the shift theorem is applied to a slightly oversized subimage. The shifted subimage is then cropped to the correct size for cross-correlation.
Our image correlation algorithm proceeds by selecting subimage S1 from image I1. Typical subimage dimensions are 8 ≤ m ≤ 64 , where m is the subimage lateral dimension in pixels. Before subpixel analysis is performed, a subimage S2 from image I2 must be located to integer pixel precision (within ± 0.5 pixels). If the sample deformation is small then the position of S1 will suffice as an initial estimate of the position of S2, but deformations of greater than about m/3 will initially require an increased m to ensure sufficient subimage overlap. A cross-correlation of subimages S1 and S2 is performed and the position of the correlation peak, (u1, v1), is calculated. The coordinate (u1, v1) should indicate the subimage displacement to integer pixel precision, but the cyclic nature of discrete Fourier transforms causes estimates of (u1, v1) to be biased towards (0, 0). To eliminate this bias, if the initial estimate of (u1,v1) ≠ (0,0), S2 is shifted by (-u1, -v1) and the cross-correlation is repeated [2, 3].
Next a successive approximation routine is used to make increasingly precise subpixel displacement estimates. The algorithm proceeds by accurately sampling the normalized cross-correlation coefficient for a range of subpixel displacements. The value of these samples is used to narrow the search area, and finer sampling is performed. In the first iteration (n = 1), subimage S2 is shifted by (ue, ve) = (±0.5, ±0.5), (0, ±0.5) and (±0.5, 0) pixels where (ue, ve) are the estimated displacements in the x and y directions respectively. The normalized cross-correlation of S1 and each of the 8 nonintegral shifted versions of S2 is calculated. The position of the highest of the these correlation coefficients indicates the true displacement to within 0.25 or 2-(n+1) pixels. Its value and position is stored, n is incremented, and successive approximation continues by subpixel shifting S2 by half this amount, (ue, ve) = (±2-n, ±2-n), (±2-n, 0) and (±2-n, 0) about the position of the highest coefficient yet obtained. Iteration is continued until a convergence condition is satisfied. S2 is subpixel shifted prior to each cross-correlation to maximize subimage overlap and avoid displacement dependent estimation bias reported in [2,3]. The use of a normalized cross-correlation is computationally expensive but ensures that the algorithm is unaffected by variations in sample reflectivity.
A disadvantage of this implementation of subpixel MDAC is the computational requirement which currently precludes video rate displacement calculation. However, data acquisition rates are limited only by the microscope camera, and a more efficient minimization procedure such as Levenberg-Marquardt  could be applied. Alternately, artificial neural networks can be employed to speed up displacement calculation with some loss of precision . Our application performs off-line analysis and can process around 800 subimages/minute with m = 32 using a 1.8 GHz Intel microprocessor.
To determine the baseline performance of the MDAC image acquisition and software, a correctly focused, nominally flat, ground glass sample was translated in 51 submicron steps. An image was acquired at each step position and the subpixel algorithm described above was used to calculate the displacement between the centrally positioned image and each of the others. The step size was chosen such that a wide range of subpixel displacements were analyzed. A ground glass sample was chosen as its fine spatial structure, flat substrate and reasonable image contrast make it suitable for correlation techniques. The trade-off between spatial resolution and precision of the MDAC technique was investigated for subimage sizes m from 8 to 64 pixels and nominal objective magnifications of x5 and x10 (Fig. 2b). The error for each translated image was calculated as the standard deviation of the MDAC displacement measurements, using between 400 and 20,000 subimages (for m = 64 and m = 8 respectively) covering the entire field of view. The data plotted is the mean error across the 50 images.
Good precision is obtained with this sample, especially for higher values of m. At m = 40, the error is around 2% of a pixel, or 24 and 12 nm at x5 and x10 magnification. The absolute displacement resolution is inversely proportional to NA at these magnifications, implying that the subimage correlation peaks have similar widths in each case. The micrographs in Fig. 2b illustrate that increasing image magnification on a finely structured sample such as ground glass reveals higher spatial frequencies. The relative performance of MDAC is maintained and the absolute performance is inversely proportional to NA for such samples.
Figure 3b shows corresponding results from two samples of practical interest to us. The images were acquired from different regions of a ground section through a microelectronic device. The upper micrographs (Fig. 3a) are an area of mold compound device packaging. The lower pair are of the silicon die. The results for the silicon region are quantitatively similar to the ground glass. The absolute precision on the silicon sample is increased by the use of higher image magnifications, though not by quite as much as ground glass. The packaging material however, does not benefit greatly from the increased NA. The micrograph at x10 appears less finely structured and this has caused the width of correlation peak in pixels to increase. To summarize, for samples which are correctly focused across the field of view, the use of SA nonintegral pixel shifting and microscope optics can achieve precision of better than 24 nm at x5 (12 nm at x10) for subimages of m ≥ 40 and less than 60 nm (40 nm) for m = 16. Absolute precision is inversely proportional to NA for samples possessing spatial structure which is finer than the resolving power of the optics. However in the absence of higher object spatial frequencies, increasing the imaging NA only reduces the field of view.
3. Defocus errors in MDAC and their correction
We have found that the displacement precision reported above is difficult to achieve with thermally deformed microelectronic devices. Even if the device is carefully ground and accurately aligned perpendicular to the optical axis, ensuring sharp focus across the entire field of view at ambient temperature, the thermal load inevitably causes both in-plane and out-of-plane deformation. This can produce spatially dependent changes in image focus which introduce measurement error . Automated focusing can provide a partial solution. We use a routine which maximizes the image content of a band of mid spatial frequencies while varying sample focus as follows. A through focus series of images are acquired and Fourier transformed to allow analysis of their relative spatial frequency content. If f max is the highest spatial frequency represented in the image, we have found experimentally that selecting the image with the highest spectral content within the band 3f max/8 and f max/2 allows optimized and repeatable sample positioning, but cannot fully compensate for thermally induced variations in sample height within a single image.
The effect of sample defocus on measurement error is plotted in Fig. 4a for a ground glass sample at x10 magnification, with m = 32. The error was calculated by acquiring a z-stack of images at 0.5 μm increments and applying MDAC to the correctly focused image and each of the defocused stack. We find that in our optical system at NA 0.25, that focus induced errors dominate when variations of sample surface height exceed about 2 μm. This limits the effectiveness of image correlation for cracked or stepped samples.
MDAC results on a tilted sample more fully illustrate the effect of out-of-plane deformation. The ground glass sample was first imaged perpendicular to the optical axis at NA 0.25, then tilted by 2° about the y-axis and re-imaged. In the absence of focus induced error, a tilt of 2° should result in an apparent compression along the x-axis of 270 nm. The result of applying subpixel image correlation to the two images is shown in Fig. 4b. It can be seen that the error is not random in our system, but due to very small, focus dependent variations in magnification. The errors are generally low in the correctly focused central region, but negative defocus (left-hand side of the plot) causes a slight decrease in magnification. The direction of the errors caused by the demagnification tend towards the center of the image, causing them to partially add to the apparent object compression caused by the tilt. Positive defocus has the opposite effect, but the increase in magnification is partly canceled by the perspective change, giving the errors a randomized appearance. The standard deviation of the calculated displacements for this pair of images is 110 nm.
Precise telecentric optical alignment can minimize the dependence of magnification on focus, but is not simple to implement across a range of microscope objectives. Instead, we modified the wide field auto-focus routine described above to operate on subimages. A z-stack of images of the flat and tilted sample were acquired and spectral image analysis was used to select only sharply focused subimages prior to applying the subpixel image correlation. This method can suppress the distortion and decorrelation caused by defocus while also estimating the surface profile of both the undeformed and deformed sample. The surface profiles calculated by this technique are in Fig. 5.
The results of this focus compensation are shown in Fig. 6a. The dominant visual feature of the plot is now the apparent compression caused by the tilt. This result compares very favorably with that in Fig. 4. The standard deviation of the measurement was calculated to be 68 nm by using the profiles of Fig. 5 to calculate and remove the image compression effect. Unlike the systematic errors visible in Fig. 4, the residual errors are quite random and could be reduced by filtering (Fig. 6b). For strain analysis, the displacement results must be smoothed and differentiated. A significant point is that the apparent strain caused by local defocus can dominate precise measurements in semiconductor packages.
Microscope optics and subpixel digital image correlation can achieve high displacement precision on flat, well focused samples. Precision scales with NA as long as sufficient surface texture is present in the sample. A sample independent image correlation algorithm which applies the Fourier shift theorem and successive approximation has been applied to microscope images and displacement precision of 20 nm has been demonstrated at NA 0.25. However, out-of-plane deformations as small as 10 % of the depth of focus (typically several microns or less) can introduce significant error to in-plane measurement by slightly defocusing the image. A focus compensation method using standard microscope optics has been described and used to analyze a rigid body tilt. It has been shown that this method can suppress defocus errors for slowly varying or static samples at the expense of image acquisition bandwidth.
The precision of image correlation techniques is highly sample dependent, making comparison with published results difficult. We find in our experiments that the techniques described in this paper perform well over a wide range of samples, whereas interpolation and curve fitting methods, while less computationally intensive, require optimization for each sample. We find that the increased computational expense is justified by the quality of the results. Future work will more thoroughly analyze the performance of this technique and apply it to thermal strain measurement in microelectronic devices.
This work was partially supported by the European Union under the Fifth Framework Program for RTD in the specific program Competitive and Sustainable Growth, project number GRD1-1999-10839 OPTIMISM under contract number G6RD-CT99-0179.
References and links
1. H. A. Bruck, S. R. McNeill, M. A. Sutton, and W. H. Peters III, “Digital image correlation using Newton-Raphson method of partial differential correction,” Exp. Mech. 29, 261–267 (1989). [CrossRef]
4. Z. Sun, J. S. Lyons, and S. R. McNeill, “Measuring microscopic deformations with digital image correlation,” Opt. Laser Eng. 27, 409–428 (1997). [CrossRef]
5. P. Zhou and K. E. Goodson, “Subpixel displacement and deformation gradient measurement using digital image/speckle correlation (DISC),” Opt. Eng. 40, 1613–1620 (2001). [CrossRef]
6. D. Vogel, V. Grosser, A. Schubert, and B. Michel, “MicroDAC strain measurements for electronics packaging structure,” Opt. Laser Eng. 36, 195–211 (2001). [CrossRef]
7. M. C. Pitter, C. W. See, and M. G. Somekh, “Subpixel microscopic deformation analysis using correlation and artificial neural networks,” Opt. Express 8, 322–327 (2001), http://www.opticsexpress.org/abstract.cfm?URI=OPEX-8-6-322. [CrossRef] [PubMed]
8. B. Han, “Recent advances of moiré and microscopic moiré interferometry for thermal deformation analyses of microelectronic devices,” Exp. Mech. 38, 278–288 (1998).
9. R. N. Bracewell, The Fourier Transform and its Applications (McGraw-Hill, Singapore, 1986), Chap. 6.
10. M. Frigo and S. G. Johnson, “FFTW: an adaptive software architecture for the FFT,” in Proc. ICASSP 3 (1998), pp. 1381–1384, http://www.fftw.org/fftw-paper-icassp.pdf.
11. M. J. Kidger, “Use of the Levenberg-Marquardt (damped least-squares) optimization method in lens design,” Opt. Eng. 32, 1731–1739 (1993). [CrossRef]