1. Introduction

The pixel count of a sampled imaging system is often interpreted to be its ability to capture finely spaced details. Nevertheless, the addressability of a capturing device does not necessarily correspond to the pixel count of the output image. For example, a single-chip camera usually has a Bayer color filter array on its image sensor, each element of which can record only one color (of red, green, or blue). Some cameras do not even have addressable photoelements aligned with the square sampling grid of the output image format. Although a three-chip camera has three separate image sensors to capture filtered red, green, and blue color images separately using a trichroic beam splitter, in some cameras the image sensors have fewer photoelements than the pixel count defined in the output image format, obscuring the pixels’ definition. Furthermore, along with the optical performance of the lens, the photoelement size characterizes the spatial resolution.

The modulation transfer function (MTF) is a reliable indicator of the spatial performance of an optical system [1]. Analogous to its use in optical systems, the MTF can be used to characterize the spatial resolution of a sampled imaging system [2]. Based on the convolution theorem, the MTF of the overall system can be described as the product of all the individual MTFs of its components when the given imaging system is linear and shift invariant [3]. To preserve the convenience of the transfer function approach when characterizing a sampled imaging system, it is assumed that the object being imaged has spatial frequency components with random phases that are distributed uniformly relative to the sampling sites [4,5].

An effective metrological method should be independent of the device architecture and technologically neutral. Therefore, in metrology, the pixel is assumed to be a point, rather than the smallest addressable element or physical pixel of the device, or to be the interval of the square sampling grid defined in the image format [6]. Thus, the unit cycles/pixel is used for the spatial frequency of a sampled imaging system with the Nyquist and sampling frequencies corresponding to 0.5 cycles/pixel and 1 cycle/pixel, respectively.

In the conventional edge-based method specified in the ISO 12233 standard [7], the spatial frequency response is measured to approximate the MTF as a function of the horizontal or vertical spatial frequency by analyzing a near-vertical or near-horizontal bi-tonal edge image, respectively. The MTF can be estimated from a 1D supersampled edge gradient obtained by projecting the pixels from a region of interest (ROI) in the image into a linear array of subpixel-wide bins that align horizontally or vertically along the square sampling grid of the image.

Modified edge-based methods are expected to apply to various edge orientations other than near-vertical or near-horizontal for the measurement of tangential and sagittal MTFs. Roland [8] proposed an ad hoc correction to the MTF derived using the ISO 12233 edge-based method. In the correction process, the spatial frequency is scaled linearly based on the estimated slant angle to yield the MTF as a function of the spatial frequency in an orientation perpendicular to the edge. Before Roland, the author of this paper and others had proposed a method to streamline multidirectional MTF measurements without ad hoc scaling of the spatial frequency [9,10]. The author et al. also developed a real-time multidirectional MTF measurement system using a slanted starburst chart, visualizing the result using a contour plot that enabled observation of the anisotropy resulting from the performance and conditions of the camera and lens (e.g., the misalignment of the optical axes of the lenses and image sensor) as well as the pixel arrangement of the image sensor and image processing (e.g., Bayer color filter array demosaicing) [11]. The system averages multiple image frames to achieve noise-robust slant angle and MTF estimates, assuming that the slant angle is maintained in each measurement instance. Based on this assumption, the system creates a look-up table (LUT) to map the pixels in an ROI to the bins, the LUT being applied to the new input frames in succession for binning without re-estimating the slant angle and projection paths. This reduces the computational overhead significantly and enables real-time MTF measurement while focusing even when the edge position moves due to focus breathing.

The essence of the edge-based method is the slant-angle-dependent projection of pixels in a square array into a 1D array of subpixel-wide bins. In the ISO 12233 edge-based method, an oversampling ratio of four is used. However, the rationale behind the use of bins that are a quarter of a pixel wide is unclear. The author of this paper demonstrated that the accuracy and precision of the edge-based method can be improved with higher oversampling ratios, especially for critical angles that have tangents of simple fractions [10]. However, a higher oversampling ratio needs more computational overhead. Furthermore, the accuracy and precision of the MTF estimates at spatial frequencies higher than the Nyquist frequency—which is important for evaluating aliasing artifacts as well as estimating the pixel pitch and active pixel size of the image sensor [12]—has not been adequately discussed.

Van den Bergh [13] showed that a lack of spacing uniformity of the samples projected on the bin array axis can result in significant errors of the MTF estimates when a simplistic interpolation is utilized. He proposed a method that reconstructs an 8× oversampled regularly-spaced edge-spread function (ESF) from the samples with irregular spacing using either a local polynomial or kernel-based interpolation. In this method, a central gradient part is determined, a weighting kernel being applied to the samples to estimate the values of the ESF at the midpoints of the bins within the gradient part. Outside the gradient part, the samples in each bin are simply averaged, and a low-pass filter is applied to the reconstructed ESF tails to improve robustness against noise. Both the polynomial and kernel-based implementations accept a parameter that controls the effective width of the support of their respective weighting kernels. For each implementation, Van den Bergh optimized the parameter using a set of simulated edge images at several slant angles with known MTFs to maintain the balance between robustness and accuracy of the MTF estimates, demonstrating reduced edge-orientation-dependent errors over the spatial frequency range from 0 to 1 cycle/pixel. However, the prerequisite parameter optimization is sophisticated, and interpolation accompanying determination of the gradient part of the ESF in each frame compromises real-time measurement performance.

In this study, I demonstrate the cause of errors in MTF estimates versus critical angles using geometric diagrams and propose a simple method to significantly improve the accuracy and precision using a variable oversampling ratio dependent on the slant angle with neither local polynomial interpolation nor kernel-based interpolation, achieving a computational overhead comparable to that of the ISO 12233 edge-based method. The demonstration is limited to analysis using synthesized edge images to cover the theoretical aspects of the proposed method as the principle of the proposed method is fundamental and effective regardless of ideal simulations or real MTF measurements.

2. Conventional edge-based methods

2.1 ISO 12233 edge-based method (ISO method)

The ISO 12233 edge-based method [7] is intended to characterize the spatial resolution of a 2D sampled imaging system that responds to light in proportion to its intensity [14,15]. Using this method, a bi-tonal edge chart is captured at a slant angle with respect to the square sampling grid of the image, the edge gradient being measured at various phases relative to the sampling sites. The ISO 12233 standard charts have near-vertical edges with a slant angle of 5°. Figure 1 shows an illustration of such an edge chart. A vertically oriented rectangular ROI is selected to enclose the near-vertical edge such that the edge passes through the short sides of the ROI, the slant angle, θ, being measured in a positive clockwise direction from the vertical orientation. After estimating the slant angle, the pixels in the ROI are projected in parallel to the estimated edge onto the horizontal axis, which is divided into subpixel-wide bins, as shown in Fig. 2(a). Here, n_bin is defined as the number of bins per pixel interval or oversampling ratio, each bin being 1/n_bin wide. Using the ISO method, n_bin is specified to be four. Each pixel treated as a metrological point is placed in a single bin after projection: it should be noted that the intra- and inter-bin distributions must be random and uniform to preserve the convenience of the transfer function approach for the sampled imaging system, as mentioned in the Introduction. The values of the pixels collected in each bin are averaged to generate a 4× supersampled 1D ESF. This oversampling avoids artifacts of signal aliasing on the MTF estimates. The derivative of the ESF yields the line spread function (LSF), and after applying a Hamming window to it for apodization, a discrete Fourier transform is performed. The modulus of the corresponding optical transfer function is then normalized to unity at zero frequency, and the MTF is estimated over a range of horizontal spatial frequencies beyond the Nyquist frequency. The slant angle must be small so that the resultant MTF can approximate the MTF as a function of the horizontal spatial frequency. To measure the MTF as a function of the vertical spatial frequency, a horizontally oriented rectangular ROI is selected to enclose a near-horizontal edge, the ROI image being rotated by 90° before the edge analysis is performed.

 

Fig. 1. Slanted edges of black square enclosed by an ROI (translucent orange).

Download Full Size | PDF

 

Fig. 2. Geometric diagram of binning using edge-based methods (n_bin = 4): (a) ISO method, (b) ISO-cosine method, and (c) OMNI method. For simplicity and illustration, a small ROI of 4 (w) × 6 (h) pixels and slant angle θ of 10° are used. The black dots indicate (metrological point) pixels, the squares representing the physical photoelements and their gray levels. The blue line segments indicate the projection paths from the pixels at the sampling grid to the positions where they land.

Download Full Size | PDF

2.2 ISO method with ad hoc spatial frequency scaling (ISO-cosine method)

To accommodate an MTF measurement using a slanted edge other than a near-vertical or near-horizontal edge, Roland proposed an ad hoc MTF correction to the ISO method [8]. The correction linearly scales the horizontal spatial frequency, ξ_x, of the MTF based on the estimated slant angle, θ (−45° ≤ θ ≤ 45°), thereby yielding the MTF as a function of the spatial frequency, ξ$_{\bot}$, in an orientation perpendicular to the edge. The binning process of the ISO method using a horizontal bin array is equivalent to that using a linear bin array perpendicular to the edge with a bin width of (1/n_bin) cos θ pixels, as shown in Fig. 2(b). The equivalent oversampling ratio is a variable of n_bin sec θ. Thus, as θ approaches ±45°, the oversampling ratio increases. As a result, in the ISO method, the supersampled ESF obtained from the horizontal bin array appears shallow, resulting in an underestimation of the MTF. To compensate for this and to obtain a corrected MTF(ξ$_{\bot}$), ξ_x values of the resultant MTF(ξ_x) are replaced by ξ_x sec θ, the ISO method with ad hoc spatial frequency scaling being referred to as the ISO-cosine method. (It should be noted that Eq. (2) in Ref. 8 (for correcting the LSF) should be LSF_corr(x) = LSF(x sec θ), not LSF_corr(x) = LSF(x cos θ).)

2.3 Method using a fixed-wide bin array perpendicular to the edge (OMNI method)

In the modified method [10]—referred to here as the OMNI method—a 1/n_bin-wide bin array is set perpendicular to the edge to accommodate the MTF measurement for slant angles other than the near-vertical or near-horizontal orientations. Figure 2(c) illustrates the binning process in the OMNI method when n_bin = 4. No ad hoc spatial frequency scaling is required using this method. Some modifications are incorporated to improve the measurement accuracy and precision. A Tukey window is applied to the LSF for apodization, eliminating leakage errors more effectively than the Hamming window applied in the ISO method. The OMNI method compensates for the underestimation of the MTF of sinc(2ξ$_{\bot}$/n_bin), which is caused by a 3-tap difference filter applied to the ESF to approximate its derivative [16,17]. The method also compensates for the underestimation of the MTF of sinc(ξ$_{\bot}$/n_bin) caused by the bin-width averaging filter in the binning process [16,17]. Finally, the OMNI method uses multiple bin arrays of different phases and averages the MTF results dependent on the binning phase to improve the accuracy and precision of the MTF estimates.

3. Fundamental difference between the ISO-cosine and OMNI methods

The fundamental difference between the ISO-cosine and OMNI methods lies in the bin width. Other differences are only apparent since the MTF corrections, apodization using a Tukey window, and multi-phase binning employed in the OMNI method are applicable to the ISO-cosine method. Figures 3(a) and 3(b) compare the binning processes of the two methods when tan θ = 1/4. Using the ISO-cosine method, the projection paths of the pixels are aligned with the bin intervals. Conversely, using the OMNI method, the interval between the projection paths is slightly smaller than that of the bin interval, the pixels landing at slightly different positions bin-to-bin. Using the ISO-cosine method, however, periodic deviations can be caused by misalignments between the projection paths of the pixels in each column and the bin array. When tan θ = 1/4, the pixel interval of each column is aligned with the bin interval simply because tan θ matches the bin width of 1/n_bin. By contrast, Fig. 3(c) shows an example when tan θ = 1/3—that is, while the projection paths of the pixels in each row align with the bin array, the bin width is narrower than the interval of the projection paths of the pixels in each column, leaving one out of every four bins empty. Such periodic intra- and inter-bin deviations from the original sampling positions are unwanted artifacts which do not meet the required assumption of random phases with respect to the sampling sites of the imaging system, resulting in errors in the MTF estimates.

 

Fig. 3. Geometric diagram of binning using edge-based methods (n_bin = 4): (a) ISO-cosine method when tan θ = 1/4, (b) OMNI method when tan θ = 1/4, and (c) ISO-cosine method when tan θ = 1/3. Refer to Fig. 2 for the details of the graphics.

Download Full Size | PDF

It has been claimed that in the ISO method the fixed oversampling ratio (n_bin) of four is the best choice for reducing noise and aliasing [18,19,20]. However, the analysis behind this claim is limited in terms of variations of the binning phase and edge angle. The author of this paper has shown that the OMNI method was able to improve the precision of MTF estimates by using higher oversampling ratios of 8, 16, and 32 [10]. However, misalignments between the pixel projection paths and the bin array in the OMNI method were not adequately examined.

4. Proposed method with a variable oversampling ratio (OMNI-sine method)

I propose a new method—called OMNI-sine—which modifies the OMNI method by adjusting the bin width based on the slant angle. Figure 4(a) shows the projection paths of the pixels in a row and column of an ROI for design considerations. The sampling interval in each row of an ROI always aligns with the bin array at intervals of cos θ pixels, where θ ranges from −45° to 45°. (If the interval is quartered, the projection corresponds to that of the ISO-cosine method.) The sampling interval of each column of the ROI, on the other hand, aligns with the bin array at intervals of |tan θ| pixels. If the bin intervals are set to the product of cos θ and |tan θ| (= |sin θ|), the bin array is simultaneously aligned with the projection paths of the pixels of each row and column. Figures 4(b) and 4(c) show examples of the binning for tan θ = 1/4 and 1/3, respectively, the overlapping projection paths aligning with the bin array. If the slant angle θ approaches 0°, however, the variable bin width also approaches zero—that is, the variable oversampling ratio approaches infinity. Therefore, the bin width is limited to vary around 1/n_bin to reduce any possible increase of empty bins and achieve a computational overhead comparable to the ISO method over the full range of slant angles.

 

Fig. 4. Geometric diagram of binning using the OMNI-sine method: (a) (cos θ)-pixel-wide intervals to maintain the pixel interval in each row and |tan θ|-pixel-wide intervals to maintain the pixel interval in each column, and (b, c) (sin θ)-pixel-wide bin intervals for tan θ = 1/4 and tan θ = 1/3, respectively. Refer to Fig. 2 for the details of the graphics.

Download Full Size | PDF

Considering the geometric symmetry of the square sampling grid, a symmetric slant angle θ_sym is defined to be the smaller of the two absolute slant angles from the vertical and horizontal orientations. θ_sym is calculated from θ by using a triangular wave function with a period of 90°:

 

Fig. 5. Symmetric slant angle θ_sym and variable oversampling ratio used in the OMNI-sine and ISO-cosine methods when n_bin = 4.

Download Full Size | PDF

5. Simulation: ISO-cosine method vs. OMNI-sine method

To compare the accuracy and precision of the ISO-cosine and OMNI-sine methods, edge images were analytically synthesized using the MATLAB Symbolic Math Toolbox. A Heaviside step function was used to represent an ideal 2D edge target with a slant angle of θ. The camera MTF was assumed to be |sinc(ξ_x, ξ_y)|³, which represents a typical MTF of a high-end digital imaging device [21]. To simulate the camera MTF and sampling, the Heaviside function was thrice convolved with a 2D rectangle function and sampled on a square grid, yielding an image containing 200 × 200 pixels with the blurred edge passing through the center. Figure 6 shows an image with θ = 5°.

 

Fig. 6. Analytically synthesized edge image for a slant angle θ of 5° assuming a camera MTF of |sinc(ξ_x, ξ_y)|³.

Download Full Size | PDF

Nine slant angles, θ, were used: 3°, 3.013° (tan θ = 1/19), 5° (ISO standard chart), 5.194° (tan θ = 1/11), 11.310° (tan θ = 1/5), 14.036° (tan θ = 1/4), 18.435° (tan θ = 1/3), 26.565° (tan θ = 1/2), and 33.690° (tan θ = 2/3), the slant angle of 3° representing a near-vertical slant angle [10,11]. The second and fourth critical angles were selected as they were close to 3° and 5°, respectively, the other critical angles being selected as a series of tangents with simple fractions. The slant angles were set manually in the test simulation to separate errors of slant angle estimates from the analysis of the different oversampling ratios between the two methods.

In each method, the bin array was set to pass through the image center (i.e., the origin) at an orientation perpendicular to the edge, n_bin being set to four. Thirty-two MTF results were obtained using 32 bin arrays of different phases: One of the 32 phases had a bin boundary at the origin, while the other 31 were shifted by intervals of 1/32-th of the bin width. Zero-count bins were scanned from the lower end of the ESF and sequentially interpolated using the average pixel value of the pixels in the lower side neighboring bin. Apodization was not applied to the LSF. The MTF corrections employed in the OMNI method were also applied to the ISO-cosine method for a fair comparison. The difference filter and bin-width averaging filter described in section 2.3 were used, the MTF underestimations caused by them being compensated for in both methods.

6. Results and discussion

Figure 7 compares the MTF results of the ISO-cosine and OMNI-sine methods for each slant angle. The 32 MTF curves computed with different binning phases are plotted for each method. The dotted curve is the true MTF of |sinc(ξ_x, ξ_y)|³ for each slant angle. Note that the true curve depends on θ.

In analyzing the precision of the MTF results, it is not surprising that there is no difference between the two methods when tan θ = 1/4 and tan θ = 1/2 since in these cases the oversampling ratios are identical (i.e., v_bin = n_bin sec θ), as shown in Fig. 5. For the other critical angles, the MTF curves estimated using the ISO-cosine method fluctuate depending on the binning phase, demonstrating the advantage of the OMNI-sine method in adjusting the bin width to maintain the pixel intervals in each column in addition to in each row in the ROI image. In particular, the 32 MTF curves computed using the OMNI-sine method are identical for the 5 critical angles greater than or equal to 11.310°, which is reasonable as v_bin is not scaled down. Two unique MTF curves were obtained when tan θ = 1/11 as there were two unique distributions of binned pixels for a v_bin of (cosec θ)/2. Similarly, four unique MTF curves were obtained for tan θ = 1/19 because of the use of (cosec θ)/4 for v_bin. Nevertheless, the unique curves were similar, appearing as a single curve in each plot. Consequently, in practice—when using the OMNI-sine method—there would be no need to estimate the MTFs using different binning phases.

 

Fig. 7. Comparison of the MTF results of the ISO-cosine and OMNI-sine methods (n_bin = 4): θ = (a) 3°, (b) 3.013° [tan θ = 1/19], (c) 5° [used in the ISO standard charts], (d) 5.194° [tan θ = 1/11], (e) 11.310° [tan θ = 1/5], (f) 14.036° [tan θ = 1/4], (g) 18.435° [tan θ = 1/3], (h) 26.565° [tan θ = 1/2], and (i) 33.690° [tan θ = 2/3]. The curves in (f) and (h) are superimposed, making the ISO-cosine results invisible.

Download Full Size | PDF

Regarding the accuracy of the OMNI-sine method, the MTF curves were slightly overestimated for tan θ = 1/4 and tan θ = 1/3, primarily because of the unnecessary compensation for bin-width averaging. As shown in Figs. 4(b) and 4(c), for these angles, the pixels were projected at identical intra-bin positions and averaging the pixel values in each bin did not work as a low-pass filter. Conversely, when tan θ = 1/2 and tan θ = 2/3, empty bins caused small deviations from the true MTF curves. Such deviations are sensitive to the algorithm used to interpolate the values for empty bins as well as the phase of the edge of an image, which is beyond the scope of this study.

To assess the possibility of improving the accuracy and precision of the OMNI-sine method, it would be necessary to conduct further detailed analysis of its accuracy and precision for near-critical angles, including near 0° and near 45°, and many other non-critical angles. When arcsin(2^−2.5) ≤ θ_sym < arcsin(2^−1.5)—that is, from 10.182° to 20.705°—v_bin is cosec θ_sym with no scaling, which is the most suitable oversampling ratio. For θ_sym < arcsin(2^−2.5), a larger v_bin value might improve the accuracy and precision of the MTF. It is expected that v_bin, as shown in Fig. 5, would be appropriate for a wide range of critical angles as well as near-critical and non-critical angles other than near 0°, near 45°, and in the vicinity of 26.565° [ = arctan(1/2)], based on the author’s further simulations, which were not included here for reasons of space.

7. Conclusions

An accurate and precise edge-based method is needed for any slant angle to conduct multidirectional MTF measurements. However, when projecting pixels from a square array into a linear array of subpixel-wide bins, the projection paths of the pixels have regular intervals that can misalign with the bin array for some critical angles. These unwanted artifacts do not meet the requirement of random phases for the sampling sites in edge-based methods, and can cause significant errors in MTF estimates. While the default integer oversampling ratio of four works in the ISO 12233 edge-based method, which was designed for analyzing near-vertical and near-horizontal edges only, the ratio choice matters when using slant angles that are not near-vertical or near-horizontal.

In this study, geometric diagrams were used to visualize how the orientation-dependent projection paths of the pixels can be misaligned with the bin array, causing periodic deviations from the original sampling positions. The proposed OMNI-sine method was designed to align the pixel intervals in each column and row with the bin array by adjusting the oversampling ratio within a limited range. Through computer simulations, the MTF estimates of the new method were found to be insensitive to the binning phase in a practical sense. Accurate and precise MTF estimates using a single binning phase significantly reduce the computational overhead, facilitating real-time MTF measurements. Further detailed analysis is needed to evaluate the accuracy and precision of the OMNI-sine method over a full range of slant angles to determine areas of improvement.