Image resolution is one of the most important performance specifications of aerial display techniques. However, there is no standard method for evaluating the aerial image resolution. In this paper, we propose a method for measuring the modulation transfer function (MTF) of an aerial imaging system based on the slanted knife edge method. We hypothesize that aerial images have a different blur function from standard camera images. In order to explore this, we simulate blurred slanted knife edge images by convolving two types of blur functions. Furthermore, the MTF curves of the aerial image formed using different retro-reflectors are compared using the proposed method.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
Aerial displays [1,2] are expected to open new markets for digital signage, entertainment, and car equipment. In particular, they have attracted attention as touchless aerial interfaces that are immune to hygiene issues. Aerial images formed as real images satisfy multiple visual cues in human depth perception to look at aerial images floating in the air, including binocular parallax, motion parallax, accommodation, and convergence. Thus, real aerial images have been considered to cause less visual fatigue than other methods. Aerial images can be formed using passive optics, such as with dihedral corner reflector arrays (DCRA) and other types of retro-reflectors [3,4]. The DCRA forms an aerial image after double reflection, while singly-reflected light causes undesired pseudo images located on both sides of the aerial image. We have previously proposed a method of aerial imaging by retro-reflection (AIRR) – a low cost method that provides a wide viewing angle and solves the problem of pseudo images as mentioned above [5,6]. In past research, we have combined AIRR with other imaging modalities to give new functionality to aerial display. Examples include multi-function aerial displays that can switch to being a deep three-layered aerial image or to being visible only from the front , achieving the perception of depth by overlaying aerial images , omnidirectional aerial display for behavioral experiments on fish , and forming multiple aerial images using only one light source . However, AIRR can have problems of low brightness and image blur, both of which are affected by the optical properties of the retro-reflector. By learning more about the limiting characteristics of retro-reflectors, we can come to understand how to improve aerial image quality [11–13].
Image resolution is one of the most important performance specifications of any display. However, there is no standard method for evaluating aerial image resolution. We have previously proposed a method which employs the contrast transfer function (CTF) to evaluate aerial image resolution . However, one of the problems in using CTF to evaluate imaging optics is that the CTF is affected by the light-source. The contrast of the aerial image cannot exceed the combined contrast of the light-source display and the passive optical components.
The purpose of this paper is to explain how to employ the slanted knife edge method [15,16] to measure the MTF of an aerial image. While the slanted knife edge method has often been used to measure the MTF of conventional cameras, aerial images formed with AIRR have different blur properties than standard camera images do. In order to better understand the behavior of the knife edge approach for aerial images, we also simulate the influence of blur functions on the measurement images, and quantify their effect on the estimates of the knife-edge slant angle with respect to the image pixel grid. Furthermore, the MTF curves of the aerial image formed using different retro-reflectors are compared with the proposed method.
2. Principle of AIRR
An AIRR display utilizes a retro-reflector to form an aerial image, the principle of which is shown in Fig. 1. The setup consists of a light source, a beam splitter, and a retro-reflector. Light rays from the light source impinge on the beam splitter. Rays reflected by the beam splitter reach the retro-reflector and are reflected back along their incident directions. The location where the retro-reflected rays converge is the position of the aerial image. As a result, the aerial image of the light source is formed at the plane-symmetrical position of the light source with respect to the beam splitter.
3. Slanted knife edge method
3.1 Slanted knife edge method
The modulation transfer function (MTF) expresses how well the optical system preserves the contrast of spatial frequencies of the object in the spatial frequencies of the image. The slanted knife edge method estimates the MTF curve for spatial frequency in one direction by calculating an image of a knife edge. The knife edge is intentionally slanted in order to stagger the sampling by the pixel grid.
The MTF is calculated based on the procedure shown in Fig. 2. An edge spread function (ESF) is estimated from the recorded knife edge image, giving the one-direction response of the imaging system to an edge object. In the conventional approach to the knife edge method, the edge position is determined along each line in the image by calculating the location of the intensity midpoint to subpixel precision. In this method, the user next applies a linear regression on the collection of raw ESF curves to estimate the slant angle θ, which is then used as the projection angle which is used to project all of the various ESF curves onto a shared axis. However, the obtained projected ESF curves cannot yet be used since the sampling points are not arranged at equal intervals. Thus, the projected curves are upsampled by a factor of four, and any ESF curve samples collected within each bin are averaged. In the end, this procedure effectively eliminates aliasing problems from the resulting average ESF curve. The line spread function (LSF) is obtained by the derivative of the ESF:
Compared to our proposed method of obtaining the CTF, the slanted knife edge method does not require test charts for each spatial frequency. Thus, the MTF curve can be calculated on a single image.
3.2 Slanted angle calculation method
In the conventional approach to the knife edge method described above, it can be difficult to calculate an accurate slant angle due to the influence of noise in the aerial image. In addition, the slant angle calculated using the recorded aerial image is affected by distortion. We propose instead to calculate the slant angle using a Radon transform (as shown in Fig. 3). The slant angle estimated this way is affected by noise and image blur, and so noise is removed by first applying a bilateral filter and binarizing the result. The edge line is estimated from this binarized image. The formula of the line is
3.3 Projection method of the edge spread function
In the slanted knife edge method, it is difficult to obtain a stable MTF curve due to the noise amplified by differentiating the ESF curve. Thus, in this study we calculate the ESF curve by projecting the image data at the estimated slant angle, as shown in Fig. 5 . The data in the edge image are projected onto the vertical to the edge and superimposed. The sampling point width of two adjacent pixels in the row correspond to p cosθ, where p is the pixel width and θ is the slant angle. The sampling point width of two adjacent pixels in the column correspond to p sinθ, where p is the pixel width and θ is the slant angle. All superimposed data are binned using a width equal to a quarter of the sampling pitch and averaged. From this procedure, we obtain an estimate of the finely-sampled ESF curve.
4. Numerical simulation results
4.1 Examination of simulated edge image with noise
Next, we perform a simulation to investigate the influence of noise on calculating the slant angle for knife edge images, by blurring ideal images convolved with two types of blur functions. We have constructed ESF curves for slanted knife edge images projected at experimentally obtained slant angles. The blurred image g(x, y) is obtained by
Figure 6 shows an edge image with an angle of 5 degrees that was obtained by convolving blur functions according to Eq. (5). The image intensity ranges from 0 to 255 for each pixel. The two types of blur function are a Gaussian function and a sinc-squared function. These two functions derive from models of the sources of blur in an AIRR system: optical aberrations and the discrete pixelated sampling induced by the grid of retro-reflectors. The noise we use is white Gaussian noise with average of 0 and variance of 10−4.
4.2 Simulation results
In the simulation, we generate ideal slanted knife edge images, blur them, and add noise prior to calculating their Radon transforms, with the results shown in Fig. 7. Table 1 shows the slant angles calculated from the blurred, noisy edge images. In each instance, the slant angle is estimated as exactly 5 degrees, regardless of the type of blur function or noise level.
We have investigated sharpness of the edge image using the calculated slant angles shown in Table 1. Figure 8(a) shows the ESF curves for slanted knife edge images that have been blurred with a Gaussian function and estimated by use of the projection method (as in Fig. 5). Figure 8(b) shows MTF estimates obtained from the ESF curves of Fig. 8(a) and calculated according to Eqs. (1) and (2). Figure 9 shows the ESF curves and MTF curves for slanted knife edge images blurred with a sinc-squared function. We see that the ESF curve can be estimated by the slanted knife edge method under sinc-square blur, and the triangular function MTF can be accurately estimated regardless of noise.
5.1 Experimental setup
In order to investigate the influence of MTF curves on aerial images, we measure the sharpness of aerial images formed by use of three different retro-reflectors: RF-Ax and Nikkalite CRG, both manufactured by Nippon Carbide, and Scotchlite 8910, manufactured by 3M. Each retro-reflector is a circular panel 150 mm in diameter. Figure 10 shows the experimental setup, together with the integrating sphere and knife edge used to produce the knife edge image. The aerial image is the 2D image of the aperture of the integrating sphere that is partially blocked by a knife edge. Figure 11 shows views of the experiment’s aerial images obtained with each of the three retro-reflector types. (Note that the aerial images are flat, and not three-dimensional.) The beam splitter is a half mirror (transmittance and reflectance ∼50%). We use a digital camera (Nikon, D5500) tilted so that its azimuth is rotated 5 degrees from the axis of the knife edge. The recording conditions of the camera are: ISO 400, F-number 4.5, and focal length 35 mm. We have selected the exposure time so that brightness of aerial images formed by using the three types of retro-reflectors are the same. The exposure times for the RF-Ax, NCRG, and 3M 8910 are 1/100, 1/60 and 1/25 seconds, respectively.
5.2 Experimental result
Aerial images of the slanted knife edge obtained with AIRR are shown in Fig. 12. The slant angles estimated from these images, using the methods of Sec. 3.2, are shown in Table 2. The estimates differ from one another by up to 0.6 degrees.
Figure 13(a) shows the ESF curves obtained for the slant angles given in Table 2, where the curves are obtained by projecting and binning the aerial edge images. Figure 13(b) shows the corresponding MTF curves, calculated according to Eqs. (1) and (2). In each MTF curve, the horizontal axis represents the spatial frequency at the position of the aerial image. The results show that the RF-Ax retro-reflector provides aerial images with the best contrast, while the 3M 8910 provides the lowest contrast.
In order to explain the difference in the MTFs measured with each retro-reflector type, we can reflect that the Nikkalite CRG and the 3M Scotchlite 8910 are retro-reflectors used in road safety applications, with the former having half the divergence angle of the latter . On the other hand, the RF-Ax is a dedicated retro-reflector used in aerial images formed by AIRR. Thus, the divergence angle of the RF-Ax is estimated to be narrower than that of the Nikkalite CRG. Thus, the aerial image resolution obtained with narrow divergence retro-reflector provides the highest contrast.
5.3 Measurement of MTF for various camera F-numbers
It has been generally recognized that the resolution of a standard camera system improves as F-number increases, and we investigate whether the same rule holds for AIRR as well. The aerial image is recorded while varying the F-number of the camera from F4.5 to F18. The exposure time is varied according to the F-number so that the brightness of the aerial image is the same in each case. The exposure time is 1/5 seconds, 1/20 seconds, and 1/80 seconds, respectively. Figure 14 shows ESF curves and MTF curves obtained using different camera F-numbers, obtained with the slanted knife edge method, and using the RF-Ax retro-reflector. From this result, we see that the MTF of AIRR is entirely independent of the imaging camera’s F-number.
Aerial image resolution is an important criterion for choosing a retro-reflector for AIRR. However, there is currently no standard method for evaluating the aerial image resolution. The slanted knife edge method is commonly used to measure the MTF of traditional imaging optical systems, and we have confirmed that it can be used for aerial images formed by AIRR as wellwhich is an imaging system comprised of discrete optical imaging elements.
We have shown that the MTF curve of an AIRR system can be measured based on the knife edge method, and that the MTF measurements for different retro-reflectors exhibit differences according to the divergence angle of each retro-reflector. We also investigated the influence of F-number of the capturing camera and reveal that the camera’s F-number does not influence the aerial image MTF curves because the blurring is dominated by the aerial image and not the capturing camera’s optics. Evaluation of the imaging property to match the human perception is beyond the scope of this paper because human contrast sensitivity is not proportional to MTF.
In future work, we intend to verify the influence of anisotropy and incident angle on retro-reflectors using this technique.
Japan Science and Technology Agency Accelerated Innovation Research Initiative Turning Top Science and Ideas into High-Impact Values (JPMJAC1601).
The authors declare no conflicts of interest.
1. J. Hong, Y. Kim, H.-J. Choi, J. Hahn, J.-H. Park, H. Kim, S.-W. Min, N. Chen, and B. Lee, “Three-dimensional display technologies of recent interest: principles, status, and issues,” Appl. Opt. 50(34), H87–H115 (2011). [CrossRef]
2. J. Geng, “Three-dimensional display technologies,” Adv. Opt. Photonics 5(4), 456–535 (2013). [CrossRef]
3. S. Maekawa, K. Nitta, and O. Matoba, “Transmissive optical imaging device with micromirror array,” Proc. SPIE 6392, 63920E (2006). [CrossRef]
4. C. B. Burckhardt, R. J. Colloer, and E. T. Doherty, “Formation and inversion of pseudoscopic images,” Appl. Opt. 7(4), 627–631 (1968). [CrossRef]
5. H. Yamamoto and S. Suyama, “Aerial 3D LED display by use of retroreflective sheeting,” Proc. SPIE 8648, 86480Q (2013). [CrossRef]
6. H. Yamamoto, Y. Tomiyama, and S. Suyama, “Floating aerial LED signage based on aerial imaging by retro-reflection (AIRR),” Opt. Express 22(22), 26919–26924 (2014). [CrossRef]
7. K. Uchida, S. Ito, and H. Yamamoto, “Multi-Functional Aerial Display by Use of Polarization-Processing Display,” Opt. Rev. 24(1), 72–79 (2017). [CrossRef]
8. Y. Terashima, S. Suyama, and H. Yamamoto, “Aerial depth-fused 3D image formed with aerial imaging by retro-reflection (AIRR),” Opt. Rev. 26(1), 179–186 (2019). [CrossRef]
9. E. Abe, M. Yasugi, H. Takeuchi, E. Watanabe, Y. Kamei, and H. Yamamoto, “Development of omnidirectional aerial display with aerial imaging by retro-reflection (AIRR) for behavioral biology experiments,” Opt. Rev. 26(1), 221–229 (2019). [CrossRef]
10. K. Chiba, M. Yasugi, and H. Yamamoto, “Multiple aerial imaging by use of infinity mirror and oblique retro-reflector,” Jpn. J. Appl. Phys. 59(SO), SOOD08 (2020). [CrossRef]
11. Y. Tokuda, A. Hiyama, M. Hirose, and T. Large, “Comparison of material combinations for bright and clear floating image by retro-reflective re-imaging technique,” Proc. IDW 21, 818–819 (2014). [CrossRef]
12. M. Nakajima, Y. Tomiyama, I. Amimori, and H. Yamamoto, “Evaluation methods of retro-reflector for polarized aerial imaging by retro-reflection,” in 11th Conference on Lasers and Electro-Optics Pacific Rim (CLEO-PR)2015, (2015), paper 25B1_2. [CrossRef]
13. M. Nakajima, K. Onuki, I. Amimori, and H. Yamamoto, “Polarization state analysis for polarized aerial imaging by retro-reflection (PAIRR),” Proc. IDW 22, 429–432 (2015).
14. N. Kawagishi, K. Onuki, and H. Yamamoto, “Comparison of divergence angle of retro-reflectors and sharpness with aerial imaging by retro-reflection (AIRR),” IEICE Trans. Electron. E100.C(11), 958–964 (2017). [CrossRef]
15. ISO 12233: 2014, “Photography - Electronic still picture imaging - Resolution and spatial frequency response”.
16. P. L. Smith, “New Technique for Estimating the MTF of an Imaging System from its Edge Response,” Appl. Opt. 11(6), 1424–1425 (1972). [CrossRef]
17. N. Kawagishi, R. Kakinuma, and H. Yamamoto, “Evaluation of image resolution of aerial image based on slanted knife edge method,” Proc. IDW 26, 746–749 (2019).