An optical see-through (OST) display is a promising augmented reality (AR) system for its capability to show the clear real scene. In most OST display systems, the virtual scene is simply added into the real scene by optical combiners. Those systems, however, do not provide depth perception cues about the interposition between the virtual and real objects, which can offer the user a more immersive AR experience [1,2]. Since the virtual scene appears as a ghost image, its contrast gets deteriorated [3]. Also, the OST is forced to consume a lot of battery under bright outdoor surroundings. Occlusion-capable OST (OC-OST) displays alleviate those issues by occluding the real objects with virtual objects. Implementing such occlusion is challenging due to the difficulties in controlling the incident light from the real scene.

In the last decades, various systems for OC-OST displays were proposed [4–11]. The common methods in those researches are to place a pixelated active device such as a liquid crystal display (LCD) or digital micromirror device (DMD) for the occlusion mask. Itoh et al. [4] placed an occlusion mask directly in front of the user’s eyes. Since this method can create only a blurred mask, they captured the partially occluded part of the real scene and appended it on the virtual scene. Kiyokawa et al. [5] presented the primary method of occlusion using the 4-f system. This method using a 4-f system has an advantage in obtaining a sharp occlusion mask, and has been applied to various researches using different optical components [6–11].

There are various issues in the present systems. The first issue is the bulky system size. The 4-f system rotates the real scene image by 180°, and an additional 4-f system to rotate once more is required, making the total system size bulky. In general, the total physical length becomes tens of centimeters. To reduce the system size, Yamaguchi and Takaki [11] used lens arrays for imaging the real scene. Compared to conventional lenses, lens arrays with the same numerical aperture (NA) have a much shorter focal length. However, this system still has an issue of a limited field of view (FoV) of only 4.3°. The FoV of OC-OST displays is determined by the NA of the lenses, which typically is low for lens arrays [12].

The second issue is the screen-door effect by pixelated devices. The grid structure such as electrodes or blank spaces between the pixels is overlapped on the real scene and acts as a noise [13]. Also, such a grid induces the diffraction of light incident from the plane out of the system’s target plane. Users will observe a degraded real scene and its chromatic diffraction noise. This results from the periodicity of pixelated devices, as the pixel structure operates as a fine diffraction grating [4].

In this Letter, we introduce a novel OC-OST display system to resolve the previous issues. We utilize the photochromic plate to substitute pixelated devices for the occlusion mask. Visible light transmittance (VLT) of photochromic material varies by incident near-UV (NUV) light. By imaging the occlusion pattern with NUV, the occlusion mask can be generated on the photochromic plate. Because the photochromic plate does not have a pixelated structure, there is no visible grid structure or diffraction effect of the real scene. Also, while adopting the lens array based 4-f system to reduce the system size, we added concave and convex lenses at both sides of the 4-f system to overcome the narrow FoV of conventional OC-OST displays.

First, we conducted an experiment for analyzing the VLT and resolution properties of the photochromic plate. The photochromic plate used is a specially ordered flattened photochromic plate (Transitions Signature Gen 8, Transitions Optical, Inc.). Figure 1(a) shows the VLT change of the photochromic plate of our research. This photochromic plate is designed to react best to 405 nm NUV light. The VLT variation graph was obtained by measuring the power of transmitted visible light (collimated 470 nm LED) and NUV power simultaneously. The NUV was turned on for 120 s and then turned off. When the NUV is on, the VLT starts to decrease drastically and gets saturated. The final VLT drops to 1.37% with $7.5{\rm{mW}}/{\rm{cm}}^2$ irradiance. This VLT is sufficiently low to be utilized for the occlusion mask. When the NUV is off, the VLT gets fully recovered with a similar slope. Although the reaction time of the photochromic plate we used is not short enough, the fast photochromic material is being developed to realize occlusion with a transition speed of milliseconds [14]. Figure 1(b) shows the expressible resolution of the photochromic plate. The USAF 1951 mask was attached on the photochromic mask and the 405 nm NUV light was directly exposed. The result verifies that the recorded line with thinner than 3 µm can be resolved, which is fairly comparable to the pixel pitch of commercial LCDs.

 

Fig. 1. (a) VLT variation of the photochromic plate according to the irradiance and the exposure time of 405 nm light source incidence. (b) USAF 1951 resolution target recorded on the photochromic plate.

Download Full Size | PDF

Figure 2 shows the schematic diagram of our setup. The green line indicates the light from the real scene. The purple optical paths represent the propagation of the NUV light for occlusion. The total system is composed of three parts: the real scene transfer, virtual scene generation, and occlusion mask imaging. The real scene transfer is composed of four lens arrays, a partition, a concave lens ${L_1}$, and a convex lens ${L_2}$. The focal length of the lens array is denoted by $f$, and the distance between each lens array is ${{2}}f$. The focal lengths of ${L_1}$ and ${L_2}$ are ${-}F$ and $F$ each. ${L_1}$ is located in front of the first lens array with distance $f$, and ${L_2}$ is located after the last lens array with the same space. By ${L_1}$, the real scene imaging plane is formed behind the original imaging plane of the lens array. The deviated imaging distance is given by ${f_{{\rm{dev}}}} = f(F + f)/F$. The photochromic plate is placed on this deviated imaging plane. Also, a partition between the lenslets is required to prevent cross talk between the adjacent lenslets [11].

 

Fig. 2. Schematic diagram of the proposed OC-OST setup. By the lens array and convex lens ${L_2}$, elemental images for occlusion are formed on the plate and block the corresponding parallel rays from the real scene.

Download Full Size | PDF

In occlusion mask imaging, the occlusion pattern is generated by the NUV light source and a DMD device. The DMD is located at the focal plane of ${L_i}$. Diverging NUV rays from each pixel of the DMD are collimated by ${L_i}$ and propagate to the lens arrays by a polarizing beam splitter (PBS). The NUV rays then generate the elemental images for occlusion, which occlude the corresponding elemental images of the real scene. For the same reason as imaging the real scene, an additional partition between the photochromic plate and the last lens array is placed. To prevent the NUV rays from entering the eye pupil, the rejection filter is installed in front of the user’s eye location. On the other side of the PBS, the laser projector and lenses are installed to construct a simple retinal projection setup [7].

In the real scene transfer, the 4-f system restores the rays in the front focal plane of the lenslets on the rear focal plane. Without the ${L_1} - {L_2}$ lens pair, the FoV of the system is determined by the $f$-number of the lens array, given by ${\rm{FoV}} = 2\mathop {\tan}\nolimits^{- 1} (p/2f)$, where $p$ is the lenslet pitch. With the concave lens ${L_1}$, the wide incident angle range of the rays turns into a narrow range and could be transferred within the low NA of the lenslets. Hence, the concave lens ${L_1}$ extends the angle of the rays that the lens array can deliver. Such modified rays are restored back into the initial state by the convex lens ${L_2}$ on the rear focal plane of the lens arrays. To verify the validity of such ray restoration, we proceeded with the light field analysis based on modified ray transfer matrices for the lens array system:

Here, $y$ and $\theta$ denote, respectively, the $y$-axis location and the incident angle of the ray when it meets ${L_1}$. The term $i$ denotes the index of the lenslet the ray passes. It can be calculated that the final ray vectors are equivalent to the initial ray vectors for rays propagating the whole system.

Figure 3 shows the MATLAB light field simulation results of the real scene where $F$ is 50 mm and $f$ is 19.05 mm, which corresponds to the experimental setup. Figure 3(a) shows the ray tracing result of the proposed system. The eye-box is formed at the distance $F$ from ${L_2}$. The lines with the same color denote the rays with the same incident angle. Figure 3(b) is the light field diagram at the eye-box location. Each black line corresponds to a sampled initial parallel ray and shows that the output rays remain parallel. The blue boundary denotes the region where the rays can enter the general human pupil diameter of 4 mm [15]. The resulting normalized brightness along the FoV can be seen in Fig. 3(c). Although the rays cannot be fully transferred due to the vignetting effect of each lenslet, the deviation of brightness is within 28%. Further analysis of eye-box and brightness deviation is presented in Supplement 1.

 

Fig. 3. (a) Ray simulation result of the system using MATLAB. $F$ is 50 mm, and $f$ is 19.05 mm. (b) Resulting light field at the eye-box. The horizontal axis indicates the distance from the center of the system, and the vertical axis indicates the incident angles of the rays at the eye-box. The blue parallelogram indicates the light field entering the 4 mm eye pupil. (c) Normalized brightness along the FoV at the eye-box.

Download Full Size | PDF

Also, the FoV of the system can be calculated. The maximum incidence angle that can be transferred by the system is given by the combination of the NAs of both lens arrays and ${L_2}$ by ${\theta _{{\max}}} = \mathop {\tan}\nolimits^{- 1} (p/2f + D/2F)$. Within the range from zero to $2{\theta _{{\max}}}$, the FoV at the eye-box is obtained by ${\rm{FoV}} = 2\mathop {\tan}\nolimits^{- 1} ((D + {d_{{\rm{pupil}}}})/2F)$, where $D$ is the diameter of the ${L_1} - {L_2}$ lens pair, and ${d_{{\rm{pupil}}}}$ is the pupil diameter. For the simulation specification, the observable FoV is simulated as roughly 33°.

To verify the simulation results, the experimental setup is implemented as shown in Fig. 4. For the real scene transfer, commercial lens arrays from Edmund Optics (product #63-230) are used. The lenslet pitch is 4 mm horizontally and 3 mm vertically. To construct a more compact system, we attached two lens arrays into one and reduced the focal length by half. The focal length of the attached lens arrays $f$ is 19.05 mm. The focal lengths of ${L_1}$ and ${L_2}$ are ${-}{{50}}\;{\rm{mm}}$ and 50 mm, respectively, and the diameter $D$ is 25.4 mm. The partition is made by a 3D printer with a thickness of 0.6 mm, and the pitch is the same as the lens array. For the virtual scene generation, the laser projector and two lenses are installed for the retinal projection system. In occlusion mask imaging, the LED array with 405 nm center wavelength is installed for the NUV light source. After the image for the occlusion mask is uploaded on the DMD, the NUV light is reflected by the DMD and propagates to the photochromic plate. To minimize the high order terms by the DMD, an aperture is installed. By ${L_2}$ and the lens array, the NUV light forms the elemental image on the photochromic plate. The VLT of the imaging region on the photochromic plate changes and the occlusion mask is formed.

 

Fig. 4. Experimental setup for the proposed system. The real scene transfer, occlusion mask imaging, and virtual scene generation parts are represented as green, blue, and yellow lines, respectively.

Download Full Size | PDF

Finally, two NUV rejection filters from Edmund Optics (product #84-748) block the NUV and keep the user’s eye safe. After building the experimental setup, we measured the irradiance of NUV light propagating toward the pupil. When all of the DMD pixels are set to reflect the NUV light toward the system, the optical irradiance of the 405 nm NUV light is measured to be $22 \;{\rm{nW}}/{\rm{cm}}^2$. This irradiance is far below the maximum permissible exposure defined by ANSI 136.1 Standard [16]. See Supplement 1 for the detailed NUV irradiance values.

Figure 5 confirms the FoV expansion by the ${L_1} - {L_2}$ lens pair. Without the lens pair, the FoV is highly limited due to the partition and the low NA of the lens arrays, 10.2° horizontally. This FoV is narrower than the theoretical FoV since the thickness of the partition further reduces the NA [11]. After installing the lens pair, the FoV expansion up to 32.8° is verified.

 

Fig. 5. Experimental result of the FoV expansion. By the ${L_1} - {L_2}$ lens pair, FoV is expanded from (a) 10.2° to (b) 32.8°.

Download Full Size | PDF

Next, the occlusion mask is generated and analyzed, as shown in Figs. 6 and 7. The colorful planet is the virtual scene. Without the occlusion mask in Fig. 6(a), the virtual scene is formed as a semi-transparent image and its contrast is degraded. As the occlusion mask is formed on the photochromic plate in Fig. 6(b), the planet surely occludes the word “PLANET” in the background, and its contrast is improved. Also, the occlusion gives depth order to the virtual scene, making it look clear in front of the real scene. Figure 7 shows the simulated and experimental modulation transfer function (MTF) graphs of the occlusion image in our system. The contrast value was calculated by generating the slanted-line pattern occlusion masks of several different spatial frequencies. The cutoff frequency of the occlusion mask is 18.6 cycles per degree for the MTF criterion of 20%, as marked as a red dot.

 

Fig. 6. Experimental result of the OST display (a) without and (b) with occlusion using photochromic plate.

Download Full Size | PDF

 

Fig. 7. MTF graph of the occlusion image. The experimental MTF well matches the simulated MTF graph. The cutoff frequency of 20% MTF is obtained as 18.6 cpd.

Download Full Size | PDF

 

Fig. 8. Real scene comparison of the OC-OST systems using (a), (c) an LCD and (b), (d) the photochromic plate for the occlusion mask. The screen-door effect and diffraction noise are alleviated in the proposed system.

Download Full Size | PDF

Figure 8 compares the real scene quality of two different OC-OST systems using an LCD (SONY LCX017) and photochromic plate as the occlusion mask. In Fig. 8(a), the screen-door effect by the black matrix of the LCD can be observed clearly throughout the real scene, even in the part where occlusion is not required. In contrast, no grid structure is seen when the photochromic plate was used as shown in Fig. 8(b). Figure 8(c) shows the diffraction in the real scene induced by the LCD occlusion mask. The system’s target distance for occlusion is infinity, and the object with markings is located 35 cm away. The rainbow-color diffraction pattern deteriorates the image quality of the markings. Such diffraction noise further increases as the pixel pitch of the LCD gets smaller for the higher resolution. As the photochromic plate does not have the black matrix, such a diffraction effect is alleviated, as shown in Fig. 8(d). These comparisons clearly show the merit of using the gridless photochromic plate for the OC-OST.

In conclusion, we propose and experimentally construct an OC-OST display using a photochromic plate. The ${L_1} {-} {L_2}$ lens pair expands the FoV significantly while using the lens arrays for imaging the real scene in a reduced size. Specifically, our system resolves the issues of the visible screen-door effect and chromatic diffraction effect of the real scene by using the photochromic plate for the occlusion mask. The occlusion pattern is well formed and improves the contrast and depth order recognition of the virtual scene. Although the deviation of the brightness of the real scene exists over the FoV, this defect can be alleviated by using lens arrays with a pitch smaller than the eye pupil. With the progress of fast photochromic materials, the proposed system can offer real-time occlusion and be applied for defect-free OC-OST.