In virtual reality (VR) and augmented reality (AR) applications, near-eye displays (NEDs) are the main devices delivering intuitive visual information to users [1]. Unlike VR NEDs, AR NEDs need to have an optical see-through (OST) configuration to superimpose virtual images onto the real world. An optical combiner is a key device enabling the OST capability, but it also makes the optical system of the AR NEDs more complicated than that of the VR ones.

Various AR NEDs can be categorized by the type of embedded optical combiners. Waveguide-type NEDs (WNEDs) have a compact form factor owing to the thin waveguide and in- and out-couplers, such as diffractive optical elements and holographic optical elements (HOEs) [2,3]. Since these grating-based components have high transparency, owing to the angular and wavelength selectivity, they can transmit the real scene better than other components. Additionally, by applying the exit pupil expansion (EPE) technique, the small eye-box of the WNED can be enlarged [4]. Taking advantage of these features, WNEDs have been actively studied and commercialized [5]. However, most of the conventional AR WNEDs suffer from a vergence-accommodation conflict (VAC), which is a cause of user’s visual discomfort [6]. The VAC originates from an optical structure of the WNED, which is based on the collimated display engine.

Figure 1 shows a schematic diagram of a conventional AR WNED using HOE couplers. A ray bundle from a center-located pixel is collimated by a lens. It is then guided into the waveguide by the in-coupler HOE, which diffracts the normally incident light to an angle larger than the critical angle. Ray bundles from different pixels, drawn with dashed and dotted lines in Fig. 1, are also guided with different angles. After multiple total internal reflections (TIRs), they are diffracted towards the user by the out-coupler HOE. To observe the virtual image, the eye should be focused at the infinite plane. If the eye focuses at a near plane, the image will be blurred due to defocus. Thus, the eye accommodation distance is always fixed at the optical infinite plane. The vergence distance, however, varies according to the three-dimensional (3D) virtual image distance, which is controlled by the binocular disparity. Consequently, there exists a discrepancy between the accommodation and vergence distance, and this causes discomfort while using the WNED.

 

Fig. 1. Schematic diagram of a conventional AR WNED with HOE couplers. Relaxed and accommodated human eyes are depicted. When the eye focuses on the near plane, the pixel image is blurred on the retinal plane.

Download Full Size | PDF

In recent studies, various techniques have been proposed to address the VAC problem in AR WNEDs. Yoo et al. fabricated a dual-focal AR WNED with a single waveguide using geometric phase lenses [7]. However, dual virtual image planes are still insufficient to fully solve the VAC problem. To present 3D virtual images with true focal cue, light field or holographic AR NEDs have been proposed [8–10]. However, these optical systems not only contained a relatively thick waveguide, which will be referred to as a lightguide in this paper, but also had a restricted eye-box since the EPE technique was not applied.

Maxwellian display, or retinal projection display, is another technique that can alleviate the VAC problem [11,12]. By restricting the effective pupil of the NED like a pinhole camera, the virtual image with deep depth of field (DoF) can be presented. The observed image is always in focus regardless of the eye lens power, which helps to mitigate the distance discrepancy. Although this technique makes NED users feel more comfortable, there is a limitation in that the eye-box size is very small due to the restriction of the exit pupil. This interrupts the smooth experience of the AR NED during the rotation of the eyeball or dislocation of the device. To enlarge this tiny eye-box, Lin et al. fabricated a two-dimensional (2D) beam deflector based on polarization-dependent components. Although they successfully achieved 3 × 3 viewpoints, the overall form factor was not as compact as the WNEDs since they used a bulky beam splitter as a combiner [13]. Kim et al. and Jeong et al. fabricated a lightguide-type Maxwellian AR NED with an enlarged eye-box using a multiplexed HOE working as multiple off-axis concave mirrors [14,15]. To avoid blank or overlapping of the virtual image that can occur in these works, Yoo et al. proposed dynamically switchable viewpoints using a polarization-dependent lens and a multiplexed HOE [16]. Furthermore, Jo et al. used multiple independent HOEs for an eye-box extended Maxwellian AR NED [17]. In the listed studies, however, the thickness of the lightguide is larger than that of the usual waveguides with the EPE technique. Unlike the usual waveguide configuration, the chief rays from all pixels have the same TIR angle in the lightguide-based configuration and are distinguished only by their spatial positions when they reach the out-coupler. If the lightguide is sufficiently thin such that spatially different information overlaps at a certain position, a ghost image or image duplication problem may occur. Therefore, the thin waveguide could not be used in the listed studies, and the EPE technique could not be applied.

In this paper, we propose a novel eye-box replicated Maxwellian AR WNED, keeping a thin waveguide form factor. The core element of the proposed system is a pin-mirror HOE (PMHOE) array out-coupler. The tiny size of the individual PMHOE reduces the effective aperture of the system, presenting virtual images with a deep DoF. The 2D array of the PMHOEs replicates the eye-box, enhancing user comfort and eye position tolerance.

Note that the small mirror HOE has been reported by Jeong et al. [18]. They introduced a reflection-type AR NED using the holographically printed freeform mirror array HOE. Although they successfully proved the Maxwellian view effect with an extended eye-box, their system has a free space projection configuration rather than a waveguide configuration. A large volume is needed for the image projection onto the HOE plane and the EPE technique cannot be applied. An AR NED with a pin-mirror array was also reported by a company [19]. Although they developed a Maxwellian display module using the pin-mirror array, the lightguide is relatively thick and the pin-mirror-embedded lightguide is not easy to fabricate. In contrast, the proposed system is based on the waveguide, achieving a thin (approximately 1 mm in our implementation) form factor and replicated eye-box. Since the PMHOE array can be simply recorded and attached to the waveguide, the proposed system has an advantage in fabrication.

Figure 2 illustrates the basic configuration of the proposed Maxwellian WNED with a single PMHOE. Unlike the conventional out-coupler HOE, the PMHOE limits the beam waist from each pixel that is out-coupled toward the eye. Since the effective exit pupil is restricted, the amount of blur in the retinal plane is reduced, giving deep DoF for the image of the pixel. As a result, the Maxwellian view effect of the virtual image is realized by the PMHOE.

 

Fig. 2. Schematic diagram of the proposed Maxwellian AR WNED with a single PMHOE. Even though the eye is focusing on the near plane, the blur is reduced compared to the conventional WNED.

Download Full Size | PDF

The size of the PMHOE is an important design factor of the proposed system. To figure out the suitable size, we conducted a simple simulation and estimated the modulation transfer function (MTF) of the proposed optics. The PMHOE and eye lens were assumed to be an ideal aperture and lens, and the on-axis virtual image was supposed to be located at the infinite plane. In Fig. 3 the MTF of the eye with a PMHOE is analyzed according to the diameter of the PMHOE and the focal length of the lens. When the eye focuses at infinity, it is obvious that the PMHOE with large diameter has high modulation depth (MD) for high frequency (blue-dashed lines, 2.0 mm diameter), because the large aperture gives good resolving power. When the eye focuses near the 40 cm plane, however, the MD decreases rapidly as the angular frequency increases. This means that this system has a shallow DoF, and the Maxwellian view cannot be realized. If the PMHOE becomes too small (red-dotted lines, 0.5 mm diameter), the deep DoF is achieved, giving negligible MTF change for different eye focus distances. Because of the diffraction from the tiny aperture, however, the high-frequency part above the cut-off frequency cannot be expressed and the MTF is generally worse. In the case of a mid-sized PMHOE (green-solid lines, 1.0 mm diameter), it has at least the same or better MTF than the small-sized PMHOE, regardless of the focal power of the eye. Although the best MTF of the mid-sized PMHOE is less than that of the large-sized PMHOE, the MTF maintains a reasonable value, 0.5 MD at 5 cycles per degree even for the most significant defocus (eye focused at 0.4 m). Therefore, in our implementation, the mid-sized 1 mm diameter of the PMHOE is selected.

 

Fig. 3. Modulation transfer functions for different PMHOE sizes and eye focus planes.

Download Full Size | PDF

A single PMHOE in the proposed system presents virtual images with a deep DoF. The eye-box, however, is limited around the lateral position of the PMHOE. In order to increase the overall eye-box, we fabricated an array of the PMHOEs. Figure 4 shows the schematic diagram of the proposed Maxwellian AR WNED with the replicated eye-box using the PMHOE array. After the ray bundles are diffracted by the first PMHOE, the rest of them propagate in the waveguide by TIRs and are diffracted by other PMHOEs. Therefore, the PMHOE array replicates the individual eye-box, increasing the overall area in which the eye can be located. Note that the eye-box replication used in the proposed system only increases the overall eye-box area while keeping the Maxwellian-view effect, unlike conventional EPE techniques. Note also that the 2D replication is easily achieved by using a 2D array of PMHOEs, while in Fig. 4 we illustrate the one-dimensional array only for simplicity. It is another advantage of the proposed system that the 2D replication does not require additional folding HOEs like conventional 2D EPE techniques. If the ray bundles from two or more PMHOEs enter the eye at the same time, however, the Maxwellian view effect can be degraded. In our system, the gap between neighboring PMHOEs is designed to be about 5 mm, which is larger than the average pupil size in bright conditions.

 

Fig. 4. Schematic diagram of the proposed Maxwellian AR WNED with a replicated eye-box using the PMHOE array.

Download Full Size | PDF

The schematic diagram and the captured photos of the experimental setup for the PMHOE array fabrication are shown in Figs. 5(a) and 5(b), respectively. After attaching a photopolymer film (Litiholo, C-RT20) onto a Littrow prism (Edmund Optics, S/N 43-649), a 3D-printed pinhole array mask, shown in Fig. 5(a), was placed in front of the photopolymer. The wavelength of the recording laser was 660 nm (Cobolt, Flamenco 500 mW). After the interference and fixing process, we detached the out-coupler and attached it to a 1-mm-thick waveguide. Thanks to the symmetry between the two couplers, the recording process for the in-coupler was conducted in a setup that was the same except for the presence of the pinhole array mask.

 

Fig. 5. (a) Schematic diagram and (b) captured photo of the experimental setup for the PMHOE array fabrication. In-coupler HOE was recorded in the same setup without a pinhole array mask.

Download Full Size | PDF

A picture of the fabricated waveguide-based optical combiner with HOE couplers is shown in Fig. 6(a). The overall size is about 25 mm × 75 mm with a thickness of 1.1 mm. The out-coupler PMHOE array is shown in Fig. 6(b). Although we recorded only nine PMHOEs in this work, more PMHOEs can be recorded or customized depending on the mask used in the fabrication. To observe the virtual image from the proposed AR WNED, we made an experimental setup as shown in Fig. 6(c). A digital light processing (DLP) projector (Qumi, Q38) is used as the display. To expand the collimated beam diameter of each pixel from the projector, we used a 4-f relay system with lenses having different focal lengths. A smartphone camera in the eye-box area is used to capture the experimental results of the implemented system.

 

Fig. 6. Captured photos of the (a) fabricated waveguide-based optical combiner, (b) recorded out-coupler PMHOE array, and (c) experimental setup for virtual image observation of the proposed AR WNED.

Download Full Size | PDF

Figure 7 and related movies show the observed experimental results from the proposed WNED. Figures 7(a) and 7(b) were taken by focusing on a plane far (infinity) from and near (40 cm) to the camera, respectively. The schematic diagram of the PMHOE array is depicted on the upper left side of each photo. In each photo, the position of the camera is illustrated as a circle filled with a red-colored cross symbol. To confirm the eye-box expansion, we located the camera and took result images at all of the PMHOEs’ positions. Since the proposed WNED is designed to present a virtual image at infinity, a clear image can be seen in Fig. 7(a), where the camera is focused at the infinite plane. Thanks to the Maxwellian view effect, clear images can also be observed in Fig. 7(b), which was taken by focusing at a near (40 cm) plane. On the right side of Fig. 7, experimental results from a conventional WNED are also shown for comparison with the proposed method. From the experimental results in Fig. 7(b), it is clear that the defocus blur is removed by the increased DoF. Finally, the in-focus images at each of the PMHOE positions for different camera focus planes demonstrate the VAC mitigation and the eye-box replication of the proposed AR WNED successfully.

 

Fig. 7. Captured photos when the camera focus plane is (a) far (infinite) from and (b) near (40 cm) the camera. The lateral position of the camera is depicted on the upper left side of each picture. Experimental results from a conventional WNED are also shown for comparison. (See Visualization 1 and Visualization 2 for the movies.)

Download Full Size | PDF

Although we experimentally verified the fundamental concept of the proposed WNED, there are still many points to be enhanced. First, the field of view (FoV) of a single PMHOE is limited by the critical angle of the waveguide and angular tolerance of the HOE. In Fig. 8(a), a k-vector diagram of the in-coupler HOE is illustrated. In the diagram, the grating vector ${\boldsymbol K}$ is determined by subtracting the object beam k-vector ${{\boldsymbol k}_{\boldsymbol o}}$ from the reference one ${{\boldsymbol k}_{\boldsymbol r}}.$ In our HOE fabrication configuration, ${{\boldsymbol k}_{\boldsymbol r}}$ only has a z-component, and ${{\boldsymbol k}_{\boldsymbol o}}$ has a diffraction angle of ${\theta _o} = {60^\circ }$ with the z-axis. If there is an angle of deviation such as ${\theta _1}$ and ${\theta _2}$ to the incident probe beam, the k-vector of the diffracted light is also changed to ${{\boldsymbol k}_1}$ and ${{\boldsymbol k}_2}$, which can be calculated by the k-vector diagram. The calculated diffraction angle in the waveguide is plotted in Fig. 8(b) (blue solid line). The angle of deviation in the air is allowed to be between ${\theta _1}$ and ${\theta _2},$ making the diffraction angle larger than the critical angle ${\theta _c}$ (blue dashed line) and smaller than 90°. As shown in Fig. 8(b), the usable angular range of the incident light is calculated to be 28.88° in the air. Since the HOE is a rigorous angular selector, however, the FoV is also affected by the angular tolerance of the HOE couplers. Based on the coupled wave theory of Kogelnik [20], the normalized diffraction efficiency (DE) considering the in- and out-coupler is plotted in Fig. 8(b) (orange solid line). The full-width at half-maximum of the normalized DE is 5.36°, which restricts the FoV of the proposed WNED. In our experimental results, however, the actual FoV of the virtual image was measured to be 9.29°, including the low brightness part due to the small DE. Since this value is insufficient for an immersive AR experience, FoV enhancement should be applied in our system. The use of multiple waveguides may be one approach. By fabricating several waveguides that cover different parts in the FoV, the overall FoV can be enhanced. A HOE multiplexing technique can also be used with a single waveguide, by recording HOEs to cover the different parts of the microdisplay. Another limitation of the proposed method is that the virtual image plane is still at the infinite plane. If this plane is located at the hyperfocal distance of the human eye, the MTF of the proposed optical system can be enhanced. We believe that this can be achieved by recording the PMHOE to have a slight negative optical power with the freeform HOE fabrication technique [21]. Finally, the resolution of the displayed images is limited by the small exit pupil size as shown in Fig. 3. We think that the subpixel shifting technique [22] can be considered to enhance the effective resolution over the MTF limit of the proposed system.

 

Fig. 8. (a) k-vector diagram of the in-coupler HOE. (b) Diffraction angle in the waveguide (sweeping blue line) and overall DE (narrow orange line) considering the in- and out-couplers according to the angle deviation of the incident light.

Download Full Size | PDF

In conclusion, we propose a Maxwellian AR WNED using the PMHOE array as an out-coupler. The proposed configuration is based on a 1-mm-thick waveguide, achieving a compact form factor. The Maxwellian view was realized by each PMHOE, and the restricted eye-box of the Maxwellian display was extended by the array of PMHOEs. The experimental results confirm that the in-focus virtual images with 9.29° FoV can be presented within the 4.5 (H) × 2 (V) times extended eye-box, regardless of the camera focus plane.