Open Access
Add to BibSonomy
Abstract
We propose a lightguide-type super multi-view near-eye display that uses a digital micromirror device and a LED array. The proposed method presents three-dimensional images with a natural monocular depth cue using a compact combiner optics which consists of a thin lightguide and holographic optical elements (HOEs). Feasibility of the proposed method is verified by optical experiments which demonstrate monocular three-dimensional image presentation over a wide depth range. We also analyze the degradation of the image quality stemming from the spectral spread of the HOEs and show its reduction by a pre-compensation exploiting an adaptive moment estimation (Adam) optimizer.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Virtual reality (VR) and augmented reality (AR) displays provide users with a three-dimensional (3D) experience, and they are regarded as the core component for reaching the true potential of the metaverse [1–3]. Despite the soaring need for immersive devices for VR and AR, commercialized products have yet to meet this demand. Near-eye display (NED) is one of the most vibrant research areas for its capability of achieving both immersive feeling and comfortable wearing in VR/AR experience. However, conventional NEDs suffer from vergence-accommodation conflict (VAC) problem, causing visual fatigue and discomfort [4–6]. VAC originates from a discrepancy between the vergence and accommodation of a human visual system. Human eyes perceive the depth of objects using various depth cues including: the rotation of both eyes that is related to vergence distance, and the focal power of the individual eye lens which is associated with an accommodation distance. The difference between the vergence and accommodation distance confuses the human visual system, causing the visual fatigue. Incompatibility between the accommodation and vergence cues of a virtual image can be eliminated by providing precise focus cues to individual monocular images. Various methods, such as Maxwellian NED [7–10], multi-focal NED [11–13], vari-focal NED [14–17], holographic NED [18,19] and lightfield NED [20–28] have been introduced to mitigate the VAC problem.
A lightfield NED provides the accurate focus cue by reconstructing the spatial-angular distribution of light rays for a 3D scene to be displayed. Integral imaging with a micro lens array (MLA) [20–23] and super multi-view (SMV) [24–28] technique have been suggested to implement the lightfield NED. In the integral imaging based lightfield NED, diverging light from each display pixel is converted to a ray pencil with a specific direction after passing through the corresponding lens in the MLA. The collection of the rays formed by all display pixels and the MLA approximates the continuous spatial-angular distribution of the light rays coming from a 3D scene. When the angular resolution of the ray collection is sufficiently high, the integral imaging based NED can produce natural 3D scenes without the VAC problem. However, due to the limited display panel resolution, the continuous ray distribution can only be approximated by a sparsely sampled discrete light rays, which generally induces a spatial resolution loss imposed by a trade-off between the spatial and angular ray sampling rate. The SMV-based lightfield NED, to the contrary, can create VAC-free 3D images with high spatial resolution, usually using a time-multiplexing scheme [29]. A SMV display creates multiple viewpoints that are densely packed with an interval smaller than an eye pupil. It allows the rays from multiple viewpoints are projected on the retina together, enabling the eye to focus on 3D images without the VAC problem. Wang et al. [25] suggested a Maxwellian-viewing-SMV NED system based on a Pancharatnam-Berry optical element (PBOE) to generate 3D images with a large depth of field (DOF). However, multiple views are displayed with a spatial-multiplexing in their method, resulting in spatial resolution loss of the 3D image. Liu et al. [26] proposed a SMV NED with an enlarged field of view (FOV) using a gating aperture array. Although they successfully demonstrated the feasibility of a wide FOV, the number of the views is limited owing to the slow refresh rate of the display panel, which impedes the expression of a natural 3D image. Ueno et al. [27] developed a SMV NED system using a time-multiplexing technique with a fast refresh rate display to offer natural 3D images with a full-parallax. Their work, however, uses a beam splitter optical combiner which makes the overall system bulky.
In this paper, we propose a lightguide type optical-see through SMV NED using a digital micromirror device (DMD) and a LED array. The proposed method combines the lightguide combiner with the SMV technique, achieving a compact form factor and a natural monocular depth cue. The combiner optics, consisting of a relatively thin lightguide, and film-shaped holographic optical elements (HOEs), contributes to the compactness of the whole system. SMV images presented by the synchronization between the DMD and LED array present the 3D images with correct monocular focus cues. In the following, we explain the principle and system configuration of the proposed method. System implementation and the optical experimental results that verify the feasibility of the proposed method are also presented. Finally, an image pre-compensation technique using an adaptive moment estimation (Adam) optimizer [30] is presented to enhance the image quality of the proposed system.
2. Basic principle
Before describing the details of the proposed lightguide type SMV NED, we explain the basic principle of a lightguide and SMV NED in this section.
2.1 Lightguide type AR NED
Figure 1 shows two configurations of the lightguide type AR NEDs. A conventional NED consists of a flat panel display, a lightguide combiner, and in/out couplers as shown in Fig. 1(a) [31]. Light emanating from each display pixel is collimated by a lens and diffracted by the in-coupler HOE that functions as a slanted mirror, coupling the light into the lightguide. After propagating through total internal reflections (TIRs) within the lightguide, the light is extracted toward the eye by the out-coupler HOE that also works as a slanted mirror with zero focusing power. Meanwhile, the light coming from the real object passes through the out-coupler HOE without being diffracted due to the high angular/spectral selectivity of the HOE [32]. One of the major issues of this configuration is that the virtual image plane is located at a fixed depth, evoking VAC problems and preventing long-term wearing.
Fig. 1. Schematic diagram of (a) the conventional NED and (b) Maxwellian NED with a lightguide combiner. Arrows with different line types represent light rays from different display pixels.
Another configuration that is called a Maxwellian NED is shown in Fig. 1(b). The Maxwellian NED uses a display with a narrow exit pupil and an out-coupler HOE having a focal power. The out-coupler HOE focuses the light from the display to a single spot in a pupil plane as depicted in Fig. 1(b). Because the effective eye pupil is reduced by the size of the focal spot for the virtual images, the user can observe always-in-focused images regardless of the focal power of the eye lens, which relaxes the VAC problem. The Maxwellian display, however, only expands the DOF of the images while the displayed images are still 2D. A true 3D display should present defocus blur of the image according to the image depth and the eye focal distance. To the contrary, a Maxwellian display only gives always-focused images due to its extended DOF, failing in presenting the defocus blur. The Maxwellian displays, therefore, only detour the VAC problem, not solving it completely.
2.2 SMV AR NED
Figure 2(a) depicts the core principle of the SMV technique. The SMV technique samples the light rays from the object only at a few specific spots, i.e., viewpoints in the eye pupil plane. The light rays sampled at the spatial positions are equivalent to the perspective views at the corresponding viewpoints. The SMV AR NED provides a user with multiple perspective views at the corresponding viewpoints in the pupil plane by exploiting the imaging technique of the Maxwellian display. A simplified SMV AR NED configuration employing a temporal-multiplexing technique is shown in Fig. 2(b). A display engine with a fast refresh rate projects the multi-view images and the combiner converges them to corresponding viewpoints in the pupil plane. These views are presented in a time-multiplexing manner such that the user observes multiple all-in-focus images almost simultaneously. The disparity between these images drives a natural accommodation response of the eye, presenting 3D images without the VAC problem. Although the SMV display techniques have long been studied, a SMV AR NED with a compact lightguide combiner has yet to be proposed to our best knowledge. In this paper, we propose a novel design of the NED using a lightguide combiner and the SMV technique to provide the user with natural 3D images in a compact system.
Fig. 2. (a) Observation of 3D image in SMV technique and (b) a simplified example configuration of the SMV AR NED with time-multiplexing technique.
3. Proposed method
3.1 System configuration of the proposed method
A schematic diagram of the proposed method is illustrated in Fig. 3. We adopt and synchronize a LED array and a digital micromirror device (DMD) to generate multi-view images. The refresh rate of the DMD and LED array is fast enough to create sufficient number of images within the eye integration time. A lightguide configuration of the Maxwellian NED is also adopted with the reflection-type in-/out-coupler HOEs in the bottom and top plane of the lightguide, respectively. The out-coupler HOE has a focal power so that each view image converges to the corresponding spot in the pupil plane while keeping a compact system size. Because the LED array with a wide spectral linewidth is used as a light source, a bandpass filter is additionally placed before the in-coupling to reduce the spectral spread by the HOEs. The spectral spread of the light originates from a grating structure recorded in the HOEs. Dispersion stemming from the grating-recorded HOE evokes the spread of the incident light depending on its wavelength, which we called the spectral spread in this paper. The spectral spread is not usually a problem in conventional lightguide NEDs shown in Fig. 1(a) because it is compensated by the symmetric grating structure of the in-/out-coupler HOEs [31]. In the Maxwellian configuration shown in Fig. 1(b), the grating structure of the in-/out- coupler HOEs are not symmetric, requiring the consideration of the spectral spread.
Fig. 3. Schematic diagram of the proposed SMV AR NED. Arrows with different line types represent different views.
The overall light path of the proposed configuration is as follows. The light emitted from each LED is spectrally filtered by the bandpass filter and collimated by the lens. The collimated beam is initially incident on the in-coupler HOE but passes through it without diffraction because the incident angle is far from the Bragg angle of the in-coupler HOE. After passing through the in-coupler HOE, the light is modulated to a perspective view image by the DMD and perpendicularly incident on the in-coupler HOE again. The incident angle is now close to the Bragg angle and thus the light is diffracted to be coupled into the lightguide. The light then propagates through TIRs within the lightguide, being finally extracted and converged to the corresponding viewpoint in the pupil plane by the out-coupler HOE. Note that the viewpoints corresponding to different LEDs are formed at slightly different positions in the pupil plane because the light from different LEDs are collimated with slightly different angles before the in-coupling. This viewpoint shift is exploited to generate the SMV 3D images as explained below. Also note that the slightly different collimation angle makes the view image beams meet the out-coupler HOE at slightly different transverse positions. This makes the view images to be observed through slightly shifted windows, which is explained with more details in sections 3.2 and 3.3.
Figure 4 shows an example operation of the proposed display system where a star image at 100 cm depth is used as a target 3D image. For simplicity, only three horizontal view images are considered in Fig. 4. To display the 3D star at 100 cm in front of the eye as shown in Fig. 4(a), the view images with the corresponding disparity are presented as shown in Fig. 4(b). Because the view images are projected to the eye retina through the corresponding small focal spots in the pupil plane, the individual view has extended DOF, remaining sharp regardless of the focal length of the eye. However, their relative position in the eye retina changes according to the eye focal length as shown in Fig. 4(c). When the eye focus is at 100 cm, three view images are perfectly superimposed together in the retina plane so that a sharp single image is observed. To the contrary, when the eye is focused at other depth, such as 10 cm, or 50 cm, three view images are now mismatched, which is seen as a blur in the observation. In this way, the proposed configuration presents multiple 2D view images to the eye through slightly different focal spots in the eye pupil plane, displaying the 3D image. The disparity between the 2D view images provides the user with natural monocular depth cue, solving the VAC issue.
Fig. 4. An operation example of the proposed method. Only 3 horizontal views are shown for simplicity. (a) Overall configuration. (b) Input view images loaded to the DMD. (c) Observed images at different eye focal distances
3.2 Detailed design of the proposed system
In the proposed method, the collimated beam from each LED meets the out-coupler HOE at a slightly different transverse position as shown in Fig. 5. The beam extent in the out-coupler HOE works as a window through which the corresponding perspective view is observed. The transverse shift of the observing windows can be calculated by
Fig. 5. Illustration of the window shift in the proposed method. Colored solid lines in the out-coupler HOE plane indicate the windows of the corresponding views.
Fig. 6. (a) Schematic diagrams for calculating the window pitch. (b) A k-vector diagram of the in-coupler HOE.
The angle ${\tau _{x,k}}\; $ of the light incident on the in-coupler HOE is obtained from the LED pitch ${p_{LED}}$ and the focal length of the collimation lens ${f_c}$ considering the refraction at the air-lightguide interface. Thereby, ${\tau _{x,k}}$ is derived as
For the incident light with an angle of ${\tau _{x,k}}$, the in-coupler HOE diffracts the light with an angle of $\theta + {\varphi _{x,k}}$ as shown in Fig. 6(b). The diffracted angle $\theta + {\varphi _{x,k}}$ is determined from the grating vector cloud of the in-coupler HOE which is normal to the HOE plane [33,34]. Therefore, we derive ${\varphi _{x,k}}$ as
The window pitch in the vertical direction pwy can be obtained from the TIR angle deviation in the vertical direction ${\varphi _{y,k}}$. The result of the 3D k-vector analysis confirms that ${\varphi _{y,k}}$ is equal to the vertical angle of incident light ${\tau _{y,k}}$. With the same assumptions as applied to pwx, we approximately derive pwy as
Figure 7 shows a simplified optics for the viewpoint pitch calculation. In the proposed method, the spacing between the viewpoints in the pupil plane needs to be smaller than the eye pupil so that multiple view images are projected to the eye. Given a LED pitch pLED in the array, the corresponding viewpoint pitch pv in the eye pupil plane can be approximately calculated by
where ${f_o}$ is the focal length of the out-coupler. In the actual implementation, the converging light diffracted by the out-coupler HOE passes through two different mediums, lightguide and air, such that the effective focal length reduces. The effective focal length fo can be obtained byThe proposed method presents the individual view with a large DOF by focusing it at a small spot, i.e., viewpoint in the eye pupil plane, following the Maxwellian display principle. In usual lightguide type NEDs, the spatial pixel distribution in the display panel is translated into angular distribution of the collimated beams inside the lightguide, thus the number of the TIRs does not need to be uniform for all rays and the beam from each display panel pixel cover the entire out-coupler HOE area. To the contrary, the Maxwellian display configuration of the proposed NED requires the one-to-one correspondence between the beam from each image pixel and the location in the out-coupler HOE region. The number of the TIRs needs to be kept the same for all rays. As can be seen in Fig. 8, failing to meet this requirement results in duplicated images.
Fig. 8. Illustration of the disconnected image when the light is diffracted twice by the out-coupler. Red bold line indicates the undesirably diffracted light.
From Fig. 8, the condition to meet the requirement can be derived by
In implementation, we selected the lightguide thickness and size of the out-coupler that satisfy the above conditions.
3.3 Multi-view image acquisition
The proposed method requires multiple perspective views to create a 3D image. Placing a pinhole camera at each viewpoint and capturing the 3D objects would be the simplest way to obtain the multi-view images. In our methods, however, each view image has a different window in the out-coupler HOE as shown in Fig. 5, which needs to be considered in the image acquisition. A simplified illustration of the multi-view images acquisition is indicated in Fig. 9.
Fig. 9. Illustration of the multi-view images acquisition through the corresponding windows. Colored rectangles represent the windows for the corresponding viewpoints.
Although the location of the window can be calculated from Eqs. (3)–(5), we exploited another method that contemplates the parameter obtained from an experiment to create the accurate multi-view images.
Figure 10 depicts the relationship between the viewpoints and the corresponding windows. Due to the window position difference with the pitch pw, the disparity in the observed images, i.e., dw in Fig. 10 is different from the original disparity in the rendered multi-view images. The disparity dw in the out-coupling HOE plane is related with the object distance dobj by
where ${d_e}$ is the eye relief. Note that when the original disparity in the rendered multi-view images is zero, or when all multi-view images are the same, the observed disparity dw becomes the window pitch pw. We utilize this property to acquire the actual window pitch ${p_w}$. The detail method is as follows. (1) Display the same image such as a grid pattern at all viewpoints. (2) Find ${d_{obj}}$ by adjusting the focal power of the camera. (3) Obtain dw from Eq. (11) that is the same as pw in this case. The obtained pw value is used in the multi-view image rendering for arbitrary depths.Fig. 10. Relationship between the disparity and the image depth when multiple viewpoints have different windows in the out-coupler HOE plane.
4. Implementation
4.1 Verification setup
Figure 11(a) shows a photo of our implementation of the proposed method. The light source is a LED array with pLED = 2 mm pitch (Adafruit, Product ID: 5362) that is driven by a Raspberry pi board. A 532 nm bandpass filter with the full width at half maximum (FWHM) of 10 nm was placed in front of a collimation lens with the focal length of fc = 175 mm. The lightguide, collimation lens, and bandpass filter were placed on a homemade custom mount fabricated by a 3D printer (Flashforge, creator3). The lightguide is made of glass with 6 mm thickness. A DMD (Texas Instruments, DLPLCR6500EVM) with a resolution of 1920 × 1080 was employed as a light modulator. The LED array and the DMD were synchronized using trigger signals generated by the Raspberry pi board. The maximum refresh rates of the LED array and the DMD in our implementation were 3,840 Hz and 9,523 Hz, respectively. In the experiment, we generated 3 × 3 views, the maximum number of the views that can be projected into the small aperture (about 2 mm) of the camera (Samsung, Galaxy S21 Ultra). Each view was presented sequentially with a refresh rate of 540 Hz, achieving 60 Hz 3D image display with 3 × 3 views. A picture of 3 × 3 viewpoints captured in the pupil plane is shown as an inset of Fig. 11(a). Because 9 views are arranged in the 3 × 3 grid, our implementation realizes the defocus image blur not only in the horizontal direction but also in vertical direction, providing full parallax 3D images.
4.2 Details of the lightguide combiner design
We designed the lightguide considering Eqs. (8), and (10). The values of the parameters exploited in the experiments is in Table 1. With these values, Eq. (10) is satisfied and the value of the left term of Eq. (8) is 18.22. To satisfy Eq. (8), we chose 20 mm as the width of the out-coupler.
Table 1. Parameters of the experimental setup
Figure 12(a) shows a picture of the experiment setup for recording the in-coupler HOE that functions as a slanted mirror. A photopolymer film (Litiholo, C-RT20) was used as a recording medium. A collimated s-polarized beam from a 532 nm laser was split into a reference and signal beam by a beam splitter (BS). The reference beam was reflected by a mirror (M2) and incident on the photopolymer attached on a 60° Littrow prism (Edmund Optics, S/N 43-649). The signal beam was reflected by another mirror (M1) and incident on the slanted plane of the prism. Since the nominal TIR angle $\theta $ in our implementation is $\; 55^\circ $, the signal beam was tilted slightly with respect to the inclined plane of the prism. Figure 12(b) portrays the recording setup of the out-coupler HOE that functions as a slanted convex lens. The recording setup of the out-coupler HOE is the same as that of the in-coupler HOE except that a lens of 75 mm focal length was placed in front of the prism to give the focal power.
Because of the discrepancy between the center wavelength of the LED and the recording laser, angular pre-compensation of the signal/reference beam is required for proper operation of the HOEs. Placing a 532 nm bandpass filter before the coupling to the lightguide may also affect the center wavelength of the input light to the HOEs. We measured the spectrum of the input light after the bandpass filter using a spectrometer (Ocean Insight, FLAME-S), and the center wavelength of the input light was measured to be 527 nm. To fabricate the in-coupler HOE diffracting 527 nm light with the angle of $55^\circ $, the pre-compensated angle of the reference and signal beam with a wavelength of 532 nm was calculated to be $1.05^\circ $ and $54.40^\circ $, respectively, exploiting a k-vector analysis [35,36]. Because the center grating vector of the out-coupler HOE is symmetric with that of the in-coupler, the angles of the two beams recording the out-coupler HOE were exactly reversed to those of in-coupler HOE.
4.3 Details of the multi-view image acquisition
Multi-view images for a 3D object were rendered using a software, i.e., Blender by placing a virtual camera at the viewpoints. As explained in section 3.3, the location of the window of each viewpoint, i.e., ${p_w}$ obtained by exploiting Eq. (11) should be considered. In our setup, ${d_{th}}$ is 6 mm, ${d_e}\; $ and ${p_v}$ are approximately 45 mm and 0.586 mm, respectively. The window pitch in horizontal/vertical axis calculated by Eqs. (3)–(5) was 0.738 mm and 0.236 mm, respectively. We also estimated ${p_w}$ experimentally using the method explained in section 3.3. We turned on all 3 × 3 LEDs and displayed a grid pattern on the DMD. We then found the in-focus depth of the image by adjusting the focus of the camera. The measured in-focus depth was 3 diopters for the horizontal lines of the grid pattern and 2 diopters for the vertical lines. The corresponding ${p_w}$ was approximately 0.526 mm in horizontal axis and 0.496 mm in vertical axis, which were considered in the view image rendering. Inconsistency between the numerically calculated value and the experimentally estimated value is expected from ignoring the spatial shift between multi-view image beams before the in-coupling, experimental errors, and the horizontal blur of the output image that will be explained in section. 5.
4.4 Preliminary experiment result
We verified by optical experiments that the proposed method produces a natural monocular depth cue. In the first experiment, we tested whether our method presents the images at a desired depth or not. The multi-view images rendered for the ‘INHA’ letters that are located at 50 cm in front of the camera were uploaded to the DMD and illuminated by the LED array with time-multiplexing manner.
As can be seen in Fig. 13, when the camera was focused on 50 cm, the observed “INHA” image becomes the sharpest. When the camera was focused on different depth such as 10 cm,100 cm, and infinity, blurry “INHA” appeared. Note that when the distance between the camera focus plane and the 50 cm depth plane where “INHA” is located was large, severe blur was observed. This ensures the “INHA” image was displayed at the designated depth properly. In the second experiment, two multi-depth images were displayed. A lion and a warning patch were placed at 30 cm depth, and an elephant and a fire patch were located at 300 cm from the camera as depicted in Fig. 14(a).
Fig. 13. (a) Original depth of the target object ‘INHA’ and the captured output images when the camera was focused at (b) 10 cm, (c) 50 cm, (d) 100 cm and (e) infinity.
Fig. 14. (a) Original depths of the target two-depth objects and the captured output images when the camera was focused at (b) 10 cm, (c) 30 cm, (d) 100 cm and (e) 300 cm. (Visualization 1)
Figure 14(b)-(e) shows the pictures captured when the focus plane of the camera was at 10 cm, 30 cm, 100, and 300 cm. Sharper images were observed when the camera was in-focus to each image. As in the single depth case shown in Fig. 13, the larger discrepancy between the camera focus plane and the image plane makes images blurrier, showing proper accommodation effects. The experimental results in Fig. 14 verify that the proposed method can cover a wide depth range from 30 cm to 300 cm which is important in practical AR/VR applications.
5. Image pre-compensation
In this section, the image quality of the system is analyzed along with its degradation factors. We also present a pre-compensation algorithm that employs the Adam optimizer for the enhancement of the image quality.
5.1 Degradation factors of the image quality
As can be seen in Figs. 13 and 14, a weak but discernable blur is still observed when the image is focused, degrading the image quality. There are a few causes of this image blur. First, the LED is not an ideal point source. The LED used in our implementation has small but finite sized active area of a circular shape. Light from individual point inside the active area of the LED is projected at a slightly different position in the retina plane, causing an image blur in the observation.
Second, the horizontal (i.e., x axis in Fig. 15) blur of the projected image was induced by a spectral spread of the in-coupler HOE. The diffraction angle of the in-coupler HOE depends on the wavelength of the input light. Different wavelength components of an individual LED light spread in the out-coupler HOE region, making the observed image hazy. Note that the spectral spread of the HOE does not increase the blur in the vertical (i.e., y axis in Fig. 15) direction. As depicted in Fig. 15, the grating vector of the HOE interacts only with the x and z components of the reference/signal beam, leaving the y component unchanged.
Fig. 15. Schematic diagram that depicts the path of the light reflected from a single pixel of the DMD and the corresponding k-vector diagram of the in/out-coupler HOE (Regarding the k-vector diagram of out-coupler HOE, only the chief ray is depicted for simplicity).
Third, the DMD can be considered as a blazed grating and thus spectral spread by the DMD also exists. A small blur of the image in the transverse direction would be generated by the DMD. Finally, a slight mismatch between the multi-view images stemming from the experimental errors worsens the image quality.
Among them, the blur originating from the finite active area of the LED is small because the radius of the LED active area is less than 1 mm in our implementation. The spectral spread by the DMD is also not severe due to its small diffraction angle. However, the spectral spread by the HOE is not negligible as explained in the following section. Therefore, we devise an iterative pre-compensation algorithm that compensates the spectral spread of the in-coupler HOE.
5.2 Pre-compensation of the input image
To enhance the quality of the output image of the proposed method, a pre-distorted input image which compensates the spectral spread by the in-coupler HOE was obtained using a point-spread function (PSF)-based iteration algorithm. As the first step, the PSF of the proposed system was approximately calculated using a k-vector analysis, only considering the effects of the spectral spread of the in-coupler HOE. Because the spectrally-induced spatial-spread of the image within the lightguide is directly related to the wavelength range, we measured the spectrum of the output image using a spectrometer. The center wavelength and the FWHM of the output image was measured to be 527 nm and 4 nm, respectively, as shown in Fig. 16(a). In this range, an infinitesimal spot of light in the in-coupler plane is laterally spread to be 0.5178 mm width in the out-coupler plane after traveling through the lightguide. Considering the DMD pixel pitch of $7.56\mu m$ in our implementation, this is equivalent to that a single pixel in the input image is spread to the horizontal line of 69 pixel width after passing through the proposed system. Employing this result, we approximate the PSF to a Gaussian function with the FWHM of the 69 pixels as indicated in Fig. 16(b).
Fig. 16. (a) Measured spectrum of the output light, (b) calculated PSF of the proposed system, and (c) the loss function of the implemented iterative pre-compensation algorithm.
Figure 16(c) shows a loss function (L2 loss) of the iterative pre-compensation algorithm that we have implemented. We obtained the pre-compensated image, employing an Adam optimizer that minimizes the L2 loss of the system. Adam optimizer, akin to stochastic gradient descent (SGD) [37,38], is a gradient-based optimization method that minimizes the loss function by updating system parameters in the direction of a negative gradient of the loss. Going further from SGD, Adam adopts the exponentially decaying average of squared gradients and gradients in the update formula, considering the gradients of the former iteration. These terms prevent convergence to local minima and enable application of an adaptive learning rate at each iteration, which speeds up the convergence to global minima [30]. In this paper, Adam optimizer is used to update the values of the input image matrix during the iteration. The number of iterations for each view image was 40 and the obtained pre-compensated images of 3 × 3 views were uploaded to the DMD and illuminated from the LED array.
Figure 17 shows the effect of the pre-compensation. The top row in Fig. 17 shows the center view out of 9 total input view images for an airplane and dump truck object which are displayed at 50 cm and 100 cm depth, respectively. The output 3D image captured by a camera focused on the corresponding depth is shown in the bottom row of Fig. 17. As can be seen in Fig. 17, the pre-compensation suppresses the blur in the output image, improving its quality. Although a basic Adam optimization is employed in our current implementation, further enhancement is expected by applying a more sophisticated algorithms such as the recently proposed camera-in-the-loop optimization [38].
6. Experimental results
Figure 18 shows the final experimental results of the proposed system with the image pre-compensation. Various images at different depths were displayed to verify the proposed method. Figure 18(a) shows captured photos of 6 output images at 4 different camera focus (10 cm, 50 cm, 100 cm, infinity). Based on the image location indicated in the rightmost column, it shows that the image becomes clearer when the camera focus matches the image location while it becomes blurrier as the camera focus deviates from the image location. Figure 18(b) is the result of the object having a continuous depth from 30 cm to 100 cm. The sharpest part of the image varies according to the focus of the camera, which proves that the continuous depth can be expressed in the proposed method successfully.
Fig. 18. Final experiment results with the image pre-compensation. (a) Discrete depth objects at 50 cm or 100 cm. (Visualization 2) (b) Continuous depth objects spanning from 30 cm to 100 cm. (Visualization 3)
7. Discussion
Although 3D image presentation with monocular depth cue has been successfully demonstrated, the performance of the proposed method is limited by some factors. First, using the LED array as a light source brings blurry images, even after the pre-compensation is applied. As mentioned in Section 5, image blurring is caused by the spectral spread of the coupler, which may be solved in two ways; employing an achromatic coupler or a light source with narrow spectral bandwidth. As a pure achromatic coupler, a cascaded mirror or a hybrid coupler combining diffractive and refractive coupler can be adopted [31]. A laser or a super-luminescent LED can be used as a light source with narrow linewidth. Second, our current implementation is limited to the monochromatic images with green color. Full color display would be possible by using a RGB LED array with appropriate spacing and offset considering the diffraction property of the DMD.
The FOV of the current implementation is about $16.7^\circ \times 9.7^\circ $ which is limited by the DMD size of $14.39 \times 8.09$mm and long eye relief of 45 mm. By using a display panel with larger size or an out-coupler HOE with shorter focal length, the FOV can be expanded further. The size of the DMD, however, is restricted by system parameters such as the thickness of the lightguide and the number of TIRs, and the TIR angle of the light, as indicated in Eq. (10). From Eq. (10), increasing the thickness of the lightguide would be the simplest way to implement a larger DMD, but it results in a large form factor of the whole system. Therefore, there is a tradeoff between the FOV and the size of the system. The upper bound of the DMD width calculated from the current setup parameters is 16.18 mm which is larger than the actual size of the implemented DMD of 14.39 mm. Assuming that the lightguide thickness is increased to 10 mm, the width of the DMD can be extended to approximately 27 mm. Note that the primary reason for the small FOV of the current system is the long eye relief. If the eye relief is decreased to 15 mm in the current setup, the obtainable FOV could increase to approximately 47° ${\times} $ 27°.
The eyebox of our implementation is about $1.37 \times 1.37$ mm and proportional to the number of the viewpoints and the viewpoint pitch. As more LEDs are used in the system, i.e., as the number of the perspective views increases, the viewpoint array region in the pupil plane expands, resulting in enlarged eyebox. Considering the refresh rate of the LED array (=3,840 Hz) and the DMD (=9,523 Hz), the maximum number of the LEDs or viewpoints is $8 \times 8$ for a monochromatic 60 Hz display or $5 \times 4$ for a full-color 60 Hz display.
In the proposed method, the main requirement of the display engine is the driving speed that satisfies the sequential presentation of the multiple-view images within the eye integration time. Although a DMD is used in our implementation, a liquid-crystal-on-silicon (LCoS) that is widely used in the NED configuration can also be an alternative. Recent technological advance of the LCoS enables the fast frame rate, making it applicable to the proposed method [39]. The small form factor, high resolution, and high brightness of the LCoS could give additional benefit in the system implementation.
Our current implementation presents the 3D images with a long DOF, and the defocus blur of the images is small. Figure 19 shows the photos of the horizontal lines located in 2D in front of the camera and captured by the camera focusing at 1D, 1.5D, 2D, 2.5D, and 3D. For comparison, we captured the images not only for the 3${\times} $3 view case but also for the single view case. As portrayed in the captured photos in Fig. 19, a defocus blur is not observed when only a single center view is projected. To the contrary, when the 3${\times} $3 views are projected, the defocus blur is noticeable especially at 1D and 3D camera focus. Bottom row of Fig. 19 shows the cross-section along the red line in the captured photos. As shown in the graphs, the width of displayed horizontal line does not change much regardless of the camera focus in the single center view case. The cross-section graph of the 3${\times} $3 view case, however, clearly shows the increased width at 1D and 3D, proving the defocus blur of the image. From the above result, the DOF of the system is estimated around 1D. The 1D DOF, however, is still large compared to the DOF of the human eye. The main reason is the small area of the viewpoint array in the pupil plane which restricts effective entrance pupil of the observer. The viewpoint array area can be simply expanded by increasing the LED spacing. Note that it was deliberately kept small in our implementation to make it fall inside the small aperture of the camera.
Fig. 19. Captured varifocal photos of the horizontal lines located in 2D in front of the camera and the cross-section graph of red lines in the images.
The total diffraction efficiency of the HOEs in the fabricated setup was measured using a 532 nm laser employed at the recording process. The lights passing through the in-coupler HOE without diffraction and the lights out-coupled from the lightguide after TIR was measured to calculate the total diffraction efficiency. The measured intensity of the out-coupled light and the non-diffracted light was 47.23 and 41.51uW/cm, respectively. In consequence, the calculated efficiency of the whole system was about 53%. The light source adopted in the method, however, is the LED with the large linewidth which further decreases the effective efficiency of the whole system, provoking the low brightness not compatible with the bright AR environment. We believe that the energy efficiency of the current system would be refined with the application of the light source with the shallow linewidth.
Finally, the background real see-through scene was missing in all the experiment results because of the large DMD board as shown in Fig. 11(b). To test the optical see-through ability of the proposed method, we rotated the lightguide by 90 degrees. The out-coupler region was out of the DMD board area after rotation, and we tested the optical see-through ability.
Figure 20 shows the results of the see-through ability test. We placed the real target at 2D and projected the virtual image at 4D as indicated in Fig. 20(a). The experiment setup in Fig. 20(b) was used for the test. As shown in Fig. 20(c)-(f), the “INHA” letter is superimposed on the real scene and the “INHA” and the real target are focused respectively according to the focus of the camera. The proposed method, therefore, can provide the AR experience with the proper depth expression. Note that not the horizontal blur but the vertical blur of the image was observed due to the 90 degrees of rotation of the lightguide. We believe that this DMD board size issue can be resolved by separating the DMD from its driver board.
Fig. 20. Experiment for testing the optical see-through ability. (a) Original depth of the real target object and virtual image. (b) Experimental setup with a 90° rotation of the lightguide. The captured output images when the camera was focused at (c) 10D, (d) 4D, (e) 2D, and (f) 0D.
8. Conclusion
In this paper, we proposed an optical see-through SMV NED with a compact combiner optics. A synchronized DMD and LED array with a fast refresh rate in the proposed system creates multi-view images, employing Maxwellian imaging principle for each view. The combiner comprised of a thin lightguide and film-shaped HOEs delivers the multiple views to the eye pupil in a compact form factor. The disparity between the views generates the full-parallax 3D images, providing a continuous depth sensation in a wide range. In experiment, a benchtop setup with a 6 mm thickness lightguide combiner was implemented. The implemented system generates 3${\times} $3 views inside the 2 mm eye pupil, presenting full-parallax 3D images in the depth range from 30 cm to 300 cm, successfully. An input image pre-compensation technique was also proposed to mitigate the image degradation owing to the spectral spread property of the HOEs, which was verified experimentally, showing the suppression of the horizontal blur.
Funding
Institute of Information and Communications Technology Planning and Evaluation (2020-0-00929); National Research Foundation of Korea (NRF-2022R1A2C2013455); Samsung (SRFC-IT1702-54).
Acknowledgements
This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIT) (No. 2022R1A2C2013455, Research on occlusion-capable holographic augmented reality 3D near-eye display, 50%), Samsung Research Funding Center of Samsung Electronics (SRFC) under Project (No. IT1702-54, Seamless Solution for Immersive Transformation, 25%) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00929, Development of Digital Hologram Window Reconstruction, 25%).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. J. Xiong, E. L. Hsiang, Z. He, T. Zhan, and S. T. Wu, “Augmented reality and virtual reality displays: emerging technologies and future perspectives,” Light: Sci. Appl. 10(1), 216 (2021). [CrossRef]
2. C. Chang, K. Bang, G. Wetzstein, B. Lee, and L. Gao, “Toward the next-generation vr/ar optics: a review of holographic near-eye displays from a human-centric perspective,” Optica 7(11), 1563–1578 (2020). [CrossRef]
3. Y. Li, Q. Yang, J. Xiong, K. Yin, and S. T. Wu, “3D displays in augmented and virtual realities with holographic optical elements,” Opt. Express 29(26), 42696–42712 (2021). [CrossRef]
4. M. Urvoy, M. Barkowsky, and P. Le Callet, “How visual fatigue and discomfort impact 3D-TV quality of experience: A comprehensive review of technological, psychophysical, and psychological factors,” Ann. Telecommun. 68(11-12), 641–655 (2013). [CrossRef]
5. D. M. Hoffman, A. R. Girshick, K. Akeley, and M. S. Banks, “Vergence-Accommodation Conflicts Hinder Visual Performance and Cause Visual Fatigue,” J. Vis. 8(3), 33 (2008). [CrossRef]
6. M. Lambooij, M. Fortuin, I. Heynderickx, and W. IJsselsteijn, “Visual discomfort and visual fatigue of stereoscopic displays: a review,” J. Imag. Sci. Tech. 53(3), 30201-1 (2009). [CrossRef]
7. C. Yoo, M. Chae, S. Moon, and B. Lee, “Retinal projection type lightguide-based near-eye display with switchable viewpoints,” Opt. Express 28(3), 3116–3135 (2020). [CrossRef]
8. T. Lin, T. Zhan, J. Zou, F. Fan, and S.-T. Wu, “Maxwellian near-eye display with an expanded eyebox,” Opt. Express 28(26), 38616–38625 (2020). [CrossRef]
9. M. H. Choi, K. S. Shin, J. Jang, W. Han, and J. H. Park, “Waveguide-type Maxwellian near-eye display using a pin-mirror holographic optical element array,” Opt. Lett. 47(2), 405–408 (2022). [CrossRef]
10. Z. Wang, X. Zhang, K. Tu, G. Lv, Q. Feng, A. Wang, and H. Ming, “Lensless full-color holographic Maxwellian near-eye display with a horizontal eyebox expansion,” Opt. Lett. 46(17), 4112–4115 (2021). [CrossRef]
11. Z. Lv, J. Liu, and L. Xu, “A multi-plane augmented reality head-up display system based on volume holographic optical elements with large area,” IEEE Photonics J. 13(5), 1–8 (2021). [CrossRef]
12. K. S. Shin, M. H. Choi, J. Jang, and J. H. Park, “Waveguide-type see-through dual focus near-eye display with a polarization grating,” Opt. Express 29(24), 40294–40309 (2021). [CrossRef]
13. C. Yoo, K. Bang, C. Jang, D. Kim, C.-K. Lee, G. Sung, and B. Lee, “Dual-focal waveguide see-through near-eye display with polarization-dependent lenses,” Opt. Lett. 44(8), 1920–1923 (2019). [CrossRef]
14. D. Dunn, P. Chakravarthula, Q. Dong, and H. Fuchs, “Mitigating vergence-accommodation conflict for near-eye displays via deformable beamsplitters,” Proc. SPIE 10676, 104 (2018). [CrossRef]
15. X. Xia, Y. Guan, A. State, P. Chakravarthula, K. Rathinavel, T. J. Cham, and H. Fuchs, “Towards a Switchable AR/VR Near-eye Display with Accommodation-Vergence and Eyeglass Prescription Support,” IEEE Trans. Visual. Comput. Graphics 25(11), 3114–3124 (2019). [CrossRef]
16. D. Yoo, S. Lee, Y. Jo, J. Cho, S. Choi, and B. Lee, “Volumetric head-mounted display with locally adaptive focal blocks,” IEEE Trans. Visual. Comput. Graphics 28(2), 1415–1427 (2022). [CrossRef]
17. A. Wilson and H. Hua, “Design and demonstration of a vari-focal optical see-through head-mounted display using freeform Alvarez lenses,” Opt. Express 27(11), 15627–15637 (2019). [CrossRef]
18. J. Kim, M. Gopakumar, S. Choi, Y. Peng, W. Lopes, and G. Wetzstein, “Holographic glasses for virtual reality,” In Proceedings of the ACM SIGGRAPH (2022), pp 1–9.
19. C. Jang, K. Bang, G. Li, and B. Lee, “Holographic near-eye display with expanded eye-box,” ACM Trans. Graph. 37(6), 1–14 (2018). [CrossRef]
20. X. Wang and H. Hua, “Depth-enhanced head-mounted light field displays based on integral imaging,” Opt. Lett. 46(5), 985–988 (2021). [CrossRef]
21. C. Yao, D. Cheng, T. Yang, and Y. Wang, “Design of an optical see-through light-field near-eye display using a discrete lenslet array,” Opt. Express 26(14), 18292–18301 (2018). [CrossRef]
22. C. Jang, K. Bang, S. Moon, J. Kim, S. Lee, and B. Lee, “Retinal 3D: augmented reality near-eye display via pupil-tracked light field projection on retina,” ACM Trans. Graph. 36(6), 1–13 (2017). [CrossRef]
23. H. Hua and B. Javidi, “A 3D integral imaging optical see-through head-mounted display,” Opt. Express 22(11), 13484–13491 (2014). [CrossRef]
24. T. Ueno and Y. Takaki, “Approximated super multi-view head-mounted display to reduce visual fatigue,” Opt. Express 28(9), 14134–14150 (2020). [CrossRef]
25. L. Wang, Y. Li, S. Liu, Y. Su, and D. Teng, “Large Depth of Range Maxwellian-Viewing SMV Near-Eye Display Based on a Pancharatnam-Berry Optical Element,” IEEE Photonics J. 14(1), 1–7 (2022). [CrossRef]
26. L. Liu, J. Cai, Z. Pang, and D. Teng, “Super multi-view near-eye 3D display with enlarged field of view,” Opt. Eng. 60(08), 085103 (2021). [CrossRef]
27. T. Ueno and Y. Takaki, “Super multi-view near-eye display to solve vergence-accommodation conflict,” Opt. Express 26(23), 30703–30715 (2018). [CrossRef]
28. X. Zhang, Y. Pang, T. Chen, K. Tu, Q. Feng, G. Lv, and Z. Wang, “Holographic super multi-view Maxwellian near-eye display with eyebox expansion,” Opt. Lett. 47(10), 2530–2533 (2022). [CrossRef]
29. Y. Takaki, “Development of super multi-view displays,” MTA 2(1), 8–14 (2014). [CrossRef]
30. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv, arXiv:1412.6980 (2014). [CrossRef]
31. B. C. Kress, Optical Architectures for Augmented-, Virtual-, and Mixed-Reality Headsets (SPIE, 2020).
32. J. Piao, G. Li, M. Piao, and N. Kim, “Full Color Holographic Optical Element Fabrication for Waveguide-type Head Mounted Display Using Photopolymer,” J. Opt. Soc. Korea 17(3), 242–248 (2013). [CrossRef]
33. H.-J. Yeom, H.-J. Kim, S.-B. Kim, H. Zhang, B. Li, Y.-M. Ji, S.-H. Kim, and J.-H. Park, “3D holographic head mounted display using holographic optical elements with astigmatism aberration compensation,” Opt. Express 23(25), 32025–32034 (2015). [CrossRef]
34. H. Kogelnik, “Coupled wave theory for thick hologram gratings,” Bell Labs Tech. J. 48(9), 2909–2947 (1969). [CrossRef]
35. J. Yeom, Y. Son, and K.-S. Choi, “Pre-compensation method for optimizing recording process of holographic optical element lenses with spherical wave reconstruction,” Opt. Express 28(22), 33318–33333 (2020). [CrossRef]
36. S. Lee, C. Jang, J. Cho, J. Yeom, and B. Lee, “Viewing angle enhancement of an integral imaging display using Bragg mismatched reconstruction of holographic optical elements,” Appl. Opt. 55(3), A95–A103 (2016). [CrossRef]
37. S. Choi, J. Kim, Y. Peng, and G. Wetzstein, “Optimizing image quality for holographic near-eye displays with michelson holography,” Optica 8(2), 143–146 (2021). [CrossRef]
38. Y. Peng, S. Choi, N. Padmanaban, and G. Wetzstein, “Neural holography with camera-in-the-loop training,” ACM Trans. Graph. 39(6), 1–14 (2020). [CrossRef]
39. K. Yin, E. L. Hsiang, J. Zou, Y. Li, Z. Yang, Q. Yang, P. C. Lai, C. L. Lin, and S. T. Wu, “Advanced liquid crystal devices for augmented reality and virtual reality displays: principles and applications,” Light: Sci. Appl. 11(1), 161 (2022). [CrossRef]
