Open Access
Add to BibSonomy
Abstract
We introduce an approach to expand the eye-box in a retinal-projection-based near-eye display. The retinal projection display has the advantage of providing clear images in a wide depth range; however, it has difficulty in practical use with a narrow eye-box. Here, we propose a method to enhance the eye-box of the retinal projection display by generating multiple independent viewpoints, maintaining a wide depth of field. The method prevents images projected from multiple viewpoints from overlapping one other in the retina. As a result, our proposed system can provide a continuous image over a wide viewing angle without an eye tracker or image update. We discuss the optical design for the proposed method and verify its feasibility through simulation and experiment.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. INTRODUCTION
Augmented reality (AR) can play an important role in connecting the digital and physical world in the Fourth Industrial Revolution. In particular, AR near-eye display (AR NED) is receiving much attention for its applicability to various industries. Unlike AR based on hand-held devices, AR NEDs, such as glasses, are worn on the head, which will help to facilitate access to virtual spaces anywhere in everyday life. To achieve this, AR NEDs should have high portability and be able to provide the real scene clearly. An image combiner allows AR NEDs to deliver virtual images simultaneously with high optical transparency. Various methods for the image combiner have been considered. Depending on which method is adopted, visual performance and a form factor of AR NEDs are primarily determined. Among them, diffractive elements have advantages of being thin and transparent. In particular, holographic optical elements (HOEs) are capable of recording the desired volume grating from the type of lens to the complex wavefront using the interference of light. For this reason, research on using the HOE for AR NEDs has been actively conducted [1].
For AR NEDs to become a part of our lives, further consideration, such as visual fatigue, should be regarded to avoid discomfort from wearing for a long time. In the NEDs, visual fatigue could arise when there is a difference between the real scene and the displayed image. When looking at an object in the real world, depth information is obtained from physical cues such as accommodation and vergence derived from the eye, as well as psychological cues. In a typical NED, the virtual image is physically located at a certain depth by a fixed focal lens. As the object is positioned away from this depth, the vergence distance provided by binocular disparity is inconsistent with the focus distance. If the difference is too large, the vergence–accommodation conflict (VAC) could cause the user to feel discomfort [2–4].
For a solution to this problem, many methods have been proposed, such as generating multiple focal planes [5–9] and adjusting the location of a focal plane according to the vergence distance [10–14]. Holographic displays that can create objects at desired depths through wavefront modulation can also effectively reduce the VAC problem [15,16]. However, an additional optical system can make a system bulky, and a hologram requires many resources for computation. AR NEDs, with the support of external real-time interaction, still need further development of hardware. The retinal projection display (RPD) adopts a relatively simple optical system and increases the depth of field (DOF) [17–21]. It can provide a vivid image in a wide depth range without retinal blur. It has been proven by researchers that this focus-free system can alleviate the VAC problem [22,23].
The RPD uses a lens to focus the image into a small point of the pupil and project it to the retina. This makes retinal blur be less sensitive to the user’s accommodation. Depending on the size of the exit-pupil and the sensor’s resolution, the depth range that can provide clear images will be determined. However, there is a problem in which the eye-box that can view images is limited because they are focused at one point. This may cause the image to be blank if the eye deviates from the eye-box by displacement due to eye rotation. In the RPD using a HOE as the image combiner, several methods have been proposed to extend the small eye-box. Previous research was to provide multiple viewpoints [24,25] or to dynamically shift the viewpoint according to the location of the pupil [19,26,27]. Figure 1(a) indicates the implementation of a dynamic eye-box by adjusting the incident angle for the HOE. It requires an eye tracker as well as a mechanical device such as a moving HOE module or steering mirror. Another approach is to increase the number of viewpoints by simultaneously recording multiple angles of the lens on the HOE, as shown in Fig. 1(b). In this case, there is a problem in that each viewpoint image overlaps or there exists a blank while the user changes the viewpoint [28]. Also, the eye tracker is essential, and the images should be updated according to the viewpoint.
Fig. 1. Eye-box expanding methods in the RPD using HOE: (a) shifting the viewpoint by projecting the images onto the HOE in different angles and (b) recording the HOE at different angles. Both methods should track the user’s gaze and provide a corresponding image. (c) The proposed method is capable of providing continuous images within the eye-box without an additional tracker.
In this paper, a novel method to expand the eye-box of the RPD is proposed. The proposed method constructs independent viewpoints that can implement retinal projection with a wide DOF without an eye tracker. As shown in Fig. 1(c), multiple HOEs were designed to form different viewpoints according to the spatial position of HOEs. Each viewpoint image represents a different area of the perspective scene and does not overlap with each other in the projection plane. Therefore, this method is relatively free of eye rotation with a continuous and natural transition between viewpoints. In Section 2, the principle of the proposed method is introduced with a theoretical analysis of the RPD. Then, we prove the concept of the proposed display system through simulation using LightTools and Zemax in Section 3. A demonstration of the experiment and a discussion of the system take place in Sections 4 and 5, respectively. We conclude with the discussion of the future works to improve the proposed system in Section 6.
2. PRINCIPLE
A. Lightguide and HOE Image Combiner
The optical system for AR NEDs can be considered in three parts: generating, relaying, and combining the images. Of these, the image combiner, which merges real scenes and virtual images, has a direct effect on the performance of the NED system [29]. This element should have a transparent property to view the external scene and serve to deliver the virtual image. In this paper, the lightguide with a HOE is utilized as an image combiner. The HOE is a transparent element that can reproduce the wavefront, which is usually recorded to act as a lens or mirror. Due to an angular selectivity of the volume gratings in the HOE, high efficiency is obtained only for the recorded wavelength [30–32]. With these advantages and thin characteristic, the HOE is suitable for the image combiner of the AR NED.
The typical designs of HOE-based AR NEDs are described in Fig. 2. The HOEs in both systems are recorded as a concave mirror that focuses the incident light, which enlarges the display image. At this time, the eye-box is defined as an area where the image can be viewed with a uniform field of view (FOV). It depends on the divergence angle of the display light and the efficiency of the HOE. The system in Fig. 2(a) is a basic configuration for the propagation of the image in free space. It is necessary to correct the image distortion caused by the oblique incidence and a difference in native resolution according to the incident position to the HOE [28]. Compared to this, the system in Fig. 2(b) relays the image to the HOE through the total internal reflection in the lightguide. Although the system becomes more complex, the spatial uniformity of the image can be improved.
Fig. 2. Configuration of the retinal projection displays. (a) Free-space propagation method in which the image from the display directly enters the image combiner from the air and (b) lightguide method in which the light propagates in the glass. (c) By guiding on-axis with a half-mirror inside the lightguide, off-axis aberration in the image combiner can be reduced. (d) In the case of on-axis incidence, the maximum field of view is determined by the eye-relief and lightguide thickness. The simulation is conducted under the guiding angle of 60°.
When a lightguide is utilized, it can be divided into on- and off-axis incidence according to the direction in which the image enters the combiner, as shown in Figs. 2(b) and 2(c). For on-axis incidence, a reflective optics is necessary to match the focal plane of the image perpendicular to the optical axis of the HOE after the total internal reflection. In this paper, a half-mirror is used inside the lightguide. The discussion in the case of the off-axis incidence is described in Section 5.A. The ${\theta _m}$ is the slanted angle of the half-mirror, and it should be half of the guiding angle ${\theta _g}$ to reflect the image vertically at the end of the lightguide, as shown in Fig. 2(c). At a guiding angle smaller than 60°, the area reflected twice from the half-mirror results in an overlap, and in the opposite case, the lightguide thickness increases. Therefore, we choose 60° for the guiding angle. The maximum FOV supported by the lightguide-based RPD for eye-relief and lightguide thickness is shown in Fig. 2(d).
B. Retinal Projection Display
The light rays entering the aperture–lens system, such as the eye lens, are focused on the image plane according to the lens formula, as shown in Fig. 3(a). In the case of being out of focus due to the change in the focal plane, defocus blur will occur on a sensor. Since the lights converge into a small exit-pupil, the RPD can be considered a system with a small aperture [33]. It results in a wide depth range in which the image can be sharply focused. As the DOF increases, the RPD can effectively reduce the VAC problem that could cause visual fatigue in AR NEDs.
Fig. 3. (a) Definition of DOF in imaging systems. It is possible to define the closest and farthest object plane where the defocus blur is within the sensor’s pixel pitch. (b) DOF and spatial frequency corresponding to the Rayleigh criterion versus aperture size when the focal length is 4.6 mm with a wavelength of 530 nm.
In the optical system with an aperture, the focus distance giving the maximum DOF can be expressed with the hyperfocal distance [34]. Here, ${d_n}$ denotes the nearest object plane of acceptable sharpness, ${d_f}$ is the farthest object plane, and $f$ is the focal length of the lens. DOF can be calculated as the difference between ${d_n}$ and ${d_f}$. When the farthest object plane is at infinity, the hyperfocal distance ${d_h}$ is defined as $2{d_n}$ and can be calculated as follows:
Here, $N$ represents the ${\rm{F}}$-number of the optical system, and the circle of confusion $c$ is considered as the pixel pitch at the sensor plane. Figure 3(b) shows the DOF for the exit-pupil size. We designed the spot size at the pupil of the RPD based on the relation between the exit-pupil and DOF. For calculation, we utilized the camera specification instead of the eye to verify the principle through experimental results. Based on a 2.9 mm aperture size and a 4.3 mm distance from the sensor with a $1.4\,\,\unicode{x00B5}{\rm m}$ pixel pitch used in the experiment, the aperture size to make the RPD system cover the wide depth range from 0.5 m to infinity can be calculated as 0.32 mm.
Conventional flat panel displays, non-holographic displays, have a large viewing angle with scattered light. To make the exit-pupil small for a wide DOF, we need to filter the light from the display, as shown in Fig. 4. In this case, the image resolution can be limited by diffraction. For diffraction through an aperture, the angular resolution by the Rayleigh criterion can be expressed by
where $\theta$ is the angular resolution, $\lambda$ is the wavelength of the light source, and ${D_{{\rm ap}}}$ is the diameter of the aperture.A factor of 1.22 is derived from a calculation of the position at the first minimum of the Airy disk. The criterion means that if the peak of the point spread function is placed on the first minimum of the other point, the two points or lines are resolved. It is used to calculate the eye’s resolution for stimuli degraded by the optics [35].Fig. 4. Illustration of a conventional spatial filtering technique for increasing the DOF by reducing the exit-pupil in the RPD.
Figure 3(b) shows that the spatial frequency increases in proportion to the aperture size. It means that the maximum resolution the RPD supports is determined by the size of exit-pupil as the aperture. The smaller the exit-pupil, the wider the DOF, but the resolution of the RPD is inversely reduced. In addition, the size of the exit-pupil is related to the numerical aperture (NA) of the lens relaying the image. Focusing with a high NA lens increases the FOV, and the exit-pupil decreases proportionally. In a conventional optical system, it is a property of light, commonly known as étendue, limiting a value between the FOV and exit-pupil [36]. This trade-off relationship can be expressed as follows:
where ${\theta _{{\rm ap}}}$ is the emitting angle at the aperture. Since étendue is conserved, a diameter ${D_{{\rm ap}}}$ corresponds to a diameter ${D_{{\rm ep}}} = {D_{{\rm ap}}}{\theta _{{\rm ap}}}/{\theta _{{\rm FOV}}}$ at the pupil. In other words, the size of the exit-pupil can be smaller by increasing the focal length ratio ${f_{{\rm relaylens}}}/{f_{{\rm eyepiece}}}$. However, a high NA eyepiece could cause severe aberrations, and the small eye-box is vulnerable to pupil movement. In this case, a method of increasing the number of viewpoints to expand the eye-box can be considered.C. Eye-Box Extended RPD with Multiple Viewpoints
Several studies have been conducted to generate multiple viewpoints to extend the eye-box. The methods utilize multiplexed HOEs in which different converging beams are recorded [24,25]. In this case, multiple viewpoints are generated simultaneously, even if a single image is relayed. Therefore, the gap distance between viewpoints should be set well to avoid interference with each other. However, it is difficult to fully cover the various eye pupil sizes that change depending on the environment. As shown in Fig. 5, the proposed method divides the HOE area to form multiple independent viewpoints. At this time, images projected from each viewpoint are continuous and do not interfere with each other in the retina. Therefore, there is no overlap or blank problem, and it has the advantage of not requiring an eye tracker for updating the images. In addition, in the case of multiplexed HOEs, the efficiency of the HOE is shared in the process of multiplexing viewpoints, but the proposed method can achieve high efficiency by using multiple HOEs.
Fig. 5. Simplified concept diagram of the proposed method. For a conventional RPD with a single viewpoint, the image could disappear due to eye rotation or movement. In contrast, the proposed method can create multiple viewpoints by segmenting the HOE and sorting the images into appropriate HOEs without overlapping.
On the other hand, as a trade-off of expanding the eye-box, the FOV at a single viewpoint decreases by the number of viewpoints. It can be alleviated by selecting a gap distance smaller than the pupil to observe several viewpoints simultaneously. For example, if the gap distance is set to 2 mm, two or three viewpoints enter the pupil together in a situation where the pupil size is larger than 4 mm. For this, the projected images should be a continuous and independent area for each viewpoint. Unlike the case of increasing the eye-box by reducing the FOV while maintaining the étendue, the proposed method maintains the size of the exit-pupil in the extended eye-box, so that a wide DOF can be obtained. Also, it can effectively increase a viewing angle for eye rotation.
Figure 6(a) shows how to set the gap distance between multiple HOEs to avoid an overlap in the sensor plane. From similar triangles, the gap distance between HOEs is given by
where ${g_{{\rm vp}}}$ is the gap distance between viewpoints, and ${d_{{\rm HOE}}}$ is the distance to the HOEs. If ${g_{{\rm HOE}}}$ and ${g_{{\rm vp}}}$ are the same, the focal plane without the overlap is located at infinity. When the focal length changes, the image positions projected from multiple viewpoints are shifted as described in Fig. 6(b). The displacement ${d_{{\rm error}}}$ at the sensor plane can be calculated as follows: where ${d_s}$ is the distance to the sensor, and ${d_f}$ is the distance to the focal plane from the lens. Figure 6(c) shows the displacement between viewpoints caused by the deviation of the focus distance from the hyperfocal distance of 1 m. We utilized the camera specification mentioned above with a 2 mm gap distance. The farther the focal plane is from the hyperfocal distance, an overlap occurs between the images, and in the opposite case, a separation occurs. At this time, the maximum displacement within the system’s DOF is ${d_s}{g_{{\rm vp}}}/{d_h}$. Considering Eq. (2), if the ${g_{{\rm vp}}}$ is less than 2 mm, the maximum displacement is within the radius of an Airy disk, as shown in Fig. 6(d). In this case, it can be seen that the proposed display system can provide the viewpoint images continuously within the DOF.Fig. 6. Condition that (a) projected images do not overlap at the sensor plane and that (b) a displacement occurs. The displacement between the projected images is due to changes in the focal length. The degrees of displacement according to the (c) focus distance and (d) gap distance are represented. The further the focal plane is from the hyperfocal distance, the more displacement that occurs. The maximum displacement within the DOF increases in proportion to the gap distance between viewpoints.
D. Overall Configuration
The overall configuration of the proposed method is shown in Fig. 7. In detail, the image produced by the spatial light modulator (SLM) is relayed to the lightguide through the spatial filtering to reduce the exit-pupil. We adopted a micro–electro-mechanical system (MEMS) mirror that rapidly changes the reflection angle of incidence light. It is located between $4f$ relay optics, which performs a filtering process and simultaneously orients the filtered image to different areas on the lightguide. Then, the images of different areas are guided to the HOEs and focused on different viewpoints. By adopting the time-multiplexing technique with the MEMS mirror, the problem of a dead area caused by the gaps between viewpoints can be mitigated. The images projected from different viewpoints are set not to overlap each other. In summary, the proposed method can effectively solve the image problem, such as an overlap or blank caused by viewpoint replication in multi-view RPD. This method can apply to the NED without an additional optical path, and the eye-box can be improved through the time-multiplexing method, which can further increase the number of viewpoints if the refresh rate of the display is supported.
Fig. 8. Raytracing simulation results of image formation using LightTools. Conventional RPD using fixed exit-pupil has a narrow eye-box, which is disadvantageous for eye rotation. In the case of a dynamic eye-box that moves the exit-pupil according to the pupil position, the update of image and precise eye tracking is necessary. The proposed method can remain continuously aligned to gaze direction during eye rotation so that it can provide an effective FOV near the central area. All methods of simulation have the same area of HOE.
3. SIMULATION
A. Simulation of Image Formation
We validated that the proposed method provides a continuous eye-box through simulation work. The previous methods using a single viewpoint were compared with the case of using multiple independent viewpoints. Due to the limitation of the raytracing simulation, it considered only the parallel rays with the ideal condition. Figure 8 shows the extended eye-box simulation under the change of gaze direction using the eye model in LightTools. The pupil size and the gap distance between the viewpoints were set to 5 mm and 2 mm, respectively, with five viewpoints. When the area of the HOE is set to ${23.2}\;{\rm{mm}} \times {8.3}\;{\rm{mm}}$ and the eye-relief is 20 mm, the maximum FOV is calculated as ${{60}}^\circ \times {{23}}^\circ$ with the case of using one viewpoint. Compared to this, the horizontal FOV of the proposed method is 38° at the center with three viewpoints. Although the maximum FOV was reduced, it shows that the proposed method can provide the continuous image over a wide viewing angle.
B. Determination of System Parameters
Before the experiment, first, we determined several boundaries for (1) maximum thickness of the lightguide and (2) target depth range. When the eye-relief of the system is fixed to a specific value, the thicker the lightguide, the wider the guided area, and thus a wide FOV can be achieved. However, a thick lightguide is not suitable for the AR NED where portability is important, so we limited it to less than 10 mm. The limitation of the lightguide thickness can be solved by adopting an image update method, which is described in Section 5.B. Then, we choose the exit-pupil size and floating distance of the display for supporting a wide depth range. As the target depth range is set to be from 0.5 m to infinity, the display focused at the hyperfocal distance of 1 m gives the maximum DOF. From Eq. (1), an optical system is designed to make the exit-pupil coincide with 0.32 mm for the two-diopter depth range. We utilized a smartphone camera as mentioned above for the calculation. Considering the aperture size of the camera sensor, three viewpoints were used with the gap distance of 1 mm.
C. Optical Design Using MEMS Mirror
Figure 9 shows the simulation results using Zemax. We utilize a polarizing beam splitter (PBS) to shorten the optical path and simply implement $4f$ relay through polarization-selective transmission and reflection. The focal length of the relay lens is 50 mm.
In detail, the collimated light from an LED source is polarized along the $y$ axis while passing through a linear polarizer. The $y$-polarized light transmits through the PBS, and then reflects on the SLM, which modulates the polarization state of the reflected light. This allows eight-bit grayscale while reflecting on the PBS with $x$-axis polarization. The modulated image is focused by the relay lens and filtered to reduce the exit-pupil size in the Fourier plane. At this point, the $x$-polarized light is converted into a circular polarization by placing the quarter-wave plate (QWP) with the optical axis at 45° from the $x$ axis. Then, as the light reflected by the MEMS mirror located in the Fourier plane again undergoes retardation on the QWP, the polarization state consequently rotates 90° from the $x$ axis to the $z$ axis. The $z$-polarized light passes through PBS and enters the lightguide.
Since HOEs are made to form different viewpoints, it is necessary to adjust a light incident position in the lightguide for each viewpoint. The MEMS mirror located between a $4f$ relay lens can control the spatial position of the reflected image while performing the filtering function. There is a spacing between the incident images equal to the spacing in the HOEs. The exit-pupil focused through HOEs is determined by multiplying the size of the MEMS mirror by the focal length ratio of the HOE and the relay lens. A MEMS mirror with an aperture of 0.8 mm is used for the simulation, where the exit-pupil is 0.32 mm, as shown in the simulation result. All the distances between the optics are optimized to focus on the hyperfocal distance of 1 m.
4. EXPERIMENT
A. Recording Multiple Independent Viewpoints
The experimental setup for recording multiple independent viewpoints in the HOE is shown in Fig. 10. The HOE was recorded by adopting the conjugate reconstruction method to avoid errors caused by changes in the optical axis of the lens for each viewpoint. Instead of recording a converging wave as a signal wave, we recorded a diverging wave generated by an objective lens with a high NA that can cover all the viewing range. The diverging waves from the objective lens are recorded in the HOE on a glass sequentially by using a linear stage. For each recording, the objective lens was moved by the gap distance of the viewpoints. The areas that we did not want to record were blocked by masks precisely manufactured using a 3D printer. In the reconstruction process, the relayed images are incident in the opposite direction of the recorded reference wave and converge to each viewpoint as the conjugated signal wave. The objective lens of 0.50 NA and a 532 nm laser are used in the recording process. The distance from the object lens to the HOE is set to 20 mm.
Fig. 10. Recording setup for generating multiple viewpoints (top left), recording process for each viewpoint (bottom), rendering image of the masks (middle right), and recorded HOE after curing process (top right).
B. Experimental Setup
The experimental setup is built to validate the design of the proposed method, as shown in Fig. 11. A cage system was used to precisely control the distance between the optics in a compact space. The system specifications are the same as those set in Section 3.C. The size of the SLM we used was $4.64\, {{\rm mm}} × 8.26 \,{\rm {mm}}$ with full high-definition (FHD) resolution (Selcos). Fiber-coupled LED Green (Thorlabs M530F2) was used as the light source, and it was collimated by a condenser lens of 20 mm focal length. A commercial MEMS mirror, A3I8.2-800AL (Mirrorcle Tech.), was used to adjust the incident light at high speed. Its clear aperture diameter was 0.8 mm, which limits the exit-pupil size.
In the system, the SLM is updated with images of each viewpoint sequentially, and the MEMS mirror changes an angle to form the appropriate viewpoints. To make the gap distance between incident images 1 mm, the angle of the MEMS mirror is set to ${-}{3.2}^\circ$, 0°, and 3.2° for each viewpoint. Since the display has a frame rate of 60 Hz, three viewpoints are reproduced at the speed of 20 Hz. For synchronization between the MEMS mirror and the SLM, OpenGL library is used in a C/C++ programmed tool.
Fig. 12. Photographs of experimental results: (a) AR scene of the test image with different focus distances and (b) resolution target.
Fig. 13. Experimental results to demonstrate the feasibility of independent viewpoints (Visualization 1). We captured the images according to the camera’s rotation angle (source image courtesy of “SimplePoly Urban,” https://www.cgtrader.com). Corresponding simulation results represent the horizontal FOV that varies with the rotation angle. It shows a continuous change for each viewpoint marked by the colors. In the case of 2 mm distance, the side images are cut off slightly while entering the lightguide (dashed white line). It is because the lightguide is designed for 1 mm gap distance instead of 2 mm.
C. Experimental Results
The experimental result was taken with a smartphone (${\rm{F}}/{1.5}$ and 4.6 mm focal length) to capture in a condition of short eye-relief. The smartphone was combined with the rotation stage to confirm the continuous change of images according to the rotation. Distance from the rotation axis was set to 12 mm equal to the eye.
Figure 12 shows the experimental results at the center viewpoint. The FOV is approximately 13° in the horizontal direction and 23° in the vertical direction. To demonstrate that the proposed system can support the desired depth range, we captured the photographs by changing the focus distance from 0.2 m to optical infinity. As presented in Fig. 12(a), it provides clear images in the desired DOF. We also evaluate the resolution of the proposed system by using a USAF 1951 target (${{969}} \times {{1024}}$ pixels). For comparison, we performed a diffraction-limited incoherent imaging simulation in the same condition [37]. Figure 12(b) shows the vertical line profiles of group number 0 in the experiment and simulation. As a result, the system performance was shown to be near the diffraction limit of the imaging system, which means that the resolution of the system is limited only by aperture diffraction and not by aberrations.
We demonstrate proofs of concept of multiple independent viewpoints by changing the rotation angle. The experiment was conducted in two cases where the gap distance was 1 mm and 2 mm. As shown in Fig. 13, it can provide continuous images corresponding to the eye rotation without any eye tracker. In the case of 1 mm distance, three viewpoints are projected onto the sensor plane simultaneously at the center. The diagonal FOV of the system is 44°. In the experiment with the gap distance of 2 mm, two viewpoints come in when the rotation angle is ${-}$6° and 6°, as shown in the figures. In conclusion, we confirmed that independent and continuous images are provided according to the gaze direction, which fits well with the simulation results.
5. DISCUSSION
A. Off-Axis Aberrations of Oblique Incident
The proposed method generates multiple viewpoints by guiding the images through the lightguide. In the case of off-axis guiding, the area entering the HOE may double, but off-axis aberrations such as coma and astigmatism occur [30,38]. Especially when an object plane such as flat panel display is tilted, the pixels in a light guiding direction have different focal lengths, as shown in Fig. 14. According to the Scheimpflug principle, the focal plane can be made oblique by tilting a lens during the image relaying [39]. However, it leads to other off-axis aberrations, so the image quality could be worse. This problem can be mitigated theoretically by modulating the wavefront. In many studies on holographic NEDs, the correction of off-axis aberrations of HOEs has been conducted [15,16]. Alternatively, pre-compensation can be performed easily using a holographic printer for the recording process of HOEs [40,41].
Fig. 14. (a) Illustration for the case where the incident object plane is not perpendicular to the optical axis of the HOE and (b) its experimental result.
B. Extended Eye-Box without Increasing Lightguide Thickness
As mentioned in Section 2.D, the proposed method can increase the number of viewpoints as long as the frame rate of the display supports it. Currently, high-speed displays such as digital micromirror devices (DMDs) or ferroelectric liquid crystal on silicon (FLCoS) are capable of generating a dozen viewpoints [42,43]. However, since each viewpoint is formed in different areas, the guiding area must be widened. It means that the thickness of the lightguide should be increased in proportion to the number of viewpoints.
Here, by adopting eye tracking technology with an image update, multiple viewpoints can be achieved without increasing the lightguide thickness. We apply the idea that the number of viewpoints entering the pupil is constant. As described in Fig. 15, the images projected to the lightguide are updated according to the current gaze position. For example, at the center gaze, the images corresponding to viewpoints from four to six enter the pupil. If the gaze direction shifts sideways from five to four, a continuous change of images can be provided by updating the input image at position C from six to three. For this, it is required to precisely control the efficiency of the HOEs for each reflection so that the guided images have uniform intensity in all areas.
Fig. 15. Schematic layout of image update strategy for expanding the eye-box in the AR NED with limited lightguide thickness.
6. CONCLUSION
In this paper, we proposed a novel method to increase the eye-box of the retinal projection type NED. Our method is based on forming multiple independent viewpoints that do not overlap each other on the retina. It has an advantage in that the image problem occurring in a multiple-viewpoint RPD does not happen even if the pupil moves, so an eye tracker is not necessary. Simulation results proved that the proposed method provides continuous and effective images with an expanded eye-box. We built a proof-of-concept system in which a FOV of 13° or more was supported at a horizontal rotation angle of ${\pm}12{^ \circ}$. A maximum diagonal FOV of 44° was achieved for the condition of three viewpoints. We also showed that a high-speed MEMS mirror can be utilized to form multiple viewpoints based on the time-multiplexing technique and as an aperture stop for a small exit-pupil. Experimental results demonstrated that the proposed method could support a wide depth range. The proposed concept of multiple independent viewpoints can also be applied in a holographic display that can easily correct aberrations. Also, by manufacturing the HOE with a holographic printer, it will be possible to increase the number of viewpoints as desired while compensating for the aberrations. We hope this work will be a meaningful approach for practical use of RPD in the future.
Funding
Institute for Information and Communications Technology Promotion Planning and Evaluation Grant funded by the Korean Government (MSIT) (2017-0-00787).
Acknowledgment
This work is supported by the Institute for Information and Communications Technology Promotion Grant funded by the Korea Government (MSIT) (development of vision assistant HMD and contents for the legally blind and low vision).
Disclosures
The authors declare no conflicts of interest.
REFERENCES
1. B. Lee, C. Yoo, and J. Jeong, “Holographic optical elements for augmented reality systems,” Proc. SPIE 11551, 1155103 (2020). [CrossRef]
2. S. Yano, S. Ide, T. Mitsuhashi, and H. Thwaites, “A study of visual fatigue and visual comfort for 3D HDTV/HDTV images,” Displays 23, 191–201 (2002). [CrossRef]
3. D. M. Hoffman, A. R. Girshick, K. Akeley, and M. S. Banks, “Vergence–accommodation conflicts hinder visual performance and cause visual fatigue,” J. Vis. 8(3), 33 (2008). [CrossRef]
4. T. Shibata, J. Kim, D. M. Hoffman, and M. S. Banks, “The zone of comfort: predicting visual discomfort with stereo displays,” J. Vis. 11(8), 11 (2011). [CrossRef]
5. K. Akeley, S. J. Watt, A. R. Girshick, and M. S. Banks, “A stereo display prototype with multiple focal distances,” ACM Trans. Graph. 23, 804–813 (2004). [CrossRef]
6. S. Liu and H. Hua, “Time-multiplexed dual-focal plane head-mounted display with a liquid lens,” Opt. Lett. 34, 1642–1644 (2009). [CrossRef]
7. D. Lanman and D. Luebke, “Near-eye light field displays,” ACM Trans. Graph. 32, 220 (2013). [CrossRef]
8. R. Narain, R. A. Albert, A. Bulbul, G. J. Ward, M. S. Banks, and J. F. O’Brien, “Optimal presentation of imagery with focus cues on multi-plane displays,” ACM Trans. Graph. 34, 59 (2015). [CrossRef]
9. S. Lee, Y. Jo, D. Yoo, J. Cho, D. Lee, and B. Lee, “Tomographic near-eye displays,” Nat. Commun. 10, 2497 (2019). [CrossRef]
10. S. Ravikumar, K. Akeley, and M. S. Banks, “Creating effective focus cues in multi-plane 3D displays,” Opt. Express 19, 20940–20952 (2011). [CrossRef]
11. R. Konrad, E. A. Cooper, and G. Wetzstein, “Novel optical configurations for virtual reality: evaluating user preference and performance with focus-tunable and monovision near-eye displays,” in Proceedings of the CHI Conference on Human Factors in Computing Systems (Association for Computing Machinery, 2016), pp. 1211–1220.
12. D. Dunn, C. Tippets, K. Torell, P. Kellnhofer, K. Akşit, P. Didyk, K. Myszkowski, D. Luebke, and H. Fuchs, “Wide field of view varifocal near-eye display using see-through deformable membrane mirrors,” IEEE Trans. Vis. Comp. Graph. 23, 1322–1331 (2017). [CrossRef]
13. K. Akşit, W. Lopes, J. Kim, P. Shirley, and D. Luebke, “Near-eye varifocal augmented reality display using see-through screens,” ACM Trans. Graph. 36, 189 (2017). [CrossRef]
14. N. Padmanaban, R. Konrad, T. Stramer, E. A. Cooper, and G. Wetzstein, “Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays,” Proc. Natl. Acad. Sci. USA 114, 2183–2188 (2017). [CrossRef]
15. A. Maimone, A. Georgiou, and J. S. Kollin, “Holographic near-eye displays for virtual and augmented reality,” ACM Trans. Graph. 36, 85 (2017). [CrossRef]
16. C. Jang, K. Bang, G. Li, and B. Lee, “Holographic near-eye display with expanded eye-box,” ACM Trans. Graph. 37, 195 (2018). [CrossRef]
17. G. Westheimer, “The Maxwellian view,” Vis. Res. 6, 669–682 (1966). [CrossRef]
18. M. von Waldkirch, P. Lukowicz, and G. Tröster, “Defocusing simulations on a retinal scanning display for quasi accommodation-free viewing,” Opt. Express 11, 3220–3233 (2003). [CrossRef]
19. C. Jang, K. Bang, S. Moon, J. Kim, S. Lee, and B. Lee, “Retinal 3D: augmented reality near-eye display via pupil-tracked light field projection on retina,” ACM Trans. Graph. 36, 190 (2017). [CrossRef]
20. Y. Wu, C. P. Chen, L. Mi, W. Zhang, J. Zhao, Y. Lu, W. Guo, B. Yu, Y. Li, and N. Maitlo, “Design of retinal-projection-based near-eye display with contact lens,” Opt. Express 26, 11553–11567 (2018). [CrossRef]
21. L. Mi, C. P. Chen, Y. Lu, W. Zhang, J. Chen, and N. Maitlo, “Design of lensless retinal scanning display with diffractive optical element,” Opt. Express 27, 20493–20507 (2019). [CrossRef]
22. R. Konrad, N. Padmanaban, K. Molner, E. A. Cooper, and G. Wetzstein, “Accommodation-invariant computational near-eye displays,” ACM Trans. Graph. 36, 88 (2017). [CrossRef]
23. P. K. Shrestha, M. J. Pryn, J. Jia, J.-S. Chen, H. N. Fructuoso, A. Boev, Q. Zhang, and D. Chu, “Accommodation-free head mounted display with comfortable 3D perception and an enlarged eye-box,” Research 2019, 1 (2019). [CrossRef]
24. S.-B. Kim and J.-H. Park, “Optical see-through Maxwellian near-to-eye display with an enlarged eyebox,” Opt. Lett. 43, 767–770 (2018). [CrossRef]
25. J. Jeong, J. Lee, C. Yoo, S. Moon, B. Lee, and B. Lee, “Holographically customized optical combiner for eye-box extended near-eye display,” Opt. Express 27, 38006–38018 (2019). [CrossRef]
26. J. Kim, Y. Jeong, M. Stengel, K. Akşit, R. Albert, B. Boudaoud, T. Greer, J. Kim, W. Lopes, and Z. Majercik, “Foveated AR: dynamically-foveated augmented reality display,” ACM Trans. Graph. 38, 99 (2019). [CrossRef]
27. K. Ratnam, R. Konrad, D. Lanman, and M. Zannoli, “Retinal image quality in near-eye pupil-steered systems,” Opt. Express 27, 38289–38311 (2019). [CrossRef]
28. C. Yoo, M. Chae, S. Moon, and B. Lee, “Retinal projection type lightguide-based near-eye display with switchable viewpoints,” Opt. Express 28, 3116–3135 (2020). [CrossRef]
29. Y.-H. Lee, T. Zhan, and S.-T. Wu, “Prospects and challenges in augmented reality displays,” Virtual Real. Intell. Hardw. 1, 10–20 (2019). [CrossRef]
30. D. Close, “Holographic optical elements,” Opt. Eng. 14, 145408 (1975). [CrossRef]
31. S. C. Barden, J. A. Arns, and W. S. Colburn, “Volume-phase holographic gratings and their potential for astronomical applications,” Proc. SPIE 3355, 866–876 (1998). [CrossRef]
32. J. Han, J. Liu, X. Yao, and Y. Wang, “Portable waveguide display system with a large field of view by integrating freeform elements and volume holograms,” Opt. Express 23, 3534–3549 (2015). [CrossRef]
33. Y. Takaki and N. Fujimoto, “Flexible retinal image formation by holographic Maxwellian-view display,” Opt. Express 26, 22985–22999 (2018). [CrossRef]
34. J. Conrad, “Depth of field in depth,” in Large Format Photography (2006), pp. 1–45.
35. M. Kalloniatis and C. Luu, “Visual acuity,” in Webvision: The Organization of the Retina and Visual System, H. Kolb, E. Fernandez, and R. Nelson, eds. (University of Utah Health Sciences Center, 1995).
36. G. Brooker, Modern Classical Optics (Oxford University, 2003), Vol. 8.
37. D. Voelz, Computational Fourier Optics: A MATLAB Tutorial (SPIE, 2011).
38. S. Lee, J. Cho, B. Lee, Y. Jo, C. Jang, D. Kim, and B. Lee, “Foveated retinal optimization for see-through near-eye multi-layer displays,” IEEE Access 6, 2170–2180 (2017). [CrossRef]
39. Y. Jo, S. Lee, D. Yoo, S. Choi, D. Kim, and B. Lee, “Tomographic projector: large scale volumetric display with uniform viewing experiences,” ACM Trans. Graph. 38, 215 (2019). [CrossRef]
40. S. Lee, B. Lee, J. Cho, C. Jang, J. Kim, and B. Lee, “Analysis and implementation of hologram lenses for see-through head-mounted display,” IEEE Photon. Technol. Lett. 29, 82–85 (2016). [CrossRef]
41. J. Jeong, C.-K. Lee, B. Lee, S. Lee, S. Moon, G. Sung, H.-S. Lee, and B. Lee, “Holographically printed freeform mirror array for augmented reality near-eye display,” IEEE Photon. Technol. Lett. 32, 991–994 (2020). [CrossRef]
42. T. Ueno and Y. Takaki, “Super multi-view near-eye display to solve vergence–accommodation conflict,” Opt. Express 26, 30703–30715 (2018). [CrossRef]
43. B. Lee, D. Yoo, J. Jeong, S. Lee, D. Lee, and B. Lee, “Wide-angle speckleless DMD holographic display using structured illumination with temporal multiplexing,” Opt. Lett. 45, 2148–2151 (2020). [CrossRef]
